eventlet patches subprocess module since 0.35.2[1]. So we no longer
need that old workaround.
[1] 614a20462a
Change-Id: I75088e04cc3ba9a8bbb0a7af74650fc1ebff4c95
Update all .py source files by
$ pyupgrade --py3-only $(git ls-files | grep ".py$")
to modernize the code according to Python 3 syntaxes.
Also add the pyupgrade hook to pre-commit to avoid merging additional
Python 2 syntaxes.
Change-Id: I9d54c11642acb341cc5db1267f5a91d4740f932c
This code worked around a bug in eventlet[1] that has been fixed in
6533958f5fcec8b1577f56834ea2dc75a4f211e3. The fix has been available in
every eventlet release since v0.27.0.
[1] https://github.com/eventlet/eventlet/issues/592
Co-Authored-By: Cyril Roelandt <cyril@redhat.com>
Change-Id: I377f664b4e5092ee0e2a95a7e277f1c8a1e18953
The fallback was added 9 years ago, in 0.1.0 release[1].
[1] c98d5edc44ad44da7196afa4b6bba0a2afea1f8e
Change-Id: I3b66c5f9df58eaeea9fc51a6c0b83af2e39a525b
Because WinStackers project has been retired, running OpenStack on
Windows operating systems is no longer maintained.
Change-Id: If4ad14a9f7f974c120c6d70c826a1bba358052a3
The fasteners lib in version 0.15.0 removed the
threading.current_thread workaround for eventlet[1] because eventlet
seemed to fixed the current_thread issues tracked in [2]. However the
fix for [2] only fixed half of the problem. The threading.current_thread
call works if it is called from thread created by eventlet.spawn.
However if the thread is created with eventlet.spawn_n then
threading.current_thread is still broken and returns the ID of the
python native thread.
The fasteners' ReaderWriterLock depends heavily on
threading.current_thread to decide which thread holds a lock and to
allow re-entry of that thread. This leads to the situation that
multiple threads created from spawn_n could take the same
ReaderWriterLock at the same time.
The fair internal lock in oslo.concurrency uses ReaderWriterLock and
as a result such lock is broken for threads created with spawn_n.
Note that this issue was raised with eventlet in [3] when the nova team
detected it via a direct usage of ReaderWriterLock in the nova test
code. As [3] did not lead to a solution in eventlet nova implemented a
nova local fix for the test code in [4].
However now we detected that oslo.concurrency is affected by this issue
as well.
This patch restores the workaround that was removed by [1].
Note that a fasteners issue [5] also opened to restore the
workaround[1].
[1] 467ed75ee1
[2] https://github.com/eventlet/eventlet/issues/172
[3] https://github.com/eventlet/eventlet/issues/731
[4] https://review.opendev.org/c/openstack/nova/+/813114
[5] https://github.com/harlowja/fasteners/issues/96
Closes-Bug: #1988311
Change-Id: Ia873bcc6b07121c9bd0b94c593567d537b4c1112
The fasteners lib in version 0.15.0 removed the
threading.current_thread workaround for eventlet[1] because eventlet
seemed to fixed the current_thread issues tracked in [2]. However the
fix for [2] only fixed half of the problem. The threading.current_thread
call works if it is called from thread created by eventlet.spawn.
However if the thread is created with eventlet.spawn_n then
threading.current_thread is still broken and returns the ID of the
python native thread.
The fasteners' ReaderWriterLock depends heavily on
threading.current_thread to decide which thread holds a lock and to
allow re-entry of that thread. This leads to the situation that
multiple threads created from spawn_n could take the same
ReaderWriterLock at the same time.
The fair internal lock in oslo.concurrency uses ReaderWriterLock and
as a result such lock is broken for threads created with spawn_n.
Note that this issue was raised with eventlet in [3] when the nova team
detected it via a direct usage of ReaderWriterLock in the nova test
code. As [3] did not lead to a solution in eventlet nova implemented a
nova local fix for the test code in [4].
However now we detected that oslo.concurrency is affected by this issue
as well.
This patch adds tests to show the scope of the problem.
Note that the coverage tox target is changed to explicitly enable native
threading otherwise it runs eventlet specific tests in a native
environment.
Also note that [5] was opened to reintroduce the workaround[1] in fasteners.
[1] 467ed75ee1
[2] https://github.com/eventlet/eventlet/issues/172
[3] https://github.com/eventlet/eventlet/issues/731
[4] https://review.opendev.org/c/openstack/nova/+/813114
[5] https://github.com/harlowja/fasteners/issues/96
Related-Bug: #1988311
Change-Id: Ibc193c855b49b95b46ebd2aac82ea89e33f885f0
We currently have no log entry when we start trying to acquire a lock.
In general this is ok, but there are cases where it can be problematic,
for example if we have a deadlock situation or if a lock takes a very
long time to be acquired.
In those scenarios looking at the logs we would see the operation
proceed normally and suddenly go completely silent without knowing that
it's waiting for a lock to be freed.
This patch adds a debug log message right before trying to acquire the
lock so we can detect those situations.
Change-Id: I1354dfb98b0927ae167802ecc4ab1d34f6b4d720
When lock_file is not present on the [oslo_concurrency] section of the
config file, we are getting a misleading error saying that it is not
preset on the DEFAULT group. This commit fixes it by providing the group
arg to the RequiredOptError constructor.
Closes-Bug: #1966320
Signed-off-by: Thiago Brito <thiago.brito@windriver.com>
Change-Id: Idee6987739917c7eae4faee15a66085a9ef7f6d2
processutils.execute() is documented to take a python_exec kwarg and
default to using sys.executable if that argument isn't set. It is
convenient (and more intuitive) for this behavior to also occur if
the argument is present with value None.
Closes-bug: #1962603
Related-bug: #1962581
Change-Id: I2e1f187feaf4bf9fbfaf04fce78efa0cba49fc07
This adds support for a non-blocking behavior of the lock : the context-based
lock now raises an AcquireLockFailedException if the lock can't be acquired on
the first try.
At a higher layer, we just caught this exception inside the 'synchronized'
decorator to prevent calling the wrapped function. In which case, we then
trace this acquisition failure.
For now at least, disabling blocking is not supported when using fair locks
because the ReaderWriterLock.write_lock() provided by the fasteners module
doesn't implements this behavior.
Change-Id: I409da79007c9ba4fb8585da881e3d56998b0b98b
Monkey patch the original current_thread to use the up-to-date _active
global variable. This solution is based on that documented at:
https://github.com/eventlet/eventlet/issues/592
Change-Id: I0a2c1e0d8a8cad99d68100d25e88e0d3a2eb8f5c
Related-Bug: #1863021
Monkey patch the original current_thread to use the up-to-date _active
global variable. This solution is based on that documented at:
https://github.com/eventlet/eventlet/issues/592
Change-Id: I876e2825da628d51c9977a4eec0c2ef05aed3229
Closes-Bug: #1863021
Now that we no longer support py27, we can use the standard library
unittest.mock module instead of the third party mock lib.
Change-Id: I792a17f4b2667713503e6bc10cfabed9905d8535
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
There may be cases were library users want to request the removal of an
external lock that may not have been created and don't want to pollute
the logs when the file is not present.
We raise the log level from info to warning, but we don't log it if the
file didn't exist, since the end result the caller wanted is there.
Change-Id: I5ce8be34c9f2c4c59ea99dabc6760c3300f743a3
Current external lock removal unit tests are not really testing the
removal as they don't bother to check that we are calling os.remove
This patch fixes this situation by mocking the appropriate methods and
checking that they are being called (or not) as expected.
Change-Id: I25efc1638106431ee36e760c0c213a1a675ec908
Following up on comments in [1], clarify and unify the docs for the
following lockutils methods:
- lock_with_prefix
- synchronized_with_prefix
- remove_external_lock_file_with_prefix
[1] I4e723ee3be1e57c543684390b607c84388c6e930
Change-Id: I4179e8732dba7367bd0c835cbd11939ca7b8cc42
There's a convenience wrapper around the lockutils.synchronized
decorator that lets oslo.concurrency consumers set up a prefix to use
for all lock files (presumably to obviate collisions with other
consumers when using jejune lock names, like 'lock'). This commit adds
an equivalent wrapper around lockutils.lock, the context manager
counterpart to the lockutils.synchronized decorator.
Note that the unit test added for lock_with_prefix is pretty bare; but
it follows the precedent set by the existing tests. Future commits
should make all these tests more thorough/robust.
Change-Id: I4e723ee3be1e57c543684390b607c84388c6e930
- Tests are based on oslotest.base.BaseTestCase, which uses the
NestedTempFile fixture, which uses the TempDir fixture, which adds a
cleanup routine to remove the base temporary directory it creates.
There's therefore no need for tests to clean up this directory, so all
the code that does that is removed.
- The eventlet incarnation of tests was trying to make use of the `env`
external without whitelisting it, resulting in an ugly red deprecation
warning. This commit adds `env` to whitelist_externals in [testenv].
- A handful of typos in the Beowulf quote are corrected.
Change-Id: I91cc52e00e0a918dadd2a3a771bd322b0f165ed2
1. Thorough replacement of git.openstack.org URLs with their
opendev.org counterparts.
2. dict_object.keys() is not required for *in* operator
Change-Id: Ie1d0c828522ae92b01f28e7017fd7fd42c2e91e7
This adds support for a "fair" variant of the lock. When there are
multiple entities within a single process that are blocked waiting
for the lock the fair lock will ensure that they acquire the lock
in FIFO order.
For now at least, when fair locks are in use we don't support
the "semaphores" argument.
If external locks are enabled, the inter-process ordering will be
determined by the underlying OS lock ordering and process scheduling.
Change-Id: I37577becff4978bf643c65fa9bc2d78d342ea35a
The functions in this library are meant to wrap subprocess, and
sometimes we do want to allow passing shell=True, so mark that line
with nosec to ignore B604.
Change-Id: Iab676449e543a4fdabee62efa9283940d30e3398
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
According to Openstack summit session [1],
stestr is maintained project to which all Openstack projects should migrate.
Let's switch to stestr as other projects have already moved to it.
[1] https://etherpad.openstack.org/p/YVR-python-pti
Change-Id: I753f32ecf3275cf49d8c93bf648a6a26bc6da8e7
pep8 under python 3 is more strict than under python 2, so to make
sure our projects meet the more strict standards we want to run the
pep8 jobs using python 3 by default.
Change-Id: Ice3cffa36e9167f2f1f30337372cf9187af5e02e
At many places, processutils.ssh_execute() is being invoked to run
a command over ssh and output returned is parsed to get appropriate
information. In this flow, unsanitized output is being expected
where processutils.ssh_execute() was invoked but found that
output like volume details(containing "password" string in its name)
is being masked away with strutils.mask_password(stdout) even though
no error occured during command execution.
This is regression issue from patch[0]. In this fix, stdout and stderr
in processutils.ssh_execute() will be masked only when
ProcessExecutionError exception is thrown i.e. command execution failed
due to some reasons.
[0] https://github.com/openstack/oslo.concurrency/commit/
ae9e05bfc3d7ec867bd6ec78c301f11c2bdd0d5f
Change-Id: I2ce344330905eef437ef3f89a2a01169a30df8ab
Closes-Bug: #1482382
This commit adds a new kwarg to the process_utils.execute() function to
specify the python executable to use when launching python to check
prlimits. This is necessary when processutils.execute() is called from
inside an API server running with uwsgi. In this case sys.executable is
uwsgi (because uwsgi links libpython.so and is actually the interpreter)
This doesn't work with the execute() function because it assumes the
cpython interpreter CLI is used for the arguments it uses to call the
prlimits module. To workaround this and enable API servers that may run
under uwsgi to use this those applications can simply pass in an
executable to use.
Longer term it might be better to migrate the prlimits usage to call
multiprocessing instead of subprocessing python. But that would require
a more significant rewrite of both processutils and prlimit to
facilitate that.
Change-Id: I0ae60f0b4cc3700c783f6018e837358f0e053a09
Closes-Bug: #1712463
Noticed that the generated docs were displaying the parameter
to trycmd() in a misleading way, fixed the syntax there and for
get_worker_count().
Change-Id: If989523be81e917ed62f525f2431f368dcf13c7a
eventlet.green.subprocess is not actually greenthread friendly on
Windows. It just uses the native subprocess.Popen in this case.
For this reason, exec calls do not yield on Windows, blocking other
greenthreads.
This change avoids this issue by wrapping the 'communicate' call
using eventlet.tpool.
We're also ensuring that subprocess.Popen uses *native* threads
internally in order to avoid deadlocks when passing data through
stdin.
Change-Id: Ic25fd1b61b5498f16e6049cbbe0877492f8aab4d
Closes-Bug: #1709586
In a case when a number of threads are concurrently accessing a
resource that has a external lock there is no indication when
one of the threads has taken the external lock.
This patch adds in a debug log to try and add some clarity.
Change-Id: Ibd7cfd3b39370b1a54f2ab50c75c6b92a876ae1a
