* See bug description for the example that didn't work. It was
caused by a simple mistake in a python expression of type
"my_set = my_set or set()" that didn't work as expected, i.e.
it created a new set even if my_set is already an empty set.
So, the proper expression that's needed is
"my_set = set() if my_set is None else my_set"
Change-Id: I2a787921449fecf3301013a770ffe712e9606baf
Closes-Bug: #1803677
* Once in a while we get DetachedInstanceError for action definitions
and it happens when they are fetched from cache. We must always
clone persistent objects before caching them.
Change-Id: I1d0cffea6775eb258dcefc0dbb8a6ee18effe597
Closes-Bug: #1803528
* Previously we used periodic jobs to refresh state of "join" tasks
and there was a guarantee that only one such job could run at a
time, so there wasn't a need in using locking. Now we allow more
than one such jobs run in parallel processes (and threads) so
we have to lock task execution and then check task state again
and update it, if needed.
Change-Id: Icaad486d9c3f830db0314dedb44664940cca0014
Closes-Bug: #1803483
Using "python setup.py check -r -s" method of checking the package has
been deprecated with the new recommendation to build the sdist and
wheel, then running "twine check" against the output.
Luckily, there is already a job that covers this that only runs when the
README, setup.py, or setup.cfg files change, making running this in the
pep8 job redundant. This covered by the test-release-openstack-python3
that is defined in the publish-to-pypi-python3 template.
More details can be found in this mailing list post:
http://lists.openstack.org/pipermail/openstack-dev/2018-October/136136.html
Change-Id: Iab0d2a2086cfecb8cc609c11de67ebbfc9d4d7d5
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
In case of creating/updating multiple workflows from one yaml,
we should not save the whole input to each workflow.
Closes-Bug: #1792975
Change-Id: I724c041ab3441805fcfa2cfc4a50afd774998cc7
Signed-off-by: Oleg Ovcharuk <vgvoleg@gmail.com>
* Removed using scheduler from action execution heartbeat checker
in favor of regular threads.
* Added the new config options "batch_size" under [action_heartbeat]
group to limit a number of action executions being processed during
one iteration the checker.
* Added a test checking that an action execution is automatically
failed by the heartbeat checker.
Closes-Bug: #1802065
Change-Id: I18c0c2c3159b9294c8af96c93c65a6edfc1de1a1
* Added the new property 'execution_integrity_check_batch_size'
under the [engine] group to limit the number of task executions
that the integrity checker may process during one iteration.
Closes-Bug: #1801876
Change-Id: I3c5074c45c476ebff109617cb15d56c54575dd4f
* action_queue module is replaced with the more generic
post_tx_queue module that allows to register operations that must
run after the main DB transaction associated with processing a
workflow event such as completing action.
* Instead of calling workflow completion check from all places
where task may possibly complete, Mistral now registers a post
transactional operation that runs after the main DB transaction
(to make sure at least one needed consistent DB read) right
inside the task completion logic. It reduces clutter significantly.
* Workflow completion check is now registered only if the just
completed task may lead to workflow completion, i.e. if it's the
last one in a workflow branch.
* Join now checks delayed calls to reduce a number of join
completion checks created with scheduler and also uses post
transactional queue for that.
Closes-Bug: #1801872
Change-Id: I90741d4121c48c42606dfa850cfe824557b095d0
The commands used by constraints need at least tox 2.0. Update to
reflect reality, which should help with local running of constraints
targets.
Change-Id: I0bb160bd02b876ed94a3804c88087289f9c3acc2
* Workflow completion algorithm use periodic scheduled jobs to
poll DB and determine when a workflow is finished. The problem
with this approach is that if Mistral runs another iteration
of such job too soon then running such jobs will create a big
load on the system. If too late, then a workflow may be in
RUNNING state for too long after all its tasks are completed.
The current implementation tries to predict a delay with which
the next job should run, based on a number of incompleted tasks.
This approach was initially taken because we switched to a
non-blocking transactional model (previously we locked the entire
workflow execution graph in order to change a state of anything)
and in this architecture, when we have parallel branches, i.e.
parallel DB transactions, we can't make a consistent read from
DB from neither of these transactions to make a reliable decision
about whether the workflow is completed or not. Using periodic
jobs was a solution. However, this approach has been proven to
work unreliably because such a prediction about delay before the
next job iteration doesn't work well on all variety of use cases
that we have.
This patch removes using periodic jobs in favor of using the
"two transactions" approach when in the first transaction we
handle action completion event (and task completion if it causes
it) and in the second transaction, if a task is completed, we
check if the workflow is completed. This approach guarantees
that at least one of the "second" transactions in parallel
branches will make needed consistent read from DB (i.e. will
see the actuall state of all needed objects) to make the right
decision.
Closes-Bug: #1799382
Change-Id: I2333507503b3b8226c184beb0bd783e1dcfa397f
The new openstacksdk mechanism forces a keystone request to find info
about endpoints. We don't need this for fake client, so skip the
__init__ of the class.
Change-Id: I5b0d89ac57c14f982a6afa638f088d365e0e4ab8
* In the latest version of cachetools lib (3.0.0) the previously
deprecated argument "missing" of cache classes has been removed.
* Disabled test_generator failing due to the changes in the
senlin client until it's fixed by https://review.openstack.org/614211
Change-Id: Iac42f592834734a6fddb743e947860b3bb7e1aba
* This patch removes the approach with DB polling needed to
determine if a "join" task is ready to run. Instead of running
a periodic scheduled job, each task completion now runs the
algorithm that finds all potentially affected join tasks
and schedules just one job (instead of a periodic job) to check
their readiness.
This solves a problem of system cascaded overloading in case of
having many very large joins (when a workflow has many joins with
many dependencies each). Previously, in such case Mistral created
too many periodic jobs that just didn't let the workflow progress
well, i.e. most CPU was used by scheduler to run those periodic
jobs that very rarely switched "join" tasks to the RUNNING state.
Change-Id: I5ebc44c7a3f95c868d653689dc5cea689c788cd0
Closes-Bug: #1799356
First attempt at creating the mistral install guide
Change-Id: I30142b46e36270b573b9ec10201907811040d94b
Signed-off-by: visnyei <andrea.visnyei@nokia.com>
When we rerun an execution we must create the "_check_and_complete"
delayed calls for all parent workflows. The problem was that we created
the delayed call only for the rerun execution and its parent.
Recursive rerun was extracted in the separated function. Because we
need to execute some additional operations, for example, create delayed
call for every a rerun execution.
Change-Id: I530094e916daf25bb9c672c445afa980ad4311ae
Closes-Bug: #1792451
Signed-off-by: Vitalii Solodilov <mcdkr@yandex.ru>
* Currently Mistral retries a DB transaction only in case of a DB
deadlock (often happens on MySql) and a connection error. Both
make sense to retry because the issue may be temporary. This
patch also adds sqlalchemy.exc.OperationalError to the list of
retriable exceptions since part of the errors wrapped into this
exception may also be temporary, such as "Too many connections"
error thrown by MySql. Some errors may not make sense to retry
though (like SQL error) but this shouldn't be a problem because
most of them will happen during development/testing time and
will be fixed before going in production and even if it happens
in a real production the worst thing that will happen is retrying
a DB transaction up to the maximum configured number of attempts,
currently hardcoded 50 times.
Change-Id: Ie2fe988cdb8e4ca88c3e51f510d87320d3fca9a6
Closes-Bug: #1796242
Client should be able to create a token using “auth_url” (e.g. ”https://keycloak:7443/auth”)
Server should be able to validate the token using “user_info_endpoint_url” (e.g. “https://cbnd:9443/something/custom”)
also be backward compatible
Change-Id: I437fde40345af52483cc764e5dc6a1f55f1b3e88
With moving away from required milestone releases, the version numbers
calculated by PBR on the master branch will not work for those testing
upgrades from the last stable release. More details can be found in the
mailing list post here:
http://lists.openstack.org/pipermail/openstack-dev/2018-October/135706.html
This is an empty commit that will cause PBR to increment its calculated
version to get around this.
PBR will see the following which will cause it to increment the version:
Sem-Ver: feature
Please merge this patch as soon as possible to support those testing
upgrades.
Change-Id: Ibd0fc3036014a4cfaa76d48c707ccb04cfc869a2
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
The OnClauseSpec required Task names to be \w+ or [a-zA-Z0-9_]
which is not enforced by the DSL, so it was possible to have
valid task names that could not be referenced in an on-clause.
YAML enforces some restrictions on characters in task names (#, !, |)
but other than that any JSON-schema valid string should be a valid Task
name
Change-Id: I3f1056cad7c67e160a082c2a0de2e3bfd476bc63
Closes-Bug: 1797439
Fix the issue where `next` link in the response of the REST API, when
used with pagination, gets corrupted and makes it unusable. After this
fix, next link should be readily usable.
Change-Id: Idf45a59e0b07d8306cc82391679fe30a9cd2f0c1
Closes-Bug: #1793344
A number of configuration options provided by oslo.messaging were
deprecated in Ocata and have now been removed. See
https://docs.openstack.org/releasenotes/oslo.messaging/unreleased.html#upgrade-notes
for more details.
* Because of the removal of a number of options from
[oslo_messaging_rabbit] some code related to them and the
corresponding tests for the Kombu RPC now don't make sense
and so they've been removed by this patch.
* Style/formatting changes in the Kombu RPC tests.
Change-Id: I37c71dbe4bb270367f5434b0b8c2557e29a9b1df
pbr provides VersionInfo.version_string() as a method to determine
the version. mistral.version.version_string can hide this and
provide a uniform string interface to config and launch.
This fixes a problem with mistral-server --version generating
an exception because the version passed to argparse was a method
instead of a string.
Change-Id: Ie468685e4360bfaec5d82b02f8cf1a27a93bcd94
Closes-Bug: 1796921
Quotes around {posargs} cause the entire string to be combined into one
arg that gets passed to stestr. This prevents passing multiple args
(e.g. '--concurrency=16 some-regex')
Change-Id: I25ebe667cce8a2a35f9b119b76bbed7851e458f5
This fixes a bad weird condition in the API server related to
cron-triggers and SIGHUP. The parent API server creates a RPC connection
when creating workflows from cron. If a SIGUP signal happens after, the
child inherits the connection, but it's non-functional.
Change-Id: Ic98e2db02abd8483591756d73e06784cc2e9cbe3
Closes-Bug: #1789680
The oslo.policy library exposes entry points so that users can
generate sample policy files and templates. The entry points do
expect some things to be done by the service in order to work,
though.
This commit adds an entry point for oslo.policy so that it can
consume an enforcer that has been initialized with mistrals
policies. The library will use this to generate useful things
for users like templates and sample policy files.
Change-Id: Ib442fbb79b5c237d634586c3169cf8c7f595da1c
Closes-Bug: 1793346