3923 Commits

Author SHA1 Message Date
tonybrad
a0cada98e1 Change mistral opendev.org to releases.openstack.org
Change-Id: Ic5faf47b13f38b54d5c9ae2d8476ba86c664b074
2019-06-11 11:09:53 +08:00
Zuul
d1d4087a3f Merge "Simple optimization of creating/updating workflows" 2019-06-10 15:27:51 +00:00
Zuul
30b660bedb Merge "Reduce number of fields in notification data" 2019-06-10 12:33:32 +00:00
Zuul
3b37eea14b Merge "Delete delayed calls for deleted entities" 2019-06-10 12:33:31 +00:00
Oleg Ovcharuk
9a2f3b5715 Simple optimization of creating/updating workflows
In case we load definition of one workflow, there is no need to convert
it to yaml and back.

Change-Id: Ia7fb914843e56650c8f6ff5e8c16d1b57001c0fc
Signed-off-by: Oleg Ovcharuk <vgvoleg@gmail.com>
2019-06-10 11:50:43 +03:00
Oleg Ovcharuk
bea98f6f85 Reduce number of fields in notification data
It is very important to prevent sending huge messages via rpc.
Notification data has a lot of potentially large fields, such as output,
spec and etc. Most of them are useless as a part of notification message
so we should omit them.

Change-Id: I0762a465742fcb1d1dc8d4d00d81cf8288b842ca
Signed-off-by: Oleg Ovcharuk <vgvoleg@gmail.com>
2019-06-10 11:19:05 +03:00
Oleg Ovcharuk
475b82c532 Delete delayed calls for deleted entities
Delayed calls for nonexistent entities should not fail; they should do
nothing and be deleted in normal way.

Change-Id: I1b818d671468b95ce8ae06416b57fd4a22cc6eb2
Signed-off-by: Oleg Ovcharuk <vgvoleg@gmail.com>
2019-06-10 11:14:31 +03:00
pengyuesheng
1e83a07696 Exclude broken ironicclient versions 2.5.2 and 2.7.1
depend on https://review.opendev.org/#/c/659612/

Change-Id: Ie1ba340c23156373562768a5da5c07fdde35e897
2019-06-10 13:23:24 +08:00
Zuul
5131444a62 Merge "Use get_task_executions_count for any_cancels method" 2019-06-08 01:07:59 +00:00
Zuul
c7a10c2554 Merge "Allow to use 'task()' yaql function in task policies" 2019-06-07 11:03:12 +00:00
Zuul
ba29d58335 Merge "Rework updating action executions heartbeats" 2019-06-06 23:03:47 +00:00
Oleg Ovcharuk
5f86cf1f5c Allow to use 'task()' yaql function in task policies
Change-Id: Ifdf032a9e1377ed73b80cc26fd659aabb417cfc5
Signed-off-by: Oleg Ovcharuk <vgvoleg@gmail.com>
2019-06-06 23:43:16 +03:00
Mike Fedosin
15355ea5e1 Use get_task_executions_count for any_cancels method
To get the result we need only a number of tasks, so there is no
need to fetch full task executions from the database.

Change-Id: I34ea985e6de08f46e1c40c64996357c8e6f3d79c
2019-06-06 15:46:34 +02:00
Zuul
f85caa02c9 Merge "Skip context evaluation for non-conditional transitions" 2019-06-06 10:09:45 +00:00
Zuul
e35407a6d6 Merge "Bump openstackdocstheme to 1.30.0" 2019-06-06 10:09:44 +00:00
Mike Fedosin
e5031e26f1 Rework updating action executions heartbeats
New function update_action_execution_heartbeat was introduced in db api
to optimize updating of heartbeats.

Now to perform the update mistral downloads an action execution object,
then modifies it, and finally  uploads back to the db using ORM.
This function removes initial object fetching and changes ORM update to
sqlalchemy core.

Change-Id: I4cb2d85c193531648540a37caca9ee868ede3a08
2019-06-06 07:04:27 +00:00
翟小君
6dc981d814 Bump openstackdocstheme to 1.30.0
Some minor updates including fixing issues with the HTML5 output
generated by Sphinx 2.0, inlining the OpenStack logo rather than
referencing it from elsewhere, and adding a new SOURCE_DATE_EPOCH option
for use in reproducible builds in new release openstackdocstheme 1.30.0.
...to pick up many improvements, including the return of table borders.

Change-Id: I22b652e1eb007616e697cb238fcd9e5f010cd693
2019-06-06 14:27:41 +08:00
pengyuesheng
0731cd1d87 Blacklist sphinx 2.1.0 (autodoc bug)
See https://github.com/sphinx-doc/sphinx/issues/6440 for upstream details
Depend-On: https://review.opendev.org/#/c/663060/

Change-Id: Ia9ab0cfb134f0f6ca168e8e746582fe1fc7c9710
2019-06-06 11:02:02 +08:00
Mike Fedosin
f09c8ebec1 Skip context evaluation for non-conditional transitions
Context evaluation is not required for non-conditional transitions,
because the list of next tasks is known in advance.
For this reason we can skip this operation, that can be quite
time-consuming, and generate a list of task directly.

Also a little cleanup was done to remove unnecessary methods.
Change-Id: Ia419c47a7d71db46a5cae557fe8bc7512390715e
2019-06-05 18:31:56 +02:00
Zuul
55c7990684 Merge "Remove _get_next_clauses" 2019-06-05 12:37:54 +00:00
Zuul
38d0abb139 Merge "Add release notes to changes to wflanguage execution object" 2019-06-05 11:22:15 +00:00
Mike Fedosin
c1e4fd8d48 Remove _get_next_clauses
A little code clean-up: this commit replaces _get_next_clauses method
from direct_workflow.py with a built-in find_outbound_task_names

Change-Id: Ic5ea956ee45b8c455d42669c0f0e386d53e22425
2019-06-05 04:15:42 +00:00
Zuul
ea4681500a Merge "Move action caching to db layer" 2019-06-04 15:05:52 +00:00
Zuul
a05ee2af0f Merge "Prepare cache for _is_upstream_task_execution" 2019-06-04 13:27:44 +00:00
Zuul
153d32549f Merge "Remove _find_task_execution_by_name" 2019-06-04 13:18:38 +00:00
apetrich
9289f7d28e Add release notes to changes to wflanguage execution object
Change-Id: I976fe5723a8bad81f1c6e30618475aa34e3e2050
depends-on: Ia6c954e688589f69a7463f1b8e02244d029e8b7a
2019-06-03 13:58:43 +02:00
Mike Fedosin
58b714eb16 Prepare cache for _is_upstream_task_execution
This trick is similar to what we do in _get_join_logical_state.
Instead of bypassing the entire workflow graph and downloading
tasks executions one by one, we prepare a lightweight cache with
a single database query and then analyze it on the spot.

Change-Id: If1adb4ea408248e766bff449637de2b1b1356738
2019-06-02 23:18:07 +02:00
Mike Fedosin
cd19e48697 Remove _find_task_execution_by_name
This method was used to find a task execution by its name in
_is_upstream_task_execution only. But this operation can be
simplified, because we already have the task execution,
and therefore there is no need to search for it again in the
database.

Change-Id: I45a175970bc2567f5ef63e161fa431c8e9245ec1
2019-06-02 23:05:52 +02:00
Mike Fedosin
92528a82ef Move action caching to db layer
Now action caching is used only in engine.actions, but actually
there are a lot of place where this cache can be useful.

This commit moves the caching code to the db layer, that allows it to
be used in all other places, where load_action_definition is called.

Change-Id: I8c470cbb424e63c16e599758bb93192985666353
2019-06-02 22:38:02 +02:00
Oleg Ovcharuk
a75b07ec5f Send task and workflow notifications out of the transaction
Change-Id: I5c0781a0e54e718e1b7e708150b06487d9297cc7
Signed-off-by: Oleg Ovcharuk <vgvoleg@gmail.com>
2019-05-31 12:28:00 +03:00
Zuul
e25da33607 Merge "Fix adhoc action lookup" 2019-05-31 08:10:03 +00:00
Zuul
654075e553 Merge "Rework finding indirectly affected created joins" 2019-05-30 09:21:50 +00:00
Mike Fedosin
c52688523c Rework finding indirectly affected created joins
Current algorithm of finding the joins requires a recursive search
through the execution graph, which leads to a large number of calls
to the database.

To optimize it we introduce a new algorithm that requires only one db
request. It downloads all potencial join task ids and names, and then
analyzes them without any additional db calls.

Change-Id: Ic73f2112406e681ae8a2aa67bcbccebe488fc03c
2019-05-29 23:58:00 +02:00
Mike Fedosin
2712b8c151 Fix adhoc action lookup
Now to find an adhoc action definition by name action hanler takes
the name in runtime context first, and then provides this value along
with workflow name and workflow spec name to resolve_action_definition
method.

This method tries to find a definition by full name, i.e.
"wb_name.action_name" if wf_name != wf_spec_name, and, if nothing was
found, looks for a definition just by name, i.e. "action_name".

There is an issue, because runtime context already contains full name
for the action and substitution of a workbook name as a prefix is not
necessary.

It leads to the fact, that for all adhoc actions mistral tries to find
them by "my_wb_name.my_wb_name.my_action_name" first, and then, when it
fails, it looks for "my_wb_name.my_action_name".

This results in double the number of queries to the database and highly
innefficient.

This commit solves the issue by not providing workflow name and workflow
spec name to resolve_action_definition method for adhoc actions.
After that, the method will search for the action immediately by its
name.
For system actions this substitution is not necessary there either, so
the code can be safely deleted as well.

Change-Id: I3b880ef29057afb041edd9095c279a944c77f2dd
2019-05-29 17:03:38 +02:00
pengyuesheng
424a0a55a0 Blacklist python-cinderclient 4.0.0
This release of the Cinder client broke support for the v3
volume-transfer APIs unless microversion 3.55 or higher was requested.
depend on https://review.opendev.org/#/c/587877/

Change-Id: I26bbb2e5ef5f8a5f963b47385c81a64c0227f7b8
2019-05-27 08:01:00 +00:00
Zuul
49a2765383 Merge "Constraint networkx to <2.3 for Python 2" 2019-05-27 08:00:06 +00:00
Hervé Beraud
831162ac0a Constraint networkx to <2.3 for Python 2
NetworkX released 2.3, dropping support for Python 2.

 https://github.com/networkx/networkx/blob/master/doc/news.rst

 Also see I0a9700926c9a0db93e782c853c33f1aaee3d4876

 fixed sphinx requirements also according to global requirements
 
 jsonschema should be uncaped also according to global requirements
 but there are conflicts with zakarclient designateclient and irconicclient
 
 so for now I put it in remark

Change-Id: Iea3dbcfa1f6ddf2c413ede5e73fa076c6edccfdf
2019-05-23 09:10:07 +00:00
Mike Fedosin
84b8e92acc Get rid of lookup utils
With the new joining mechanism lookup utils for task executions
become useless and can be removed from the codebase.
Now all requests to the database will be perforemed directly from
a workflow controller.

Action defention cache was moved to mistral/engine/actions.py,
because it's needed only there.

Change-Id: If0d4403f5c61883ecfec4cfa14b98cc39aae5618
2019-05-22 17:11:49 +02:00
Zuul
fefdcd77cd Merge "Optimize searching of upstream task executions" 2019-05-22 07:08:31 +00:00
Renat Akhmerov
a477fe49cd Fix how "has_next_tasks" is calculated for task executions
* There was a weird typo in the list generator expression made in
  https://review.opendev.org/#/c/652575 that led to calculating
  a field value in the wrong way. Fixed. The added test was
  previously failing.

Closes-Bug: #1829841

Change-Id: I6b8ea8dfa623acbb56f649630ddee62e43d484eb
2019-05-21 17:24:52 +07:00
Mike Fedosin
8549aeaf66 Optimize searching of upstream task executions
Now to find upstream task executions mistral does a lookup for each
inbound task spec. This is not optimal, because it leads to a significant
number of db requests.

To optimize this behavior we prepare a list of task names in advance,
and then search executions using 'in' filter.

Change-Id: Ia7bf62c45b889f753671bdda048f91c46af41039
2019-05-20 18:09:55 +02:00
Zuul
26b9cc2bac Merge "Rework joining mechanism" 2019-05-20 03:44:57 +00:00
Zuul
f5d1ebd1bb Merge "Pass auth context to the event publishers" 2019-05-17 13:46:37 +00:00
Mike Fedosin
ff00c9c778 Rework joining mechanism
Current joining mechanism in some cases can be expensive because
it uses a multi-step recursive search, which leads to a huge amount
of db requests.

This work changes this behavior by precaching required task
executions to prevent hammering the database during the lookup.

Change-Id: I2d1b7e72c728a14c85b015dfdb0f8800b95f3749
2019-05-17 12:24:12 +02:00
apetrich
dbc4c91150 Remove deprecated nova commands
novaclient.cert was removed 18 months ago
4bc4078fcb
this commit removes those calls.

Change-Id: Ie5159ba52b6a682e00027ce6bf6d3c5c3f4f1eb9
2019-05-16 16:25:37 +02:00
Zuul
e51730b511 Merge "Reduce the number of "on-xxx" evaluations" 2019-05-16 08:53:12 +00:00
Zuul
6dcc048725 Merge "Adds private_key parameter in the standard ssh actions" 2019-05-16 06:40:47 +00:00
Zuul
1f704c8fd7 Merge "inspect.getargspec is deprecated on py3" 2019-05-15 10:02:59 +00:00
Renat Akhmerov
83c541acbf Reduce the number of "on-xxx" evaluations
* Mistral evaluates expressions under "on-xxx" clauses more than
  once during the processing of a workflow. It's now been fixed by
  storing crucial information about the result of those expressions
  after it was first obtained in the DB.
* Added two boolean fields "has_next_tasks" and "error_handled" in
  the TaskExecution class and the required migration. These fields
  allow not to calculate expressions under "on-xxx" clauses many
  times which leads to reducing execution time in case of heavy
  expressions and/or their data contexts.
* Minor style changes.

Closes-Bug: #1824121
Change-Id: Ib236ba7a72d8e578f9c52460d2a7d8d4540f9c37
2019-05-15 07:39:05 +00:00
Jose Castro Leon
eb59216281 Removes insecure parameter from barbican client
This removes the parameter insecure in the client creation as this
no longer exists in the client. It also changes the client class
to be called as other clients in openstack using sessions

Change-Id: I3cb7ed4255f8996b8bfd9a3e1edba6de50f4e492
Closes-Bug: #1800819
2019-05-13 06:54:39 +00:00