123 Commits

Author SHA1 Message Date
Renat Akhmerov
1a4c599a4d Improve join by removing periodic jobs
* This patch removes the approach with DB polling needed to
  determine if a "join" task is ready to run. Instead of running
  a periodic scheduled job, each task completion now runs the
  algorithm that finds all potentially affected join tasks
  and schedules just one job (instead of a periodic job) to check
  their readiness.
  This solves a problem of system cascaded overloading in case of
  having many very large joins (when a workflow has many joins with
  many  dependencies each). Previously, in such case Mistral created
  too many periodic jobs that just didn't let the workflow progress
  well, i.e. most CPU was used by scheduler to run those periodic
  jobs that very rarely switched "join" tasks to the RUNNING state.

Change-Id: I5ebc44c7a3f95c868d653689dc5cea689c788cd0
Closes-Bug: #1799356
2018-10-23 14:01:39 +07:00
hardikj
08ba20c8d2 Fix next link in get resource list rest API
Fix the issue where `next` link in the response of the REST API, when
used with pagination, gets corrupted and makes it unusable. After this
fix, next link should be readily usable.

Change-Id: Idf45a59e0b07d8306cc82391679fe30a9cd2f0c1
Closes-Bug: #1793344
2018-10-12 16:04:00 +05:30
Dougal Matthews
f0b49a82b4 Add a release note for Ic98e2db02abd8483591756d73e06784cc2e9cbe3
Change-Id: I484b47f9e0d94c37079961d18061e2e9e121af92
2018-10-01 10:56:08 +01:00
Renat Akhmerov
dfdff78315 Fix how Mistral calculates workflow output
* Workflow output sometimes is not calculated correctly due to
  the race condition between different transactions: the one that
  checks workflow completion (i.e. calls "check_and_complete") and
  the one that processes action execution completion (i.e. calls
  "on_action_complete"). Calculating output sometimes was based on
  stale data cached by the SQLAlchemy session. To fix this, we just
  need to expire all objects in the session so that they are
  refreshed automatically if we read their state in order to make
  required calculations. See the bug description for more details
  on how the problem was observed.
* Added another test for direct workflow that formally checks
  calculation of workflow output. It doesn't pretend to test the
  aforementioned issue (it can be reproduced only with a big number
  of attempts, and/or under load). It's for the sake of the test
  module completeness.

Change-Id: I4a7e7fd9a4bbb6e93df169b4b40bc2d83ccfce89
Closes-Bug: #1792090
2018-09-12 16:48:43 +07:00
Pierre Gaxatte
5a70238ede Add release note for auth_context bugfix
Bug 1785654 has been fixed in https://review.openstack.org/#/c/599606
but there was no release note. This change adds it.

Change-Id: I7bea9553c41336ba08d6fec551055290530769a9
2018-09-05 14:12:45 +02:00
Zuul
78dcb2d419 Merge "Performance: remove unnecessary workflow execution update" 2018-09-03 13:25:39 +00:00
Renat Akhmerov
8f40b01cb6 Performance: remove unnecessary workflow execution update
* This fix drastically improves performance since it eliminates
  an unnecessary update of the workflow execution object when
  processing "on_action_complete" operation. W/o this fix
  all such transactions would have to compete for the workflow
  executions table that causes lots of DB deadlocks (on MySQL)
  and transaction retries. In some cases the number of retries
  even exceeds the limit (currently hardcoded 50) and such tasks
  can be fixed only with the integrity checker over time.
  See the code comment in the "dispatcher.py" module that explains
  why this specific change eliminates the update of the workflow
  execution object.
* Style changes in api.py and aciton_execution_checker.py

Closes-Bug: #1790079

Change-Id: I08cb561e252d31e35fcfb61984d87a7bfc387a4d
2018-09-03 11:47:11 +07:00
Dougal Matthews
b16a4ce465 Explicitly convert X-Target-Insecure to a boolean
We will now only accept the string values "False" or "True". Previously
any given value was interpreted as a string and thus True.

Closes-Bug: #1666565
Change-Id: Ibd105c881dbe16cd4516bfb775c8f5f43c961b45
2018-08-22 09:48:30 +01:00
7b5bffe7c0 Update reno for stable/rocky
Change-Id: Ic8776b1a940da5379f9663964af1015385000949
2018-08-09 12:11:31 +00:00
Zuul
bef9aa5bc3 Merge "Improve std.email action" 2018-07-31 14:32:33 +00:00
Zuul
443cd6fc11 Merge "Support Manila actions in Mistral" 2018-07-31 11:30:40 +00:00
Zuul
08a61b6f10 Merge "Add a policy to control the right to publish resources" 2018-07-31 08:51:12 +00:00
Jose Castro Leon
3c430ef0a2 Improve std.email action
Adds support for cc and bcc addresses to send mails as copy to
administrators and also html formatting. If the html body is specified
the mail will be sent as multipart.

Closes-Bug: #1783349

Change-Id: I2b90354c33052c4b7ae3a98a08e7df1055524a25
2018-07-31 08:43:03 +02:00
hardikj
834747b5d9 Add namespace support for workbooks
This patch brings namespace support to workbooks.
Namespace of the workbook is inherited by workflows.

Implements: blueprint mistral-namespace-for-actions-workbooks
Change-Id: I2c66b3961915f0f35a9c468eb6dd0c0c70995234
2018-07-26 17:13:58 +05:30
Jose Castro Leon
a2756a34c2 Support Manila actions in Mistral
Adds manila actions into the available list of actions for mistral

Change-Id: Ic0a3c24f72d91a8a87ffcf81db763058bcbf8566
Closes-Bug: #1783291
2018-07-24 15:11:08 +02:00
wangxiyuan
dcb4355e59 Use register_session_conf_options API
Using this API is needed to correctly initialize the configuration.

[keystone_authtoken] group is used for keystonemiddleware to validating
token.

The new [keystone] group is used for keystoneauth to initialization
the keystone session.

Co-Authored-By: wangxiyuan<wangxiyuan@huawei.com>

Change-Id: Ie3ab512b0ab42c0f97b3099e0787f4edee08ddce
Partial-Bug: #1775140
2018-07-20 09:22:59 +08:00
Renat Akhmerov
d3305c709e Release note for adding "oslo_rpc_executor" config option
Change-Id: Iddd02f3f428cc0ab68a7a301cf5f5db19f6a083a
Closes-Bug: #1696469
2018-07-19 12:08:04 +07:00
Pierre Gaxatte
522f388709 Add a policy to control the right to publish resources
This change intends to add the following policies:
- actions:publicize
- workflows:publicize

A refactor of the policy tests is included to increase the coverage.

Change-Id: I8bd637f7de63f02c63f3c304cb6f5198fc0c5f42
Closes-Bug: 1779293
2018-07-05 11:46:52 +02:00
Cong Phuoc Hoang
49ccf8ea7b Add missing Tacker actions to Mistral
Add missing Tacker actions to Mistral that includes vnf forwarding
graph (vnffg), vnffg descriptor, network service (ns) and ns
descriptor actions

Change-Id: I0b0f214ad3f3990fa0af98923d34efebf6e2f8f3
2018-06-29 10:40:26 +09:00
Kupai József
ce18c0bf67 A mechanism to close stuck running action executions
If an executor dies while running an action execution, then the
execution will remain in RUNNING state (because the dead executor
can't signal the error).

Implements blueprint: action-execution-reporting

Change-Id: I51b4db6aa321d0e53bbb85a74f8ebaea0376d22e
2018-06-12 14:15:15 +00:00
Renat Akhmerov
360fd8bd61 Release note for adding YAQL engine options
Change-Id: I8de0c011c94b9c399e12e010e1da6dd6f4459ebc
2018-06-01 17:07:10 +07:00
Toure Dunnon
d3c6bf7767 Release note for using "passive_deletes=True"
This update includes a small change to mention memory
savings for the new configuration option.

Change-Id: Ic8d55adbfb802b8aff2000c2e4edf279c53f6bde
2018-05-17 10:43:32 -04:00
Zuul
a88ff9f80a Merge "Support Qinling actions in Mistral" 2018-05-17 06:18:48 +00:00
Lingxian Kong
4fe4198ac8 Support Qinling actions in Mistral
An example can be found in the task description.

Remove the cap on networkx library according to:
13644d63c6 (diff-d7d5c6fa7118ea10d88f3afeaef4da77)

Depends-On: https://review.openstack.org/#/c/567971/
Change-Id: I168a32b8c73b4b30224ce8f27bcde9cfa2b8d2ad
Story: 2002014
Task: 19654
2018-05-16 21:27:42 +12:00
Renat Akhmerov
696e82702c Release note for using "passive_deletes=True"
Related-Bug: #1757966

Change-Id: I013d4837603839364f8d699f7ce97e4400879d38
2018-05-16 13:58:40 +07:00
Zuul
6652a01efc Merge "Support actions for zun" 2018-05-03 17:07:27 +00:00
Zuul
1e577ae739 Merge "Release note for workflow environment optimizations" 2018-05-02 10:29:56 +00:00
Zuul
a7910a698c Merge "Only allow for deleting completed executions" 2018-04-27 14:22:51 +00:00
Renat Akhmerov
cb1cabbfe6 Release note for workflow environment optimizations
Change-Id: Ieff4887384a4f8fa0eb52a6255f3e4402dbf1d8e
Related-Bug: #1757966
2018-04-27 20:08:57 +07:00
pengdake
11896ba733 Support actions for zun
Add ZunAction for zun. Zun is an OpenStack Container service. It aims to
provide an API service for running application containers without the
need to manage servers or clusters.

Change-Id: Ibbe4777528b5af6fdb3a6c2ed3345f8f9c804bef
Signed-off-by: pengdake <19921207pq@gmail.com>
2018-04-27 09:52:19 +08:00
Zuul
94e20726b2 Merge "Add release note for jinja expression bug" 2018-04-26 11:23:58 +00:00
hardikj
a70df9e991 Add release note for jinja expression bug
Change-Id: I1b89ef4bdd081e0a32c6442b4d142fa23dceb43c
2018-04-26 15:27:25 +05:30
Adriano Petrich
af84fa9181 Only allow for deleting completed executions
A force parameter was added in case deleting a running execution is
needed but that will spew all kinds of errors afterwards
The default is force = False

Change-Id: I19d822800a1ee056682daeaa7ddf0f400392525d
Closes-bug: 1598135
Depends-On: Id935265fcbe0a3072ba2d693edf54b892173fbdf
2018-04-26 09:37:18 +00:00
Zuul
85a24b20a6 Merge "Added 'safe-rerun' policy to task-defaults section" 2018-04-25 15:56:53 +00:00
Zuul
dbbcb27491 Merge "Added Vitrage actions into Mistral Implements: blueprint mistral-vitrage-actions" 2018-04-25 10:33:36 +00:00
Jaewook Oh
f25fb43177 Added Vitrage actions into Mistral
Implements: blueprint mistral-vitrage-actions

Change-Id: I11068c1f1a0d7ee6973e7c736ba92c475f7d0902
2018-04-24 22:41:04 +09:00
Vitalii Solodilov
5f89e2e71f Added JavaScript evaluator which doesn't require a compilation
* Added new JavaScript evaluator py_mini_racer. Advantages:
    * is distributed as wheel package
    * supports differences platforms
    * live project
* BUILD_V8EVAL was removed because it was replaced by py_mini_racer in
Mistral Docker image
* Added stevedore integration to javascript evaluators
* Refreshed javascript tests. Add test for py_mini_racer evaluator
* Install py_mini_racer library in during mistral test
* Refreshed javascript action doc

Change-Id: Id9d558b9b8374a2c2639e10cb1868f4e67f96e86
Implements: blueprint mistral-add-py-mini-racer-javascript-evaluator
Signed-off-by: Vitalii Solodilov <mcdkr@yandex.ru>
2018-04-23 17:04:18 +00:00
Vitalii Solodilov
a29c7d9f7f Added 'safe-rerun' policy to task-defaults section
Change-Id: Ib6641bda008a813660af5584c9319fcaddcd416f
Signed-off-by: Vitalii Solodilov <mcdkr@yandex.ru>
2018-04-23 20:46:43 +04:00
Renat Akhmerov
5073274dd1 Release note for batched evaluation of final workflow context
Change-Id: I861d99c5a4aa6bf1f80b2eacf5dd7d0a110aef61
2018-04-20 18:32:19 +07:00
Zuul
0b1e0bd626 Merge "Release note for not persisting '__task_execution' in DB" 2018-04-19 10:45:21 +00:00
Renat Akhmerov
e42c515a5f Release note for not persisting '__task_execution' in DB
Change-Id: I55bdc141e0d8ca73df06aa8dfaec49e8798366f9
2018-04-18 13:11:59 +07:00
Dougal Matthews
15e95d9b3c Correct the string formatting in a info log message
It should be "%s," not just "%,". The error observed was something
like...

ValueError: unsupported format character ',' (0x2c) at index 91

Closes-Bug: #1764696
Change-Id: I601c8640cec857746c2c17f94116da890f4d65e0
2018-04-17 09:41:35 +00:00
Vitalii Solodilov
e7da5b6dc8 Add a release note for new indexes
Change-Id: I53b70b741134326301e40e3b25dd9733adca265e
Signed-off-by: Vitalii Solodilov <mcdkr@yandex.ru>
2018-04-08 20:22:44 +04:00
Renat Akhmerov
8b30743d0a Add a release note for the 'pause' command fix
Change-Id: Ia609cb319a01b3685d91515c8d9d81e3006a001e
Related-Bug: #1714054
2018-03-28 09:06:48 +00:00
Dougal Matthews
422c89a05a Add a release note for execution events noitifications
This is for I9820bdc4792a374dad9ad5310f84cd7aaddab8ca

Change-Id: I3157e6414a9b58f74178450f40a2e79f41dc5de1
2018-03-12 18:25:59 +00:00
wangqi
6683e154bb Remove duplicate word 'the'
Change-Id: I1bcce8034fa7af6af99a5eb3e8292c714da89e51
2018-02-24 04:13:19 +00:00
Zuul
1f83e5d5a2 Merge "Cache action definitions" 2018-02-15 10:14:19 +00:00
2de17cb760 Update reno for stable/queens
Change-Id: Ic780f945537ed07b0ba58bd58eaddf5a1a1693a3
2018-02-14 13:42:32 +00:00
Mike Fedosin
1ae082794a Cache action definitions
Now to perform some action mistral gets its definition from
the database first. It's not really optimal, because if there are
a lot of similar action calls, mistral will reread the same data
from db. It increases the whole execution time and the load on the
database.

To improve the performance it's suggested to cache read definitions
and take them from the cache instead of the database in the subsequent
times.

Cache ttl can be configured with ``action_definition_cache_time``
option from [engine] group. The default value is 60 seconds.

Change-Id: I330b7cde982821d4f0a06cdd2954499ac0b7be37
2018-02-12 16:16:54 +01:00
Pradeep Kilambi
16de7e004f Remove ceilometer actions from mistral
ceilometer api and client have been deprecated for over
two releases and now removed completely. Lets drop these
actions and update the requirements.

Change-Id: Ica2b835a885b9b4705996f91080afc12587bd314
2018-01-11 14:28:16 -05:00