167 Commits

Author SHA1 Message Date
3846b9e6ed Update master for stable/wallaby
Add file to the reno documentation build to show release notes for
stable/wallaby.

Use pbr instruction to increment the minor version number
automatically so that master versions are higher than the versions on
stable/wallaby.

Sem-Ver: feature
Change-Id: I58e157deef1aac29cf7d53af54d4726824419cc4
2021-03-26 18:42:22 +00:00
Ghanshyam Mann
7cf2014c2e [goal] Deprecate the JSON formatted policy file
As per the community goal of migrating the policy file
the format from JSON to YAML[1], we need to do two things:

1. Change the default value of '[oslo_policy] policy_file''
config option from 'policy.json' to 'policy.yaml' with
upgrade checks.

2. Deprecate the JSON formatted policy file on the project side
via warning in doc and releasenotes.

Also replace policy.json to policy.yaml ref from doc and tests.

[1]https://governance.openstack.org/tc/goals/selected/wallaby/migrate-policy-format-from-json-to-yaml.html

Change-Id: I3b9aeb3379a76f7e40dab0c46e27f4447a0c3d03
2020-12-22 16:47:46 +07:00
ali
9be4f8e119 created a new Api endpoints and added dynamic actions
* added dynamic actions:
     these actions are created and modified in runtime,
     each action needs a code source to be imported from and a
     class name.

 - there are 2 new endpoints:
    - /v2/code_sources/:
       used to add new code sources to mistral.
    - /v2/dynamic_actions/:
      used to add dynamic actions to mistral in runtime

 - a new Action provider (DynamicActionProvider) was added:
    it provides the actions created from the dynamic actions api.

Change-Id: I9fe8c28ffdef71016d9dc13aea60a288c8ebaa0a
Signed-off-by: ali <ali.abdelal@nokia.com>
2020-11-18 10:15:26 +00:00
0a74fb365b Update master for stable/victoria
Add file to the reno documentation build to show release notes for
stable/victoria.

Use pbr instruction to increment the minor version number
automatically so that master versions are higher than the versions on
stable/victoria.

Change-Id: Id9fc114b680e47e494f9ec8869c176152ccf764a
Sem-Ver: feature
2020-09-25 12:28:23 +00:00
Renat Akhmerov
068e5cc0e9 Add release notes for the action provider transition
Change-Id: Icc79a07386de08e396351e78f27ccbde72f93a2e
2020-09-25 14:45:34 +07:00
Andreas Jaeger
4b52d4d458 Switch to newer openstackdocstheme and reno versions
Switch to openstackdocstheme 2.2.1 and reno 3.1.0 versions. Using
these versions will allow especially:
* Linking from HTML to PDF document
* Allow parallel building of documents
* Fix some rendering problems

Update Sphinx version as well.

Set openstackdocs_pdf_link to link to PDF file. Note that
the link to the published document only works on docs.openstack.org
where the PDF file is placed in the top-level html directory. The
site-preview places the PDF in a pdf directory.

Disable openstackdocs_auto_name to use 'project' variable as name.

Change pygments_style to 'native' since old theme version always used
'native' and the theme now respects the setting and using 'sphinx' can
lead to some strange rendering.

Remove docs requirements from lower-constraints, they are not needed
during install or test but only for docs building.

openstackdocstheme renames some variables, so follow the renames
before the next release removes them. A couple of variables are also
not needed anymore, remove them.

Remove python 3 versioning from lower-constraints and
requirements to pass requirements-check.

See also
http://lists.openstack.org/pipermail/openstack-discuss/2020-May/014971.html

Change-Id: Id478711abb1783422ebb4ee9937d1b7202fdf2f3
2020-05-21 15:50:23 +02:00
b56576eef6 Update master for stable/ussuri
Add file to the reno documentation build to show release notes for
stable/ussuri.

Use pbr instruction to increment the minor version number
automatically so that master versions are higher than the versions on
stable/ussuri.

Change-Id: Ie516ee33ca126c6d9750ecdfc1fbb99051fc4a97
Sem-Ver: feature
2020-04-23 12:21:21 +00:00
Eyal
8bdf341af7 Remove OpenStack actions from mistral
Depends-on: https://review.opendev.org/#/c/703296/
Depends-On: https://review.opendev.org/#/c/704280/
Change-Id: Id62fdabe7699e7c3b2977166e253cfc77779e467
2020-02-26 10:12:01 +02:00
Renat Akhmerov
253fd978c5 Release note for "convert_output_data" config option
Change-Id: I99d30434c9d337529edae041427de05ffcf2ffd6
Closes-Bug: #1863857
2020-02-19 09:27:25 +00:00
ali
06e776e6e3 Added a new API to fetch sub-execution of an execution or a task
*the new endpoints are:

   v2/executions/{ex_id}/executions
   v2/tasks/{task_id}/executions

  it returns a list of execution resources, by default the API will
  return all sub-executions

  the API can take 3 parameters:

  errors_only: returns only the error routes
               - default is False
  include_output: includes the output field for the returned executions
                  - default is False
  max_depth: the max depth for the returned executions
             - if a negative value is given, then the API will return
               all sub-executions
             - default is -1

 Partially-Implements: blueprint mistral-execution-origin

Change-Id: I2a4c5e6890dbb4de868ce885d51341b15e359233
Signed-off-by: ali <ali.abdelal@nokia.com>
2020-01-23 08:50:28 +00:00
ali
20c3408692 Add namespaces to Ad-Hoc actions
added namespace for the actions, actions can have the same name if they
 are not in the same namespace, when executing an action, if an action
 with that name is not found in the workflow namespace or given
 namespace mistral will look for that action in the default namespace.

  * action base can only be in the same namespace,or in the
    default namespace.
  * Namespaces are not part of the mistral DSL.
  * The default namespace is an empty string ''.
  * all actions will be in a namespace, if not specified, they will be
    under default namespace

Depends-On: I61acaed1658d291798e10229e81136259fcdb627
Change-Id: I07862e30adf28404ec70a473571a9213e53d8a08
Partially-Implements: blueprint create-and-run-workflows-within-a-namespace
Signed-off-by: ali <ali.abdelal@nokia.com>
2020-01-07 08:10:53 +00:00
Renat Akhmerov
d1b8c8bc2c Add a release note for graceful scale-in feature
Change-Id: I9874085555db097c7ae1ae3c2c7cad73b8543ff5
2019-12-09 13:00:22 +07:00
Ghanshyam Mann
fa2042a821 [ussuri][goal] Drop python 2.7 support and testing
OpenStack is dropping the py2.7 support in ussuri cycle.

Mistral is ready with python 3 and ok to drop the
python 2.7 support.

Complete discussion & schedule can be found in
- http://lists.openstack.org/pipermail/openstack-discuss/2019-October/010142.html
- https://etherpad.openstack.org/p/drop-python2-support

Ussuri Communtiy-wide goal:
https://governance.openstack.org/tc/goals/selected/ussuri/drop-py27.html

Change-Id: Ica51b3b36f3784b4b04731406b6bf8320d5de4b8
2019-11-16 14:07:57 +00:00
Oleh Huzei
948ceb150b Add release note for fix error of cron trigger run
It relates to the change I793fbfec03032d9ff7137c11becb6d1c18ec54bc,
commit 51b7dd0c6d.

Change-Id: I2571246947ab5ae06731fc7d35de7a5954325a82
2019-10-16 11:32:28 -04:00
Renat Akhmerov
8f02e8a7d7 Change the action error message format
* Some users rely on the presence of the root error related to
  running an action and it's not convenient that it is now in
  the end of the string, e.g. if we look at the corresponding
  task execution "state_info" field. This patch includes the cause
  error message in the beginning of the resulting error string
  returned by the action executor so that it's clearly visible.
  This message can be also truncated in some cases (depending on
  the config option) so we need to make sure we keep the cause
  error message.

Closes-Bug: #1847984
Change-Id: Ieb10c10401380410665c418f4688681e929b1e23
2019-10-14 17:08:44 +07:00
e4f62dab1d Update master for stable/train
Add file to the reno documentation build to show release notes for
stable/train.

Use pbr instruction to increment the minor version number
automatically so that master versions are higher than the versions on
stable/train.

Change-Id: Icae14d39d714d0da3a945aeca3dcb70f73635162
Sem-Ver: feature
2019-09-27 20:40:17 +00:00
Renat Akhmerov
ac41f94d11 Add an ability to disable workflow text validation
* For the sake of the service performance, it may make sense to
  disable validation of the workflow language syntax if it is
  affordable for a particular use case. For example, if all
  workflows are auto-generated by a 3rd party system and tested
  thoroughly (either by running them with Mistral or at least
  validating them via the special Mistral endpoint) then we can
  safely disable validation of the language syntax when uploading
  workflow definitions. For production systems it makes a big
  difference if workflow texts are large (thousands of tasks).
  This patch adds the boolean parameter "skip_validation" for API
  requests like "POST /v2/workflows" to disable validation, if
  needed, and the new configuration property "validation_mode"
  to set a desired validation mode.
  The option is an enumeration and has the following valid values:
    1) "enabled" - enabled for all API requests unless it's
       explicitly disabled in the request itself
    2) "mandatory" - enabled for all API requests regardless
       of the flag in the request
    3) "disabled" - disabled for all API requrests regardless
       of the flag in the request
  "mandatory" is choosen as the default value for this new
  property to keep compatibility with the previous versions.
* Minor style changes.

Closes-Bug: #1844242

Change-Id: Ib509653d38254954f8449be3031457e5f636ccf2
2019-09-27 04:33:04 +00:00
apetrich
9289f7d28e Add release notes to changes to wflanguage execution object
Change-Id: I976fe5723a8bad81f1c6e30618475aa34e3e2050
depends-on: Ia6c954e688589f69a7463f1b8e02244d029e8b7a
2019-06-03 13:58:43 +02:00
Renat Akhmerov
a477fe49cd Fix how "has_next_tasks" is calculated for task executions
* There was a weird typo in the list generator expression made in
  https://review.opendev.org/#/c/652575 that led to calculating
  a field value in the wrong way. Fixed. The added test was
  previously failing.

Closes-Bug: #1829841

Change-Id: I6b8ea8dfa623acbb56f649630ddee62e43d484eb
2019-05-21 17:24:52 +07:00
Zuul
6dcc048725 Merge "Adds private_key parameter in the standard ssh actions" 2019-05-16 06:40:47 +00:00
Marc Gariepy
279fcc5b41 Add release notes for reply-to feature
Change-Id: I4b8a730940c24edd903f1f9a28807573c6d4449a
2019-04-18 08:01:55 -04:00
Renat Akhmerov
b0829f943b Fix an expression context for all_errors_handled()
* "__task_execution" wasn't included in this case into the
  expression data context so the function task() didn't work
  properly

Change-Id: I3cacae90f9031d09a5e6d8153d728ddc01e1bb21
Closes-Bug: #1823875
2019-04-10 11:49:48 +07:00
Zuul
0775d2cc81 Merge "Add release note for I04ba85488b27cb05c3b81ad8c973c3cc3fe56d36" 2019-04-05 00:43:52 +00:00
Vlad Gusev
5d2be3458d Add release note for I04ba85488b27cb05c3b81ad8c973c3cc3fe56d36
Change-Id: I3ad0a7769e2d61665d5ea76b90485e582993ddd2
2019-04-04 16:01:28 +00:00
Renat Akhmerov
5e0133c83e Add "convert_input_data" config property for YAQL expressions
Closes-Bug: #1822060
Change-Id: I521ad62c3311a2e4de714af4c31463bdff7dbd59
2019-04-04 08:49:03 +00:00
Renat Akhmerov
d332d7fefe Add release notes for engine optimizations
Change-Id: I9a87c1e2a0bc5a1ba54d7f93f6b1d077088e5023
2019-03-27 14:25:49 +07:00
3e42cffd6c Update master for stable/stein
Add file to the reno documentation build to show release notes for
stable/stein.

Use pbr instruction to increment the minor version number
automatically so that master versions are higher than the versions on
stable/stein.

Change-Id: Icb8261ce3371296b8a87e5bfa527a5f64a25157c
Sem-Ver: feature
2019-03-22 18:48:52 +00:00
Vlad Gusev
ed7c351d0b Release note for fixing event-engines HA
Add missing release note for https://review.openstack.org/#/c/548044/

Change-Id: I297054f052b79143b2d0882286fb3e22d19c70c5
2019-02-15 13:51:38 +03:00
lijunjie
c7509a9c85 Fix the misspelling of "default"
Change-Id: I0a96eebec595ed510abde253a517c4b73cf5ec7b
2019-01-10 17:22:36 +08:00
MaoyangLiu
efca6f2e95 fix typo mistakes
Change-Id: I274bb4b2e6ac97c6ed6d85367f96b72f3ecd06c5
2019-01-07 09:22:56 +00:00
Jose Castro Leon
9bbc1d41d0 Adds private_key parameter in the standard ssh actions
This method allows to specify a private key and avoids its storage
in the filesystem of the executors. This can be used later in
combination of a secrets_retrieve to use keys stored in barbican.

Change-Id: Ide438a7f6d24c8bdc9eb2c82e935fd39a6acc2c6
Closes-Bug: #1806703
2018-12-05 08:10:57 +01:00
Renat Akhmerov
0fb96bbd2b Add loging for sending an action to executor
Closes-Bug: #1805579
Change-Id: I4dc2b1e059d66d16725417ea5aa27b978e2163f0
2018-11-28 14:14:16 +07:00
Renat Akhmerov
3153f1ca58 Set admin security context in the action execution checker thread
Change-Id: I4c7abff04b8797618294d9e13fbec4b5876bcdb6
Closes-Bug: #1805080
2018-11-26 14:09:59 +00:00
Renat Akhmerov
c9e08a8839 Fix "join" when the last indirect inbound task failed
* See bug description for the example that didn't work. It was
  caused by a simple mistake in a python expression of type
  "my_set = my_set or set()" that didn't work as expected, i.e.
  it created a new set even if my_set is already an empty set.
  So, the proper expression that's needed is
  "my_set = set() if my_set is None else my_set"

Change-Id: I2a787921449fecf3301013a770ffe712e9606baf
Closes-Bug: #1803677
2018-11-16 15:35:18 +07:00
Renat Akhmerov
90ddf442ee Clone cached action definitions
* Once in a while we get DetachedInstanceError for action definitions
  and it happens when they are fetched from cache. We must always
  clone persistent objects before caching them.

Change-Id: I1d0cffea6775eb258dcefc0dbb8a6ee18effe597
Closes-Bug: #1803528
2018-11-15 18:39:21 +07:00
Zuul
ce4dc70be5 Merge "Refactor action execution checker without using scheduler" 2018-11-10 16:23:21 +00:00
Zuul
c48efec6db Merge "Add batch size for integrity checker" 2018-11-10 15:09:42 +00:00
Zuul
3e5c102ee5 Merge "Simplify workflow and join completion logic" 2018-11-10 11:36:31 +00:00
Zuul
2737586560 Merge "Fix how action result is assigned to task 'state_info' field" 2018-11-09 14:12:52 +00:00
Renat Akhmerov
43a8bddc24 Fix how action result is assigned to task 'state_info' field
Closes-Bug: #1802477

Change-Id: Ia8848b3bb0417f66422c4995b64be7a803dde1e7
2018-11-09 16:14:25 +07:00
Renat Akhmerov
2d74e6ebac Refactor action execution checker without using scheduler
* Removed using scheduler from action execution heartbeat checker
  in favor of regular threads.
* Added the new config options "batch_size" under [action_heartbeat]
  group to limit a number of action executions being processed during
  one iteration the checker.
* Added a test checking that an action execution is automatically
  failed by the heartbeat checker.

Closes-Bug: #1802065
Change-Id: I18c0c2c3159b9294c8af96c93c65a6edfc1de1a1
2018-11-09 14:17:28 +07:00
Renat Akhmerov
3b4136ff1e Add batch size for integrity checker
* Added the new property 'execution_integrity_check_batch_size'
  under the [engine] group to limit the number of task executions
  that the integrity checker may process during one iteration.

Closes-Bug: #1801876
Change-Id: I3c5074c45c476ebff109617cb15d56c54575dd4f
2018-11-09 14:17:27 +07:00
Renat Akhmerov
80a1bed67b Simplify workflow and join completion logic
* action_queue module is replaced with the more generic
  post_tx_queue module that allows to register operations that must
  run after the main DB transaction associated with processing a
  workflow event such as completing action.
* Instead of calling workflow completion check from all places
  where task may possibly complete, Mistral now registers a post
  transactional operation that runs after the main DB transaction
  (to make sure at least one needed consistent DB read) right
  inside the task completion logic. It reduces clutter significantly.
* Workflow completion check is now registered only if the just
  completed task may lead to workflow completion, i.e. if it's the
  last one in a workflow branch.
* Join now checks delayed calls to reduce a number of join
  completion checks created with scheduler and also uses post
  transactional queue for that.

Closes-Bug: #1801872
Change-Id: I90741d4121c48c42606dfa850cfe824557b095d0
2018-11-09 14:17:20 +07:00
Renat Akhmerov
b413aa087e Allow None for 'params' when starting a workflow execution
Change-Id: Ic28352d9acbe9e3f53a9d33a4ff0a5f99261f53f
Closes-Bug: #1793651
2018-11-08 21:01:21 +07:00
Renat Akhmerov
1a4c599a4d Improve join by removing periodic jobs
* This patch removes the approach with DB polling needed to
  determine if a "join" task is ready to run. Instead of running
  a periodic scheduled job, each task completion now runs the
  algorithm that finds all potentially affected join tasks
  and schedules just one job (instead of a periodic job) to check
  their readiness.
  This solves a problem of system cascaded overloading in case of
  having many very large joins (when a workflow has many joins with
  many  dependencies each). Previously, in such case Mistral created
  too many periodic jobs that just didn't let the workflow progress
  well, i.e. most CPU was used by scheduler to run those periodic
  jobs that very rarely switched "join" tasks to the RUNNING state.

Change-Id: I5ebc44c7a3f95c868d653689dc5cea689c788cd0
Closes-Bug: #1799356
2018-10-23 14:01:39 +07:00
hardikj
08ba20c8d2 Fix next link in get resource list rest API
Fix the issue where `next` link in the response of the REST API, when
used with pagination, gets corrupted and makes it unusable. After this
fix, next link should be readily usable.

Change-Id: Idf45a59e0b07d8306cc82391679fe30a9cd2f0c1
Closes-Bug: #1793344
2018-10-12 16:04:00 +05:30
Dougal Matthews
f0b49a82b4 Add a release note for Ic98e2db02abd8483591756d73e06784cc2e9cbe3
Change-Id: I484b47f9e0d94c37079961d18061e2e9e121af92
2018-10-01 10:56:08 +01:00
Renat Akhmerov
dfdff78315 Fix how Mistral calculates workflow output
* Workflow output sometimes is not calculated correctly due to
  the race condition between different transactions: the one that
  checks workflow completion (i.e. calls "check_and_complete") and
  the one that processes action execution completion (i.e. calls
  "on_action_complete"). Calculating output sometimes was based on
  stale data cached by the SQLAlchemy session. To fix this, we just
  need to expire all objects in the session so that they are
  refreshed automatically if we read their state in order to make
  required calculations. See the bug description for more details
  on how the problem was observed.
* Added another test for direct workflow that formally checks
  calculation of workflow output. It doesn't pretend to test the
  aforementioned issue (it can be reproduced only with a big number
  of attempts, and/or under load). It's for the sake of the test
  module completeness.

Change-Id: I4a7e7fd9a4bbb6e93df169b4b40bc2d83ccfce89
Closes-Bug: #1792090
2018-09-12 16:48:43 +07:00
Pierre Gaxatte
5a70238ede Add release note for auth_context bugfix
Bug 1785654 has been fixed in https://review.openstack.org/#/c/599606
but there was no release note. This change adds it.

Change-Id: I7bea9553c41336ba08d6fec551055290530769a9
2018-09-05 14:12:45 +02:00
Zuul
78dcb2d419 Merge "Performance: remove unnecessary workflow execution update" 2018-09-03 13:25:39 +00:00