Move wsgi app in wsgi folder

It will be easier to locate and use as a wsgi application like this:

mistral.wsgi:application

Also call oslo_service init_backend very early to avoid loading eventlet
backend.

Finally, amend the doc and allow using MISTRAL_CONFIG_DIR and
MISTRAL_CONFIG_FILE to provide similar functionalities as --config-dir
and --config-file options from CLI.

Change-Id: I5d4178b7de49c89a3d15467186b165e518cfe87e
Signed-off-by: Arnaud M <arnaud.morin@gmail.com>

doc/source/admin/install/install.rst

@@ -35,10 +35,12 @@ The Workflow service consists of the following components:
 
 ``Mistral Notifier`` service
   Send notifications based on state of workflow and task executions.
+  `This service is optional`.
 
 ``Mistral Event Engine`` service
   Create workflow executions based on external events (like RabbitMQ, HTTP,
   kafka, etc.).
+  `This service is optional`.
 
 The mistral project is also providing the following python libraries:
 
@@ -163,11 +165,11 @@ Update the database to the latest revision:
 .. code-block:: console
 
     # For MySQL / MariaDB / PostgreSQL
-    $ mistral-db-manage --config-file /etc/mistral/mistral.conf upgrade head
+    $ mistral-db-manage upgrade head
 
     # For SQLite - do not use sqlite in production!
     # e.g. connection = 'sqlite:////var/lib/mistral.sqlite'
-    $ python tools/sync_db.py --config-file /etc/mistral/mistral.conf
+    $ python tools/sync_db.py
 
 Before starting the Mistral server, run the *mistral-db-manage populate*
 command. It creates the DB with all the standard actions and standard workflows
@@ -175,7 +177,7 @@ that Mistral provides to all Mistral users.:
 
 .. code-block:: console
 
-    $ mistral-db-manage --config-file /etc/mistral/mistral.conf populate
+    $ mistral-db-manage populate
 
 For more detailed information on the *mistral-db-manage* script, see
 the :doc:`Mistral Upgrade Guide </admin/upgrade_guide>`.
@@ -188,7 +190,12 @@ To run the Mistral components, execute the following command in a shell:
 
 .. code-block:: console
 
-    $ mistral-server --server all --config-file /etc/mistral/mistral.conf
+    $ mistral-server --server all
+
+.. note::
+
+    in this situation API will start only one worker! If you need more than
+    worker for you API, you should start the API with uWSGI (see below)
 
 Running Mistral components separately
 -------------------------------------
@@ -198,14 +205,14 @@ server, e.g. to start only the engine:
 
 .. code-block:: console
 
-    $ mistral-server --server engine --config-file /etc/mistral/mistral.conf
+    $ mistral-server --server engine
 
 The --server command line option can be a comma delimited list, so you can
 build combination of components, like this:
 
 .. code-block:: console
 
-    $ mistral-server --server engine,executor --config-file /etc/mistral/mistral.conf
+    $ mistral-server --server engine,executor
 
 The valid options are:
 
@@ -216,6 +223,102 @@ The valid options are:
 * event-engine
 * notifier
 
+Running Mistral API with uWSGI
+------------------------------
+
+The WSGI application
+~~~~~~~~~~~~~~~~~~~~
+
+One downside of running ``mistral-server --server api`` directly is that it
+will start only one process (worker) to handle HTTP requests.
+
+While this may be enough for small/dev deployments, it may not for production.
+
+In that situation, Mistral provides a WSGI application at
+``mistral.wsgi:application`` that can be used with any WSGI server.
+
+The below example uses uWSGI
+
+
+Using uWSGI
+~~~~~~~~~~~
+
+Install uWSGI:
+
+.. code-block:: console
+
+    $ pip install uwsgi
+
+
+Create a uWSGI configuration file (e.g., ``/etc/uwsgi/mistral.ini``):
+
+.. code-block:: cfg
+
+    [uwsgi]
+    # Listen on port 8989 and start as a full web server
+    http-socket = 0.0.0.0:8989
+
+    # Stats on port 9191
+    stats = 0.0.0.0:9191
+
+    # App to start
+    virtualenv = /opt/openstack/mistral/
+    module = mistral.wsgi:application
+
+    # load apps in each worker instead of the master
+    lazy-apps = true
+
+    # Number of processes
+    processes = 4
+
+    # Will kill processes that run more that 60s
+    harakiri = 60
+
+    # Enable threads
+    enable-threads = true
+
+    # Gracefully manage processes
+    master = true
+
+    # Thunder-lock - serialize accept() usage (if possible)
+    thunder-lock = true
+
+
+Start uWSGI:
+
+.. code-block:: console
+
+    $ uwsgi --ini /etc/uwsgi/mistral.ini
+
+
+Passing Configuration Options
+------------------------------
+
+By default, Mistral will use its standard configuration file search paths:
+
+* ``/etc/mistral/mistral.conf``
+* ``/etc/mistral/mistral.conf.d/``
+* ``/etc/mistral.conf.d/``
+* many others, see:
+  https://docs.openstack.org/oslo.config/latest/configuration/options.html
+
+You can also provide ``config-dir`` or ``config-file`` options to
+``mistral-server`` command line to provide a custom file/folder:
+
+.. code-block:: console
+
+    $ mistral-server --config-dir /etc/mycustomdir/
+
+Note that, when using ``uwsgi``, you won't be able to provide such params. In
+that situation, you can use ``MISTRAL_CONFIG_DIR`` and/or
+``MISTRAL_CONFIG_FILE`` environment variable instead:
+
+.. code-block:: cfg
+
+    [uwsgi]
+    ...
+    env = MISTRAL_CONFIG_DIR=/etc/mycustomdir/
+
 .. _install-osa:
 
 Deploying with OpenStack-Ansible

mistral/api/wsgi.py

@@ -12,6 +12,15 @@
 #    See the License for the specific language governing permissions and
 #    limitations under the License.
 
+# NOTE(amorin) Deprecated wsgi entry point
+# use wsgi/ instead now
+
+# NOTE(amorin)
+# Hardcode the threading backend to avoid using eventlet until this will
+# eventually become the default
+import oslo_service.backend as service_backend
+service_backend.init_backend(service_backend.BackendType.THREADING)
+
 from mistral.api import app
 
 application = app.init_wsgi()

mistral/config.py

@@ -21,6 +21,7 @@ Configuration options registration and useful routines.
 """
 
 import json
+import os
 
 from keystoneauth1 import loading
 from oslo_config import cfg
@@ -859,7 +860,13 @@ def list_opts():
     ]
 
 
-def parse_args(args=None, usage=None, default_config_files=None):
+def parse_args(args=None, usage=None):
+    # NOTE(amorin) allow overwritting config dir and file from env
+    conf_d = os.environ.get('MISTRAL_CONFIG_DIR')
+    default_config_dirs = [conf_d] if conf_d else None
+    conf_f = os.environ.get('MISTRAL_CONFIG_FILE')
+    default_config_files = [conf_f] if conf_f else None
+
     default_log_levels = log.get_default_log_levels()
     default_log_levels.extend(_DEFAULT_LOG_LEVELS)
     log.set_defaults(default_log_levels=default_log_levels)
@@ -871,7 +878,8 @@ def parse_args(args=None, usage=None, default_config_files=None):
         project='mistral',
         version=version.version_string,
         usage=usage,
-        default_config_files=default_config_files
+        default_config_dirs=default_config_dirs,
+        default_config_files=default_config_files,
     )
 
 

mistral/wsgi/__init__.py

@@ -0,0 +1,29 @@
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""WSGI application entry-point for Mistral API."""
+
+# NOTE(amorin)
+# Hardcode the threading backend to avoid using eventlet until this will
+# eventually become the default
+import oslo_service.backend as service_backend
+service_backend.init_backend(service_backend.BackendType.THREADING)
+
+import threading
+
+from mistral.api import app
+
+application = None
+lock = threading.Lock()
+with lock:
+    if application is None:
+        application = app.init_wsgi()

releasenotes/notes/config-dir-file-6d36f8d2f2ebb5f9.yaml

@@ -0,0 +1,12 @@
+---
+other:
+  - |
+    Adding a new wsgi entrypoint to allow using it within WSGI servers (like
+    uWSGI) using ``mistral.wsgi:application``.
+
+    Note that **this is the only way to start more than one mistral API
+    worker**.
+
+    Also adding two environment variables ``MISTRAL_CONFIG_DIR`` and
+    ``MISTRAL_CONFIG_FILE`` to allow setting config-dir and config-file when
+    using the new wsgi entrypoint.