Merge "Update docker build"

releasenotes/notes/update-mistral-docker-image-0c6294fc021545e0.yaml

@@ -0,0 +1,9 @@
+---
+features:
+  - The Mistral docker image and tooling has been updated to significantly
+    ease the starting of a Mistral cluster. The setup now supports
+    all-in-one and multi-container deployments. Also, the scripts were
+    cleaned up and aligned with the Docker best practice.
+
+fixes:
+  - Javascript support in docker image.

tools/docker/DOCKER_README.rst

@@ -1,10 +1,10 @@
 Using Mistral with docker
 =========================
 
-In order to minimize the work needed to the current Mistral code, or be able
+In order to minimize the work needed to run the current Mistral code, or
-to spin up independent or networked Mistral instances in seconds, docker
+be able to spin up independent or networked Mistral instances in seconds,
-containers are a very good option. This guide describes the process to
+Docker containers are a very good option. This guide describes the process
-launch an all-in-one Mistral container.
+to launch an all-in-one Mistral container.
 
 
 Docker installation
@@ -24,65 +24,66 @@ Build the Mistral image
 -----------------------
 
 The `build.sh` script takes care of creating the `mistral-all` image locally.
-
+This is image is configured to use  RabbitMQ for transport and MySQL as database
-
+backend. It is possible to run Mistral with Sqlite as database backend but
-Running Mistral
+it is very unreliable, thus, MySQL was selected as the default database backend
----------------
+for this image.
-
-Start a RabbitMQ container::
-
-  docker run -d --name rabbitmq rabbitmq
-
-Start Mistral::
-
-  docker run -d  --link rabbitmq:rabbitmq -p 8989:8989 --name mistral mistral-all
-
-To execute commands inside the container::
-
-  docker exec -it mistral bash
-
-E.g. to list workflows, issue::
-
-  mistral workflow-list
-
-
-Running Mistral From A Volume
------------------------------
-
-A scenario you may find useful for development is to clone a Mistral git repo
-and link it into the container via a volume. This will allow you to make changes
-to the Mistral source on your local machine and execute them immediately in the
-container.
-
-The following example illustrates launching the container from the local
-directory of a git repo clone of Mistral.::
-
-  docker run -d --link rabbitmq:rabbitmq -v $(pwd):/opt/stack/mistral:Z -p 8989:8989 --name mistral mistral-all
-
-You might want to mount an additional drive to move files easily between your
-development computer and the container.  An easy way to do this is to mount an
-additional volume that maps to /home/mistral/ in the container.
-
-Since the directory is already being used to store the mistral.conf and
-mistral.sqlite files, you will want to copy these to the local directory you
-intend to use for the mount. This example assumes the directory to mount is
-"/tmp/mistral".  You should change this to the actual directory you intend to
-use.::
-
-  docker cp mistral:/home/mistral/mistral.conf /tmp/mistral/mistral.conf
-  docker cp mistral:/home/mistral/mistral.sqlite /tmp/mistral/mistral.sqlite
-
-  docker run -d --link rabbitmq:rabbitmq -v $(pwd):/opt/stack/mistral:Z -v /tmp/mistral:/home/mistral:Z -p 8989:8989 --name mistral mistral-all
 
 
 Running Mistral with MySQL
 --------------------------
 
-Other than the simplest use cases will very probably fail with various errors
-due to the default Sqlite database. It is highly recommended that, for
-example, MySQL is used as database backend.
-
 The `start_mistral_rabbit_mysql.sh` script sets up a rabbitmq container, a
 mysql container and a mistral container to work together.
 
-Check out the script for more detail.
+The script can be invoked with::
+
+  start_mistral_rabbit_mysql.sh [single|multi]
+
+`single` mode (this is the default) will create
+
+ - rabbitmq container,
+ - the mysql container,
+ - and the mistral container that runs all Mistral services.
+
+
+`multi` mode will create
+
+ - rabbitmq,
+ - mysql,
+ - mistral-api,
+ - one mistral-engine,
+ - two mistral-executors
+
+Check out the script for more detail and examples for different setup options.
+
+Using Mistral
+-------------
+
+Depending on the mode, you may need to use the `mistral` or the `mistral-api`
+container.
+
+With the `multi` option execute commands inside the container::
+
+  docker exec -it mistral-api bash
+
+E.g. to list workflows, issue::
+
+  mistral workflow-list
+
+The script also configures the containers so that the Mistral API will be
+accessible from the host machine on the default port 8989. So it is also
+possible to install the `mistral-pythonclient` to the host machine and
+execute commands there.
+
+Configuring Mistral
+-------------------
+
+The Mistral configuration is stored in the Docker image. The changes to the
+configuration should be synchronized between all deployed containers to
+ensure consistent behavior. This can be achieved by mounting the configuration
+as a volume::
+
+  export EXTRA_OPTS='-v <path to local mistral.conf>:/etc/mistral/mistral.conf:ro'
+  start_mistral_rabbit_mysql.sh multi
+

tools/docker/Dockerfile

@@ -1,22 +1,53 @@
-FROM ubuntu:16.04
+FROM krallin/ubuntu-tini:16.04
-MAINTAINER hardik.parekh@nectechnologies.in
 
-ADD . /opt/stack/mistral
+MAINTAINER Andras Kovi <akovi@nokia.com>
 
-RUN /opt/stack/mistral/tools/docker/Dockerfile_script.sh
+RUN export DEBIAN_FRONTEND=noninteractive && \
+    apt-get -qq update && \
+    apt-get install -y  \
+      curl \
+      git \
+      libffi-dev \
+      libssl-dev \
+      libxml2-dev \
+      libxslt1-dev \
+      libyaml-dev \
+      mc \
+      python-dev \
+      python-pip \
+      python-setuptools \
+      swig \
+      cmake \
+      crudini \
+      libuv1 \
+      libuv1-dev
+
+RUN pip install -v v8eval && python -c 'import v8eval'
+
+RUN apt-get install -y libmysqlclient-dev && \
+    pip install mysql-python
+
+RUN pip install -U tox python-mistralclient pip
+
+COPY . /opt/stack/mistral
+
+RUN curl -o /tmp/upper-constraints.txt http://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt && \
+    sed -i '/^mistral.*/d' /tmp/upper-constraints.txt &&\
+    pip install -e /opt/stack/mistral
+
+RUN mkdir -p /etc/mistral
+
+RUN oslo-config-generator \
+      --config-file /opt/stack/mistral/tools/config/config-generator.mistral.conf \
+      --output-file /etc/mistral/mistral.conf
+
+RUN INI_SET="crudini --set /etc/mistral/mistral.conf" && \
+    $INI_SET DEFAULT js_implementation v8eval && \
+    $INI_SET DEFAULT transport_url rabbit://guest:guest@rabbitmq:5672/ && \
+    $INI_SET database connection mysql://root:strangehat@mysql:3306/mistral && \
+    $INI_SET oslo_policy policy_file /opt/stack/mistral/etc/policy.json && \
+    $INI_SET pecan auth_enable false
 
 EXPOSE 8989
 
-VOLUME ["/opt/stack/mistral"]
+CMD mistral-server --server all
-VOLUME ["/home/mistral"]
-WORKDIR /home/mistral
-CMD mistral-server --server all --config-file /home/mistral/mistral.conf
-
-ENV TINI_SHA 066ad710107dc7ee05d3aa6e4974f01dc98f3888
-
-# Use tini as subreaper in Docker container to adopt zombie processes
-RUN curl -fsSL https://github.com/krallin/tini/releases/download/v0.5.0/tini-static -o /bin/tini \
-  && chmod +x /bin/tini \
-  && echo "$TINI_SHA  /bin/tini" | sha1sum -c -
-
-ENTRYPOINT ["/bin/tini", "--"]

tools/docker/Dockerfile_script.sh

@@ -1,48 +0,0 @@
-#! /bin/bash -xe
-#Mistral Installation.
-export DEBIAN_FRONTEND=noninteractive
-apt-get -qq update
-apt-get install -y  \
-		curl \
-		git \
-		libffi-dev \
-		libssl-dev \
-		libxml2-dev \
-		libxslt1-dev \
-		libyaml-dev \
-		mc \
-		python-dev \
-		python-pip \
-		python-setuptools \
-		sudo
-
-sudo pip install tox==1.6.1 python-mistralclient
-
-cd /opt/stack/mistral
-pip install -r requirements.txt
-pip install .
-
-mkdir -p /home/mistral
-cd /home/mistral
-oslo-config-generator --config-file /opt/stack/mistral/tools/config/config-generator.mistral.conf --output-file /home/mistral/mistral.conf
-python /opt/stack/mistral/tools/sync_db.py --config-file /home/mistral/mistral.conf
-
-#Configure Mistral.
-python -c "
-import ConfigParser
-c = ConfigParser.ConfigParser()
-c.read('/home/mistral/mistral.conf')
-c.set('database','connection','sqlite:////home/mistral/mistral.sqlite')
-c.set('DEFAULT', 'transport_url', 'rabbit://guest:guest@rabbitmq:5672/')
-c.set('oslo_policy', 'policy_file', '/opt/stack/mistral/etc/policy.json')
-c.set('pecan', 'auth_enable', 'false')
-with open('/home/mistral/mistral.conf', 'w') as f:
-  c.write(f)
-"
-
-# install pyv8 to be able to run javscript actions (note that this breaks
-# portability because of architecture dependent binaries)
-
-curl -k  "https://raw.githubusercontent.com/emmetio/pyv8-binaries/master/pyv8-linux64.zip" > /tmp/pyv8.zip
-unzip /tmp/pyv8.zip -d /tmp/
-cp /tmp/*PyV8* /usr/lib/python2.7/dist-packages/

tools/docker/start_mistral_rabbit_mysql.sh

@@ -1,39 +1,111 @@
-#! /bin/bash -xe
+#! /bin/bash -e
 
-docker rm -f mistral-mysql mistral-rabbitmq mistral | true
+if [ "${1}" == "--help" ]; then
+    echo '
+    Synopsis:
 
-docker run -d --name mistral-mysql -e MYSQL_ROOT_PASSWORD=strangehat mysql
+       start_mistral_rabbit_mysql.sh [single|multi|clean]
-docker run -d --name mistral-rabbitmq rabbitmq
 
-docker run -d --link mistral-mysql:mysql --link mistral-rabbitmq:rabbitmq --name mistral mistral-all
+    Environment variables:
+
+    EXTRA_OPTS : extra parameters to be used for all mistral containers (e.g. -v)
+    MYSQL_ROOT_PASSWORD : password for the MySQL server
+    SCRATCH : remove all existing containers (RabbitMQ and MySQL are not removed by default)
+    '
+    exit 0
+fi
+
+MODE=${1:-single}
+
+export MYSQL_ROOT_PASSWORD=${MYSQL_ROOT_PASSWORD:-strangehat}
+
+MISTRAL_CONTAINERS=$(docker ps -a --format '{{.ID}} {{.Names}}' | grep mistral || true)
+
+if [ -z "$SCRATCH" -a "$MODE" != 'clean' ]; then
+    MISTRAL_CONTAINERS=$(echo "$MISTRAL_CONTAINERS" | grep -v rabbitmq | grep -v mysql | cat)
+fi
+
+if [ -n "$MISTRAL_CONTAINERS" ]; then
+    echo "Removing existing containers: $MISTRAL_CONTAINERS"
+    KILLED_CONTAINERS=$(echo "$MISTRAL_CONTAINERS" | awk '{print $1}')
+    docker kill -s 9 $KILLED_CONTAINERS
+    docker rm $KILLED_CONTAINERS
+fi
+
+if [ "$MODE" == 'clean' ]; then
+    echo "Clean complete"
+    exit 0
+fi
+
+if [ -z "$(docker ps -aq --filter "Name=mistral-mysql")" ]; then
+    docker create --name mistral-mysql -e MYSQL_ROOT_PASSWORD=${MYSQL_ROOT_PASSWORD} mysql
+fi
+docker start mistral-mysql
+if [ -z "$(docker ps -aq --filter "Name=mistral-rabbitmq")" ]; then
+    docker create --name mistral-rabbitmq rabbitmq
+fi
+docker start mistral-rabbitmq
+
+while true; do
+  sleep 5
+  docker exec mistral-mysql \
+    mysql -u root -pstrangehat \
+    -e "CREATE DATABASE IF NOT EXISTS mistral;
+        USE mistral;
+        GRANT ALL ON mistral.* TO 'root'@'%' IDENTIFIED BY '${MYSQL_ROOT_PASSWORD}'" \
+  && break || true
+done
 
 sleep 10
 
-docker exec mistral-mysql mysql -u root -pstrangehat -e "CREATE DATABASE mistral; USE mistral; GRANT ALL ON mistral.* TO 'root'@'%' IDENTIFIED BY 'strangehat'"
+docker run -dit --link mistral-mysql:mysql --name mistral-db-setup mistral-all cat
+docker exec mistral-db-setup python /opt/stack/mistral/tools/sync_db.py
+docker kill -s 9 mistral-db-setup
+docker rm mistral-db-setup
 
-docker exec mistral apt-get install -y libmysqlclient-dev
+function run_mistral() {
-docker exec mistral pip install mysql-python
+    NAME=${1:-mistral}
-docker exec mistral cp mistral.conf mistral.conf.orig
+    shift || true
-docker exec mistral python -c "
+    LINKS='--link mistral-mysql:mysql --link mistral-rabbitmq:rabbitmq'
-import ConfigParser
+    docker run \
-c = ConfigParser.ConfigParser()
+            -d \
-c.read('/home/mistral/mistral.conf')
+            --name $NAME \
-c.set('DEFAULT', 'transport_url', 'rabbit://guest:guest@rabbitmq:5672/')
+	        $LINKS \
-c.set('database','connection','mysql://root:strangehat@mysql:3306/mistral')
+	        ${EXTRA_OPTS} \
-c.set('pecan', 'auth_enable', 'false')
+            ${OPTS} \
-with open('/home/mistral/mistral.conf', 'w') as f:
+            mistral-all "$@"
-  c.write(f)
+}
-"
 
-docker exec mistral python /opt/stack/mistral/tools/sync_db.py --config-file /home/mistral/mistral.conf
+unset OPTS
 
-docker restart mistral
+case "$MODE" in
 
-echo "
+single)
-Enter the container:
+    # Single node setup
-  docker exec -it mistral bash
+    # The CMD of the mistral-all image runs the `mistral-server --server all` command.
+    OPTS="-p 8989:8989" run_mistral
 
-List workflows
+    echo "
-  docker exec mistral mistral workflow-list
+    Enter the container:
+      docker exec -it mistral bash
+
+    List workflows
+      docker exec mistral mistral workflow-list
+
+    "
+    ;;
+
+multi)
+    # Multinode setup
+    OPTS="-p 8989:8989" run_mistral "mistral-api" mistral-server --server api
+    run_mistral "mistral-engine" mistral-server --server engine
+    run_mistral "mistral-executor-1" mistral-server --server executor
+    run_mistral "mistral-executor-2" mistral-server --server executor
+
+    echo "
+    List workflows
+      docker exec mistral-api mistral workflow-list
+    "
+    ;;
+esac
 
-"