Merge "Update docker build"
This commit is contained in:
Jenkins
Gerrit Code Review
commit
05711f6f33
@ -0,0 +1,9 @@
|
||||
---
|
||||
features:
|
||||
- The Mistral docker image and tooling has been updated to significantly
|
||||
ease the starting of a Mistral cluster. The setup now supports
|
||||
all-in-one and multi-container deployments. Also, the scripts were
|
||||
cleaned up and aligned with the Docker best practice.
|
||||
|
||||
fixes:
|
||||
- Javascript support in docker image.
|
||||
@ -1,10 +1,10 @@
|
||||
Using Mistral with docker
|
||||
=========================
|
||||
|
||||
In order to minimize the work needed to the current Mistral code, or be able
|
||||
to spin up independent or networked Mistral instances in seconds, docker
|
||||
containers are a very good option. This guide describes the process to
|
||||
launch an all-in-one Mistral container.
|
||||
In order to minimize the work needed to run the current Mistral code, or
|
||||
be able to spin up independent or networked Mistral instances in seconds,
|
||||
Docker containers are a very good option. This guide describes the process
|
||||
to launch an all-in-one Mistral container.
|
||||
|
||||
|
||||
Docker installation
|
||||
@ -24,65 +24,66 @@ Build the Mistral image
|
||||
-----------------------
|
||||
|
||||
The `build.sh` script takes care of creating the `mistral-all` image locally.
|
||||
|
||||
|
||||
Running Mistral
|
||||
---------------
|
||||
|
||||
Start a RabbitMQ container::
|
||||
|
||||
docker run -d --name rabbitmq rabbitmq
|
||||
|
||||
Start Mistral::
|
||||
|
||||
docker run -d --link rabbitmq:rabbitmq -p 8989:8989 --name mistral mistral-all
|
||||
|
||||
To execute commands inside the container::
|
||||
|
||||
docker exec -it mistral bash
|
||||
|
||||
E.g. to list workflows, issue::
|
||||
|
||||
mistral workflow-list
|
||||
|
||||
|
||||
Running Mistral From A Volume
|
||||
-----------------------------
|
||||
|
||||
A scenario you may find useful for development is to clone a Mistral git repo
|
||||
and link it into the container via a volume. This will allow you to make changes
|
||||
to the Mistral source on your local machine and execute them immediately in the
|
||||
container.
|
||||
|
||||
The following example illustrates launching the container from the local
|
||||
directory of a git repo clone of Mistral.::
|
||||
|
||||
docker run -d --link rabbitmq:rabbitmq -v $(pwd):/opt/stack/mistral:Z -p 8989:8989 --name mistral mistral-all
|
||||
|
||||
You might want to mount an additional drive to move files easily between your
|
||||
development computer and the container. An easy way to do this is to mount an
|
||||
additional volume that maps to /home/mistral/ in the container.
|
||||
|
||||
Since the directory is already being used to store the mistral.conf and
|
||||
mistral.sqlite files, you will want to copy these to the local directory you
|
||||
intend to use for the mount. This example assumes the directory to mount is
|
||||
"/tmp/mistral". You should change this to the actual directory you intend to
|
||||
use.::
|
||||
|
||||
docker cp mistral:/home/mistral/mistral.conf /tmp/mistral/mistral.conf
|
||||
docker cp mistral:/home/mistral/mistral.sqlite /tmp/mistral/mistral.sqlite
|
||||
|
||||
docker run -d --link rabbitmq:rabbitmq -v $(pwd):/opt/stack/mistral:Z -v /tmp/mistral:/home/mistral:Z -p 8989:8989 --name mistral mistral-all
|
||||
This is image is configured to use RabbitMQ for transport and MySQL as database
|
||||
backend. It is possible to run Mistral with Sqlite as database backend but
|
||||
it is very unreliable, thus, MySQL was selected as the default database backend
|
||||
for this image.
|
||||
|
||||
|
||||
Running Mistral with MySQL
|
||||
--------------------------
|
||||
|
||||
Other than the simplest use cases will very probably fail with various errors
|
||||
due to the default Sqlite database. It is highly recommended that, for
|
||||
example, MySQL is used as database backend.
|
||||
|
||||
The `start_mistral_rabbit_mysql.sh` script sets up a rabbitmq container, a
|
||||
mysql container and a mistral container to work together.
|
||||
|
||||
Check out the script for more detail.
|
||||
The script can be invoked with::
|
||||
|
||||
start_mistral_rabbit_mysql.sh [single|multi]
|
||||
|
||||
`single` mode (this is the default) will create
|
||||
|
||||
- rabbitmq container,
|
||||
- the mysql container,
|
||||
- and the mistral container that runs all Mistral services.
|
||||
|
||||
|
||||
`multi` mode will create
|
||||
|
||||
- rabbitmq,
|
||||
- mysql,
|
||||
- mistral-api,
|
||||
- one mistral-engine,
|
||||
- two mistral-executors
|
||||
|
||||
Check out the script for more detail and examples for different setup options.
|
||||
|
||||
Using Mistral
|
||||
-------------
|
||||
|
||||
Depending on the mode, you may need to use the `mistral` or the `mistral-api`
|
||||
container.
|
||||
|
||||
With the `multi` option execute commands inside the container::
|
||||
|
||||
docker exec -it mistral-api bash
|
||||
|
||||
E.g. to list workflows, issue::
|
||||
|
||||
mistral workflow-list
|
||||
|
||||
The script also configures the containers so that the Mistral API will be
|
||||
accessible from the host machine on the default port 8989. So it is also
|
||||
possible to install the `mistral-pythonclient` to the host machine and
|
||||
execute commands there.
|
||||
|
||||
Configuring Mistral
|
||||
-------------------
|
||||
|
||||
The Mistral configuration is stored in the Docker image. The changes to the
|
||||
configuration should be synchronized between all deployed containers to
|
||||
ensure consistent behavior. This can be achieved by mounting the configuration
|
||||
as a volume::
|
||||
|
||||
export EXTRA_OPTS='-v <path to local mistral.conf>:/etc/mistral/mistral.conf:ro'
|
||||
start_mistral_rabbit_mysql.sh multi
|
||||
|
||||
|
||||
@ -1,22 +1,53 @@
|
||||
FROM ubuntu:16.04
|
||||
MAINTAINER hardik.parekh@nectechnologies.in
|
||||
FROM krallin/ubuntu-tini:16.04
|
||||
|
||||
ADD . /opt/stack/mistral
|
||||
MAINTAINER Andras Kovi <akovi@nokia.com>
|
||||
|
||||
RUN /opt/stack/mistral/tools/docker/Dockerfile_script.sh
|
||||
RUN export DEBIAN_FRONTEND=noninteractive && \
|
||||
apt-get -qq update && \
|
||||
apt-get install -y \
|
||||
curl \
|
||||
git \
|
||||
libffi-dev \
|
||||
libssl-dev \
|
||||
libxml2-dev \
|
||||
libxslt1-dev \
|
||||
libyaml-dev \
|
||||
mc \
|
||||
python-dev \
|
||||
python-pip \
|
||||
python-setuptools \
|
||||
swig \
|
||||
cmake \
|
||||
crudini \
|
||||
libuv1 \
|
||||
libuv1-dev
|
||||
|
||||
RUN pip install -v v8eval && python -c 'import v8eval'
|
||||
|
||||
RUN apt-get install -y libmysqlclient-dev && \
|
||||
pip install mysql-python
|
||||
|
||||
RUN pip install -U tox python-mistralclient pip
|
||||
|
||||
COPY . /opt/stack/mistral
|
||||
|
||||
RUN curl -o /tmp/upper-constraints.txt http://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt && \
|
||||
sed -i '/^mistral.*/d' /tmp/upper-constraints.txt &&\
|
||||
pip install -e /opt/stack/mistral
|
||||
|
||||
RUN mkdir -p /etc/mistral
|
||||
|
||||
RUN oslo-config-generator \
|
||||
--config-file /opt/stack/mistral/tools/config/config-generator.mistral.conf \
|
||||
--output-file /etc/mistral/mistral.conf
|
||||
|
||||
RUN INI_SET="crudini --set /etc/mistral/mistral.conf" && \
|
||||
$INI_SET DEFAULT js_implementation v8eval && \
|
||||
$INI_SET DEFAULT transport_url rabbit://guest:guest@rabbitmq:5672/ && \
|
||||
$INI_SET database connection mysql://root:strangehat@mysql:3306/mistral && \
|
||||
$INI_SET oslo_policy policy_file /opt/stack/mistral/etc/policy.json && \
|
||||
$INI_SET pecan auth_enable false
|
||||
|
||||
EXPOSE 8989
|
||||
|
||||
VOLUME ["/opt/stack/mistral"]
|
||||
VOLUME ["/home/mistral"]
|
||||
WORKDIR /home/mistral
|
||||
CMD mistral-server --server all --config-file /home/mistral/mistral.conf
|
||||
|
||||
ENV TINI_SHA 066ad710107dc7ee05d3aa6e4974f01dc98f3888
|
||||
|
||||
# Use tini as subreaper in Docker container to adopt zombie processes
|
||||
RUN curl -fsSL https://github.com/krallin/tini/releases/download/v0.5.0/tini-static -o /bin/tini \
|
||||
&& chmod +x /bin/tini \
|
||||
&& echo "$TINI_SHA /bin/tini" | sha1sum -c -
|
||||
|
||||
ENTRYPOINT ["/bin/tini", "--"]
|
||||
CMD mistral-server --server all
|
||||
|
||||
@ -1,48 +0,0 @@
|
||||
#! /bin/bash -xe
|
||||
#Mistral Installation.
|
||||
export DEBIAN_FRONTEND=noninteractive
|
||||
apt-get -qq update
|
||||
apt-get install -y \
|
||||
curl \
|
||||
git \
|
||||
libffi-dev \
|
||||
libssl-dev \
|
||||
libxml2-dev \
|
||||
libxslt1-dev \
|
||||
libyaml-dev \
|
||||
mc \
|
||||
python-dev \
|
||||
python-pip \
|
||||
python-setuptools \
|
||||
sudo
|
||||
|
||||
sudo pip install tox==1.6.1 python-mistralclient
|
||||
|
||||
cd /opt/stack/mistral
|
||||
pip install -r requirements.txt
|
||||
pip install .
|
||||
|
||||
mkdir -p /home/mistral
|
||||
cd /home/mistral
|
||||
oslo-config-generator --config-file /opt/stack/mistral/tools/config/config-generator.mistral.conf --output-file /home/mistral/mistral.conf
|
||||
python /opt/stack/mistral/tools/sync_db.py --config-file /home/mistral/mistral.conf
|
||||
|
||||
#Configure Mistral.
|
||||
python -c "
|
||||
import ConfigParser
|
||||
c = ConfigParser.ConfigParser()
|
||||
c.read('/home/mistral/mistral.conf')
|
||||
c.set('database','connection','sqlite:////home/mistral/mistral.sqlite')
|
||||
c.set('DEFAULT', 'transport_url', 'rabbit://guest:guest@rabbitmq:5672/')
|
||||
c.set('oslo_policy', 'policy_file', '/opt/stack/mistral/etc/policy.json')
|
||||
c.set('pecan', 'auth_enable', 'false')
|
||||
with open('/home/mistral/mistral.conf', 'w') as f:
|
||||
c.write(f)
|
||||
"
|
||||
|
||||
# install pyv8 to be able to run javscript actions (note that this breaks
|
||||
# portability because of architecture dependent binaries)
|
||||
|
||||
curl -k "https://raw.githubusercontent.com/emmetio/pyv8-binaries/master/pyv8-linux64.zip" > /tmp/pyv8.zip
|
||||
unzip /tmp/pyv8.zip -d /tmp/
|
||||
cp /tmp/*PyV8* /usr/lib/python2.7/dist-packages/
|
||||
@ -1,39 +1,111 @@
|
||||
#! /bin/bash -xe
|
||||
#! /bin/bash -e
|
||||
|
||||
docker rm -f mistral-mysql mistral-rabbitmq mistral | true
|
||||
if [ "${1}" == "--help" ]; then
|
||||
echo '
|
||||
Synopsis:
|
||||
|
||||
docker run -d --name mistral-mysql -e MYSQL_ROOT_PASSWORD=strangehat mysql
|
||||
docker run -d --name mistral-rabbitmq rabbitmq
|
||||
start_mistral_rabbit_mysql.sh [single|multi|clean]
|
||||
|
||||
docker run -d --link mistral-mysql:mysql --link mistral-rabbitmq:rabbitmq --name mistral mistral-all
|
||||
Environment variables:
|
||||
|
||||
EXTRA_OPTS : extra parameters to be used for all mistral containers (e.g. -v)
|
||||
MYSQL_ROOT_PASSWORD : password for the MySQL server
|
||||
SCRATCH : remove all existing containers (RabbitMQ and MySQL are not removed by default)
|
||||
'
|
||||
exit 0
|
||||
fi
|
||||
|
||||
MODE=${1:-single}
|
||||
|
||||
export MYSQL_ROOT_PASSWORD=${MYSQL_ROOT_PASSWORD:-strangehat}
|
||||
|
||||
MISTRAL_CONTAINERS=$(docker ps -a --format '{{.ID}} {{.Names}}' | grep mistral || true)
|
||||
|
||||
if [ -z "$SCRATCH" -a "$MODE" != 'clean' ]; then
|
||||
MISTRAL_CONTAINERS=$(echo "$MISTRAL_CONTAINERS" | grep -v rabbitmq | grep -v mysql | cat)
|
||||
fi
|
||||
|
||||
if [ -n "$MISTRAL_CONTAINERS" ]; then
|
||||
echo "Removing existing containers: $MISTRAL_CONTAINERS"
|
||||
KILLED_CONTAINERS=$(echo "$MISTRAL_CONTAINERS" | awk '{print $1}')
|
||||
docker kill -s 9 $KILLED_CONTAINERS
|
||||
docker rm $KILLED_CONTAINERS
|
||||
fi
|
||||
|
||||
if [ "$MODE" == 'clean' ]; then
|
||||
echo "Clean complete"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
if [ -z "$(docker ps -aq --filter "Name=mistral-mysql")" ]; then
|
||||
docker create --name mistral-mysql -e MYSQL_ROOT_PASSWORD=${MYSQL_ROOT_PASSWORD} mysql
|
||||
fi
|
||||
docker start mistral-mysql
|
||||
if [ -z "$(docker ps -aq --filter "Name=mistral-rabbitmq")" ]; then
|
||||
docker create --name mistral-rabbitmq rabbitmq
|
||||
fi
|
||||
docker start mistral-rabbitmq
|
||||
|
||||
while true; do
|
||||
sleep 5
|
||||
docker exec mistral-mysql \
|
||||
mysql -u root -pstrangehat \
|
||||
-e "CREATE DATABASE IF NOT EXISTS mistral;
|
||||
USE mistral;
|
||||
GRANT ALL ON mistral.* TO 'root'@'%' IDENTIFIED BY '${MYSQL_ROOT_PASSWORD}'" \
|
||||
&& break || true
|
||||
done
|
||||
|
||||
sleep 10
|
||||
|
||||
docker exec mistral-mysql mysql -u root -pstrangehat -e "CREATE DATABASE mistral; USE mistral; GRANT ALL ON mistral.* TO 'root'@'%' IDENTIFIED BY 'strangehat'"
|
||||
docker run -dit --link mistral-mysql:mysql --name mistral-db-setup mistral-all cat
|
||||
docker exec mistral-db-setup python /opt/stack/mistral/tools/sync_db.py
|
||||
docker kill -s 9 mistral-db-setup
|
||||
docker rm mistral-db-setup
|
||||
|
||||
docker exec mistral apt-get install -y libmysqlclient-dev
|
||||
docker exec mistral pip install mysql-python
|
||||
docker exec mistral cp mistral.conf mistral.conf.orig
|
||||
docker exec mistral python -c "
|
||||
import ConfigParser
|
||||
c = ConfigParser.ConfigParser()
|
||||
c.read('/home/mistral/mistral.conf')
|
||||
c.set('DEFAULT', 'transport_url', 'rabbit://guest:guest@rabbitmq:5672/')
|
||||
c.set('database','connection','mysql://root:strangehat@mysql:3306/mistral')
|
||||
c.set('pecan', 'auth_enable', 'false')
|
||||
with open('/home/mistral/mistral.conf', 'w') as f:
|
||||
c.write(f)
|
||||
"
|
||||
function run_mistral() {
|
||||
NAME=${1:-mistral}
|
||||
shift || true
|
||||
LINKS='--link mistral-mysql:mysql --link mistral-rabbitmq:rabbitmq'
|
||||
docker run \
|
||||
-d \
|
||||
--name $NAME \
|
||||
$LINKS \
|
||||
${EXTRA_OPTS} \
|
||||
${OPTS} \
|
||||
mistral-all "$@"
|
||||
}
|
||||
|
||||
docker exec mistral python /opt/stack/mistral/tools/sync_db.py --config-file /home/mistral/mistral.conf
|
||||
unset OPTS
|
||||
|
||||
docker restart mistral
|
||||
case "$MODE" in
|
||||
|
||||
echo "
|
||||
Enter the container:
|
||||
docker exec -it mistral bash
|
||||
single)
|
||||
# Single node setup
|
||||
# The CMD of the mistral-all image runs the `mistral-server --server all` command.
|
||||
OPTS="-p 8989:8989" run_mistral
|
||||
|
||||
List workflows
|
||||
docker exec mistral mistral workflow-list
|
||||
echo "
|
||||
Enter the container:
|
||||
docker exec -it mistral bash
|
||||
|
||||
List workflows
|
||||
docker exec mistral mistral workflow-list
|
||||
|
||||
"
|
||||
;;
|
||||
|
||||
multi)
|
||||
# Multinode setup
|
||||
OPTS="-p 8989:8989" run_mistral "mistral-api" mistral-server --server api
|
||||
run_mistral "mistral-engine" mistral-server --server engine
|
||||
run_mistral "mistral-executor-1" mistral-server --server executor
|
||||
run_mistral "mistral-executor-2" mistral-server --server executor
|
||||
|
||||
echo "
|
||||
List workflows
|
||||
docker exec mistral-api mistral workflow-list
|
||||
"
|
||||
;;
|
||||
esac
|
||||
|
||||
"
|
||||
|
||||
Loading…
Reference in New Issue
Block a user