49 Commits

Author SHA1 Message Date
Dan Prince
84626c82cc Add docker_puppet_tasks initialization on primary node
This patch adds a new (optional) section to the docker post.j2.yaml
that collects any 'docker_puppet_tasks' data from enabled
services and applies it on the primary role node (the
first node in the primary (first) role).

The use case for this is although we are generally only using
puppet for configuration there are several exceptions that we
desire to make use of today for parity with baremetal. This
includes things like database creation and keystone endpoint
initialization which we rely on configuration via hiera variables
controlled by the puppet services.

Change-Id: Ic14ef48f26de761b0d0eabd0e1c0eae52d90e68a
2017-02-15 13:09:59 -05:00
Dan Prince
ad2ea290be docker: new hybrid deployment architecture and configuration
This patch implements a new docker deployment architecture that
should us to install docker services in a stepwise manner alongside
of baremetal puppet services. This works by using Yaql to select
docker specific services (docker/services/*.yaml) vs the puppet
specific ones and then applying the selected Json to relevant Heat
software deployments for docker and baremetal puppet in a stepwise
fashion.

Additionally the new architecture
leverages new composable services interfaces from Newton to
allow configuration of per-service container configuration
sets (directories that are bind mounted into kolla containers) by
using the Kolla containers themselves. It does this by spinning up
a throw away "configuration only" version of the container being
configured itself, then running the puppet apply in that container and
copying the generated config files into /var/lib/config-data. This
avoids having to install all of the OpenStack dependency packages
in the heat-agent-container itself (our previous approach) and should
allow us to configure a much wider variety of container config files
that would otherwise be impossible with the previous shared approach.

The new approach (combined) should allow us to configure containers in
both the undercloud and overcloud and incrementally add CI coverage to
services as we containerize them.

Co-Authored-By: Martin André <m.andre@redhat.com>
Co-Authored-By: Ian Main <imain@redhat.com>
Co-Authored-By: Flavio Percoco <flavio@redhat.com>

Change-Id: Ibcff99f03e6751fbf3197adefd5d344178b71fc2
2017-02-15 12:56:44 -05:00
Jiri Stransky
246370e490 Containers: Add required EndpointMap parameter
This parameter is passed in by the parent overcloud.yaml template, so we
have to listen accept it in docker/post.j2.yaml, otherwise the
deployment fails.

Change-Id: Ia3fdcfa01d52006a6e9fd0bb02c7379411f3d900
Closes-Bug: #1664569
2017-02-14 14:44:30 +01:00
Dan Prince
9d82796da2 docker: eliminate copy-json.py in favor of json-file
This patch rewires how we configure the Kolla external config files
via Heat templates and uses a more simple json-file heat hook to
directly write out Kolla config files to disk.

By using a heat hook instead of a shell script we can avoid
Json conversion issues.  Additionally, This generic json file hook will
be useful for other ad-hoc Json file configuration within the TripleO
docker architecture.

Co-Authored-By: Martin André <m.andre@redhat.com>
Change-Id: I8c72a4a9a7022f722bfe1cef3e18517605720cce
Depends-On: I2b372ac2e291339e436202c9fe58a681ed6a743f
Depends-On: Id3f779b11e23fd3122ef29b7ccbae116667d4520
2017-01-29 08:16:19 -05:00
Martin André
4320ee55ba Simplify passing config to ovs agent container
The mechanism to pass config files to the neutron-ovs-agent container
was overly complex and not at all justified. This commit removes a few
useless parameters and aligns the neutron-ovs-agents with the rest of
the containers.

Change-Id: Ib9a5985ac9d098731c2fb798d6c9e03cba4b87dd
2017-01-17 13:12:29 +01:00
Jenkins
3de1aa200a Merge "Use provided qemu.conf in libvirt container" 2017-01-13 16:04:22 +00:00
Jenkins
6b8c34b27a Merge "Bump template version for all templates to "ocata"" 2017-01-03 10:31:44 +00:00
Steven Hardy
3c6ec654b4 Bump template version for all templates to "ocata"
Heat now supports release name aliases, so we can replace
the inconsistent mix of date related versions with one consistent
version that aligns with the supported version of heat for this
t-h-t branch.

This should also help new users who sometimes copy/paste old templates
and discover intrinsic functions in the t-h-t docs don't work because
their template version is too old.

Change-Id: Ib415e7290fea27447460baa280291492df197e54
2016-12-23 11:43:39 +00:00
Martin André
356b961145 Use provided qemu.conf in libvirt container
This allows us to workaround an issue where the default setting for
qemu changed to logging to virtlogd which is not yet containerized.

Change-Id: I9d25b1299c4f02068d1073c5b78d4c7a2099721e
Depends-On: I9a9705e5c79eec3bbaf02b491498886b858b7a95
Closes-Bug: #1652119
2016-12-22 19:14:07 +01:00
Steve Baker
5c272e9de3 Use overcloud-full instead of atomic-image
This switches to using overcloud-full as the OS image for
containerized compute. It includes the following changes:
- install docker, until this change lands
  I1eab2a6de721c8f3c21c7df0019f2d4d1cc3775f
- agent image pull has been removed. This avoids a race between docker
  starting and the current call to pull. This relies on "docker run"
  to do the initial pull and leaves open the option of some other
  prefetch mechanism to do the initial pull
- rely on unit Conflicts= to ensure heat-docker-agents and
  os-collect-config do not run at the same time
- tweaks to host bind mounts
- removal of commands which only apply to atomic

Co-Authored-By: Martin André <m.andre@redhat.com>
Change-Id: I2e82634785834a877a4dbdbdcd788a9ac1c14a9d
2016-12-19 18:13:57 +01:00
Steve Baker
e6bc520581 Add bind mounts for agent state
These ensure that software configuration tasks are not re-run when the
heat-agents container is restarted.

Change-Id: Ieb84fe1f6dd849737ff22f51daa12ddc467dcdde
2016-12-19 18:13:57 +01:00
Steve Baker
bb73874310 docker: don't use custom run-os-net-config
The script run-os-net-config[1] copies in ifcfg-* from the host before
running os-net-config. Apparently it was done this way because the
other scripts in /etc/sysconfig/network-scripts/ differed between host
and agent container. This should be less of an issue now that host and
heat-agents run centos-7 (even when the host is atomic)

tripleo-heat-templates recently changed to running os-net-config in a
deployment script instead of an os-refresh-config script [2]. This
means that our current run-os-net-config approach is currently
resulting in os-net-config being executed twice.

Another issue with run-os-net-config is that it copies ifcfg-* from
host to container, but not back again. This means that rebooting the
server will result in unconfigured interfaces until os-net-config is
somehow run again.

This change bind mounts /etc/sysconfig/network-scripts/ from the host
and uses the conventional approach to running os-refresh-config.

This may fix the issue where compute nodes are losing network
connectivity, so
Closes-Bug: #1646897

[1] http://git.openstack.org/cgit/openstack/tripleo-common/tree/heat_docker_agent/run-os-net-config
[2] I0ed08332cfc49a579de2e83960f0d8047690b97a

Change-Id: I763fc8d8e3eb10ac64d33e46c92888d211003e72
2016-12-08 20:09:25 +00:00
Ian Main
6e866224fd Containerized Services for Composable Roles
This change modifies the template interface to support containers and
converts the compute services to composable roles.

Co-Authored-By: Dan Prince <dprince@redhat.com>
Co-Authored-By: Flavio Percoco <flavio@redhat.com>
Co-Authored-By: Martin André <m.andre@redhat.com>
Co-Authored-By: Steve Baker <sbaker@redhat.com>
Change-Id: I82fa58e19de94ec78ca242154bc6ecc592112d1b
2016-11-22 11:42:49 +01:00
Martin André
e7cb607872 Bind mount files to run DiD in latest atomic host
The /usr/bin/docker is a shell script in latest atomic host, pointing
to either docker-latest or docker-current binary. Bind mount the
required files from atomic host to be able to run docker in docker
inside heat-agents container.

Co-Authored-By: Flavio Percoco <flavio@redhat.com>
Change-Id: I97e29f65beb3a3f89c1b42c339e2e89f0fc1d519
2016-10-19 14:31:17 +02:00
Martin André
6d41f69033 Have docker start script honor configuration
The test was always evaluate as true which resulted in
insecure_registry line being set even when DockerNamespaceIsRegistry
was set to false.

Change-Id: Iacb73a4908a6a27082b94fe919734e644ed47b19
2016-10-19 14:31:17 +02:00
Martin André
704a78d342 Be more inclusive in insecure registry regex
The regex failed to match the INSECURE_REGISTRY string used in latest
atomic host image due to it expecting a whitespace after
--insecure-registry.

Change-Id: Ib8f288d844b4d94b0f6309bfd04bb05930d8c4c5
2016-10-06 11:14:20 +02:00
Flavio Percoco
89870f1c94 Update heat-agents setup files
This patch moves the image pull step out of the service heat-agent
service script to ease the service init process and to make it more
reliable. By doing this outside of the service script, it's possible to
know when the `firstboot` script failed and report back.

It also updates the firstboot yaml file to point to the
`tripleoupstream` org.

Co-Authored-By: Flavio Percoco <flavio@redhat.com>
Co-Authored-By: Martin André <m.andre@redhat.com>
Change-Id: I2f0b8092ec69320ee370e1d7d20b8c15c95a1d0d
2016-07-29 14:18:37 +02:00
Flavio Percoco
271aebffb6 Add default value for RoleData
This just adds a default value for `RoleData` in
docker/compute-post.yaml

Change-Id: I96a01dc22e03980b93b32f0f9990f35b83ecfb24
2016-07-29 14:18:37 +02:00
Flavio Percoco
88f83fe8cc Add StepConfig to docker compute-post.yaml
Change I84c97a76159704c2d6c963bc4b26e365764b1366 missed
`docker/compute-post.yaml`.

Change-Id: I680eabf2f316c7fccc9d53d75dc16139c9959c64
2016-07-08 13:32:52 +02:00
Steven Hardy
59b6afcd35 Replace NodeConfigIdentifiers with DeployIdentifier
We added NodeConfigIdentifiers to trigger config to be re-applied on
update, but then later added DeployIdentifier which forces config to
*always* be applied on update, so we can simplify things by just
referencing the DeployIdentifier directly.

Change-Id: I79212def1936740825b714419dcb4952bc586a39
2016-07-04 21:24:16 +01:00
Jenkins
5918136397 Merge "Use docker-cmd hook." 2016-05-27 18:35:14 +00:00
Jenkins
30c8663b9f Merge "Add database directory mount for openvswitchdb" 2016-05-16 08:54:31 +00:00
Ian Main
326199706a Use docker-cmd hook.
This patch switches to use docker-cmd without changing the heat
templates.

Change-Id: I4a6a42819e83e3b70bf1e37c09d155c5cf8a7ee4
2016-05-06 15:05:51 +00:00
Ryan Hallisey
e4578feaef Nova needs the proper volumes to use Cinder
Kolla has been using ceph.  For a while, cinder had
iscsi build into it, but it was removed.  In order to
get this to work with containers again, nova-compute and
libvirt containers need /dev and /lib/udev mounted into their
containers.

We also need to copy nova's rootwrap.conf into the nova.
It was missing this config file.

Change-Id: Ie77f56b4576d5393ad3756b0f5ecc3eeff844d1f
2016-04-04 06:33:44 -04:00
Ryan Hallisey
98f19c17a6 Remove hack the pulls latest docker
Atomic is set to Docker 1.8.2. We no longer need to pull the
latest Docker to make our template work.

Change-Id: I8ab4e135ed4891763f8ced596116b14101466160
Co-Authored-By: Ian Main <imain@redhat.com>
2016-03-30 15:29:39 +00:00
Ryan Hallisey
d2710b871b Centos gives /dev/pts/ptmx the wrong perms
In order to use cinder, we need to be able to use
/dev/pts/ptmx.  Centos sets this to 000 when on Fedora
it's 666.

Change-Id: I76dc5adc64d2da0d27204ea31175244bc1b94428
2016-03-30 15:29:18 +00:00
Ian Main
a340ea6e2d Add database directory mount for openvswitchdb
openvswitchdb creates a db.conf file in /etc/openvswitch.  This just
maps it to the host so we don't lose data.

Change-Id: Ic773ba94522f108a765a09849e2f442ef3ca3bcf
2016-03-11 22:35:43 +00:00
Ryan Hallisey
a0b9dacf95 Remove unused Neutron Agents container
The Neutron Agents is currently not used. Refactor the heat templates
to accommodate for this change.

Change-Id: Ice3c5ce723fa16cfb66c2b0afbe51d7b282c3210
2016-03-09 16:23:07 +00:00
Ryan Hallisey
41a333f874 Allow the containerized compute node to spawn larger VMs
Atomic's root partition & logical volume defaults to 3G.
In order to launch larger VMs, we need to enlarge the root
logical volume and scale down the docker_pool logical volume.
We are allocating 80% of the disk space for vm data and the
remaining 20% for docker images.

Change-Id: If3fff78f476de23c7c51741a49bae227f2cdfe3e
Co-authored-by: Ian Main <imain@redhat.com>
Co-authored-by: Jeff Peeler <jpeeler@redhat.com>
2016-03-09 16:23:07 +00:00
Jenkins
43e0a3df21 Merge "Remove DNS hack." 2016-02-16 11:13:40 +00:00
Jeff Peeler
255f4fd69e Minor fixes to allow local docker registry usage
Changed the heat-docker-agents namespace to use the namespacing
specified in the environment file, which reduces modifications required
on the user when using a local registry.

Changed the start agents script to handle using a local registry both
with a namespace and without.

Change-Id: I16cc96b7ecddeeda07de45f50ffc6a880dabbba6
2016-02-12 15:35:03 -05:00
Ian Main
e7dc316bff Remove DNS hack.
With a properly configured undercloud the DNS is fine.  We can remove
the 8.8.8.8 dns setting.

Change-Id: I8ba98e76f95fd0a6f3f34cb5578e6c3ea7a1d15e
2016-02-12 10:11:08 -05:00
Ryan Hallisey
2928886f88 Allow container template to recognize an update
The deployment resource looks for a change in name when
running an update.  If there is no change in containers,
docker will recognize that and the deployment will return.
If there is a new available container, docker will swap out
the old running container for a new one.

Change-Id: I60d45b5ef45714e6e0140dfc80c14d6a12701f32
2016-01-20 12:39:34 +00:00
Jenkins
795e4290dd Merge "Set the name property for all deployment resources" 2016-01-18 23:16:15 +00:00
Ryan Hallisey
6aa2330b70 Use new heat-docker-agents images
Hosted at tripleoupstream/heat-docker-agents.

Change-Id: I2133a7cb789a34c60b87339d816d29d353cb015f
2016-01-05 08:24:08 -05:00
Ryan Hallisey
db16fd6b59 Network Isolation support for containerized compute
The template will all neutron-agents to be configured so that it can
run the network isolation templates on the containerized compute node.

Co-Authored-By: Dan Prince <dpince@redhat.com>
Change-Id: I7837ed7ed3e807ec5c1276904893695918bef293
2016-01-04 20:41:41 +00:00
Ryan Hallisey
6380ed84d2 Convert JSON generations from bash to python
Python script in the heat template will handle JSON generation
for the containers.

Change-Id: I296fd4a4948f3f937e3a108bc926af6415b350c4
2015-12-17 20:55:15 +00:00
Steve Baker
1733d74392 Set the name property for all deployment resources
There are two reasons the name property should always be set for deployment
resources:
- The name often shows up in logs, files and API calls, the default
  derived name is long and unhelpful
- Sorting by name determines the merge order of os-apply-config, and the
  execution order of puppet/shell scripts (note this is different to
  resource dependency order) so leaving the default name results in an
  undetermined order which could lead to unpredictable deployment of
  configs

This change simply sets the name to the resource name, but a future change
should prepend each name with a run-parts style 2 digit prefix so that the
order is explicitly stated. Documentation for extraconfig needs to clearly
state what prefix is needed to override which merge/execution order.

For existing overcloud stacks, heat currently replaces deployment resources
when the name changes, so this change
Depends-On: I95037191915ccd32b2efb72203b146897a4edbc9

Change-Id: Ic4bcd56aa65b981275c3d4214588bfc4de63b3b0
2015-12-10 14:48:04 +13:00
Ryan Hallisey
0eafa814d5 Point registry at tripleoupstream
The tripleoupstream registry contains images that are built
every time there is a change in delorean.

The gate also needs this.

Change-Id: If460853284588f637de820afa54069f773f2e6f7
2015-11-20 13:21:39 -05:00
Jenkins
99f2910211 Merge "Nova-libvirt needs to bind to /sys/fs/cgroup" 2015-11-20 17:28:57 +00:00
Jenkins
159e78db98 Merge "Add local docker registry support" 2015-11-20 17:28:48 +00:00
Jenkins
338a2bcfb3 Merge "Update docker compute environment to use json config" 2015-11-20 17:26:46 +00:00
Jenkins
ab68379056 Merge "Pin docker version for atomic at 1.8.2" 2015-11-12 15:57:14 +00:00
Ryan Hallisey
266d123286 Change the Atomic image name so it's less specific
The atomic image name in glance was being set to 'fedora-atomic'.
The glance image can be any form of atomic distro so we shouldn't
name this specifically 'fedora-atomic', but instead 'atomic-image'.

Change-Id: Ic539b82b92e3fdd834750e591d8622b7dc85fc6d
2015-11-10 13:28:30 -05:00
Ryan Hallisey
1264b1bba2 Nova-libvirt needs to bind to /sys/fs/cgroup
Nova-compute was mounting in /sys/fs/cgroup when the libvirt
container is the one that actually needs it.

Change-Id: Iae5c7ad24083a8547474611d72f015c4fd23a073
2015-11-10 18:21:22 +00:00
Jeff Peeler
317186995a Pin docker version for atomic at 1.8.2
docker-latest now points to 1.9.0, which isn't the version we're
targetting. More importantly, docker-1.9.0 doesn't work since it
complains about /etc not being able to be relabeled. Not sure how to fix
that, but we can save that battle for another time.

Change-Id: I947b7569d9cf40a409253336e51b4dec5ada36f8
2015-11-09 17:46:01 -05:00
Ryan Hallisey
092bcd9283 Add local docker registry support
Create a set of environment variables that allows us to configure
a docker registry for deployment.  This patch assumes there is a
local docker registry already setup with the images loaded in place.

Change-Id: Iaafaf23eb3fa8b24bcd8f73bb38c552bea629607
Signed-off-by: Ian Main <imain@redhat.com>
Co-Authored-By: Ryan Hallisey <rhallise@redhat.com>
2015-11-02 19:36:51 +00:00
Ryan Hallisey
3a9186d658 Update docker compute environment to use json config
In liberty, Kolla copies around files and runs the service given
a specified command, by reading a json file.

This will update the existing work to follow that template by
creating a json file for each of the services and pushing it
into the containers.

Change-Id: I5085d1896ea965fd8854765b055068a5ad30bcfd
Co-Authored-By: Jeff Peeler <jpeeler@redhat.com>
2015-11-02 19:30:37 +00:00
Dan Prince
65958395f4 Docker compute role configured via Puppet
This change adds a containerized version of the overcloud compute node for
TripleO. Configuration files are generated via OpenStack Puppet modules
which are then used to externally configure kolla containers for
each OpenStack service.

See the README-containers.md file for more information on how to set this up.

This uses AtomicOS as a base operating system and requires that we bootstrap
the image with a container which contains the required os-collect-config agent
hooks to support running puppet, shell scripts, and docker compose.

Change-Id: Ic8331f52b20a041803a9d74cdf0eb81266d4e03c
Co-Authored-By: Ian Main <imain@redhat.com>
Co-Authored-By: Ryan Hallisey <rhallise@redhat.com>
2015-10-08 07:34:26 -04:00