When config volumes of same types are merged, the volumes to be mounted
for the continer is not merged. In order to run neutron container with
ovs agent, '/run/openvswitch' directory mount is required to execute
ovs-vsctl commands. This patch merges the volumes list of the same
config volumes instead of taking only the first one.
Closes-Bug: #1766759
Change-Id: I70972ad2fc339c9316befd74dda8555982f23316
This updates the pull function here so that it matches
how paunch also handle's images, only pulling them if
the existing image doesn't already exist on the host.
Change-Id: I90ea41ccdfdb0b9206a63901554d002a5ec0fd3a
... so we can know how long take resources configuration in Puppet
catalogs, and more easily debug why we have timeouts.
Change-Id: If3fae8837140caae91120e46b4880146ffe22afc
We faced issue where on some environments docker-puppet.py picks up
/etc/hosts as modified, even though that shouldn't be the case (LP bug
1709689). On the last occasion we found the cause to be desynced time
on the host machine of a virtual setup, and subsequent NTP sync of
overcloud nodes (causing a 5 hour skip back in time) racing with
docker-puppet.py.
Still, more info to debug these kinds of issues would be nice to
have. Printing the timestamp of origin_of_time, saving it to a
persistent directory and making sure /etc/localtime is mounted into
docker-puppet containers should make finding the root cause on such
occasions easier.
Change-Id: I2ea197673b470379ead295058b6952cce3a69606
Closes-Bug: #1737954
If docker-puppet.py fails on any config_volume, it can be difficult to
reproduce the failure given all the other entries in docker-puppet.json.
Often to reproduce a single failure, one has to modify the json file,
and remove all other entries, save the result to a new file, then pass
that new file as $CONFIG.
This commit adds the ability to specify $CONFIG_VOLUME, which will cause
docker-puppet.py to only run the configuration for the specified entry
in docker-puppet.json whose config_volume value matches the user
specified value.
Change-Id: I2889647a27a8b891696a6a3e7f78b59a015c2c79
Closes-Bug: #1737043
When new module are added, we may miss the symlink in
/etc/puppet/modules. And for consistency as we mount the
/usr/share/openstack-puppet/modules directory it’s better to add it
to the modulepath.
Change-Id: I963aede41403ebbe3b9afb55a725b304a30a0cbb
Closes-Bug: #1736980
When SELinux is enforcing, use the docker volume mount flag
:z for the docker-puppet tool's bind-mounted volumes in RW mode.
Note, if a volume mount with a Z, then the label will be specific
to the container, and not be able to be shared between containers.
Volumes from /etc/pki mounted RO do not require the context changes.
For those RO volumes that do require it, use :ro,z.
For deploy-steps, make sure ansible file resources in /var/lib/
are enforced the same SELinux context attributes what docker's :z
provides.
Partial-bug: #1682179
Related-bug: #1723003
Change-Id: Idc0caa49573bd88e8410d3d4217fd39e9aabf8f2
Signed-off-by: Bogdan Dobrelya <bdobreli@redhat.com>
puppet run on never fails, even when it should, since we moved
to the ansible way of applying it. The reason is the current following code:
- name: Run puppet host configuration for step {{step}}
command: >-
puppet apply
--modulepath=/etc/puppet/modules:/opt/stack/puppet-modules:/usr/share/openstack-puppet/modules
--logdest syslog --logdest console --color=false
/var/lib/tripleo-config/puppet_step_config.pp
The above is missing the --detailed-exitcodes switch and so puppet will never
really error out on us and the deployment will keep on running all the
steps even though a previous puppet manifest might have failed. This
cause extra hard-to-debug failures.
Initially the issue was observed on the puppet host runs, but this
parameter is missing also from docker-puppet.py, so let's add it there
as well as it makes sense to return proper error codes whenever we call
puppet.
Besides this being a good idea in general, we actually *have* to do it
because puppet does not fail correctly without this option due to the
following puppet bug:
https://tickets.puppetlabs.com/browse/PUP-2754
Depends-On: I607927c2ee5c29b605e18e9294b0f91d37337680
Co-Authored-By: Damien Ciabrini <dciabrin@redhat.com>
Change-Id: Ie9df4f520645404560a9635fb66e3af42b966f54
Closes-Bug: #1723163
We need to account for all the mounted config volumes when generating
the TRIPLEO_CONFIG_HASH in order for paunch to know to restart the
container when any one of the config_volume gets updated.
Change-Id: I473a71f49bd446694da48bb5b7b0a49126df7845
Closes-Bug: #1721306
Some services only mount this directory, not /var/lib/config-data/$service
so handle this case in the docker-puppet code that maps the mounted
volumes to the services when adding the config hash to the container
environment.
Change-Id: I3bdb7609f322458584ac9597ffbfefb057b84646
Closes-Bug: #1720208
Logging in docker puppet was changed. This just makes it a bit more verbose so
you get a sense of what's going on.
Change-Id: Ibda35b67f608a95eb39cb84f0457751d9a61d216
Use a more restrictive mode for these files, as some may contain sensitive data
which shouldn't be world readable
Closes-Bug: #1714986
Change-Id: Ib1e79b1d4e25d6e329938402b1ca776bdab81bdd
Get the path from the CONFIG_VOLUME_PREFIX environment variable.
This is useful for debugging and generate configuration files to
a different directory.
Change-Id: Ib85e3898804312ebb6677a5fa189fbfc357ce27c
Running puppet apply with --logdest syslog results in all the output
being redirected to syslog. You get no error messages. In the case where this fails, the subsequent debug task shows nothing useful
as there was no stdout/stderr.
Also pass --logdest console to docker-puppet's puppet apply so that
we get the output for the debug task.
Related-Bug: #1707030
Change-Id: I67df5eee9916237420ca646a16e188f26c828c0e
Signed-off-by: Bogdan Dobrelya <bdobreli@redhat.com>
Updates hieradata for changes in https://review.openstack.org/471950.
Creates a new service - NovaMigrationTarget. On baremetal this just configures
live/cold-migration. On docker is includes a container running a second sshd
services on an alternative port.
Configures /var/lib/nova/.ssh/config and mounts in nova-compute and libvirtd
containers.
Change-Id: Ic4b810ff71085b73ccd08c66a3739f94e6c0c427
Implements: blueprint tripleo-cold-migration
Depends-On: I6c04cebd1cf066c79c5b4335011733d32ac208dc
Depends-On: I063a84a8e6da64ae3b09125cfa42e48df69adc12
* Debug ansible 'puppet apply' stderr joined stdout, split
by lines.
* Do 'puppet apply' w/o colors, logdest syslog, and given a wanted
modulepath instead of the module puppet, that can't support those
options.
* Bind-mount syslog socket for docker-puppet.py to pass puppet logs
to host OS syslog.
* Fix logging handlers for multiprocess workers in docker-puppet.py.
Related-bug: #1698172
Closes-bug: #1700086
Change-Id: I84112a836e968aa5c3596a6544e0392980529963
Signed-off-by: Bogdan Dobrelya <bdobreli@redhat.com>
This change enables the puppet cron resource in docker-puppet.py and adds user
crontabs to the paths copied from the config containers.
Only the nova crontab is configured for now. Other services will require
similar changes to run their crontabs.
Partial-Bug: 1701254
Change-Id: I2d1d0f0d77908a132472cf4bc475f8bd526af504
Depends-On: Ie16fb4539481a3c192cff8220a97daa4c70467fc
This solves a problem with bind-mounts when the containers are holding
files descriptors open.
At the same time this makes the template more robust to puppet changes
since new config files will be available in the containers without
needing to update the templates.
Partial-Bug: #1698323
Change-Id: Ia4ad6d77387e3dc354cd131c2f9756939fb8f736
The checksum is changing each run because the mtime is different, so force
a specific date such that we only compare the directory contents.
Change-Id: I5ed2b50176f902d7af12b96e650b67b736d59a4a
If you want debug logging you can set the new DockerPuppetDebug
heat parameter to 'True'.
Change-Id: Iae7bb67379351ea15d61c331867d7005f07ba98e
Closes-bug: 1700570
This should help determine what exactly needs to be bind mounted in the
container and should also help limit the size of collected logs in CI,
as collecting the entire /etc directory from each container can grow
pretty quickly in size and is not that useful.
Related-Bug: #1698172
Change-Id: Ie2bded39cdb82a72f0c28f1c552403cd11b5af45
Also attempts to move the workaround for bug #1696283 to before the
puppet apply call.
Closes-Bug: #1696622
Change-Id: I3a195466a5039e7641e843c11e5436440bfc5a01
The configuration generated by docker-puppet may change on update,
so checksum the combined files from the config-data directories,
to enable detecting those that have changed and restarting the
appropriate containers - we need to merge this checksum into
the environment passed to the containters, as this will cause
paunch to correctly restart containers when the configuration
generated changes, even if the rest of the json definition
provided by heat does not.
Change-Id: I40d9080cf3ad708ef4ed91e46d2b2ae1138bb9c3
This helps a bit with debugging issues, and the container will be
deleted on the next run when the same volume is configured.
Change-Id: I4f2f219bd7e40abafd0eb31c1275fdd8ed4db4da
Variables are now passed in with --env in the docker run call.
This will allow docker-puppet.sh to be baked into the image instead of
having it as a custom entrypoint.
Change-Id: Icbaefe033becc6b2226535f28ee202917bdc1074
Log prepared docker command
Use logger stdout instead of print command
Log stderr as debug as well
Change-Id: I3d48fbf4fa3381d325e3be3788b041e06d4bb294
Signed-off-by: Bogdan Dobrelya <bdobreli@redhat.com>
The containers also need to trust the CA's that the overcloud node
trusts, else we'll get SSL verification failures.
bp tls-via-certmonger-containers
Change-Id: I7d3412a6273777712db2c90522e365c413567c49
This is cluttering up the logs with useless error messages, making it
more difficult than necessary to debug the CI job.
Change-Id: Icbdc4c74d99fea39b8722955dab56e5f538849aa
For both containers and classic deployments, allow to configure
policy.json for all OpenStack APIs with new parameters (hash,
empty by default).
Example of new parameter: NovaApiPolicies.
See environments/nova-api-policy.yaml for how the feature can be used.
Note: use it with extreme caution.
Partial-implement: blueprint modify-policy-json
Change-Id: I1144f339da3836c3e8c8ae4e5567afc4d1a83e95
This allows to optionally add volumes, where we could use a heat
conditional to either put the volume path we want or put an empty string
which should be safely skipped.
Change-Id: I68f91ffdd8ceb14735adad1322fcf124c47b160c