43 Commits

Author SHA1 Message Date
Sergii Golovatiuk
67489e63f0 Increase the step for gnocchi_db_sync
swift-proxy and gnocchi both run on step_4. However, swift_proxy should
be ready before gnocchi. This patch moves gnocchi_db_sync to step_5 to
be sure that swift-proxy is started before gnocchi. gnocchi_db_sync has
higher start_order to start before other gnocchi services.

Change-Id: I0044c78c4a6d751fc9d0cf6b54ed634032933934
Closes-Bug: 1781162
2018-07-12 23:51:31 +01:00
Zuul
4dfb598bcc Merge "Upgrades: Refactor playbooks to set facts" 2018-06-14 11:31:56 +00:00
Lukas Bezdicka
56bec75c02 Upgrades: Refactor playbooks to set facts
To not to redefine variable multiple times in each service we
run check only once and we set fact. To increase readability of
generated playbook we add block per strep in services.

Change-Id: I2399a72709d240f84e3463c5c3b56942462d1e5c
2018-06-08 11:46:12 +02:00
Pradeep Kilambi
b3b027e731 Add a way to override base path when file driver is used
The goal is to be able to point the Gnocchi file driver directory
to an NFS share.

A new parameter GnocchiFileBasePath allows to customize
the bind mount to /var/lib/gnocchi.

Change-Id: I868a368161f4a529e5e7dc3593dc6862e3196247
2018-06-05 07:49:22 +02:00
Mehdi Abaakouk
398bcc7f79 gnocchi: add missing /var/lib/gnocchi
When file driver is selected, Gnocchi write data into /var/lib/gnocchi
bug the data are loss when the container is restarted.

This changes adds the volumes to all Gnocchi containers, so file
driver will work as expected across restart.

Closes-bug: #1773376

Change-Id: I086af01fed4eed609ea6801c483d76c819398979
2018-05-25 19:14:00 +02:00
Carlos Camacho
44ef2a3ec1 Change template names to rocky
The new master branch should point now to rocky.

So, HOT templates should specify that they might contain features
for rocky release [1]

Also, this submission updates the yaml validation to use only latest
heat_version alias. There are cases in which we will need to set
the version for specific templates i.e. mixed versions, so there
is added a variable to assign specific templates to specific heat_version
aliases, avoiding the introductions of error by bulk replacing the
the old version in new releases.

[1]: https://docs.openstack.org/heat/latest/template_guide/hot_spec.html#rocky
Change-Id: Ib17526d9cc453516d99d4659ee5fa51a5aa7fb4b
2018-05-09 08:28:42 +02:00
Emilien Macchi
77aa6763f6 Undercloud upgrades will use upgrade_tasks
Instead of using host_prep_tasks (which are part of deployment tasks),
we'll use the upgrade tasks that are now well known and tested in
previous releases, when the we containerized the overcloud.

Depends-On: Id25e6280b4b4f060d5e3f78a50ff83aaca9e6b1a
Change-Id: Ic199c7d431e155e2d37996acd0d7b924d14af2b7
2018-05-05 06:52:34 +00:00
Emilien Macchi
d86025593b Handle undercloud upgrades via host_prep_tasks
Using host_prep_tasks interface to handle undercloud teardown before we
run the undercloud install.
The reason of not using upgrade_tasks is because the existing tasks were
created for the overcloud upgrade first and there are too much logic
right now so we can easily re-use the bits for the undercloud. In the
future, we'll probably use upgrade_tasks for both the undercloud and
overcloud but right now this is not possible and a simple way to move
forward was to implement these tasks that work fine for the undercloud
containerization case.

Workflow will be:
- Services will be stopped and disabled (except mariadb)
- Neutron DB will be renamed, then mariadb stopped & disabled
- Remove cron jobs
- All packages will be upgraded with yum update.

Change-Id: I36be7f398dcd91e332687c6222b3ccbb9cd74ad2
2018-04-12 18:14:28 -07:00
Zuul
f2e4073265 Merge "FFU: Fix gnocchi FFU tasks" 2018-03-27 18:12:39 +00:00
Lukas Bezdicka
a4a6fb71c5 FFU: Fix gnocchi FFU tasks
We need to register facts in order to stop overriding values on
skipping.

Change-Id: I250c8774dd146a99732b5407a52e983b6342bad8
2018-03-19 18:01:20 -04:00
Zuul
f4a66ba188 Merge "Enable gnocchi-statsd health check" 2018-03-16 14:29:47 +00:00
Martin Mágr
02eb6193e3 Enable gnocchi-statsd health check
This patch enables health check execution for gnocchi-statsd docker container.

Change-Id: I6d0aa53aa9e3fbb4fd47300f8fa6a32574cfda9f
Depends-On: Ib82cb849540694106a869ec81694f1159967ee79
2018-02-26 14:34:32 +00:00
Zuul
d8279a9d7d Merge "FFU: Introduce gnocchi fast-forward-upgrade tasks" 2018-02-21 19:26:15 +00:00
Pradeep Kilambi
4be6bb155a FFU: Introduce gnocchi fast-forward-upgrade tasks
this upgrades the gnocchi services for gnocchi from N -> Q

bp fast-forward-upgrades

Change-Id: I3786f76172dfe3571a4893220ab97506aed1552a
2018-02-20 14:36:31 +00:00
Giulio Fidente
0b1afb48e5 Allows for configuration of the Ceph cluster name
To be able to support multiple Ceph cluster, an initial step is
to allow for configuration of each cluster name.

Depends-On: I8d5293eaaf104b6374dfa13992a67ddc37397f10
Implements: blueprint custom-ceph-cluster-name
Change-Id: I1b4d51ca6a2d08fa7a68eea680eb104eff732057
2018-02-20 11:35:01 +01:00
Lukas Bezdicka
0cb5c847f3 Always evaluate step first in conditional
If we use variables defined in later step in conditional before
checking which step are we on we will fail.

Resolves: rhbz#1535457
Closes-Bug: #1743764
Change-Id: Ic21f6eb5c4101f230fa894cd0829a11e2f0ef39b
2018-02-09 17:12:29 +01:00
marios
dec003def8 Convert tags to when statements for Q major upgrade workflow
This converts "tags: stepN" to "when: step|int == N" for the direct
execution as an ansible playbook, with a loop variable 'step'.
The tasks all include the explicit cast |int.

This also adds a set_fact task for handling of the package removal
with the UpgradeRemovePackages parameter (no change to the interface)

The yaml-validate also now checks for duplicate 'when:' statements

Q upgrade spec @ Ibde21e6efae3a7d311bee526d63c5692c4e27b28
Related Blueprint: major-upgrade-workflow
[0]: 394a92f761/tripleo_common/utils/config.py (L141)
Change-Id: I6adc5619a28099f4e241351b63377f1e96933810
2018-01-08 13:57:47 +02:00
Carlos Camacho
927495fe3d Change template names to queens
The new master branch should point now to queens instead of pike.

So, HOT templates should specify that they might contain features
for queens release [1]

[1]: https://docs.openstack.org/heat/latest/template_guide/hot_spec.html#queens

Change-Id: I7654d1c59db0c4508a9d7045f452612d22493004
2017-11-23 10:15:32 +01:00
Zuul
5da47d2e4f Merge "Set file mode permission for Ceph keyrings in containers" 2017-11-21 01:00:07 +00:00
Zuul
6a72a9f1b4 Merge "Drop step_config as top level docker requirement" 2017-11-17 08:33:29 +00:00
Zuul
a4631060f6 Merge "Write readme.txt into old log directories" 2017-11-15 22:42:34 +00:00
Dan Prince
a307fe7ffc Drop step_config as top level docker requirement
Step config is only required within the puppet_configs section
of docker/services/*. This patch drops the top level 'step_config'
and updates the unit tests accordingly.

Change-Id: I7dc7cfae3ef1965ec95b1d9ef23e7f162418c034
2017-11-15 16:01:16 -05:00
John Fulton
ce7b65f443 Set file mode permission for Ceph keyrings in containers
Pass mode parameter to ceph-ansible for Ceph keyrings on container
host. Pass mode and ownership parameter to each Ceph client container
using kolla_config. ACLs are set for Cinder if it is not running in
containers.

Change-Id: I11618b3fd696739ad9b86618a1f3f96570c61a30
Partial-Bug: #1720787
2017-11-15 15:03:41 +00:00
Jiri Stransky
85ec193403 Write readme.txt into old log directories
This should help operators find the new log files. We do have them
documented, but not everybody reads every word in the docs :)

The readme creation has ignore_errors: true so that if the directory
isn't present at all (e.g. on deployed server environments, which
don't have openstack packages installed), we don't fail the deployment
when we're not able to create the readme.

Change-Id: I6b36db7b7ce8b3e4da566eb7828d0c3b8646a14f
Partial-Bug: #1730957
2017-11-14 10:35:11 +01:00
Jose Luis Franco Arza
4f7753420a Add validation task in docker services [Gnocchi]
Docker services are missing the pre-upgrade validation task
in the upgrade_tasks section which verifies if the service
is running before going on with the upgrade.

Change-Id: I3db9c13709ef0dcc71fbaf51b80c498d7998daac
Partial-Bug: #1704389
2017-11-03 10:41:47 +01:00
Juan Badia Payno
5dbe1121e9 docker: add logging(source & groups)
The services that docker depends on, have logging_sources and logging_groups;
but those are not set on the docker outputs so they are not used when dockers
are deployed.

Added logging_source & logging_groups as docker optional parameters in
tools/yaml-validate.py

Closes-Bug: #1718110
Change-Id: I8795eaf4bd06051e9b94aa50450dee0d8761e526
2017-09-27 07:37:14 +00:00
Dan Prince
9d8e496f3e Run gnocchi statsd and metrcd at step 5
Running these daemons at step 5 should avoid seeing error messages in
the gnocchi-statsd log files on startup which starts at step4.

Change-Id: Idb82f864a2e1c623dab7a2a87054443036670453
Closes-bug: #1713182
2017-09-05 19:00:36 +00:00
Pradeep Kilambi
71059a08cc Mount ceph config on gnocchi statsd
gnocchi-statsd needs access to ceph config. lets mount the
ceph config files so it doesnt throw conf_read_file errors.

Change-Id: I1426d580c8d8d60e986ca859f89eeb8799ab6bd2
2017-08-18 21:22:34 +00:00
Damien Ciabrini
0cb45d65c6 Generate MySQL client config if service requires database
Services that access database have to read an extra MySQL configuration file
/etc/my.cnf.d/tripleo.cnf which holds client-only settings, like client bind
address and SSL configuration. The configuration file is thus used by
containerized services, but also by non-containerized services that still
run on the host.

In order to generate that client configuration file appropriately both on the
host and for containers, 1) the MySQLClient service must be included by the
role; 2) every containerized service which uses the database must include the
mysql::client profile in the docker-puppet config generation step.

By including the mysql::client profile in each containerized service, we ensure
that any change in configuration file will be reflected in the service's
/var/lib/config-data/{service}, and that paunch will restart the service's
container automatically.

We now only rely on MySQLClient from puppet/services, to make it possible to
generate /etc/my.cnf.d/tripleo.cnf on the host, and to set the hiera keys that
drive the generation of that config file in containers via docker-puppet.

We include a new YAML validation step to ensure that any service which depends
on MySQL will initialize the mysql::client profile during the docker-puppet
step.

Change-Id: I0dab1dc9caef1e749f1c42cfefeba179caebc8d7
2017-07-27 13:41:13 -04:00
Jenkins
2185b83560 Merge "Use a single configuration file for specifying docker containers." 2017-07-15 06:19:13 +00:00
Ian Main
e76d84f784 Use a single configuration file for specifying docker containers.
This removes the default container names from all the templates
and uses a single environment file to specify the full container
name and registry from which to pull.  Also does away with most
of DockerNamespace.

Change-Id: Ieaedac33f0a25a352ab432cdb00b5c888be4ba27
Depends-On: Ibc108871ebc2beb1baae437105b2da1d0123ba60
Co-Authored-By: Dan Prince <dprince@redhat.com>
Co-Authored-By: Steve Baker <sbaker@redhat.com>
2017-07-14 22:23:02 +00:00
Giulio Fidente
baf6eee501 Adds network/cidr mapping into a new service property
Makes it possible to resolve network subnets within a service
template; the data is transported into a new property ServiceData
wired into every service which hopefully is generic enough to
be extended in the future and transport more data.

Data can be consumed in service templates to set config values
which need to know what is the subnet where a deamon operates (for
example the Ceph Public vs Cluster network).

Change-Id: I28e21c46f1ef609517175f7e7ee19e28d1c0cba2
2017-07-14 13:44:04 +02:00
Martin André
cf18e865d1 Copy only generated puppet files into the container
This solves a problem with bind-mounts when the containers are holding
files descriptors open.

At the same time this makes the template more robust to puppet changes
since new config files will be available in the containers without
needing to update the templates.

Partial-Bug: #1698323
Change-Id: Ia4ad6d77387e3dc354cd131c2f9756939fb8f736
2017-07-10 11:13:25 +02:00
Martin André
a474ae82d5 Add heat parameter for all of config_volume images
This commit consistently defines a heat template parameter in the form
of DockerXXXConfigImage where XXX represents the name of the
config_volume that is used by docker-puppet.

The goal is to mitigate hard to debug errors where the templates would
set different defaults for the image docker-puppet.py uses to run, for
the same config_volume name.

This fixes a couple of inconsistencies on the way.

Change-Id: I212020a76622a03521385a6cae4ce73e51ce5b6b
Closes-Bug: #1699791
2017-06-28 10:48:53 +02:00
Martin André
af3828437e Make container names consistent
This commit change the container names to consistently use the `_` char
as a word separator and make the kolla external config file match the
container name to make operators' life easier.

Change-Id: Ibac9d76dde474b94c3cb86031ead0fd0327e126f
2017-06-09 09:04:44 +02:00
Martin André
93bb4648f9 Add missing type for RoleParameters parameter
This was forgotten in I72376a803ec6b2ed93903cc0c95a6ffce718b6dc and
broke containerized deployment.

Change-Id: I599a87bf06efbfefd3067c77ed6ca866505900f9
Closes-Bug: #1690870
2017-05-15 19:06:09 +02:00
Saravanan KR
a096ddab34 Add role specific information to the service template
When a service is enabled on multiple roles, the parameters for the
service will be global. This change enables an option to provide
role specific parameter to services and other templates.

Two new parameters - RoleName and RoleParameters, are added to the
service template. RoleName provides the role name of on which the
current instance of the service is being applied on. RoleParameters
provides the list of parameters which are configured specific to the
role in the environment file, like below:

  parameters_default:
      # Default value for applied to all roles
      NovaReservedHostMemory: 2048
      ComputeDpdkParameters:
          # Applied only to ComputeDpdk role
          NovaReservedHostMemory: 4096

In above sample, the cluster contains 2 roles - Compute, ComputeDpdk.
The values of ComputeDpdkParameters will be passed on to the templates
as RoleParameters while creating the stack for ComputeDpdk role. The
parameter which supports role specific configuration, should find the
parameter first in in the RoleParameters list, if not found, then the
default (for all roles) should be used.
Implements: blueprint tripleo-derive-parameters

Change-Id: I72376a803ec6b2ed93903cc0c95a6ffce718b6dc
2017-05-15 10:06:46 +05:30
Flavio Percoco
58a8b282c2 Mount hostpath logs on /var/log
Some containers are using the logs named volume for collecting logs
written to `/var/log`. We should make this consistent for all the
containers.

This patch also cleans up some mounts that weren't needed for some
services. For example, glance-api doesn't need `/run` to be mounted.

Other changes:
* Rework log volumes to hostpath mounts to omit slow COW writes.
* Add kolla_config's permission and host_prep_tasks create and
  manage hostpath mounted log dirs permissions.
* Rework data owning init containers to kolla_config permissions
* When a step wants KOLLA_BOOTSTRAP or DB sync, use logs data owning
  init containers to set permissions for logs. This is required
  because kolla bootsrap and DB sync runs before the kolla config
  stage and there is yet permissions set for logs.
* In order to address hybrid cases for host services vs containerized
  ones to access logs having different UIDs, persist containerized
  services' logs into separate directories (an upgrade impact)
* Ensure host prep tasks to create /var/log/containers/ and /var/lib/
  sub-directories for services
* Fix missing /etc/httpd, /var/www config-data mounts for zaqar/ironic
* Fix YAML indentation and drop strings quotation.

Co-authored-by: Bogdan Dobrelya <bdobreli@redhat.com>
Partial blueprint containerized-services-logs

Change-Id: I53e737120bf0121bd28667f355b6f29f1b2a6b82
2017-05-05 12:30:17 +02:00
Juan Antonio Osorio Robles
c7471e7503 Move containers common volumes from yaql to list_concat
list_concat was introduced recently and is able to replace the yaql
calls for concatenating lists.

Change-Id: Id3a80a0e1e4c25b6d838898757c69ec99d0cd826
2017-05-02 15:00:34 +03:00
Juan Antonio Osorio Robles
e81ddeb685 Introduce common resources for docker templates
This enables common resources that the docker templates might need.
The initial resource only is common volumes, and two volumes are
introduced (localtime and hosts).

Change-Id: Ic55af32803f9493a61f9b57aff849bfc6187d992
2017-04-18 11:20:06 +03:00
Pradeep Kilambi
4713f2b951 Add upgrade tasks for gnocchi container services
Change-Id: I43c35bbf959e5dcdd7e87a8f6a604d5fe5b4f2a9
2017-04-10 13:53:53 +00:00
Martin André
91e7a548cb Remove kolla_config copy from services
Simplify the config of the containerized services by bind mounting in
the configurations instead of specifying them all in kolla config.

This is change is useful to limit the side effects of generating the
config files and running the container is two separate steps as config
directories are now bind-mounted inside the container instead of having
files being copied to the container. We've seen examples of Apache's
mod_ssl configuration file present on the container preventing it to
start when puppet configured apache not to load the ssl module (in case
TLS is disabled).

Co-Authored-By: Ian Main <imain@redhat.com>
Change-Id: I4ec5dd8b360faea71a044894a61790997f54d48a
2017-04-03 18:24:49 +02:00
Pradeep Kilambi
b800b141bf Containerize gnocchi services
Closes-bug: #1668928

Change-Id: I291df31be97c3d55cddb3924482aa5976a79c2b1
2017-03-13 11:41:09 -04:00