110 Commits

Author SHA1 Message Date
Zuul
198d54b0b3 Merge "Add custom role for realtime compute" 2018-01-26 05:27:19 +00:00
Sven Anderson
06bab12f5a Add custom role for realtime compute
This change introduces the ComputeRealTime role, that can be used
Create an initial version of a custom role for real-time compute
nodes.

Partially Implements: blueprint tripleo-realtime

Change-Id: I935cf8a74415b28f50eac041d6d3f92dc1ec1391
2018-01-24 22:43:00 +00:00
Jose Luis Franco Arza
39b7b930ba Add yaml validations for FFU.
A new set of tasks are being added,
the 'fast_forward_upgrade_tasks', which
share structure with 'upgrade_tasks' hence
the same set of validations are being applied
to check the correct tasks structure.

Change-Id: I26898e2df68d80d0721e0467c5220f34835f6ba4
2018-01-23 10:57:48 +01:00
Zuul
1af7729939 Merge "Convert tags to when statements for Q major upgrade workflow" 2018-01-13 09:39:38 +00:00
marios
dec003def8 Convert tags to when statements for Q major upgrade workflow
This converts "tags: stepN" to "when: step|int == N" for the direct
execution as an ansible playbook, with a loop variable 'step'.
The tasks all include the explicit cast |int.

This also adds a set_fact task for handling of the package removal
with the UpgradeRemovePackages parameter (no change to the interface)

The yaml-validate also now checks for duplicate 'when:' statements

Q upgrade spec @ Ibde21e6efae3a7d311bee526d63c5692c4e27b28
Related Blueprint: major-upgrade-workflow
[0]: 394a92f761/tripleo_common/utils/config.py (L141)
Change-Id: I6adc5619a28099f4e241351b63377f1e96933810
2018-01-08 13:57:47 +02:00
John Fulton
a85718428a Add missing CephClient service to ControllerNoCeph role
Change-Id: Ice5b52c1cedaf5662ea9a9058ba155c3e9d84258
Closes-Bug: #1741270
2018-01-04 11:42:42 -05:00
John Fulton
e5cda83954 Add new roles for Ceph containerization
With the move to containers, Ceph OSDs may be combined with other
Ceph services and dedicated Ceph monitors on controllers will be
used less. Popular Ceph roles which include OSDs are Ceph file,
object and nodes which can run all Ceph services. This pattern
will also apply to HCI roles. This change adds the following
pre-composed roles to make it easier for users to use these
patterns:

- CephAll: Standalone Storage Full Role (OSD + MON + RGW + MDS + MGR + RBD Mirroring)
- CephFile: Standalone Scale-out File Role (OSD + MDS)
- CephObject: Standalone Scale-out Object Role (OSD + RGW)
- HciCephAll: HCI Full Stack Role (OSD + MON + Nova + RGW + MDS + MGR + RBD Mirroring)
- HciCephFile: HCI Scale-out File Role (OSD + Nova + MDS)
- HciCephObject: HCI Scale-out Object Role (OSD + Nova + RGW)
- HciCephMon: HCI Scale-out Block Full Role (OSD + MON + MGR + Nova)
- ControllerNoCeph: OpenStack Controller without any Ceph Services

Change-Id: Idce7aa04753eadb459124d6095efd1fe2cc95c17
2017-12-24 07:07:11 +00:00
James Slagle
34b7a81f07 Add -c for clean_templates
Add a new option to process-templates.py that will clean the working
template directory of any generated template files.

Change-Id: I7283d25260c1501964c240c89dd4f658d5c14a3b
2017-12-05 15:56:43 -05:00
Zuul
410027d64f Merge "Add name property where missing" 2017-12-05 18:07:49 +00:00
Zuul
7d0d74891f Merge "Add NovaMigrationTarget service to SR-IOV Compute role" 2017-12-05 16:15:52 +00:00
James Slagle
7a3fc67559 Add name property where missing
All SoftwareDeployment resources should use the name property when using
config-download.

This also adds a validation to check that the name property is set in
yaml-validate.py

Change-Id: I621e282a2e2c041a0701da0296881c615f0bfda4
Closes-Bug: #1733586
2017-12-04 18:01:52 -05:00
Andreas Jaeger
3a4698d8e8 Avoid tox_install.sh for constraints support
We do not need tox_install.sh, pip can handle constraints itself
and install the project correctly. Thus update tox.ini and remove
the now obsolete tools/tox_install.sh file.

This follows https://review.openstack.org/#/c/508061 to remove
tools/tox_install.sh.

Change-Id: I0b36f95fa92f15159156815bc6d8e1add8902182
2017-12-01 07:47:29 +01:00
Saravanan KR
44e1b941d3 Add NovaMigrationTarget service to SR-IOV Compute role
The service NovaMigrationTarget is missing in SR-IOV compute role,
but is required for migration of instances. Added the missing
service to the role. And added validation to avoid such mistakes.
Closes-Bug: #1730275

Change-Id: I49d310b0c61331eef2d2bf5fd05cf67b34095bbb
2017-11-30 11:00:15 +05:30
Carlos Camacho
927495fe3d Change template names to queens
The new master branch should point now to queens instead of pike.

So, HOT templates should specify that they might contain features
for queens release [1]

[1]: https://docs.openstack.org/heat/latest/template_guide/hot_spec.html#queens

Change-Id: I7654d1c59db0c4508a9d7045f452612d22493004
2017-11-23 10:15:32 +01:00
Steven Hardy
9ce86956ff Add external_post_deploy_tasks interface
This adds another interface like external_deploy_tasks, but instead
of running on each deploy step, the tasks are run after the deploy
is completed, so it's useful for per-service bootstrapping such
as is under development for octavia in:

https://review.openstack.org/#/c/508195
https://review.openstack.org/#/c/515402/

These reviews could potentially be reworked to use this interface,
which would avoid the issue where the configuration needs to happen
after all the openstack services are deployed and configured.

As an example, here is how you could create a temp file post deploy:

    external_post_deploy_tasks:
        - name: Test something happens post-deploy
          copy:
            dest: /tmp/debugpostdeploy
            content: "done"

Change-Id: Iff3190a7d5a238c8647a4ac474821aeda5f2b1f8
2017-11-22 18:39:05 +00:00
Dan Prince
a307fe7ffc Drop step_config as top level docker requirement
Step config is only required within the puppet_configs section
of docker/services/*. This patch drops the top level 'step_config'
and updates the unit tests accordingly.

Change-Id: I7dc7cfae3ef1965ec95b1d9ef23e7f162418c034
2017-11-15 16:01:16 -05:00
Oliver Walsh
61fcfca045 Refactor cellv2 host discovery logic to avoid races
The compute service list is polled until all expected hosts are reported or a
timeout occurs (600s).

Adds a cellv2_discovery flag to puppet services. Used to generate a list of
hosts that should have cellv2 host mappings.

Adds a canonical fqdn and that should match the fqdn reported by a host.

Adds the ability to upload a config script for docker config instead of using
complex bash on-liners.

Closes-bug: 1720821
Change-Id: I33e2f296526c957cb5f96dff19682a4e60c6a0f0
2017-11-08 23:20:46 +00:00
Giulio Fidente
c10aa7a043 Update CephPools format in the docker templates to fit ceph-ansible
The format which ceph-ansible uses to describe the list of pools
to be created in the cluster is different from the one which
puppet-ceph uses; this commit updates the description and the
the docker templates accordingly.

Change-Id: I1e5b2c3cbf6ae02c19a2275ca119fed6e173319d
Closes-Bug: #1720373
2017-10-25 09:00:44 +02:00
Zuul
31488edbc4 Merge "Add external deployment tasks executed on undercloud" 2017-10-22 13:02:39 +00:00
Zuul
b1786cc1b0 Merge "Config download support for standalone deployments" 2017-10-16 13:46:34 +00:00
Jiri Stransky
80eff5f4d7 Add external deployment tasks executed on undercloud
Services can define external_deploy_tasks, which are meant to be
executed on the undercloud node. They are step-based as the other
Ansible tasks we have, and they get executed during each deployment
step before the puppet and docker tasks.

These tasks can be used to perform complex actions from the
undercloud, such as executing nested installers like kubespray or
ceph-ansible. This should allow deploying overcloud with a single
Ansible playbook, and without creating Ansible->Mistral->Ansible loop.

Implements: blueprint ansible-config-download
Change-Id: I3dcafb96f5cea5fdcebe2b2012b61a38b0568834
Depends-On: I8491540edf78711f3229eabeda22a17cd55e99c8
2017-10-13 17:24:54 +02:00
Juan Antonio Osorio Robles
850f90bdc8 Add validation to logging templates
This adds a simple validation that checks that the required outputs
are present in the templates for logging to stdout or files. It also
disables checking the usual required parameters (EndpointMap,
ServiceNetMap, etc.) since these are not used.

bp logging-stdout-rsyslog

Change-Id: I1d7d0faa5f9488cfba08107fc87ecd213f07d063
2017-10-13 09:29:54 +00:00
James Slagle
a0e6d30ca2 Config download support for standalone deployments
Presently, "openstack overcloud config download" does not support all
Deployment resources, only those included in the RoleData and are
natively of type group:ansible.

This patch adds support for also pulling all the deployment data for
OS::Heat::SoftwareDeployment (singular) resources applied to individual
servers of any group type. Those resources are mapped to a new nested
stack via the config-download-environment.yaml environment.

The nested stack has the same interface as a SoftwareDeployment but only
creates a OS::Heat::Value resource. The "config download" code will be
updated in a separate patch to read the deployment data from these Value
resources and apply them via ansible.

The related tripleo-common patch (which depends on this patch) is:
I7d7f6b831b8566390d8f747fb6f45e879b0392ba

implements: blueprint ansible-config-download
Change-Id: Ic2af634403b1ab2924c383035f770453f39a2cd5
2017-10-12 22:34:09 +00:00
marios
a953bda0ae Adds pacemaker update_tasks for Pike minor update workflow
Adds update_tasks for the minor update workflow. These will be
collected into playbooks during an initial 'update init' heat
stack update and then invoked later by the operator as ansible
playbooks.

Current understanding/workflow:
 Step=1: stop the cluster on the updated node
 Step=2: Pull the latest image and retag the it pcmklatest
 Step=3: yum upgrade happens on the host
 Step=4: Restart the cluster on the node
 Step=5: Verification: test pacemaker services are running.

https://etherpad.openstack.org/p/tripleo-pike-updates-upgrades

Related-Bug: 1715557
Co-Authored-By: Damien Ciabrini <dciabrin@redhat.com>
Co-Authored-By: Sofer Athlan-Guyot <sathlang@redhat.com>
Change-Id: I101e0f5d221045fbf94fb9dc11a2f30706843806
2017-10-05 14:35:26 +00:00
Juan Badia Payno
5dbe1121e9 docker: add logging(source & groups)
The services that docker depends on, have logging_sources and logging_groups;
but those are not set on the docker outputs so they are not used when dockers
are deployed.

Added logging_source & logging_groups as docker optional parameters in
tools/yaml-validate.py

Closes-Bug: #1718110
Change-Id: I8795eaf4bd06051e9b94aa50450dee0d8761e526
2017-09-27 07:37:14 +00:00
Jenkins
87ade779b8 Merge "Add pep8 check that generated environments are up to date" 2017-09-23 04:37:06 +00:00
Jenkins
904ae85bd0 Merge "Fix upgrades that use Management network" 2017-09-22 21:33:02 +00:00
Jenkins
60cf6a5ff2 Merge "Support for Ocata-Pike live-migration over ssh" 2017-09-22 21:32:28 +00:00
Dan Sneddon
5b9fbc2b2b Fix upgrades that use Management network
Upgrades from older versions using Management network fail.
This patch enables the management network even though it is not
enabled in any of the role definitions. This will allow upgrades
to complete using existing network environment files, without
requiring operators to switch to the new method for defining
which networks are attached to roles. Eventually these older
environment files will be removed.

Change-Id: Iadd12a559f0ad6918958a1355f189187fd327363
Closes-bug: 1717123
2017-09-20 15:24:17 -07:00
Jenkins
52e1a0c943 Merge "Adds post_upgrade_tasks for any service post-upgrade ansible tasks" 2017-09-20 11:16:26 +00:00
Giulio Fidente
09137304b9 Rename service_workflow_tasks into workflow_tasks
Using the service_ prefix seems incoherent with its use in
service_config_settings (vs config_settings).

Change-Id: Ia39f181415bee0071409dabddfa0c5c312915e1f
2017-09-13 17:15:17 +02:00
marios
2e182bffee Adds post_upgrade_tasks for any service post-upgrade ansible tasks
This adds a new config/deployment per role that will come after any
post deploy steps. It drives the same ansible config as the
upgrade_tasks but instead collects the post_upgrade_tasks for any
service in the given role.

The workflow is upgrade_tasks, then post deploy steps (either
puppet/ or docker/ depending on the env) and then the
post_upgrade_tasks added here.

This is added to the pacemaker/cinder-volume.yaml service for now
see the bug below for more info

Change-Id: Iced34fecf02ebddc91df9302de54d2f4c2cab680
Closes-Bug: 1706951
2017-09-12 18:43:16 +03:00
Jenkins
ada47ab7ff Merge "OpenStack containerized qdrouterd service" 2017-09-11 21:48:38 +00:00
Jenkins
e0867ddffa Merge "Stricter heat template version validation" 2017-09-11 11:51:22 +00:00
Ben Nemec
1c9553c37a Add pep8 check that generated environments are up to date
This check ensures that if a parameter is changed that would affect
a generated environment then the environment must be updated before
pep8 will pass.  It will also catch any mistaken hand edits to the
generated files.

bp generated-environments

Change-Id: I2d12992ed55f963285422e1282a4cee06e989b6d
2017-09-07 15:30:31 -05:00
Lars Kellogg-Stedman
e3f25dfbf8 teach yaml-validate.py how to be --quiet
This adds the --quiet (-q) option to yaml-validate.py.  With '-q',
yaml-validate.py will only print warnings or errors. With '-qq',
yaml-validate.py will print only errors. This makes it much easier to
spot problems when working on templates.

This commit does not change the default behavior of yaml-validate.py.

Change-Id: I358f1d4edde03714627d98361b44e6b90ce5e93c
2017-09-07 14:17:27 -04:00
Oliver Walsh
17fd16b9f2 Support for Ocata-Pike live-migration over ssh
In Ocata all live-migration over ssh is performed on the default ssh port (22).
In Pike the containerized live-migration over ssh is on port 2022 as the
docker host's sshd is using port 22.

To allow live migration during upgrade we need to temporarily pin the Pike
computes to port 22 and in the final converge we can switch over to port 2022.

This also changes the default port to 2022 for baremetal computes in Pike to
enable live-migration between baremetal and containerized computes.

Change-Id: Icb9bfdd9a99dc1dce28eb95c50a9a36bffa621b1
Depends-On: I0b80b81711f683be539939e7d084365ff63546d3
Closes-Bug: 1714171
2017-09-07 12:20:34 +01:00
Jose Luis Franco Arza
0be3317f45 Add yaml validation for upgrade_tasks section.
In every ansible task defined within upgrade_tasks it is
necessary to specify the tag 'tags' which are used during
the ansible execution for the upgrade_tasks serialization.

Adding the 'tags' check per upgrade_tasks step into
the YAML validation will allow us to catch if any
service upgrade task is missing this flag.

Change-Id: I8f56a87cc2e9ffc0d827bbb729f6bc3f6ca7550b
2017-09-06 15:21:01 +02:00
John Eckersberg
6a991f6102 OpenStack containerized qdrouterd service
Add the qdrouterd container as an infrastructure component
that provides a messaging backend for olso.messaging. Currently
the qdr role aliases the rabbitmq service.

This patch:
* Add qdrouterd to docker services
* Update environments docker file
* Add global_config_settings to yaml validate

Change-Id: Ief8c09a2728b6e1a1127a53b6df2affecc0ce3c4
2017-09-06 09:05:38 -04:00
James Slagle
cba00abb75 Separate config_volume for ringbuilder
Use a separate config_volume for swift_ringbuilder puppet_config tasks.
This is necessary so that the swift_ringbuilder and swift-storage
services don't both rsync files to the same bind mounted directory.

The rsync command from docker-puppet.py uses --delete-after, so when
they both use the same config_volume, they can end up deleting the files
generated by the other (depending on the order of execution).

Even though a separate config_volume is used, the rings must still end up
in /etc/swift for the swift services containers.  An additional
container init task is used to copy the ring files into
/var/lib/config-data/puppet-generated/swift/etc/swift so that they will
be present when the actual swift services containers are started.

Change-Id: I05821e76191f64212704ca8e3b7428cda6b3a4b7
Closes-Bug: #1710952
2017-08-28 16:04:45 +00:00
Oliver Walsh
d9fa1f6d4a Stricter heat template version validation
The current check just validates that the heat template versions are using the
release alias instead of the date format.

This patch add stricter validation to ensure the heat template versions used
are supported by the release.

Also emits a warning if the most current template
version is not used.

Change-Id: I9e1d9b05fd5aa91c6a26d032043f386b6a9b072d
2017-08-16 18:48:52 +01:00
Jenkins
1ea7c35f4f Merge "Make UpgradeLevelNovaCompute parameters consistent" 2017-08-03 21:53:09 +00:00
Jenkins
b3b9e953a9 Merge "Make many networking parameters consistent" 2017-08-03 14:19:44 +00:00
Jenkins
39a6e47109 Merge "Render isolated network templates using jinja2" 2017-08-03 04:30:48 +00:00
Ben Nemec
7f84409a6a Make UpgradeLevelNovaCompute parameters consistent
There is logic in nova-base.yaml that depends on the default for
this parameter being '', and the nova-compute service only needs it
set to auto during upgrade.  That will be done by [1] anyway, so it
doesn't matter what the default is.  It's also not clear to me that
the nova-compute task is even needed now that we're post-Ocata, but
that's not a change I feel comfortable making.

1: https://github.com/openstack/tripleo-heat-templates/blob/master/environments/major-upgrade-composable-steps.yaml

Change-Id: Iccfcb5b68e406db1b942375803cfedbb929b4307
Partial-Bug: 1700664
2017-08-02 16:20:12 -05:00
Ben Nemec
c05e72cd72 Make many networking parameters consistent
These are mostly the low hanging fruit that only required a few
minor changes to fix.  There are more that require a lot of changes
or might be more controversial that will be done later.

Change-Id: I55cebc92ef37a3bb167f5fae0debe77339395e62
Partial-Bug: 1700664
2017-08-02 16:20:08 -05:00
Ben Nemec
4502b7cba6 Make RoleParameters and key_name descriptions consistent
The key_name default is ignored because the parameter is used in
some mutually exclusive environments where the default doesn't
need to be the same.

Change-Id: I77c1a1159fae38d03b0e59b80ae6bee491d734d7
Partial-Bug: 1700664
2017-08-02 16:18:25 -05:00
Damien Ciabrini
0cb45d65c6 Generate MySQL client config if service requires database
Services that access database have to read an extra MySQL configuration file
/etc/my.cnf.d/tripleo.cnf which holds client-only settings, like client bind
address and SSL configuration. The configuration file is thus used by
containerized services, but also by non-containerized services that still
run on the host.

In order to generate that client configuration file appropriately both on the
host and for containers, 1) the MySQLClient service must be included by the
role; 2) every containerized service which uses the database must include the
mysql::client profile in the docker-puppet config generation step.

By including the mysql::client profile in each containerized service, we ensure
that any change in configuration file will be reflected in the service's
/var/lib/config-data/{service}, and that paunch will restart the service's
container automatically.

We now only rely on MySQLClient from puppet/services, to make it possible to
generate /etc/my.cnf.d/tripleo.cnf on the host, and to set the hiera keys that
drive the generation of that config file in containers via docker-puppet.

We include a new YAML validation step to ensure that any service which depends
on MySQL will initialize the mysql::client profile during the docker-puppet
step.

Change-Id: I0dab1dc9caef1e749f1c42cfefeba179caebc8d7
2017-07-27 13:41:13 -04:00
Jenkins
bf906d75a9 Merge "Deploy Ceph in containers using ceph-ansible via external workflow" 2017-07-27 06:18:28 +00:00
Giulio Fidente
d11e256eed Deploy Ceph in containers using ceph-ansible via external workflow
Add docker profiles to deploy Ceph in containers via ceph-ansible. This is
implemented by triggering a Mistral workflow during one of the overcloud
deployment steps, as provided by [1].

Some new service-specific parameters are available to determine the workflow to
execute and the ansible playbook to use. A new `CephAnsibleExtraConfig`
parameter can be used to provide arbitrary config variables consumed by `ceph-ansible`.

The pre-existing template params consumed up until the Pike release to
drive `puppet-ceph` continue to work and are translated, when possible, into
the equivalent `ceph-ansible` variable.

A new environment file is added to enable use of ceph-ansible;
the pre-existing puppet-ceph implementation remains unchanged and usable
for non-containerized deployments.

1. https://review.openstack.org/#/c/463324/

Change-Id: I81d44a1e198c83a4ef8b109b4eb6c611555dcdc5
2017-07-26 17:37:35 -04:00