162 Commits

Author SHA1 Message Date
Zuul
a189f37216 Merge "yaml-validate: sets isn't python3" 2018-05-14 23:16:57 +00:00
Emilien Macchi
af7a4ab5b8 yaml-validate: sets isn't python3
Run builtin "set" instead of python2 Sets module.

Change-Id: I2f3db189b23e5578f2f02a3a7189a7aa8323acb6
2018-05-09 15:42:43 +00:00
Carlos Camacho
44ef2a3ec1 Change template names to rocky
The new master branch should point now to rocky.

So, HOT templates should specify that they might contain features
for rocky release [1]

Also, this submission updates the yaml validation to use only latest
heat_version alias. There are cases in which we will need to set
the version for specific templates i.e. mixed versions, so there
is added a variable to assign specific templates to specific heat_version
aliases, avoiding the introductions of error by bulk replacing the
the old version in new releases.

[1]: https://docs.openstack.org/heat/latest/template_guide/hot_spec.html#rocky
Change-Id: Ib17526d9cc453516d99d4659ee5fa51a5aa7fb4b
2018-05-09 08:28:42 +02:00
Zuul
6fe62ce661 Merge "Fix sample env data" 2018-05-02 20:18:27 +00:00
Zuul
e24316c4ec Merge "Ensure tasks aren't empty dicts" 2018-05-02 05:05:47 +00:00
Alex Schultz
03c8cbcdc2 Fix sample env data
I934561612d26befd88a9053262836b47bdf4efb0 renamed the rabbit ssl
parameters that we use in the same environment generate but since the
script did not fail, it made it past CI. This change fixes the
RabbitClientUseSsl parameter in the environment to match the new
RpcUseSsl flag and updates the check script to fail if this happens
again.

Change-Id: I47c63875c6934bca2903883787467fc1804ba5da
Closes-Bug: #1768358
2018-05-01 16:05:02 -06:00
Alex Schultz
960fc05b8a Ensure tasks aren't empty dicts
Ansible doesn't like it when tasks are empty dicts so let's make sure
we aren't passing an empty dict in to the ansible tasks. Unfortunately
since we're validating non-heat processed yaml, the list_concats end
up being dicts to the yaml validator so we can't just ensure they are
always lists.

Change-Id: I13dcc4f3d4ea2ac4c7ab62887dfea293246f6530
Related-Bug: #1768019
2018-05-01 15:52:55 -06:00
Emilien Macchi
88bfa38ae5 undercloud-upgrade: use deploy_steps_tasks for yum update
Use deploy_steps_tasks instead of external_deploy_tasks so we execute
the playbook on nodes that match "overcloud" group, which is the case of
the containerized undercloud.

Also add deploy_steps_tasks to tools/yaml-validate.py part of
OPTIONAL_DOCKER_SECTIONS.

Change-Id: Iaa6a05bf864cdb54a000ef74e6c5ab8e627ab0cf
2018-04-30 16:38:32 -07:00
Zuul
5e0775a1ca Merge "Restrict tenant network to geneve" 2018-04-28 10:23:16 +00:00
Zuul
f4fa117cc7 Merge "Add dry-run mode for process templates" 2018-04-27 12:54:37 +00:00
venkata anil
ae4aac8cb5 Restrict tenant network to geneve
OVN doesn't support VXLAN networks. VLAN tenant networks
have limitations and should be blocked till they are fixed.
So we restrict tenant network to geneve.

Change-Id: I4cabde39c252a605d769e137ae402f6fbc5a3041
Closes-Bug: 1767070
2018-04-27 15:40:03 +05:30
Zuul
c71ffb5740 Merge "Add validation on role names" 2018-04-26 09:58:53 +00:00
Zuul
80b12bcbe9 Merge "Move remnants of environments/services-docker" 2018-04-25 18:23:02 +00:00
Dan Prince
3b56c9e501 Add validation on role names
Also, fixes role templates that weren't passing this new validation.

Closes-bug: #1756346

Change-Id: I387386a0d7cf47fe11f24a68b9f2bb2ee2f0b7bd
2018-04-25 15:42:22 +00:00
Zuul
1bec57e977 Merge "Add validation for SoftwareConfig outputs" 2018-04-25 08:44:14 +00:00
Zuul
822bd996b3 Merge "Support separate oslo.messaging services for RPC and Notification" 2018-04-25 04:43:46 +00:00
Zuul
f9a4653f03 Merge "Fail validation on new workflow_tasks" 2018-04-24 14:41:40 +00:00
Zuul
f05507195b Merge "Suppress warnings with -qq" 2018-04-24 14:41:38 +00:00
James Slagle
87c76e8d7a Fail validation on new workflow_tasks
Adds a new validation to check role_data for workflow_tasks. If found,
the validation will fail. Existing templates with workflow_tasks are
excluded for now until they are fully migrated to external_deploy_tasks.

Depends-On: I0cf206e8ceeecfca17dd99a83b8c2eaa50e66712
Change-Id: I830a5cda660f5f73ca3a98f4a886abce6989daf0
implements: blueprint deprecate-workflow-tasks
2018-04-24 04:14:26 +00:00
James Slagle
d696030293 Suppress warnings with -qq
The quiet arg (-q) is cumulative and should suppress warnings when -qq is
specified per the help text. These warnings were previously added and
did not honor -qq.

Depends-On: I0cf206e8ceeecfca17dd99a83b8c2eaa50e66712
Change-Id: Ic9fe4d7934fa6ad9c4d689a5402f43231c245ada
2018-04-24 04:14:03 +00:00
Bogdan Dobrelya
4053399860 Move remnants of environments/services-docker
Move new files, which made it into environments/services-docker.
Ensure YAML validate will not pass for environments/services-docker
any more.

Change-Id: If16cf6bdafa8e10480134d356a7d7787f1c0bd72
Signed-off-by: Bogdan Dobrelya <bdobreli@redhat.com>
2018-04-23 12:01:33 +00:00
Andrew Smith
78bc457585 Support separate oslo.messaging services for RPC and Notification
This commit introduces oslo.messaging services in place of a single
rabbitmq server. This will enable the separation of rpc and
notifications for the continued use of a single backend (e.g.
rabbitmq server) or a dual backend for the messaging communications.

This patch:
* add oslo_messaging_rpc and oslo_messaging_notify services
* add puppet services for rpc and notification
  (rabbitmq and qdrouterd servers)
* add docker services to deploy rpc (rabbitmq or qdrouterd)
  and notify (rabbitmq or shared)
* retains rabbit parameters for core services
* update resource registries, service_net_map, roles, etc.
* update ci environment container scenarios
* add environment generator for messaging
* add release note

Depends-On: Ic2c1a58526febefc1703da5fec12ff68dcc0efa0
Depends-On: I154e2fe6f66b296b9b643627d57696e5178e1815
Depends-On: I03e99d35ed043cf11bea9b7462058bd80f4d99da
Needed-By: Ie181a92731e254b7f613ad25fee6cc37e985c315
Change-Id: I934561612d26befd88a9053262836b47bdf4efb0
2018-04-22 04:33:44 +00:00
Zuul
d8b4385971 Merge "Omit hiera YAML validations for external deploys" 2018-04-21 07:10:16 +00:00
James Slagle
a134b717dd Add validation for SoftwareConfig outputs
SoftwareConfig/StructuredConfig outputs aren't supported with
config-download given that Heat doesn't know what the output values will
since Ansible is applying all configuration after the stack is complete.

This validation will report a warning whenever it finds use of outputs on
these resource types.

After config-download is the default and the Heat driven method is no
longer supported, we can switch this warning to an error.

Change-Id: I44d5ee3bab3d05ab0a59261d15ea915c75b35713
2018-04-20 13:27:22 +00:00
Bogdan Dobrelya
68c60892a1 Omit hiera YAML validations for external deploys
Closes-bug: #1765641

Change-Id: I3e98521f6396acc9e4e1a7b32faec85e988ab0fe
Signed-off-by: Bogdan Dobrelya <bdobreli@redhat.com>
2018-04-20 11:11:19 +02:00
Bogdan Dobrelya
b079bcc3e1 Track global failures flag for YAML validations
Closes-bug: #1764741

Change-Id: Id0313cde5e1d86103940ff4f77f09b79d0d04ec3
Signed-off-by: Bogdan Dobrelya <bdobreli@redhat.com>
2018-04-19 16:44:21 +02:00
Bogdan Dobrelya
7226efce60 Add dry-run mode for process templates
Sometimes it is nice to know which templates
should be rendered into which files. For example,
if we need to define additional checks in the client.

Change-Id: I10d4dffcd3802f62fc824c808728c0b5b4f1002c
Signed-off-by: Bogdan Dobrelya <bdobreli@redhat.com>
2018-04-18 16:41:22 +02:00
Bogdan Dobrelya
0b44170e73 Add validation for hiera interpolation in services
Walk through services' templates role_data to identify
missing hiera interplolation of networks.

Use additionally provided interfaces for validations:
 * search in dicts by keys or values matching some regex,
   entering into lists as an option;
 * safe get values by the discovered paths casted as lists,
   like get_param/get_attr works for heat templates.

Add PyYAML missing to the requirements.txt.

Closes-bug: #1764315

Change-Id: Idef66ee96cbd67d23760a1cce9537ecc157c3429
Signed-off-by: Bogdan Dobrelya <bdobreli@redhat.com>
2018-04-18 10:11:39 +02:00
Emilien Macchi
a5e5041762 pep8: include no-tls-endpoints-public-ip.yaml in validation
no-tls-endpoints-public-ip.yaml is a new file that needs to be validated
among other TLS environments, so we can make sure that EndpointMap will
be constructed correctly with all needed endpoints.

Change-Id: I5e83b37d8fa757065a6dab87d6eeac1c345efd32
2018-04-13 14:05:13 +00:00
Zuul
dddd649ba6 Merge "remove OS::TripleO::Services::CephClient from CephAll" 2018-03-28 16:49:54 +00:00
binhong.hua
58d1e814c6 remove OS::TripleO::Services::CephClient from CephAll
CephClient should be removed from the CephAll role.
The only thing it does is the key set which is already
handled by the ceph mon profile.
if not will cause Duplicate declaration: Class[Ceph::Keys]

Change-Id: I77bbec1edd21cd6a4212a381a1a7712adc4b604f
Related-Bug: 1722633
2018-03-26 02:35:53 +00:00
Zuul
32740eb2ac Merge "Delete rendered nic configs during --clean" 2018-03-23 08:37:29 +00:00
Zuul
0901d8ee5f Merge "Add pre_upgrade_rolling_tasks" 2018-03-21 19:33:59 +00:00
Zuul
b8044a36b6 Merge "Fix newton compat mode for deployed server." 2018-03-21 11:52:27 +00:00
Zuul
2918341f8b Merge "Add check for old-style nic config files" 2018-03-20 08:14:09 +00:00
Zuul
7c466fb751 Merge "docker: add support for TripleO UI" 2018-03-17 11:38:01 +00:00
Bob Fournier
0017b64560 Add check for old-style nic config files
Since old-style nic config files can no longer be used in Queens
or Rocky (the format changed in Ocata), add a check in yaml-validate
to detect that old-style files are in use and list conversion script
that can be used.

Made some changes to the script to ask before overwriting the nic
config file and save a datestamped copy as backup.  In addition,
the script now takes an optional parameter to define location
of run-os-net-config.sh.

Change-Id: Ic56c48fa35ab2f4c1762c0e370be03fbf2e7671c
Closes-Bug: 1753812
2018-03-16 15:34:43 -04:00
Jiri Stransky
ae085825e2 Add pre_upgrade_rolling_tasks
The resultin pre_upgrade_rolling_steps_playbook will be executed in a
node-by-node rolling fashion at the beginning of major upgrade
workflow (before upgrade_steps_playbook).

The current intended use case is special handling of L3 agent upgrade
when moving Neutron services into containers. Special care needs to be
taken in this case to preserve L3 connectivity of instances (with
regard to dnsmasq and keepalived sub-processes of L3 agent).

The playbook can be run before the main upgrade like this:

openstack overcloud upgrade run --roles overcloud --playbook pre_upgrade_rolling_steps_playbook.yaml

Partial-Bug: #1738768
Change-Id: Icb830f8500bb80fd15036e88fcd314bf2c54445d
Implements: blueprint major-upgrade-workflow
2018-03-16 12:37:19 +01:00
Honza Pokorny
7cbe28b5ef docker: add support for TripleO UI
Co-Authored-By: Martin André <m.andre@redhat.com>
Co-Authored-By: Dan Prince <dprince@redhat.com>
Co-Authored-By: Emilien Macchi <emilien@redhat.com>
Partially-Implements: bp tripleo-ui-undercloud-container
Change-Id: I1109d19e586958ac4225107108ff90187da30edd
2018-03-15 23:43:17 +01:00
Sofer Athlan-Guyot
5015cc6dd8 Fix newton compat mode for deployed server.
The ControlPlanePort interface changed in the deployed-server
ressource.  It now takes an additional fixed_ips parameters.

This add the required parameter and is required for FFU testing in CI.

Adjust the validation to take the fixed_ips discrepancies between two
different templates interfaces in deployed-neutron-port and
ctlplane-port.

Change-Id: I58af23129bcba04a367d0169dcafd53d33ab42f2
Closes-Bug: #1755837
2018-03-15 15:52:53 +00:00
Lukas Bezdicka
26c55d15cd FFU: Introduce post FFU steps and use them for qeens switch
In last step of FFU we need to swich repos before running upgrade.
We do so by introducing post FFU steps and running the switch in
them. We also update heat agents and os-collect-config on nodes.

Change-Id: I649afc6fa384ae21edc5bc917f8bb586350e5d47
2018-03-15 14:33:19 +01:00
Zuul
3a2d7fbc3a Merge "Minor update steps for ODL" 2018-03-14 13:20:04 +00:00
Janki Chhatbar
98faacad44 Minor update steps for ODL
Updating OpenStack (within release) means updating ODL from v1 to v1.1.
This is done by "openstack overcloud update" which collects
update_tasks. ODL needs 2 different steps to achieve this
minor update. These are called Level1 and Level2. L1 is
simple - stop ODL, update, start. This is taken care by paunch
and no separate implementation is needed. L2 has extra steps
which are implemented in update_tasks and post_update_tasks.

Updating ODL within the same major release (1->1.1) consists of either
L1 or L2 steps. These steps are decided from ODLUpdateLevel parameter
specified in environments/services-docker/update-odl.yaml.

Upgrading ODL to the next major release (1.1->2) requires
only the L2 steps. These are implemented as upgrade_tasks and
post_upgrade_tasks in https://review.openstack.org/489201.

Steps involved in level 2 update are
 1. Block OVS instances to connect to ODL
 2. Set ODL upgrade flag to True
 3. Start ODL
 4. Start Neutron re-sync and wait for it to finish
 5. Delete OVS groups and ports
 6. Stop OVS
 7. Unblock OVS ports
 8. Start OVS
 9. Unset ODL upgrade flag

These steps are exactly same as upgrade_tasks.
The logic implemented is:
follow upgrade_tasks; when update_level == 2

Change-Id: Ie532800663dd24313a7350b5583a5080ddb796e7
2018-03-08 17:20:47 +05:30
James Slagle
9e0c55a7ac Delete rendered nic configs during --clean
Also delete the newly rendered per-role sample nic config templates when
process-templates.py is called with --clean.

Change-Id: Ic77b3c9e243839201065260b5807d145c4bb9972
2018-03-05 15:50:05 -05:00
Jose Luis Franco Arza
11ff6d0668 Add extra yaml validations in upgrade tasks.
Extra checks:
 - Check that only tags=['common', 'validate',
  'pre-upgrade'] are accepted.
 - Fail if tags not defined.
 - Fail if 'step|int == ' condition inside 'when'
   is not the evaluated first.
 - Suggest the use of lists to append conditions
   inside 'when'.

Change-Id: I15f6d4cb6f2a13d04580779a93a02daf86f8b412
2018-02-21 11:10:42 +01:00
Dan Sneddon
1dec175241 Render NIC config templates with jinja2
This change converts the existing NIC templates to jinja2 in
order to dynamically render the ports and networks according
to the network_data.yaml. If networks are added to the
network_data.yaml file, parameters will be added to all
NIC templates. The YAML files (as output from jinja with
the default network_data.yaml) are present as an example.

The roles in roles_data.yaml are used to produce NIC configs
for the standard and custom composable roles. In order to
keep the ordering of NICs the same in the multiple-nics
templates, the order of networks was changed in the
network_data.yaml file. This is reflected in the network
templates, and in some of the files that is the only
change.

The roles and roles_data.yaml were modified to include
a legacy name for the NIC config templates for the
built-in roles Controller, Compute, Object Storage,
Block Storage, Ceph Storage, Compute-DPDK, and
Networker roles. There will now be a file produced
with the legacy name, but also one produced with the
<role>-role.j2.yaml format (along with environment
files to help use the new filenames).

Note this change also fixes some typos as well as
a number of templates that had VLANs with device:
entries which were ignored.

Closes-Bug: 1737041
Depends-On: I49c0245c36de3103671080fd1c8cfb3432856f35
Change-Id: I3bdb7d00dab5a023dd8b9c94c0f89f84357ae7a4
2018-02-13 00:19:37 -08:00
Bogdan Dobrelya
60a209b45e Ignore same file errors for templates processor
When copying templates or files with the
process-templates.py's shutil, ignore cases when
the source and the destination are same files.

This allows the following scenario:
  - Symlink t-h-t from the installed package to a work dir
  - Process j2 templates with overwrite in the work dir

Required-by: https://review.openstack.org/#/c/542875

Change-Id: I9a9c32f05fde325709998f4fe8bc7fef6c25b5c5
Signed-off-by: Bogdan Dobrelya <bdobreli@redhat.com>
2018-02-12 11:11:30 +01:00
Bogdan Dobrelya
e6946bc5f2 Fix custom output dir for templates processor
Templates processor fails to locate *.j2 files,
when a custom output dir is specified.

Ensure *.j2 templates are on their expected search
paths for upcoming pasring and rendering

Change-Id: Idbc93e27574c66a9a5a73e3fcd7e88647282f201
Closes-bug: #1748425
Signed-off-by: Bogdan Dobrelya <bdobreli@redhat.com>
2018-02-09 14:48:30 +01:00
Zuul
47da89d2ca Merge "Added realtime NFV roles for OVS-DPDK and SR-IOV" 2018-02-08 07:01:23 +00:00
Jan Provaznik
96b82d149e Add support for ceph-nfs manila backend
If ceph-nfs (ganesha) service is enabled, it's set up by ceph-ansible
and it can be used as a manila backend. Manila can be configured to use
ceph either directly (manila-cephfsnative-config-docker.yaml env file)
or through ganesha (environments/manila-cephfganesha-config-docker.yaml
env file).

Change-Id: Ib408c7827e5fba0c1b01388db26363806fc64370
Partially-Implements: blueprint nfs-ganesha
2018-02-06 19:04:39 +00:00