228 Commits

Author SHA1 Message Date
Cédric Jeanneret
fb7ea6734e Flatten rabbitmq service - step 1
This flattens rabbitmq and removes puppet parts. The next step will
move the flattened templates to their final location.

It's split in two steps in order to make reviews easier on that big change.

Change-Id: I30f0802770d86d64e2ec6fa93dc9a608d4b15d69
2019-02-05 15:44:40 +01:00
Zuul
10d612bb12 Merge "UX - Useful error msg if role is not in roles data" 2019-01-31 07:54:14 +00:00
Zuul
70068d220d Merge "Look for parameters in parameter_groups" 2019-01-29 02:35:33 +00:00
Zuul
560ec36685 Merge "Add network data for the undercloud" 2019-01-25 18:05:32 +00:00
Thomas Herve
ae2ccb5f41 Remove workflow_tasks
I don't think it's used anymore.

Change-Id: I928b53d7388e460da3b26306b9f3c548808d329e
2019-01-24 11:08:53 +01:00
Zuul
d385118e8d Merge "Address python3 string issues with subprocess" 2019-01-24 00:36:38 +00:00
Zuul
ac8ebf638c Merge "Remove deprecated TLS-related environment files" 2019-01-23 22:17:10 +00:00
Bogdan Dobrelya
a619d990c0 Address python3 string issues with subprocess
Follows up Id0060a3abbcda8edb6124eb096cb824aaea48396.
This patch updates our Popen calls to enable universal newlines for
calls that we parse or consume the output for. Without
univeral_newlines=True, the output is treated as bytes under python3
which leads to issues later where we are using it as strings.

See https://docs.python.org/3/glossary.html#term-universal-newlines
Signed-off-by: Bogdan Dobrelya <bdobreli@redhat.com>

Change-Id: I1a82c14d8c894cb6ea6c0c322c3fe5b71b34a11e
2019-01-23 16:08:57 +01:00
Harald Jensås
c1116e59c1 Add network data for the undercloud
The undercloud needs the External network for the external VIP
address. It uses the _from_pool template creating a fake neutron
port.

Other networks are not required, and should be removed to optimize
the number of heat resources and plan rendering.

NOTE: Skips validation of network_data_undercloud.yaml against
network_data.yaml, as is already done to openshift and routed
networks networks data examples.

Related Bug: #1809313
Change-Id: Ib11a134df93e59947168b40bc71fb1da9172d4ac
2019-01-21 19:35:37 +01:00
Harald Jensås
53027484ae Skip templating disabled networks
Ignore disabled networks when rendering templates.

Add's the ctlplane network to maps to ensure we don't
end up with no keys/values in map_replace functions.

Also some Jinja cleanup:
 - Reduce the number of times we iterate over networks
   where we can.
 - Add's indentation to make the code easier to read.

Related-Bug: #1809313
Depends-On: I2e8135bc9389d3bf1a6ef01e273515af5c488a9a
Change-Id: Ifeb2d2d1acb43c16a5bf29e95965776494d61fef
2019-01-21 19:35:37 +01:00
Harald Jensås
3fa6349089 UX - Useful error msg if role is not in roles data
If the user specify a role name that does not exist
in the provided roles data the scipt exits with a
StopIteration error. Catch it and raise RuntimeError
with user friendly error message.

Closes-Bug: #1812530
Change-Id: I704316f66c197668a7d8e373efe00889776d2a85
2019-01-20 04:18:13 +01:00
Dan Prince
6fefd102bf Look for parameters in parameter_groups
Look for used parameters in parameter_groups

This patch searches for parameter use in the parameter_groups section
as well and should eliminate some false positive warnings with
our validations.

Change-Id: I67c5ad2b6e865d454489702e5eb263a9508d26d2
2019-01-18 10:09:46 -05:00
Juan Antonio Osorio Robles
a72f8d4ae9 Remove deprecated TLS-related environment files
The ones in environments/ssl/ are preferred instead. These have been
available since pike.

Change-Id: I84a7b354ede46d6ec88964e5dcbd5678d89c8c0f
Depends-On: I5a905ec7499a6faa08cbcacfccb19a6e424e4a80
2019-01-18 09:57:48 +00:00
Dan Prince
144b74d3b4 Look for used parameters in conditionals
This patch searches for parameter use in conditions as well
and should eliminate a lot of false positive warnings with
our validations.

Change-Id: I33eba9d46d8c83b7a34c39fdfcd35b62f52c8752
2019-01-16 13:07:48 -05:00
Harald Jensås
c740b54214 Remove default role-name from merge network param script
The merge-new-params-nic-config-script.py previosly had the
'Controller' role as the default for --role-name. It is not
obvious that this parameter must be changed when merging
nic config templates.

Remove the default and make the argument required. Improves
UX since user error is less likely.

Making the mistake of using a Role with too many networks
is'nt as forgiving since we now only pass parameters for
the role.networks.

Related-Bug: #1800811
Change-Id: Iff9e364db66ad09a30ac10a7814a3c01d50caf58
2019-01-12 13:16:18 +00:00
Zuul
825ae19190 Merge "Designate - Use net_cidr_map for rndc_allowed_addresses" 2019-01-10 21:13:25 +00:00
Zuul
86755894f7 Merge "Apache - Use net_cidr_map for proxy_ips" 2019-01-10 21:13:22 +00:00
Zuul
8f5fb5144d Merge "flatten sshd service configuration" 2019-01-08 06:50:55 +00:00
Zuul
845bc3e845 Merge "Remove MongoDB" 2019-01-07 18:39:49 +00:00
Harald Jensås
5666a4fe0d Designate - Use net_cidr_map for rndc_allowed_addresses
Change I68e064d23ec5d43f59146d974cae604d2c5fdb52 makes
NetCidrMapValue a list of ip networks.

The designate service configures the dns backend security
to contol from wich addresses updates are allowed. We
should use the list of cidr's associated with the
DesignateApiNetwork to allow all nodes in the network to
remotely control the nameserver.

Partial: blueprint tripleo-routed-networks-templates
Change-Id: I5c5cd51c8f127e8879c5528883c3abd261f4a5b3
2019-01-06 18:20:58 +01:00
Harald Jensås
da1de3aafc Apache - Use net_cidr_map for proxy_ips
Change I68e064d23ec5d43f59146d974cae604d2c5fdb52 makes
NetCidrMapValue a list of ip networks.

Pass the list of cidr's from the ApacheNetwork entry in
the cidr map to 'apache::mod::remoteip::proxy_ips:'.

Partial: blueprint tripleo-routed-networks-templates
Change-Id: Ieb6aff9889136f0ccbec32e36b46140aa7826019
2019-01-06 18:20:38 +01:00
Zuul
0e68a0e30f Merge "L3 routed networks - subnet fixed_ips (3/3)" 2019-01-04 20:38:01 +00:00
Emilien Macchi
be07f991b6 Remove MongoDB
MongoDB support was stopped in Pike, it is not used anywhere now.
Therefore, in Stein are removing it to clean things up.

Change-Id: I4ec8f35b1dd71c25cfb41cc54105ac743ef67745
2019-01-04 15:17:00 +00:00
Harald Jensås
2f2d8183e6 L3 routed networks - subnet fixed_ips (3/3)
When using neutron routed networks we need to specify
either the subnet or a ip address in the fixed-ips-request
when creating neutron ports.

a) For the Vip's:

Adds VipSubnetMap and VipSubnetMapDefaults parameters in
service_net_map.yaml. The two maps are merged, so that the
operator can override the subnet where VIP port should be
hosted. For example:

parameter_defaults:
  VipSubnetMap:
    ctlplane: ctlplane-leaf1
    InternalApi: internal_api_leaf1
    Storage: storage_leaf1
    redis: internal_api_leaf1

b) For overcloud node ports:

Enrich 'networks' in roles defenition to include both
network and subnet data. Changes the list to a map
instead of a list of strings. New schema:

- name: <role_name>
  networks:
    <network_name>
      subnet: <subnet_name>

For backward compatibility a conditional is used to check
if the data is a map or not. In either case the internal
list of role networks is created as '_role_networks' in
the jinja2 templates.

When the data is a map, and the map contains the 'subnet'
key the subnet specified in roles_data.yaml is used as
the subnet in the fixed-ips-reqest when ports are created.
If subnet is not set (or role.networks is not a map) the
default will be {{network.name_lower}}_subnet.

Also, since the fixed_ips request passed to Vip ports are no
longer [] by default, the conditinal has been updated to
test for 'ip_address' entries in the request.

Partial: blueprint tripleo-routed-networks-templates
Depends-On: I773a38fd903fe287132151a4d178326a46890969
Change-Id: I77edc82723d00bfece6752b5dd2c79137db93443
2019-01-03 19:07:20 +01:00
Zuul
35f01e48c3 Merge "Process the templates even if j2_excludes file is not present" 2019-01-03 05:43:14 +00:00
David J Peacock
67e74a676c flatten sshd service configuration
This change realigns the sshd baremetal puppet service yaml config
files into a common hierachy as with the rest of this blueprint.

This change also removes container functionality, since this was a
temporary measure to proxy live-migration connections from
non-containerized to containerized compute nodes during upgrade.

Change-Id: I87e112a0f1973fa3b0e959777e00071c2bbf7c9c
Related-Blueprint: services-yaml-flattening
2018-12-19 13:04:08 -05:00
Zuul
1201d46ee3 Merge "YAML NIC Config 2 script - fix comment indentation" 2018-12-12 05:50:32 +00:00
Alex Schultz
623924972f Update yaml-validate for deployment/ folders
Since we're looking at flattening the services into a deployment/
folder, we need to update the validation script to also handle this
directory structure. Additionally this change updates the service name
validation to ensure that the service name in matches the start of the
filename itself.

Change-Id: Ibb140a38b69a8780adf69362e0f437b3426f360d
Related-Blueprint: service-yaml-flattening
2018-12-06 08:06:10 -07:00
Saravanan KR
435e846779 Process the templates even if j2_excludes file is not present
It is possible to use the process_templates script to generate the
user's jinja templates based on role and network data. But the script
expects the presence of j2_excludes file. Making it as optional, allows
users to generate templates in user's template directory itself, like

  $ /usr/share/openstack-tripleo-heat-templates/tools/process-templates.py \
         -p ~/templates/ \
         -r ~/templates/roles_data.yaml \
         -n ~/templates/network_data.yaml

Closes-Bug: #1806351
Change-Id: I375cd9ff9b40bbdad34d0732ec8abd25fbdde46e
2018-12-03 15:02:39 +05:30
zhulingjie
5edaac5246 Python 3 compatibility: convert raw_input to input
Change-Id: I86699cc6d880c137e80183d7196874ec0f7893e7
Closes-Bug: #1797760
2018-11-12 16:03:47 +00:00
Zuul
90d022a129 Merge "Added all keystone log files to fluentd" 2018-11-09 06:42:08 +00:00
Martin André
306162694a Add network data for use with openshift deployments
Provide a network data file with only the External, InternalApi and the
Storage networks for use with OpenShift deployments.

Take out the IPv6 addresses from the network data until OpenShift can
be deployed in an IPv6 environment.

This also disables the validation for the network_data_openshift.yaml
since it expects network_data files to only add new networks to the
existing pre-defined overcloud networks and we're doing the opposite
here -- only keeping a subset of the networks.

Change-Id: I6d8c08c12ae9002e6386f26c5a8bcf70b8eda4b7
2018-11-05 09:42:10 +01:00
Harald Jensås
f7a359cd0e Merge new params - nic-config templates
Utility script to merge new parameters into existing nic-config
templates. Uses process-templates.py rendered 'single-nic-vlans'
templates as reference and appends any parameters that is not
already present in the existing NIC template.

New NIC template parameters were introduced in:
  https://review.openstack.org/#/c/580236/

When upgrading the existing NIC templates have to have these
new parameters merged.

Change-Id: I474e57878212d2cb7c2b392a5fdf4e449f783a66
2018-10-23 19:59:37 +02:00
Zuul
1bd8888d79 Merge "Rename no-tls environment" 2018-10-20 02:03:33 +00:00
Juan Badia Payno
69626cc3a6 Added all keystone log files to fluentd
Add all the keystone log files to fluentd, so
fluentd is aware of all the keystone log files

Added the new parameters to the exclusion ones

Depends-On: Ifd5fbf6509addf4a564ff83c4551525c9a139ff4
Depends-On: Id1d58637967ffb0e9bd0a83c3cbca699432f5378
Change-Id: I48c957496f7fb36d2128c545d5bcd1499e9e9bf6
2018-10-17 15:29:36 +02:00
Zuul
58f6604f47 Merge "Remove unused tls-cert-inject.yaml template" 2018-10-17 11:56:50 +00:00
Steven Hardy
b278f6c476 Remove unused tls-cert-inject.yaml template
This is no longer handled as the TLS handling tasks were converted
to ansible, and in the context of this series we need to remove it
because it references bootstrap_nodeid

Partial-Bug: #1792613
Change-Id: Ib32177b116f148f007574847320566e32240cf96
2018-10-12 11:12:25 +01:00
Juan Antonio Osorio Robles
3ecbf827ec Rename no-tls environment
It was using a wrong name, which came by accident since it was
introduced to the sample environment generator.

Change-Id: I154af6d0b7ebf5cd339d5d06eaaf9b1ab66814b0
Related-Bug: #1796022
2018-10-12 11:16:35 +03:00
Juan Antonio Osorio Robles
cb3c72f37d Remove references to logging_source
This has been unused for a while, and even deprecation was scheduled
(although the patch never merged [1]). So, in order to stop folks
getting confused with this, it's being removed.

[1] https://review.openstack.org/#/c/543871/

Change-Id: Iada64874432146ef311682f26af5990469790ed2
2018-10-08 13:43:47 +03:00
Harald Jensås
0b58798c8e YAML NIC Config 2 script - fix comment indentation
When translating comments into yaml the last_non_comment_spaces
must be set to the current spaces prior to insering the comment.

Change-Id: Ib5bbb47cfce7d6c0ac0990a3c9384f5143dd1263
2018-10-06 10:17:01 +00:00
Bob Fournier
47f47c1dea In process-templates script write output files to provided dir when using base path
When running the process-templates script with both the -o OUTPUT_DIR and
-p BASE_PATH options the output files were not being written to the output
directory.  This fix splits out the file path from the base path to properly
write the files.

Change-Id: I845e8a2cbd2b12a4a1552b2cfa3ac013466da6bd
Closes-Bug: #1794769
2018-09-28 15:06:44 +02:00
Juan Antonio Osorio Robles
90234f4f2a Remove references to logging_group
This has been unused for a while, and even deprecation was scheduled
(although the patch never merged [1]). So, in order to stop folks
getting confused with this, it's being removed.

[1] https://review.openstack.org/#/c/543871/

Change-Id: Icc6b51044ccc826f5b629eb1abd3342813ed84c0
2018-08-29 13:43:30 +03:00
Zuul
5100f950ef Merge "Fix logic around heat output exclusions handling" 2018-08-20 19:26:42 +00:00
Zuul
5fadfd093f Merge "Add host routes to subnets" 2018-08-14 19:40:21 +00:00
James Slagle
5710f5b35f Fix logic around heat output exclusions handling
The logic in the validation was incorrect in that it would actually
error (return 1) if args.quiet >= 2, even though the file was excluded.

This commit fixes that bug, as well as improves the message around
excluded files, which should not be a warning, only informative.

Change-Id: I2cd8cd84a7ebb952e3c39f99a460177b9be9c2e1
2018-08-13 14:15:07 -04:00
Jiri Stransky
6364f2286c Update and upgrade tasks for services deployed via external deploy tasks
Composable service templates can now define external_update_tasks and
external_upgrade_tasks. They are meant for update/upgrade logic of
services deployed via external_deploy_tasks. The external update
playbook first executes external_update_tasks and then
external_deploy_tasks, the procedure for upgrades works
analogously. All happens within a single playbook, so variables or
fact overrides exported from the update/upgrade tasks will be
available to the deploy tasks during the update/upgrade procedure.

Partial-Bug: #1783949
Change-Id: Ib2474e8f69711cd6610a78884d5032ffd19ad249
2018-08-02 15:04:15 +02:00
Martin Mágr
b76d7623ac QDR for metrics collection purposes
This patch adds composable new service (QDR) for containerized deployments.
Metrics QDR will run on each overcloud node in 'edge' mode. This basically
means that there is a possibility that there will be two QDRs running
on controllers in case that oslo messaging is deployed. This is a reason why
we need separate composable service for this use case.

Depends-On: If9e3658d304c3071f53ecb1c42796d2603875fcd
Depends-On: I68f39b6bda02ba3920f2ab1cf2df0bd54ad7453f
Depends-On: I73f988d05840eca44949f13f248f86d094a57c46
Change-Id: I1353020f874b348afd98e7ed3832033f85a5267f
2018-07-31 21:55:45 +00:00
Harald Jensås
4e44547533 Add host routes to subnets
This change adds a new routes field to the network
definition in network_data.yaml. This field contains
a list of network routes in JSON, e.g.
  [{'destination':'10.0.0.0/16','nexthop':'10.0.0.1'}].

This list is used to set the ``host_routes`` property
of each networks subnet.

Co-Authored-By: Dan Sneddon <dsneddon@redhat.com>
Partial: blueprint tripleo-routed-networks-templates
Depends-On: Ifc5aad7a154c33488a7613c8ee038c92ee6cb1a7
Change-Id: I33b34f1445f4203fbf25edeb093b37c7494c664f
2018-07-30 09:42:19 +02:00
Cédric Jeanneret
d37308d63c Fix python3 support in yaml-validate script.
Python3 drops the "iteritems" method for dict in favor to
"items".

Change-Id: I8d21233e917b5d36be385d59f1a7c9a0588aea2d
Closes-Bug: 1783988
2018-07-27 15:20:32 +02:00
Zuul
87c03bf6b8 Merge "Add a ComputePPC64LE role" 2018-07-23 00:33:07 +00:00