131 Commits

Author SHA1 Message Date
Zuul
f5152da976 Merge "Switch public endpoints to use FQDNs by default" 2018-05-14 23:27:04 +00:00
Carlos Camacho
44ef2a3ec1 Change template names to rocky
The new master branch should point now to rocky.

So, HOT templates should specify that they might contain features
for rocky release [1]

Also, this submission updates the yaml validation to use only latest
heat_version alias. There are cases in which we will need to set
the version for specific templates i.e. mixed versions, so there
is added a variable to assign specific templates to specific heat_version
aliases, avoiding the introductions of error by bulk replacing the
the old version in new releases.

[1]: https://docs.openstack.org/heat/latest/template_guide/hot_spec.html#rocky
Change-Id: Ib17526d9cc453516d99d4659ee5fa51a5aa7fb4b
2018-05-09 08:28:42 +02:00
Juan Antonio Osorio Robles
8e104b3c54 Switch public endpoints to use FQDNs by default
This is in preparation for TLS by default, since the TLS certificate will
use FQDNs for the SubjectAltName, and that will be verified.
This required for us to change both CloudDomain and CloudName to be
required parameters, and not default them to use localdomain. This is to
avoid folks in real deployments using them in their clouds.

Change-Id: Ic70dd323b33596eaa3fc18bdc69a7c011ccd7fa1
2018-05-08 18:16:27 +03:00
Zuul
1e2cdd60aa Merge "Support SshKnownHostsDeployment with config-download" 2018-03-29 21:45:09 +00:00
Zuul
3eb0c62e47 Merge "Remove unused minor update code" 2018-03-19 12:34:21 +00:00
James Slagle
088d5c12f0 Support SshKnownHostsDeployment with config-download
Add support for the SshKnownHostsDeployment resources to
config-download. Since the deployment resources relied on Heat outputs,
they were not supported with the default handling from tripleo-common
that relies on the group_vars mechanism.

Instead, this patch refactors the templates to add the known hosts
entries as global_vars to deploy_steps_playbook.yaml, and then includes
the new tripleo-ssh-known-hosts role from tripleo-common to apply the
same configuration that the Heat deployment did.

Since these deployments no longer need to be triggered when including
config-download-environment.yaml, a mapping is added that can be
overridden to OS::Heat::None to disable the deployment resources when
using config-download.

The default behavior when not using config-download remains unchanged.

Closes-Bug: #1746336
Change-Id: Ia334fe6adc9a8ab228f75cb1d0c441c1344e2bd9
2018-03-19 07:50:06 -04:00
Jiri Stransky
a782462a1a Remove unused minor update code
Since Pike, minor updates are done via the composable services
framework. The old shell script approach hasn't been used/tested for 2
releases now, and should be dropped.

Also drop the UpdateWorkflow interface. Before we started doing
upgrades via Ansible, we used this pluggable resource interface to
perform oneshot operations like migrations to WSGI or AODH
services. Nowadays this interface is not referenced from anywhere and
we'd probably rather do similar operations via Ansible tasks.

Change-Id: I6c5eafe76eb53bc38d100a9ba132dd8fe6dd2d5f
2018-03-15 18:27:14 +01:00
Thomas Herve
0ddfff79ee Do not use the 3rd argument of yaql groupBy
Due to an incompatible change in yaql, it's hard to use the aggregration
of groupBy as the behavior is completely different depending on the
version. Let's try to not rely on it.

Change-Id: I2887011f6baf4867d422579b116b5e143acf5679
Related-Bug: #1750032
2018-02-19 16:32:42 +01:00
Emilien Macchi
e897da3b69 Update YAQL queries with groupBy
YAQL introduced a backward incompatible change in one of its minor
versions:
3fb9178401 (diff-f36776b660e5fe4f88e3295e5b751396R215)

It changes the expected behavior of groupBy() aggregator, so we need to
update our queries otherwise it fails with a "list index out of range"
error.

Change-Id: I2ca2ebb2c8d22aeedbcb6920072db5b6dba3311b
Closes-Bug: #1750032
Co-Authored-By: Alex Schultz <aschultz@redhat.com>
2018-02-17 01:41:12 +00:00
Harald Jensas
77c82f6645 Add subnet property to ctlplane network for server resources
The subnet property is added to puppet/role.role.j2.yaml as
`{{role}}ControlPlaneSubnet`. Roles with a different subnet specified
can be used to deploy a routed network architecture by using one
role per routed network.

When enabling the neutron segments plug-in to support routed-networks
the neutron IPAM code will defer ipallocation unless the port create
request contain enough details. (Ref: LP Bug: #1695740) By adding the
subnet to port create request this change enables tripleo deployment
on an undercloud with Neutron segments plug-in and routed networks.

This depends on a Heat change that improves network logic in server
resource to not replace the current port if new props match what is
on the current interface. Without this adding the subnet property on
update/upgrades would cause a port replacement, which in turn would
cause IPAM info in undercloud neutron to miss-match the deployed
overcloud nodes.

Depends-On: Iab75ec49b962617943017dcaf1b04b89f91a982e
Change-Id: I33804bfd105a13c25d6057e8414e09957939e8af
Implements: blueprint tripleo-routed-networks-deployment
2018-02-02 09:53:03 +00:00
Zuul
4be52761af Merge "Allow defaults for Image/Flavor/RoleParameters in roles_data.yaml" 2018-01-24 02:28:57 +00:00
Steven Hardy
d006711426 Allow defaults for Image/Flavor/RoleParameters in roles_data.yaml
This allows specific roles, e.g ComputeRealTime to specify defaults
where the services are the same as some existing roles but a different
image and/or configuration are needed.

Inspired by discussion of this requirement in:
  https://review.openstack.org/#/c/531739/

RoleParametersDefaults is merged with the user provided parameters
with precendence to user parameters, as this is a special parameter,
which contains a map of the actual parameters to be applied to a
role.

Partially Implements: blueprint tripleo-realtime
Change-Id: I6497144340d3b9276e6ed141d3bc655bfbbeb53c
2018-01-19 11:06:36 +05:30
James Slagle
d4a5876e57 Also pass blacklisted hostnames
Workflows may need access to the list of blacklisted hostnames so they
can filter on that value. This change adds that input to the workflow
execution environment.

Change-Id: I41de32b324a406633699d17933ae05417b28c57b
Partial-Bug: #1743046
2018-01-15 15:26:11 +01:00
James Slagle
79570ed2b9 Workflow execution blacklist support
Workflows triggered from deploy-steps.j2 were not honoring the
blacklist, particularly ceph-ansible. This patch starts to address that
issue by passing in a list of blacklisted ip addresses to the workflow
execution environment that the workflow can make use of to filter
against ctlplane_service_ips.

Change-Id: Ic158171c629e82892e480f1e6903a67457f86064
Partial-Bug: #1743046
2018-01-15 15:25:49 +01:00
Zuul
f253c0d08d Merge "Add EndpointMapOverride parameter" 2017-12-04 23:09:48 +00:00
Zuul
fb8cad4b97 Merge "Add parameter ExtraHostFileEntries" 2017-11-28 11:40:41 +00:00
Steven Hardy
0f49e8eb8b Add EndpointMapOverride parameter
This allows overriding the calculated EndpointMap, which is useful
in some cases such as deploying compute-only stacks which reference
some existing endpoints from a controlplane stack.

The values can be generated like:

openstack stack output show controlplane EndpointMap

Or with https://review.openstack.org/#/c/521969/ applied to heatclient:

openstack stack output show controlplane EndpointMap --format yaml \
 | grep -A 1000 output_value | sed "s/^/  /" \
 | sed "1s/^/parameter_defaults:\n  EndpointMapOverride:\n/" | sed "/output_value/d"

Change-Id: Ie1185b99db1b0db93acaf0deae05bd7b707b442f
2017-11-23 11:59:00 +00:00
Steven Hardy
f2915552b9 Add parameter ExtraHostFileEntries
This allows passing hosts entries from an external stack, or
that references other external nodes.  This is useful in the
case where you want to split e.g controlplane nodes from compute
or similar.

As an example you could do something like this to generate an
environment file:

openstack stack output show controlplane HostsEntry | grep controller\
 | sed "s/|//g" | sed "s/^ */    - /" \
 | sed "1s/^/parameter_defaults:\n  ExtraHostFileEntries:\n/"

Change-Id: If26f87e3384e242ae637650adce7c5dba1611cdf
2017-11-23 11:58:25 +00:00
Carlos Camacho
927495fe3d Change template names to queens
The new master branch should point now to queens instead of pike.

So, HOT templates should specify that they might contain features
for queens release [1]

[1]: https://docs.openstack.org/heat/latest/template_guide/hot_spec.html#queens

Change-Id: I7654d1c59db0c4508a9d7045f452612d22493004
2017-11-23 10:15:32 +01:00
Oliver Walsh
61fcfca045 Refactor cellv2 host discovery logic to avoid races
The compute service list is polled until all expected hosts are reported or a
timeout occurs (600s).

Adds a cellv2_discovery flag to puppet services. Used to generate a list of
hosts that should have cellv2 host mappings.

Adds a canonical fqdn and that should match the fqdn reported by a host.

Adds the ability to upload a config script for docker config instead of using
complex bash on-liners.

Closes-bug: 1720821
Change-Id: I33e2f296526c957cb5f96dff19682a4e60c6a0f0
2017-11-08 23:20:46 +00:00
Steven Hardy
87735ac4bc Expose role tags via heat RoleTags output
This will enable easier consumption of the tags e.g via
the dynamic ansible inventory or workflows that run ansible

Change-Id: I8810240d012ff369991e5e6098e36aa3713e2615
2017-10-31 11:32:15 +00:00
Tim Rozet
9285cb5fc9 Fixes dynamic networks falling back to ctlplane
Currently when a network in network_data is disabled it no port
definitions for that network will be created per role.  This results in
no fallback to the ctlplane IP because overriding a type in
network-isolation to noop.yaml does nothing when the port does not exist
for the role.

This patch changes the IPs when a network is disabled to be the same IPs
as ctlplane and fixes the issue, along with removing the need to use
noop.yaml override for ports (non-vip).

Closes-Bug: 1721542

Change-Id: I301370fbf47a71291614dd60e4c64adc7b5ebb42
Signed-off-by: Tim Rozet <trozet@redhat.com>
2017-10-05 09:59:49 -04:00
Jenkins
12d437db66 Merge "fluentd: log configuration was not generated correctly" 2017-09-06 20:49:15 +00:00
Steven Hardy
4a4f678308 Fix hardcoded references to deprecated *ExtraConfig parameters
These were missed in the previous refactor in role.role.j2.yaml,
we shouldn't reference these via hard-coded values or they become
mandatory in the roles_data.yaml

Change-Id: I014e7d6679c5733b17243d647eaad228c276585a
Closes-Bug: #1711656
2017-09-01 11:14:06 +01:00
Lars Kellogg-Stedman
d9db0c5f4f fluentd: log configuration was not generated correctly
fluentd hiera elements were being set in all_nodes.json, but then were
overwritten by values in <role>.json (e.g., controller.json). This
commit removes the values from all-nodes.json and ensures that they
are set correctly in <role>.json.

Closes-Bug: #1713240
Change-Id: I2b4c74c2a807f8e2fed57112f06b3791701bbe95
2017-08-26 10:28:34 -04:00
Steven Hardy
c6207379db Move deprecated SchedulerHints logic to overcloud.j2.yaml
The changes in puppet/role.role.j2.yaml should have been made
to overcloud.j2.yaml, because we don't want the hard-coded reference
to the deprecated name in the parent template.  Note we need to
pass this value from the parent template so the %index% substitution
works, which is required for predictable placement via *SchedulerHints

Partial-Bug: #1711656
Change-Id: Ided1802daac48d737f53caa7093df814ba101dd0
2017-08-23 11:16:24 +01:00
Jenkins
0c6437eb5b Merge "Render VIPs dynamically based on network_data.yaml" 2017-08-16 15:30:29 +00:00
Steven Hardy
46279be9cb Add RoleConfig output
This exposes the deploy workflow for all roles from deploy-steps
via overcloud.j2.yaml - which means we can write it via the new
openstack overcloud config download command and/or run the workflow
outside of heat via mistral

With https://review.openstack.org/#/c/485732/ applied to
tripleoclient it becomes possible to do:

openstack overcloud config download --config-dir tmpconfig
cd tmpconfig/tripleo-EvEZk0-config
ansible-playbook -b -i /usr/bin/tripleo-ansible-inventory deploy_steps_playbook.yaml

This runs the deploy steps, exactly the same as normally run via heat
via ansible-playbook for all overcloud nodes (--limit can be used to restrict
to specific nodes/roles).

Change-Id: I96ec09bc788836584c4b39dcce5bf9b80e914c71
2017-08-12 10:40:41 +00:00
Steven Hardy
d8e2531820 Convert compute-role.yaml to role.role.j2.yaml
Add some special-casing for backwards compatibility, such that the
Compute role can be rendered via j2 for support of composable networks.

Change-Id: Ieee446583f77bb9423609d444c576788cf930121
Partially-Implements: blueprint composable-networks
2017-08-11 15:06:34 +01:00
Dan Sneddon
b19b88bd1c Render VIPs dynamically based on network_data.yaml
This change modifies the templates to dynamically define the VIPs
based on network_data.yaml. If a network is defined and marked
with "vip: true" in network_data.yaml, it will be included in the
overcloud.yaml which defines the deployment-level resources.

This should make it possible to create custom networks and
use them for services which use high-availability through VIPs.

Also, extraconfig/nova_metadata/krb-service-pricipals.yaml
was modified to dynamically produce the FQDN map for VIPs on
isolated networks, to match overcloud.j2.yaml.

Depends-On: If074f87494a46305c990a0ea332c7b576d3c6ed8
Depends-On: Iab8aca2f1fcaba0c8f109717a4b3068f629c9aab
Partially-implements: blueprint composable-networks
Closes-bug: 1667104
Change-Id: I71339a6ac41133e95dbc3f93abb7a9fdeb0f2da0
2017-08-04 09:44:31 -04:00
Jenkins
45c95100cf Merge "Add environment for setting a custom domain name" 2017-08-03 21:52:28 +00:00
Ben Nemec
c05e72cd72 Make many networking parameters consistent
These are mostly the low hanging fruit that only required a few
minor changes to fix.  There are more that require a lot of changes
or might be more controversial that will be done later.

Change-Id: I55cebc92ef37a3bb167f5fae0debe77339395e62
Partial-Bug: 1700664
2017-08-02 16:20:08 -05:00
Ben Nemec
736b1e1bf3 Add environment for setting a custom domain name
Just setting CloudDomain won't make the domains used consistent.
There are a number of CloudName parameters that must be set as well.
This change adds a sample environment that includes all of those
parameters so it is easy to set everything consistently.

Also fixes the description of CloudNameCtlplane to reflect the
actual use for that parameter.

Change-Id: I56d1c1c5619f83c16c4e8350aa84fccc3d748425
2017-07-27 09:07:29 -06:00
Giulio Fidente
68e582323e Handles {controller,NovaCompute}ExtraConfig deprecation in ovecloud.j2
We missed to parse and merge {controller,NovaCompute}ExtraConfig data
in change [1].

Also fixes whitespaces handling in docker-steps.j2 and
puppet-steps.j2 previously updated by [2].

1. Id37de5864138edd5476c097a8a1f0763faeaf768
2. I36a642fbc2076ad9e4a10ffc56d6d16f3ed6f27a

Change-Id: Ia9983bc991eb79e479855993c1c8819ddfb52e38
2017-07-17 11:54:54 +02:00
Giulio Fidente
8b81b363fd Add role_merged_configs into workflow executions environmentxi
Merges per-role config settings into merged_config_settings which
is wired into the workflow executions environment.

Useful to consume role config settings from within a workflow.

Change-Id: Id37de5864138edd5476c097a8a1f0763faeaf768
2017-07-14 18:41:32 +02:00
Giulio Fidente
baf6eee501 Adds network/cidr mapping into a new service property
Makes it possible to resolve network subnets within a service
template; the data is transported into a new property ServiceData
wired into every service which hopefully is generic enough to
be extended in the future and transport more data.

Data can be consumed in service templates to set config values
which need to know what is the subnet where a deamon operates (for
example the Ceph Public vs Cluster network).

Change-Id: I28e21c46f1ef609517175f7e7ee19e28d1c0cba2
2017-07-14 13:44:04 +02:00
Steven Hardy
b108289aa6 Remove special-case of memcache node ips for ipv6
This should be handled in puppet-tripleo, as is done for some other
services e.g ceph.  This has also been identified as a possible
performance problem due to the nested get_attr calls.

Change-Id: I7e14f0219c28c023c4e8e1d4693f0bfa9674d801
Related-Bug: #1684272
Depends-On: Iccb9089db4b382db3adb9340f18f6d2364ca7f58
2017-07-13 13:39:36 -07:00
Jenkins
25df46b3b8 Merge "Use ServerOsCollectConfigData value in output" 2017-07-13 11:28:09 +00:00
Jenkins
c76feac1a9 Merge "Add DeployedServerEnvironmentOutput" 2017-07-13 02:04:21 +00:00
Zane Bitter
b5c110f1c1 Add dependency relationship between nested get_attr targets
Starting with Pike, Heat will do attribute resolution in a single pass. A
consequence of this is that when the result of a get_attr is passed to
another get_attr call, there must be a dependency relationship between the
resources so that the inner attribute is resolved first before we try to
determine which attributes are required from the resource in the outer
call.

There are two uses of nested dep_attr in the overcloud template. One (which
hopefully can be removed soon) is in the allNodesConfig resource. In this
case, the {{primary_role_name}}IpListMap already depends on the
ServiceNetMap.

The second is in the KeystoneAdminVip output. This patch makes the VipMap
depend on the ServiceNetMap so that attributes can be resolved in a single
pass in that case.

Change-Id: I438a79748b9b408ec1101271d96c60d84028b57e
2017-07-11 15:52:37 -04:00
James Slagle
b524e0faf5 Use ServerOsCollectConfigData value in output
Just use the value from the ServerOsCollectConfigData resource in the
output instead of recalculating the value for each role via jinja.

Change-Id: I4e3bf4f25c9a8f677d5d177eb409594193a86405
2017-07-10 09:36:22 -04:00
James Slagle
d0acf56606 Add DeployedServerEnvironmentOutput
Add a new output, DeployedServerEnvionmentOutput, that can be used as
the contents of an environment file to input into a services only stack
when using split-stack. The parameter simplifies the manual steps needed
to deploy split-stack.

By default, the resource that generates the output is mapped to
OS::Heat::None.

implements blueprint split-stack-default
Change-Id: I6004cd3f56778f078a69a20e93a0eba0c574b3db
2017-07-10 09:36:22 -04:00
Steven Hardy
afbbbdc894 Add ServerIdMap output
This exposes the nova server IDs for each role, and the bootstrap node
so that we can add this data to the tripleo dynamic ansible inventory

Change-Id: I2fc48eec77210805c0139fa4abcbf4dd721e7c37
2017-06-29 10:32:07 +01:00
Jenkins
f7062d83d7 Merge "Provides a list of per-service ctlplane IPs to the workflows env" 2017-06-27 10:04:14 +00:00
Jenkins
e2da9f0338 Merge "Allows use of Mistral workflows during deployment steps" 2017-06-27 02:30:00 +00:00
Jenkins
d24690765c Merge "Add VipMap output" 2017-06-26 23:59:14 +00:00
Jenkins
8c778263a8 Merge "Add split-stack environments" 2017-06-26 23:59:05 +00:00
Jenkins
eda12ae632 Merge "Add os-collect-config data as an output" 2017-06-26 23:58:51 +00:00
Giulio Fidente
9c1940e461 Provides a list of per-service ctlplane IPs to the workflows env
Adds in the execution environment of the workflow steps a list of
per-service network IPs. This can be used by the workflows to
execute actions against the nodes hosting a given service.

Change-Id: Id7c735d53f04f6ad848b2f9f1adaa3c84ecd2fcd
Implements: blueprint tripleo-ceph-ansible
2017-06-26 16:32:02 +02:00
Giulio Fidente
71f1338816 Allows use of Mistral workflows during deployment steps
Introduces a general mechanism meant to allow for the execution
of workflows during the deployment steps.

Services can define workflow actions to be triggered during a step
in the newly added service_workflow_tasks section. The syntax is:

  service_workflow_tasks:
    step2:
      - name: my_action_name
        action: std.echo
        input:
          output: 'hello world'

Implements: blueprint tripleo-ceph-ansible
Depends-On: If02799e7457ca017cc119317dfb2db7198a3559f
Depends-On: Ibc5707f9f06266fe84ad1dd91dcb984157871d30
Change-Id: I36a642fbc2076ad9e4a10ffc56d6d16f3ed6f27a
2017-06-26 16:28:23 +02:00