11119 Commits

Author SHA1 Message Date
Jason Anderson
5c760df3d5 [horizon] Move 'balance' HAProxy keyword
The "balance" keyword is not valid in a frontend section. From the
HAProxy documentation[1]:

balance <algorithm> [ <arguments> ]
balance url_param <param> [check_post [<max_wait>]]
  Define the load balancing algorithm to be used in a backend.
  May be used in sections :   defaults | frontend | listen | backend
                                 yes   |    no    |   yes  |   yes

When running HAProxy using the "split" template style, where a
frontend/backend pair are used instead of one listen section, HAProxy
will emit warnings for the Horizon config due to this.

[1]: https://www.haproxy.org/download/1.5/doc/configuration.txt

Closes-Bug: #1872540
Change-Id: I91cee275d91a51944298618493f4ea0cd80282cc
2020-04-16 09:25:18 +00:00
Zuul
969159cc17 Merge "Fix live migration to use migration int. address" 2020-04-12 06:14:09 +00:00
Zuul
ee21a7a503 Merge "Allow operators to use "ceilometer-upgrade" parameters" 2020-04-12 05:45:48 +00:00
Rafael Weingärtner
6fcccdae59 Allow operators to use "ceilometer-upgrade" parameters
Allow operators to use custom parameters with the ceilometer-upgrade
command. This is quite useful when using the dynamic pollster subsystem;
that sub-system provides flexibility to create and edit pollsters configs,
which affects gnocchi resource-type configurations. However, Ceilometer
uses default and hard-coded resource-type configurations; if one customizes
some of its default resource-types, he/she can get into trouble during
upgrades. Therefore, the only way to work around it is to use the
"--skip-gnocchi-resource-types" flag. This PR introduces a method for
operators to execute such customization, and many others if needed.

Depends-On: https://review.opendev.org/#/c/718190/
Change-Id: I92f0edba92c9e3707d89b3ff4033ac886b29cf6d
2020-04-11 14:09:06 +00:00
Zuul
b9f07bc3c4 Merge "Set mistral to use internalURL in admin-openrc.sh" 2020-04-11 10:39:40 +00:00
Zuul
573390303c Merge "CI: do 12 attempts to connect to test instance" 2020-04-10 20:17:06 +00:00
Marcin Juszkiewicz
bfd661a600 CI: tell Nova which CPU to emulate on AArch64
AArch64 does not have a way to get cpu features from libvirt.

Change-Id: Ieed404e17e8a9829f38c03f7ee7fdb3caa3919e8
2020-04-10 16:50:22 +00:00
Marcin Juszkiewicz
6f24582eb9 CI: do 12 attempts to connect to test instance
AArch64 needs a bit more time to boot testing instance. So give it more
time by doubling amount of connect attempts.

Change-Id: I87ca65691dfbac84349e8af24d2f36f1db1c8be1
2020-04-10 16:50:04 +00:00
Radosław Piliszek
20f39e4a36 Ignore .vscode/ in Git
This is the directory used by Visual Studio Code for customized
workspace preferences.

Change-Id: I8bf38a5b3b8a4fe89e6b0c611e8dcb4b9a9f3302
2020-04-10 15:55:42 +02:00
Xing Zhang
1763afc255
Set mistral to use internalURL in admin-openrc.sh
mistralclient osc plugin does not support cacert and insecure [1]
mistralclient interface support fixed in [2]

[1] https://bugs.launchpad.net/python-mistralclient/+bug/1715091
[2] https://review.opendev.org/#/q/topic:bug/1854339

Change-Id: I44726b12358bc3c5898ba952371fb838693aca2c
2020-04-10 21:20:29 +08:00
Zuul
9d217e92aa Merge "Introduce /etc/timezone to Debian/Ubuntu containers" 2020-04-10 10:38:37 +00:00
Zuul
6cd0641edf Merge "Perform host configuration during upgrade" 2020-04-09 19:53:13 +00:00
Dincer Celik
4b5df0d866 Introduce /etc/timezone to Debian/Ubuntu containers
Some services look for /etc/timezone on Debian/Ubuntu, so we should
introduce it to the containers.

In addition, added prechecks for /etc/localtime and /etc/timezone.

Closes-Bug: #1821592
Change-Id: I9fef14643d1bcc7eee9547eb87fa1fb436d8a6b3
2020-04-09 18:53:36 +00:00
John Garbutt
628c27ce9e Fix live migration to use migration int. address
In kolla ansible we typically configure services to communicate via IP
addresses rather than hostnames. One accidental exception to this was
live migration, which used the hostname of the destination even when
not required (i.e. TLS not being used for libvirt).

To make such hostnames work, k-a adds entries to /etc/hosts in the
bootstrap-servers command. Alternatively users may provide DNS.

One problem with using /etc/hosts is that, if a new compute host is
added to the cloud, or an IP address is changed, that will not be
reflected in the /etc/hosts file of other hosts. This would cause live
migration to the new host from an old host to fail, as the name cannot
be resolved.

The workaround for this was to update the /etc/hosts file (perhaps via
bootstrap-servers) on all hosts after adding new compute hosts. Then the
nova_libvirt container had to be restarted to pick up the change.

Similarly, if user has overridden the migration_interface, the used
hostname could point to a wrong address on which libvirt would not
listen.

This change adds the live_migration_inbound_addr option to nova.conf. If
TLS is not in use for libvirt, this will be set to the IP address of the
host on the migration network. If TLS is enabled for libvirt,
live_migration_inbound_addr will be set to migration_hostname, since
certificates will typically reference the hostname rather than the
host's IP. With libvirt TLS enabled, DNS is recommended to avoid the
/etc/hosts issue which is likely the case in production deployments.

Change-Id: I0201b46a9fbab21433a9f53685131aeb461543a8
Closes-Bug: #1729566
2020-04-09 18:17:07 +00:00
Zuul
2d8edc374d Merge "Add support for encrypting backend Keystone HAProxy traffic" 2020-04-09 16:10:50 +00:00
James Kirsch
b475643c11 Add support for encrypting backend Keystone HAProxy traffic
This patch introduces an optional backend encryption for Keystone
service. When used in conjunction with enabling TLS for service API
endpoints, network communcation will be encrypted end to end, from
client through HAProxy to the Keystone service.

Change-Id: I6351147ddaff8b2ae629179a9bc3bae2ebac9519
Partially-Implements: blueprint add-ssl-internal-network
2020-04-09 09:22:55 +00:00
Mark Goddard
1d70f509e3 Perform host configuration during upgrade
This is a follow up to I001defc75d1f1e6caa9b1e11246abc6ce17c775b. To
maintain previous behaviour, and ensure we catch any host configuration
changes, we should perform host configuration during upgrade.

Change-Id: I79fcbf1efb02b7187406d3c3fccea6f200bcea69
Related-Bug: #1860161
2020-04-08 17:03:22 +01:00
Zuul
fa49143f10 Merge "Update hacking for Python3" 2020-04-06 06:43:46 +00:00
Zuul
7c92e56cfd Merge "Separate per-service host configuration tasks" 2020-04-05 16:40:27 +00:00
Zuul
3122e6151c Merge "manila share container name variable" 2020-04-05 16:40:24 +00:00
linpeiwen
fa16190910 manila share container name variable
manila share container name variable is fixed in some places,
but in the defaults directory, manila share container_name variable
is variable. If the manila share container_name variable is changed
during deployment, it will not be assigned to container name,
but a fixed 'manila_share' name.

Change-Id: Iea23c62518add8d6820b76b16edd3221906b0ffb
2020-04-04 22:53:31 -04:00
Zuul
fa65e0402b Merge "CI: Avoid tox (and clean up gate setup)" 2020-04-04 16:06:42 +00:00
Andreas Jaeger
454489762c Update hacking for Python3
The repo is Python 3 now, so update hacking to version 3.0 which
supports Python 3.

Fix problems found by updated hacking version.

Remove hacking and friends from lower-constraints, they are not needed
during installation.

Change-Id: I7ef5ac8a89e94f5da97780198619b6facc86ecfe
2020-04-04 17:00:17 +02:00
Zuul
e631367122 Merge "Remove default(omit) from openstack_cacert in templates" 2020-04-04 12:50:52 +00:00
Radosław Piliszek
ed1b74a35b CI: Avoid tox (and clean up gate setup)
We don't need tox nor tell our users to use it when doing kolla
builds.
Tox is going away from base infra images.
It's already gone from aarch64 (arm64) ones.

Change-Id: I2eb5203ad93c011b8806f4b6fb56db081c14a2cb
2020-04-03 18:04:52 +01:00
Mark Goddard
0edad7138c Remove default(omit) from openstack_cacert in templates
The use of default(omit) is for module parameters, not templates. We
define a default value for openstack_cacert, so it should never be
undefined anyway.

Change-Id: Idfa73097ca168c76559dc4f3aa8bb30b7113ab28
2020-04-03 14:49:11 +01:00
Zuul
0c1b6c8a4b Merge "Use unittest.mock instead of PyPI mock" 2020-04-03 13:29:25 +00:00
Zuul
450e3d209b Merge "Be less confusing about custom Docker registry" 2020-04-03 11:45:25 +00:00
Zuul
9b6cbcfe9b Merge "CI: Always use upper-constraints" 2020-04-03 11:45:22 +00:00
Zuul
5b86b94307 Merge "Avoid unconditional fact gathering" 2020-04-02 16:50:26 +00:00
Radosław Piliszek
5648ad6e9d Use unittest.mock instead of PyPI mock
Now that py2 is gone, oslotest dropped dependency on mock and will
soon affect Ussuri CI [1], let's use unittest.mock built in py3.

This also fixes py38 jobs and proactively prevents py36 and py37
failing due to [1]. This is because we never included mock in
test-requirements (but in lower-constraints where it does not
really belong at all) and instead relied on oslotest to bring
it in.

[1] https://review.opendev.org/716322

Change-Id: I30e82e2d87418272a71c7ee089a8acdaf8872158
2020-04-02 18:21:01 +02:00
Zuul
47e2baf9f4 Merge "Support setting Kafka storage volume" 2020-04-02 15:11:05 +00:00
Mark Goddard
fdea19a305 Separate per-service host configuration tasks
Currently there are a few services that perform host configuration
tasks. This is done in config.yml. This means that these changes are
performed during 'kolla-ansible genconfig', when we might expect not to
be making any changes to the remote system.

This change separates out these host configuration tasks into a
config-host.yml file, which is included directly from deploy.yml.

One change in behaviour is that this prevents these tasks from running
during an upgrade or genconfig. This is probably what we want, but we
should be careful when any of these host configuration tasks are
changed, to ensure they are applied during an upgrade if necessary.

Change-Id: I001defc75d1f1e6caa9b1e11246abc6ce17c775b
Closes-Bug: #1860161
2020-04-02 13:51:56 +00:00
Mark Goddard
e0ba55a85d Avoid unconditional fact gathering
One way to improve the performance of Ansible is through fact caching.
Rather than gather facts in every play, we can configure Ansible to
cache them in a persistent store. An example Ansible configuration for
doing this is as follows:

[defaults]
gathering = smart
fact_caching = jsonfile
fact_caching_connection = ./facts
fact_caching_timeout = 86400

This does not affect Kolla Ansible however, since we use the setup
module which unconditionally gathers facts regardless of the state of
the cache. This gets worse with large inventories limited to a small
batch of hosts via --limit or serial, since the limited hosts must
gather facts for all others.

One way to detect whether facts exist for a host is via the
'module_setup' variable, which exists only when facts exist. This change
uses the 'module_setup' fact to determine whether facts need to be
gathered for hosts outside of the batch. For hosts in the batch, we
switch from using the setup module to gather_facts on the play, which
can use the 'smart' gathering logic.

Change-Id: I04841fb62b2e1d9e97ce4b75ce3a7349b9c74036
Partially-Implements: blueprint performance-improvements
2020-04-02 12:39:46 +00:00
Zuul
c791943356 Merge "Fix HAProxy prechecks during scale-out with limit" 2020-04-02 09:38:57 +00:00
Zuul
1a9dfa3101 Merge "kolla-toolbox container name variable" 2020-04-02 09:38:56 +00:00
Zuul
5ffbdfdff7 Merge "Add clients ca_file in heat" 2020-04-02 09:38:54 +00:00
Zuul
7387220132 Merge "Add glance_ca_certificates_file when using self sign cert in glance" 2020-04-02 09:38:52 +00:00
Radosław Piliszek
c033ddca08 Fix ovs fw driver for the other ovs agent
In [1] only neutron-openvswitch-agent was fixed and not xenapi.
That merged in Ussuri and went cleanly into Train.
In Stein and Rocky, the backport was not clean and
accidentally fixed xenapi instead of the regular one.

Neither the original bug nor its incomplete fix were released,
except for Rocky. :-(
Hence this patch also removes the confusing reno instead of
adding a new one.

[1] https://review.opendev.org/713129

Change-Id: I331417c8d61ba6f180bcafa943be697418326645
Closes-bug: #1869832
Related-bug: #1867506
2020-04-01 20:15:07 +02:00
Radosław Piliszek
641b22b2ef CI: Always use upper-constraints
In stable branches we are getting hit by more py2-incompats.
Let's pin u-c in all CI pip invocations.

Change-Id: Ie2bcc7c115cd2aaf4639d90803216011b346daf3
2020-03-30 20:42:08 +02:00
Doug Szumski
b758883418 Support setting Kafka storage volume
Not everyone wants Kafka data stored on a Docker volume. This
change allows a user to flexibly control where the data is stored.

Change-Id: I2ba8c7a85c7bf2564f954a43c6e6dbb3257fe902
2020-03-30 11:00:03 +02:00
Radosław Piliszek
06a6b56f14 Be less confusing about custom Docker registry
Docker registry password is, by default, sourced from
passwords.yml file.

Cleans up globals.yml to make it clearer.
Also follows the "present defaults" behaviour of the other vars.

Change-Id: Icc993e82a6a435f948e3d17e410eb14717cb0e2d
2020-03-28 09:50:39 +01:00
linpeiwen
5659177026 keystone roles container name variable
keystone and keystone_fernet container name variable is fixed
in some places, but in the defaults directory, keystone
and keystone_fernet container_name variable is variable.
If the keystone and keystone_fernet container_name variable is
changed during deployment, it will not be assigned to keystone
and keystone_fernet, but a fixed 'keystone' and 'keystone_fernet' name.

Change-Id: Ifc8ac69e6abc4586f0e4fd820b9022aea9f76396
2020-03-26 23:38:34 -04:00
LinPeiWen
8721ca35d8 kolla-toolbox container name variable
kolla-toolbox container name variable is fixed in some places,
but in the defaults directory, kolla-toolbox container_name variable
is variable. If the kolla-toolbox container_name variable is changed
during deployment, it will not be assigned to kolla-toolbox,
but a fixed 'kolla-toolbox' name.

Change-Id: I9579017761ff47477dba597282be9ae6fab4242a
2020-03-26 08:58:54 -04:00
Jeffrey Zhang
34a331ab2f Add clients ca_file in heat
This patch fix creating statck resource failure in heat.

Change-Id: I00c23f8b89765e266d045cc463ce4d863d0d6089
Closes-Bug: #1869137
2020-03-26 18:15:33 +08:00
Jeffrey Zhang
04382c8031 Add glance_ca_certificates_file when using self sign cert in glance
Change-Id: I9395ae32378f4ff1fd57be78d7daec7745579e04
Closes-Bug: #1869133
2020-03-26 15:12:45 +08:00
Mark Goddard
f3350d4e13 Fix HAProxy prechecks during scale-out with limit
Deploy HAProxy on one or more servers. Add another server to the
inventory in the haproxy group, and run the following:

kolla-ansible prechecks --limit <new host>

The following task will fail:

    TASK [haproxy : Checking if kolla_internal_vip_address and
    kolla_external_vip_address are not pingable from any node]

This happens because ansible does not execute on hosts where
haproxy/keepalived is running, and therefore does not know that the VIP
should be active.

This change skips VIP prechecks when not all HAProxy hosts are in the
play.

Closes-Bug: #1868986

Change-Id: Ifbc73806b768f76f803ab01c115a9e5c2e2492ac
2020-03-25 17:10:48 +00:00
Zuul
c9ca7e5529 Merge "Fix kolla-ansible stop with heterogeneous hosts" 2020-03-25 15:09:30 +00:00
Zuul
6af7efe349 Merge "Fix telegraf with zookeeper (wrong port variable reference)" 2020-03-25 12:55:33 +00:00
LinPeiWen
8a206699d4 mariadb container name variable
mariadb container name variable is fixed in some places,
but in the defaults directory, mariadb container_name variable
is variable. If the mariadb container_name variable is changed
during deployment, it will not be assigned to container_name,
but a fixed 'mariadb' name.

Change-Id: Ie8efa509953d5efa5c3073c9b550be051a7f4f9b
2020-03-25 01:17:29 -04:00