295 Commits

Author SHA1 Message Date
Radosław Piliszek
53ea3fe4af Trivial fix: log stderr of init-runonce as well
Missed by me in a recent merge.

TrivialFix
Signed-off-by: Radosław Piliszek <radoslaw.piliszek@gmail.com>

Change-Id: I83b1e84a43f014ce20be8677868be3f66017e3c2
2019-07-09 15:38:47 +02:00
Zuul
887938bbcb Merge "Exit on failure in init-runonce" 2019-07-09 07:33:46 +00:00
Zuul
65783c90dd Merge "CI: Pull images before upgrade" 2019-07-08 09:21:57 +00:00
Zuul
db55408620 Merge "Fix conditionals in CI playbook" 2019-07-07 10:52:01 +00:00
Mark Goddard
f11d3c694a CI: Pull images before upgrade
This is the documented procedure.

Change-Id: I09ca99e92b112621d66b564a88b13658632242f5
2019-07-04 18:11:16 +00:00
Zuul
56c3603586 Merge "CI: Keep stderr in ansible logs" 2019-07-04 07:45:54 +00:00
Zuul
6aba50e66a Merge "CI: Use template-overrides.j2 from kolla" 2019-07-03 19:22:21 +00:00
Radosław Piliszek
b9aa8b38f4 CI: Keep stderr in ansible logs
Otherwise ara had only the stderr part and logs only the
stdout part which made ordered analysis harder.

Additionally add -vvv for the bootstrap-servers run.

Change-Id: Ia42ac9b90a17245e9df277c40bda24308ebcd11d
Signed-off-by: Radosław Piliszek <radoslaw.piliszek@gmail.com>
2019-07-02 20:44:33 +02:00
Radosław Piliszek
20ab480ca5 CI: Use template-overrides.j2 from kolla
Some kolla-ansible jobs failed due to using external mirrors
instead of local ones.
This was due to not using the template override provided by kolla.
This patch fixes that.

Depends-On: https://review.opendev.org/668226
Change-Id: I27f714fdf05e521aa8ce25c5683a452ceb35eeb8
Signed-off-by: Radosław Piliszek <radoslaw.piliszek@gmail.com>
2019-07-01 17:00:53 +00:00
Radosław Piliszek
a0bdc3669a Add note to CI config regarding registry during upgrade
Change-Id: Ifc898015b9b523ef4c50fc969e464f05762f2151
Signed-off-by: Radosław Piliszek <radoslaw.piliszek@gmail.com>
2019-07-01 18:45:30 +02:00
Zuul
470108a1c0 Merge "Revert "CI - remove unnecessary logic when building images for upgrade"" 2019-07-01 15:48:47 +00:00
Mark Goddard
acac12798c Revert "CI - remove unnecessary logic when building images for upgrade"
This reverts commit 8ce5ffd0c21c221d88bacca5fec03ca042dfed85.

Change-Id: I81ce7c007ff267ebbbb721bcdb7eebc0dd575bf8
2019-07-01 11:12:58 +00:00
Mark Goddard
bc08b44fd1 Exit on failure in init-runonce
Previously we sourced this script in tests/deploy.sh, but this was
recently changed. Following that change we lost the errexit setting,
meaning we ignore errors in init-runonce.

Adding errexit in the script itself means that all callers get error
handling.

Also log init-runonce output.

TrivialFix

Change-Id: I9b35bd5f0f76eec26ddd968d093a3a5fd55a7ce2
2019-06-28 14:31:24 +00:00
Mark Goddard
3b218fd0db Fix conditionals in CI playbook
These were not templated, so always evaluated to true. This shouldn't be
causing any issues.

Change-Id: I7b8e407e688ba201c4f7d1a94bbd41af0918e7df
2019-06-27 10:32:22 +01:00
Zuul
693a30275f Merge "Add CI job for ironic" 2019-06-24 15:15:41 +00:00
Radosław Piliszek
8ce5ffd0c2 CI - remove unnecessary logic when building images for upgrade
Docker registry being insecure is handled by docker_registry_insecure
which is set to true by default when docker_registry is set.
The removed code had no effect because docker_registry is not changed
anyway for base (pre-upgrade) install.

This change makes config more readable and also prevents a potential
conflict with the zun profile if ever used in upgrade mode.

Change-Id: I9b5ae8c5b534fa6cce9dbaca8af191e2ca79d19f
Signed-off-by: Radosław Piliszek <radoslaw.piliszek@gmail.com>
2019-06-21 11:31:30 +00:00
Zuul
6cae4dedfe Merge "Remove nova-consoleauth" 2019-06-17 16:28:45 +00:00
Jeffrey Zhang
4e032923c0 Remove nova-consoleauth
The nova-consoleauth service was deprecated during the Rocky release [1]
and has not been necessary since unless you're using cells v1. As Kolla
has never supported cells v1, which is finally being removed during
Train [2], we can get ahead of the curve and stop deploying
nova-consoleauth immediately.

[1] https://specs.openstack.org/openstack/nova-specs/specs/rocky/implemented/convert-consoles-to-objects.html
[2] https://blueprints.launchpad.net/nova/+spec/remove-cells-v1/

Change-Id: I099080979f5497537e390f531005a517ab12aa7a
2019-06-16 16:39:07 +08:00
Mark Goddard
845040ad3f Add CI job for ironic
Adds four new CI jobs for testing centos/ubuntu binary/source deploys
with ironic enabled. These are run only when there are changes to the
ironic role.

Performs some simple testing by creating a node using the fake-hardware
hardware type and creating a server.

Change-Id: Ie669e57ce2af53257b4ca05f45193cb73f48827a
Depends-On: https://review.opendev.org/664011
2019-06-11 10:22:04 +01:00
Carlos Goncalves
f427920daf Remove Neutron LBaaS support
The project has been retired and there will be no Train release [1].
This patch removes Neutron LBaaS support in Kolla.

[1] https://review.opendev.org/#/c/658494/

Change-Id: Ic0d3da02b9556a34d8c27ca21a1ebb3af1f5d34c
2019-06-07 13:50:19 +01:00
Mark Goddard
d2ae42ce6d Add support for idempotent container stop and removal
This is useful when removing a container that is no longer supported.

Change-Id: I08d79ce7dd2f3d11e466930de85412017cd5f747
2019-06-07 13:50:01 +01:00
Zuul
ff2b2f44ba Merge "Fix keystone fernet key rotation scheduling" 2019-06-06 16:33:03 +00:00
Zuul
7ab67590f7 Merge "Add unit test for keystone fernet cron generator" 2019-06-06 16:29:19 +00:00
Zuul
2208b0214e Merge "Adds Qinling Ansible role" 2019-06-03 20:29:41 +00:00
Mark Goddard
78ee0287a7 Test Ceph upgrade in CI
Add CI jobs for testing an upgrade of a multinode system with Ceph
enabled. As for the existing upgrade job, we upgrade from the previous
release to the current release.

Change-Id: I931772ca4c63757769467a57c80dc0726a11167a
Depends-On: https://review.opendev.org/658163
2019-06-03 08:24:32 +00:00
Gaetan Trellu
edb3489820 Adds Qinling Ansible role
Qinling is an OpenStack project to provide "Function as a Service".
This project aims to provide a platform to support serverless functions.

Change-Id: I239a0130f8c8b061b531dab530d65172b0914d7c
Implements: blueprint ansible-qinling-support
Story: 2005760
Task: 33468
2019-05-31 10:25:28 -04:00
Mark Goddard
6c1442c385 Fix keystone fernet key rotation scheduling
Right now every controller rotates fernet keys. This is nice because
should any controller die, we know the remaining ones will rotate the
keys. However, we are currently over-rotating the keys.

When we over rotate keys, we get logs like this:

 This is not a recognized Fernet token <token> TokenNotFound

Most clients can recover and get a new token, but some clients (like
Nova passing tokens to other services) can't do that because it doesn't
have the password to regenerate a new token.

With three controllers, in crontab in keystone-fernet we see the once a day
correctly staggered across the three controllers:

ssh ctrl1 sudo cat /etc/kolla/keystone-fernet/crontab
0 0 * * * /usr/bin/fernet-rotate.sh
ssh ctrl2 sudo cat /etc/kolla/keystone-fernet/crontab
0 8 * * * /usr/bin/fernet-rotate.sh
ssh ctrl3 sudo cat /etc/kolla/keystone-fernet/crontab
0 16 * * * /usr/bin/fernet-rotate.sh

Currently with three controllers we have this keystone config:

[token]
expiration = 86400 (although, keystone default is one hour)
allow_expired_window = 172800 (this is the keystone default)

[fernet_tokens]
max_active_keys = 4

Currently, kolla-ansible configures key rotation according to the following:

   rotation_interval = token_expiration / num_hosts

This means we rotate keys more quickly the more hosts we have, which doesn't
make much sense.

Keystone docs state:

   max_active_keys =
     ((token_expiration + allow_expired_window) / rotation_interval) + 2

For details see:
https://docs.openstack.org/keystone/stein/admin/fernet-token-faq.html

Rotation is based on pushing out a staging key, so should any server
start using that key, other servers will consider that valid. Then each
server in turn starts using the staging key, each in term demoting the
existing primary key to a secondary key. Eventually you prune the
secondary keys when there is no token in the wild that would need to be
decrypted using that key. So this all makes sense.

This change adds new variables for fernet_token_allow_expired_window and
fernet_key_rotation_interval, so that we can correctly calculate the
correct number of active keys. We now set the default rotation interval
so as to minimise the number of active keys to 3 - one primary, one
secondary, one buffer.

This change also fixes the fernet cron job generator, which was broken
in the following cases:

* requesting an interval of more than 1 day resulted in no jobs
* requesting an interval of more than 60 minutes, unless an exact
  multiple of 60 minutes, resulted in no jobs

It should now be possible to request any interval up to a week divided
by the number of hosts.

Change-Id: I10c82dc5f83653beb60ddb86d558c5602153341a
Closes-Bug: #1809469
2019-05-17 14:05:48 +01:00
Mark Goddard
25ac955a4e Add unit test for keystone fernet cron generator
Before making changes to this script, document its behaviour with a unit
test.

There are two major issues:

* requesting an interval of more than 1 day results in no jobs
* requesting an interval of more than 60 minutes, unless an exact
  multiple of 60 minutes, results in no jobs

Change-Id: I655da1102dfb4ca12437b7db0b79c9a61568f79e
Related-Bug: #1809469
2019-05-17 09:13:31 +01:00
Zuul
a16576e9c0 Merge "Do some Train TODOs" 2019-04-23 06:02:10 +00:00
OpenDev Sysadmins
92d8d22cf6 OpenDev Migration Patch
This commit was bulk generated and pushed by the OpenDev sysadmins
as a part of the Git hosting and code review systems migration
detailed in these mailing list posts:

http://lists.openstack.org/pipermail/openstack-discuss/2019-March/003603.html
http://lists.openstack.org/pipermail/openstack-discuss/2019-April/004920.html

Attempts have been made to correct repository namespaces and
hostnames based on simple pattern matching, but it's possible some
were updated incorrectly or missed entirely. Please reach out to us
via the contact information listed at https://opendev.org/ with any
questions you may have.
2019-04-19 19:29:02 +00:00
Zuul
b28ffeb27d Merge "Remove RabbitMQ support from Bifrost" 2019-04-15 11:30:57 +00:00
Mark Goddard
2b7a9dc207 Fix periodic CI jobs
Periodic jobs don't have zuul.change defined, since there is no change
being tested. This causes an early failure when referencing zuul.change
to set the image tag for built images. In periodic jobs we'll never need
to build images because there is no dependent kolla change under test.

Change-Id: I6d9d81cf17b7d0d7aaf87cd96418c904c46681f2
2019-04-14 11:23:33 +01:00
Mark Goddard
33564a0097 Remove RabbitMQ support from Bifrost
During the Train cycle, Bifrost switched to using JSON-RPC by default
for Ironic's internal communication [1], avoiding the need to install
RabbitMQ. This simplifies things, so we may as well remove our custom
configuration of RabbitMQ.

[1] https://review.openstack.org/645093

Change-Id: I3107349530aa753d68fd59baaf13eb7dd5485ae6
2019-04-10 11:30:50 +01:00
Mark Goddard
bb9d51e25b Do some Train TODOs
Make an early start on the TODOs for the Train cycle.

1. Remove the task that removes the vitrage_collector container, which
   was added in the Stein cycle to clean up this container which is no
   longer deployed.

2. Remove globals.yml configuration in CI to disable Heat for upgrade
   jobs. Heat is now enabled in the previous release (Stein).

3. Remove the deprecated variable cinder_iscsi_helper, which was renamed
   to cinder_target_helper in Stein.

Change-Id: I774bf395e0bdd4db9c20c6289a22cf059fa42e1a
2019-04-08 12:25:27 +01:00
Mark Goddard
8c4ab41ffa Check configuration file permissions in CI
Typically, non-executable files should have 660 or 600 and executable
files and directories should have 770. All should be owned by the
'config_owner_user' and 'config_owner_group' variables.

This change adds a script to check the owner and permissions of config
files under /etc/kolla, and runs it at the end of CI jobs.

Change-Id: Icdbabf36e284b9030017a0dc07b9dc81a37758ab
Related-Bug: #1821579
2019-04-03 08:48:00 +00:00
Zuul
1b2503b5d4 Merge "Test upgrades in CI" 2019-03-28 12:31:08 +00:00
Zuul
03d3885a56 Merge "Add cyborg to kolla-ansible" 2019-03-28 08:20:13 +00:00
Mark Goddard
c23c9b2c25 Test upgrades in CI
This patch adds two new jobs:

* kolla-ansible-centos-source-upgrade
* kolla-ansible-ubuntu-source-upgrade

These jobs first deploy a control plane using the previous release of
Kolla Ansible, then upgrade to the current release.

Because we can't change the branch of the git repository on the Zuul
executor, we change the branch of the kolla-ansible repository on the
primary node to the branch of the previous release, in this case
stable/rocky. A new remote-template role has been added that supports
generating templates using a remote template source, to generate config
files using the previous kolla-ansible branch.

If the change being tested depends on a kolla change for the current
branch, then we build images. Rather than using the current
kolla-ansible version to tag the images, we now tag them with
change_<gerrit change ID>. This is because the version of kolla-ansible
will change from the previous release to the current one as we upgrade
the system.

Finally, it should be noted that the 'previous_release' variable in the
Zuul config needs to be updated with each release, since this sets the
release of kolla-ansible that is installed initially.

Depends-On: https://review.openstack.org/645089/
Depends-On: https://review.openstack.org/644250/
Depends-On: https://review.openstack.org/645816/
Depends-On: https://review.openstack.org/645840/
Change-Id: If301e0affcd55360fefe3b105f023ae5c47b0853
2019-03-27 16:13:49 +00:00
Zuul
e4693e8dcd Merge "Wait for cinder volume to become available in CI" 2019-03-25 11:52:09 +00:00
Zuul
33a92b9f7d Merge "Add ceilometer_ipmi container into ceilometer role" 2019-03-22 12:02:22 +00:00
Mark Goddard
e956cd87c8 Wait for cinder volume to become available in CI
Fixes a race condition where sometimes a volume would still be in the
'creating' state when trying to attach it to a server.

Invalid volume: Volume <id> status must be available or downloading to
reserve, but the current status is creating.

Change-Id: I0687ddfd78c384650cb361ff07aa64c5c3806a93
2019-03-21 17:35:13 +00:00
Zuul
4ad5122d3f Merge "Add cinder-lvm CI job" 2019-03-18 10:54:05 +00:00
caoyuan
16900c2e37 Add ceilometer_ipmi container into ceilometer role
refer to [0]

[0]: https://docs.openstack.org/ceilometer/latest/install/install-compute-rdo.html
Co-Authored-By: zhulingjie <easyzlj@gmail.com>

Change-Id: I4cda336dedb3d807b80d13bcc219268a8d667b4d
2019-03-14 16:24:15 +08:00
Zuul
ce08b31f5e Merge "Support the prometheus elasticsearch exporter" 2019-03-13 17:00:50 +00:00
Erol Guzoglu
14ab9a7c4e Support the prometheus elasticsearch exporter
This patch implements the support for the elasticsearch-exporter in
kolla-ansible

The configuration and prechecks are reused from the other exporters

Depends-On: Id138f12e10102a6dd2cd8d84f2cc47aa29af3972
Change-Id: Iae0eac0179089f159804490bf71f1cf2c38dde54
2019-03-11 17:25:51 +03:00
Victor Coutellier
3bb4c19981 Fix kolla-docker possible undefined variable
It is possible to reference undefined variable in kolla-docker module if
DockerWorker object initialization fail, so the current behaviour will
crash the playbook with the unwanted error message :

UnboundLocalError: local variable 'dw' referenced before assignment

Change-Id: Ic8d26b11f93255220888b5406f8ab4a6f81736c2
Closes-Bug: #1819361
2019-03-10 19:13:00 +01:00
Bai Yongjun
ed2fd243d1 Add cyborg to kolla-ansible
Because kolla-ansible not have cyborg so should add it.

Implements: blueprint add-cyborg-to-kolla-ansible

Depend-On: I497e67e3a754fccfd2ef5a82f13ccfaf890a6fcd

Change-Id: I6f7ae86f855c5c64697607356d0ff3161f91b239
2019-03-08 10:46:53 +08:00
Mark Goddard
c41c9a0ce0 Add cinder-lvm CI job
Tests Cinder with an LVM/iSCSI backend, including attaching a volume to
an instance.

Change-Id: I6963b5e81d7ac88dd1bdfb94b62d9d63028b9786
2019-03-01 14:36:30 +00:00
Michal Nasiadka
3f01c7c7cd Add haproxy stats to gate logs
Change-Id: Iebd98acf03418817d3707c4a117771b73da80166
2019-02-21 12:55:54 +01:00
Zuul
da98030909 Merge "add ulimit support for kolla_docker" 2019-01-27 10:17:30 +00:00