514ddc8eaa
Setup a native Zuul v3 grenade-base job that defines base folders and base devstack settings. The grenade play checks out repositories in two locations, old and new, and sets up devstack config in old and new as well. Define a grenade job that sets up devstack services. This job runs: - devstack from grenade_from_branch (without tempest) - grenade from grenade_to_branch - projects in old from grenade_from_branch and then tries to run grenade and then tempest. The configure-grenade-branches role sets the base/target branch variables (grenade_{from,to}_branch) and must be updated when cutting a new branch. Also, define the native versions of the grenade-postgresql and grenade-multinode jobs, replacing non-native jobs (like neutron-grenade and neutron-grenade-multinode) when possible. Even though Python 3 is now the default, define grenade-py3 for compatibility reasons. Finally, define a basic grenade-forward job which should be used for forward upgrade testing. Forward testing requires the user to set the destination branch, and it is relevant for stable branches only, so disable it from master. At least for legacy grenade jobs, Zuul seems to be taking care of Depends-On on the stable branch. It is worth noting that tls-proxy is set to False as it happens with the current legacy jobs. It does not work by just flipping it to true. Co-Authored-By: Luigi Toscano <ltoscano@redhat.com> Depends-On: https://review.opendev.org/637523 Depends-On: https://review.opendev.org/649275 Change-Id: Iefe8d1d7d13bb56cbc9e80fb009d19218f8b1a64
265 lines
8.5 KiB
ReStructuredText
265 lines
8.5 KiB
ReStructuredText
==============================
|
|
Modular Grenade Architecture
|
|
==============================
|
|
|
|
Grenade was originally created to demonstrate some level of upgrade
|
|
capacity for OpenStack projects. Originally this just included a small
|
|
number of services.
|
|
|
|
Proposed new basic flow:
|
|
|
|
- setup_grenade
|
|
- all the magic setup involved around err traps and filehandle redirects
|
|
- setup devstack trees
|
|
- setup_base
|
|
- run stack.sh to build the correct base environment
|
|
- verify_base
|
|
- for project in projects; do verify_project; done
|
|
- resources.sh create
|
|
- resources.sh verify pre-upgrade
|
|
- shutdown
|
|
- for project in projects; do shutdown; done
|
|
- snapshot.sh pre_upgrade (NOT YET IMPLEMENTED)
|
|
- resources.sh verify_noapi pre-upgrade
|
|
- upgrade ...
|
|
- resources.sh verify post-upgrade
|
|
- verify_target
|
|
- resources.sh destroy
|
|
|
|
|
|
|
|
Modular Components
|
|
==================
|
|
|
|
Assuming the following tree in target projects::
|
|
|
|
devstack/ - devstack plugin directory
|
|
upgrade/ - upgrade scripts
|
|
settings - adds settings for the upgrade path
|
|
upgrade.sh
|
|
snapshot.sh - snapshots the state of the service, typically a
|
|
database dump (NOT YET IMPLEMENTED)
|
|
from-juno/ - per release
|
|
within-juno/
|
|
from-kilo/
|
|
within-kilo/
|
|
resources.sh
|
|
|
|
|
|
This same modular structure exists in the grenade tree with::
|
|
grenade/
|
|
projects/
|
|
10_ceilometer/
|
|
settings
|
|
upgrade.sh
|
|
|
|
resources.sh
|
|
=================
|
|
|
|
resources.sh is a per-service resource create / verify / destroy
|
|
interface. What a service does inside a script is up to them.
|
|
|
|
You can assume your resource script will only be called if your
|
|
service is running in an upgrade environment. The script should return
|
|
zero on success for actions, and nonzero on failure.
|
|
|
|
Calling Interface
|
|
-----------------
|
|
|
|
The following is the supported calling interface
|
|
|
|
- resources.sh early_create
|
|
|
|
creates a set of sample resources that should survive very early in
|
|
the upgrade process. This should only be used for horizontal
|
|
resources that impact other services, that *have* to be available
|
|
before they do any of their setup. For instance setup of neutron
|
|
networks.
|
|
|
|
Do not use the phase unless you really know why ``create`` will not
|
|
work for you.
|
|
|
|
- resources.sh create
|
|
|
|
creates a set of sample resources that should survive
|
|
upgrade. Script should exit with a nonzero exit code if any
|
|
resources could not be created.
|
|
|
|
Example: create an instance in nova or a volume in cinder
|
|
|
|
- resources.sh verify (pre-upgrade|post-upgrade)
|
|
|
|
verify that the resources were created. Services are running at this
|
|
point, and the APIs may be expected to work. The second argument
|
|
indicates whether we are pre-upgrade or post-upgrade.
|
|
|
|
Example: use the nova command to verify that the test instance is
|
|
still ACTIVE, or the cinder command to verify that the volume is
|
|
still available.
|
|
|
|
- resources.sh verify_noapi
|
|
|
|
verify that the resources are still present. This is called in the
|
|
phase where services are stopped, and APIs are expected to not be
|
|
accessible. Resource verification at this phase my require probing
|
|
underlying components to make sure nothing has gone awry during
|
|
service shutdown. The second argument indicates whether we are
|
|
pre-upgrade or post-upgrade.
|
|
|
|
Example: check with libvirt to make sure the instance is actually
|
|
created and running. Bonus points for being able to ping the
|
|
instance, or otherwise check its live-ness. With cinder, checking
|
|
that the LVM volume exists and looks reasonable.
|
|
|
|
- resources.sh destroy
|
|
|
|
Resource scripts should be responsible and cleanup all their
|
|
resources when asked to destroy.
|
|
|
|
Calling Sequence
|
|
----------------
|
|
|
|
The calling sequence during a grenade run looks as follows:
|
|
|
|
- # start old side
|
|
- create (create will be called during the working old side)
|
|
- verify pre-upgrade
|
|
- # shutdown all services
|
|
- verify_noapi pre-upgrade
|
|
- # upgrade and start all services
|
|
- verify post-upgrade
|
|
- destroy
|
|
|
|
The important thing to remember is verify/verify_noapi will be called
|
|
multiple times, with multiple different versions of OpenStack. Those
|
|
phases of the script must not be rerunnable multiple times.
|
|
|
|
While create / destroy are only going to be called once in the current
|
|
interface, bonus points for also making those idempotent for
|
|
resiliancy in testing.
|
|
|
|
**Per-release upgrade scripts**
|
|
|
|
There are times when :ref:`exceptional <upgrade-exceptions>` manual upgrade
|
|
steps must be performed to get from one release to the next, or even within
|
|
the same release. Grenade supports this with per-release scripts found in
|
|
each project, e.g.::
|
|
|
|
projects/
|
|
60_nova/
|
|
from-ocata/
|
|
upgrade-nova
|
|
|
|
Regarding the sequence of when these per-release scripts are called, any
|
|
``within-$base`` script should be run *before* installing new code, and any
|
|
``from-$base`` script should be run *after* installing new code but before
|
|
starting the services with the new code. This is because configuration or
|
|
database changes may be needed before the upgraded code is started.
|
|
|
|
Supporting Methods
|
|
------------------
|
|
|
|
In order to assist with the checks listed the following functions
|
|
exist::
|
|
|
|
resource_save project key value
|
|
resource_get project key
|
|
|
|
This allow resource scripts to have memory, and keep track of things
|
|
like the allocated IP addresses, IDs, and other non deterministic data
|
|
that is returned from OpenStack API calls.
|
|
|
|
Environment
|
|
-----------
|
|
|
|
Resource scripts get called in a specific environment already set:
|
|
|
|
- TOP_DIR - will be set to the root of the devstack directory for the
|
|
BASE version of devstack incase this is needed to find files like a
|
|
working ``openrc``
|
|
|
|
- GRENADE_DIR - the root directory of the grenade directory.
|
|
|
|
The following snippet will give you access to both the grenade and
|
|
TARGET devstack functions::
|
|
|
|
source $GRENADE_DIR/grenaderc
|
|
source $GRENADE_DIR/functions
|
|
|
|
|
|
Best Practices
|
|
--------------
|
|
|
|
Do as many actions as non admin as possible. As early as you can in
|
|
your resource script it's worth allocating a user/project for the
|
|
script to run as. This ensures isolation against other scripts, and
|
|
ensures that actions don't only work because admin gets to bypass
|
|
safeties.
|
|
|
|
Test side effects, not just API actions. The point of these resource
|
|
survival scripts is to test that things created beyond the API / DB
|
|
interaction still work later. Just testing that data can be stored /
|
|
retrieved from the database isn't very interesting, and should be
|
|
covered other places. The value in the resource scripts is these side
|
|
effects. Actual VMs running, actual iscsi targets running, etc. And
|
|
ensuring these things are not disrupted when the control plane is
|
|
shifted out from under them.
|
|
|
|
Out of Tree Plugins
|
|
===================
|
|
|
|
A grenade plugin can be hosted out of tree in a project tree, similar
|
|
to external devstack plugins. There are a few subtle differences when
|
|
this happens.
|
|
|
|
The plugin structure will live under ``$project/devstack/upgrade/``
|
|
directory.
|
|
|
|
The plugin is enabled by adding::
|
|
|
|
enable_grenade_plugin <$project> <giturl> [branch]
|
|
|
|
To ``pluginrc`` in the ``GRENADE_DIR``. An additional rc file was
|
|
required due to sequencing of when plugin functions become available.
|
|
|
|
Note: when running a job based on the ``grenade-base`` job,
|
|
for each devstack plugin defined using the ``devstack_plugins``,
|
|
the corresponding grenade plugin is enabled automatically.
|
|
|
|
|
|
Changing Devstack Localrc
|
|
-------------------------
|
|
|
|
There is also a mechanism that allows a ``settings`` file change the
|
|
devstack localrc files with the ``devstack_localrc`` function.
|
|
|
|
::
|
|
devstack_localrc <base|target> arbitrary stuff to add
|
|
|
|
Which will take all the rest of the stuff on that line and add it to
|
|
the localrc for either the base or target devstack.
|
|
|
|
Please note that ``devstack_localrc`` only works when grenade
|
|
performs the configuration of the devstack settings and runs devstack
|
|
against the base target. When GRENADE_USE_EXTERNAL_DEVSTACK is set
|
|
to True, as it happens on the Zuul grenade jobs where devstack is
|
|
configured and executed before grenade, the function has no effect.
|
|
|
|
Example settings
|
|
----------------
|
|
|
|
The following is a reasonable example ``settings`` for out of tree
|
|
plugin::
|
|
|
|
register_project_for_upgrade heat
|
|
register_db_to_save heat
|
|
devstack_localrc base enable_service h-api h-api-cfn h-api-cw h-eng heat
|
|
devstack_localrc target enable_service h-api h-api-cfn h-api-cw h-eng heat
|
|
|
|
This registers the project for upgrade, symbolicly enables the heat
|
|
database for dump during upgrade, and adds the heat services into the
|
|
service list for base and target.
|
|
|
|
It's expected that most ``settings`` files for out of tree plugins
|
|
will need equivalent lines.
|