FIPS changes to allow FIPS to run on multinode Ubuntu jobs

There are two problems we need to solve when enabling FIPS for
multinode Ubuntu jobs.

1. Ubuntu nodes require a subscription to be activated with a
   subscription key that is stored as a secret in project-config.

2. Because enabling FIPS requires a reboot, we need to execute the
   enable-fips playbook before the multinode playbook.  If not,
   resources set up by the multinode playbook may not survive the
   reboot.

To solve these problems, we have created a new base job for
OpenStack multinode jobs.  (openstack-multinode-fips).  We expect
to use this job as the base job for OpenStack multinode jobs
instead of the multinode job in zuul-jobs.

The openstack-multinode-fips inherits from openstack-fips, which
is a job defined in project-config that access the UA subscription
key and activates the UA subscription.  This solves problem #1.

It then executes the enable-fips and multinode playbooks
(FIPS first!), which in turn, invoke the enable-fips and multinode
roles in zuul-jobs.  This solves problem #2.

This has the unfortunate result of having to duplicate the logic
of the multinode/pre.yaml playbook in zuul-jobs here in
openstack-zuul-jobs instead, but I can't see a way around that.

Note that unless the variable enable_fips is defined to be true
in the job, all the FIPS logic is essentially a no-op.

If enable_fips is set to True, then jobs will also need to specify
nslookup_target for the post-reboot-tasks role invoked by the
enable_fips role.

Depends-On: I8a88d6a9bcf5725986b00b063e03686d3225b48e
Change-Id: I080df90af850088893976c8649aa528638c6f373

playbooks/fips/README.rst

@@ -0,0 +1,22 @@
+The enable-fips playbook can be invoked to enable FIPS mode on jobs.
+
+This playbook will call the enable-fips role in zuul-jobs, which will
+turn FIPS mode on and then reboot the node.  To get consistent results,
+this role should be run very early in the node setup process, so that
+resources set up later are not affected by the reboot.
+
+In practice, this means that the playbook is invoked as part of a base job
+like openstack-multinode-fips for example.  In order to avoid duplicating
+complex inheritance trees, we expect to use this base job for most jobs.
+
+As most jobs will not require fips, a playbook variable enable_fips - which
+defaults to False - is provided.  To enable FIPS mode, a job will simply need
+to set enable_fips to True as a job variable.
+
+**Job Variables**
+
+.. zuul:jobvar:: enable_fips
+   :default: False
+
+   Whether to run the playbook and enable fips.  Defaults to False.
+

playbooks/fips/enable-fips.yaml

@@ -1,3 +1,4 @@
 - hosts: all
   roles:
     - enable-fips
+  when: enable_fips | default(false)

playbooks/multinode/enable-multinode.yaml

@@ -0,0 +1,30 @@
+- name: Configure a multi node environment
+  hosts: all
+  tasks:
+    # TODO: Add groups['all'] | length > 1 conditional when the truncated JSON
+    # issue is resolved, see:
+    # - https://github.com/ansible/ansible/issues/30385
+    # - https://review.opendev.org/#/c/504238/
+    - name: Set up multi-node known hosts
+      include_role:
+        name: multi-node-known-hosts
+
+    - name: Set up multi-node hosts file
+      include_role:
+        name: multi-node-hosts-file
+
+    - name: Set up multi-node firewall
+      include_role:
+        name: multi-node-firewall
+
+- name: Configure multi-node networking switch and peer nodes
+  hosts:
+    - switch
+    - peers
+  roles:
+    - multi-node-bridge
+
+- name: Copy the build ssh key to authorized_keys on all hosts for root
+  hosts: all
+  roles:
+    - copy-build-sshkey

zuul.d/jobs.yaml

@@ -862,6 +862,7 @@
     pre-run: playbooks/fips/enable-fips.yaml
     vars:
       nslookup_target: 'opendev.org'
+      enable_fips: True
 
 - job:
     name: openstack-tox-functional-py38-fips
@@ -872,6 +873,7 @@
     pre-run: playbooks/fips/enable-fips.yaml
     vars:
       nslookup_target: 'opendev.org'
+      enable_fips: True
 
 - job:
     name: openstack-tox-validate
@@ -2105,3 +2107,14 @@
         wheel_python: python2
       wheel-cache-ubuntu-xenial-python3:
         wheel_python: python3
+
+- job:
+    name: openstack-multinode-fips
+    abstract: true
+    parent: openstack-fips
+    description: |
+      Base job for OpenStack jobs to enable mutinode jobs while
+      being able to enable FIPS by specifying enable_fips as True.
+    pre-run:
+      - playbooks/fips/enable-fips.yaml
+      - playbooks/multinode/enable-multinode.yaml