Merge "Add precheck to fail if RabbitMQ HA needs configuring" into stable/xena

ansible/roles/rabbitmq/tasks/precheck.yml

@@ -198,3 +198,27 @@
     - enable_outward_rabbitmq | bool
     - rabbitmq_enable_tls | bool
     - key | length == 0
+
+- block:
+    - name: List RabbitMQ policies
+      become: true
+      command: "docker exec rabbitmq rabbitmqctl list_policies --silent"
+      register: rabbitmq_policies
+      changed_when: false
+      check_mode: false
+
+    - name: Check if RabbitMQ HA needs to be configured
+      assert:
+        that: "'ha-all' in rabbitmq_policies.stdout"
+        fail_msg: >
+          om_enable_rabbitmq_high_availability is True but no mirroring policy has been found.
+          Currently the procedure to migrate from transient non-mirrored queues to durable mirrored queues is manual.
+          Please follow the process described here: https://docs.openstack.org/kolla-ansible/latest/reference/message-queues/rabbitmq.html#high-availability.
+          Note that this process may take several hours on larger systems, and may cause a degredation in performance at large scale.
+          If you do not wish to enable this feature, set om_enable_rabbitmq_high_availability to False.
+
+  run_once: true
+  when:
+    - container_facts['rabbitmq'] is defined
+    - om_enable_rabbitmq_high_availability | bool
+  tags: rabbitmq-ha-precheck

doc/source/reference/message-queues/rabbitmq.rst

@@ -118,3 +118,29 @@ availability. These are durable queues and classic queue mirroring. Setting the
 flag ``om_enable_rabbitmq_high_availability`` to ``true`` will enable both of
 these features. There are some queue types which are intentionally not mirrored
 using the exclusionary pattern ``^(?!(amq\\.)|(.*_fanout_)|(reply_)).*``.
+
+After enabling this value on a running system, there are some additional steps
+needed to migrate from transient to durable queues.
+
+1. Stop all OpenStack services which use RabbitMQ, so that they will not
+   attempt to recreate any queues yet.
+
+2. Reconfigure RabbitMQ to enable classic queue mirroring.
+
+   .. code-block:: console
+
+      kolla-ansible reconfigure --tags rabbitmq --skip-tags rabbitmq-ha-precheck
+
+3. Reset the state on each RabbitMQ node with the following commands. Each
+   command must be run on all RabbitMQ nodes before moving on to the next
+   command. This will remove all queues.
+
+   .. code-block:: console
+
+      rabbitmqctl stop_app
+      rabbitmqctl force_reset
+      rabbitmqctl start_app
+
+4. Reconfigure the OpenStack services using ``kolla-ansible reconfigure``, at
+   which point they will start again and recreate the appropriate queues as
+   durable.