Add additional commented RabbitMQ policy to manage segment sizes

The default RabbitMQ segment sizes of 500MB mean you may need to
acquire many days or weeks worth of messages in each fanout/stream
queue before the x-max-age setting takes effect. Subject to the
number of streams in use across the deployment, this could consume
many tens of GB of disk space.

This example shows an additional policy which has been tested in
order to combat this in the absence of suitable per-stream settings
in oslo.messaging. The downside is that this must be applied before
any services create their streams.

Change-Id: I69699f99a88b0ba8c8ee283ecbd3b4ed0c98a373

doc/source/user/messaging/messaging.rst

@@ -1,6 +1,6 @@
-========================
-Hybrid messaging example
-========================
+=======================
+Messaging configuration
+=======================
 
 This section provides an overview of hybrid messaging deployment
 concepts and describes the necessary steps for a working
@@ -87,3 +87,32 @@ to the rabbitmq server backend.
    :start-after: under the License.
 
 .. _oslo-messaging.yml: https://github.com/openstack/openstack-ansible/blob/master/inventory/group_vars/all/oslo-messaging.yml
+
+
+Managing RabbitMQ stream policy
+-------------------------------
+
+When deploying RabbitMQ with support for quorum and stream queues, the
+retention behaviour for messages changes. Stream queues maintain an append only
+log on disk of all messages received until a retention policy indicates they
+should be disposed of. By default, this policy is set with a per-stream
+`x-max-age` of 1800 seconds. However, as noted in the `RabbitMQ docs`_, this
+only comes into effect ones a stream has accumulated enough messages to fill a
+segment, which has a default size of 500MB.
+
+If you would like to reduce disk usage, an additional policy can be applied via
+OpenStack Ansible as shown below:
+
+.. literalinclude:: ../../../../inventory/group_vars/all/infra.yml
+   :language: yaml
+   :start-at: rabbitmq_policies
+   :end-before: ## Galera options
+
+Note however, that this policy will only apply if it is in place before any
+stream queues are created. If these already exist, they will need to be
+manually deleted and re-created by the relevant OpenStack service.
+
+This issue is being tracked in an `oslo.messaging bug`_.
+
+.. _RabbitMQ docs: https://www.rabbitmq.com/docs/streams#retention
+.. _oslo.messaging bug: https://bugs.launchpad.net/oslo.messaging/+bug/2089845

inventory/group_vars/all/infra.yml

@@ -39,6 +39,22 @@ rabbitmq_policies:
         ((oslomsg_rabbit_quorum_queues | default(True) or not rabbitmq_queue_replication) and rabbitmq_install_method | default('') != 'distro'
           ) | ternary('present', 'absent')
       }}
+# The following is an example of an additional policy which applies to fanout/stream queues only
+# By default, each stream uses RabbitMQ's 500MB segment size, and no messages will be discarded
+# until that size is reached. This may result in undesirable disk usage.
+# If using this policy, it must be applied BEFORE any stream queues are created.
+# See also https://bugs.launchpad.net/oslo.messaging/+bug/2089845 and https://www.rabbitmq.com/docs/streams#retention
+#  - name: CQv2F
+#    pattern: '^.*_fanout'
+#    priority: 1
+#    tags:
+#      queue-version: 2
+#      stream-max-segment-size-bytes: 1000000
+#    state: >-
+#      {{
+#        ((oslomsg_rabbit_quorum_queues | default(True) or not rabbitmq_queue_replication) and rabbitmq_install_method | default('') != 'distro'
+#          ) | ternary('present', 'absent')
+#      }}
 
 ## Galera options
 galera_client_package_state: "{{ package_state }}"