Introduce a RabbitMQ driver documentation for admin.
Describing:
- some RabbitMQ and AMQP specifications (exchanges, queues, routing-key)
- the heartbeat specification and the type of used threads
- the driver options
Change-Id: I8fd1624834510f8dee81ab9342c708d726b8f827
This is an experimental feature.
The proposed changes will fix related issues when we run
heartbeat under apache/httpd enviornment with the apache MPM `prefork`
[1] engine and mod_wsgi or uwsgi in a monkey patched environment.
Propose changes to allow user to choose to run the rabbitmq health check
heartbeat in a standard python thread.
Issue
=====
We facing an issue with the rabbitmq driver heartbeat
under apache MPM `prefork` module and mod_wsgi when nova_api monkey
patched the stdlib by using eventlet.
nova_api calling eventlet.monkey_patch() [2] when it runs under mod_wsgi.
This impacts the AMQP heartbeat thread,
which is meant to be a native thread. Instead of checking AMQP sockets
every 15s, It is now suspended and resumed by eventlet. However,
resuming greenthreads can take a very long time if mod_wsgi isn't
processing traffic regularly, which can cause rabbitmq to close the AMQP
connection.
Root Cause
==========
The oslo.messaging RabbitMQ driver and especially the heartbeat
suffer to inherit the execution model of the service which consume him.
In this scenario nova_api need green threads to manage cells and edge
features so nova_api monkey patch the stdlib to obtain async features,
and the oslo.messaging rabbitmq driver endure these changes.
I think the main issue here is that nova_api want async and use eventlet green
threads to obtain it.
Solution
========
We want to allow user to isolate the heartbeat execution model
from the parent process inherited execution model by passing the
`heartbeat_in_pthread` option through the driver config.
While we use MPM `prefork` we want to avoid to use libevent and epoll.
If the `heartbeat_in_pthread` option is given we want to force to use the
python stdlib threading module to run the
rabbitmq heartbeat to avoid issue related to a non "standard"
environment. I mean "standard" because async features isn't the default
config in mostly case, starting by apache which define `prefork` is the
default engine.
This is an experimental feature, we can help us to ensure to run heartbeat
through a classical python thread
Specifications
==============
- https://review.opendev.org/661314
[1] https://httpd.apache.org/docs/2.4/fr/mod/prefork.html
[2] 3c5e2b0e9f
Change-Id: If8846599efc48fe18ecfb99c04e2c38f9a45b9ed
There is no need to explicitly list the choices in the help text.
The oslo.config sample generator will include the choices automatically[0]
Also tweaks the wording of text to make it clear that it is the allowed
values which vary based on kafka version.
Change-Id: I4116e8871436097dea650f56e7b187358367d92e
0: 2488c1e1ce/oslo_config/generator.py (L263)
There is a typographical errors in amqpdriver.py. Correcting spelling
from acknowlege to acknowledge.
Change-Id: I4a80d8c6b162a99176eadb052f6201dc38dbc5f9
Some options are now automatically configured by the version 1.20:
- project
- html_last_updated_fmt
- latex_engine
- latex_elements
- version
- release.
Change-Id: Ib5e22f6a5374f05e576bbc00a209209fdb09acad
Lots of exchanges create problems during failover under high
load. Please see bug report for details.
This is a step 2 patch.
Step 1 was: only using default exchange
when publishing.
Step 2 is to update consumers to only listen on default exchange,
happening now in T release.
Change-Id: Ib2ba62a642e6ce45c23568daeef9703a647707f3
Closes-Bug: #1789177
Use a sensible header style, fix some syntax highlighting, and generally
tidy things up.
Change-Id: I0b141b968ed8db10ff41a626569dd185edbdc641
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
With this feature, it is possible to specialize the parameters to send.
`options = oslo_messaging.TransportOptions(at_least_once=True)`
TransportOptions is used in every single driver,
for example in RabbitMQ driver is used to handle the mandatory flag.
Notes:
- The idea of creating a new class TransportOptions is because I'd like
to have an abstract class not related only to the RPCClient
- at_least_once is the first parameter, when needed we can add the
others.
Implements: blueprint transport-options (second point)
The blueprint link is [1]
To test it you can use [2]
1- https://blueprints.launchpad.net/oslo.messaging/+spec/transport-options
2- https://github.com/Gsantomaggio/rabbitmq-utils/
tree/master/openstack/mandatory_test
Change-Id: I1858e4a990507d3c2bac2ef7fbef75d8c2dbfce2
When the message is large, in order to improve the efficiency of
kafka, we need to compress the message before send it, so we need to
support kafka message compression.
Change-Id: I9e86d43ad934c1f82dc3dcf93d317538f9d2568e
Implements: blueprint support-kafka-compression
With this new parameter is possible to pass other parameters
from the client to the drivers.
So it is possible to tune the driver behavior.
For example can be used to send the mandatory flag in RabbitMQ
Note:
- The transport_options parameter is not actually used (yet).
- This part of blueprint transport-options (first part)
Implements: blueprint transport-options
The blueprint link is
https://blueprints.launchpad.net/oslo.messaging/+spec/transport-options
Change-Id: Iff23a9344c2c87259cf36b0d14c0a9fc075a2a72
It seems that versions are deleted from www.apache.org pretty quickly.
They stick around longer on archive.apache.org so we won't have to
be constantly chasing the latest version in our functional tests.
Change-Id: I047edac67699dd598f8dfd0f859b3772f6068bd3
Bandit 1.6.0 accidentally changed how the exclusion list option is
handled and breaks our use of it. Cap to the previous version until
Bandit has fixed the problem.
Sphinx 2.0 no longer works on python 2.7, so we need to start
capping it there as well.
Change-Id: Ie6b379f2c99862c37891ac03c52464e07bc2b2cc
In a clustered rabbitmq when a node disappears, we get a
ConnectionRefusedError because the socket get disconnected.
The socket access yields a OSError because the heartbeat
tries to reach an unreachable host (No route to host).
Catch these exceptions to ensure that we call ensure_connection for switching
the connection destination.
POC is available at github.com:4383/rabbitmq-oslo_messging-error-poc
Example:
$ git clone git@github.com:4383/rabbitmq-oslo_messging-error-poc
$ cd rabbitmq-oslo_messging-error-poc
$ python -m virtualenv .
$ source bin/activate
$ pip install -r requirements.txt
$ sudo podman run -d --hostname my-rabbit --name rabbit rabbitmq:3
$ python poc.py $(sudo podman inspect rabbit | niet '.[0].NetworkSettings.IPAddress')
And in parallele in an another shell|tmux
$ podman stop rabbit
$ # observe the output of the poc.py script we now call ensure_connection
Now you can observe some output relative to the connection who is
modified and not catched before these changes.
Related to: https://bugzilla.redhat.com/show_bug.cgi?id=1665399
Closes-Bug: #1828841
Change-Id: I9dc1644cac0e39eb11bf05f57bde77dcf6d42ed3
Log messages are no longer being translated. This removes all use of
the _LE, _LI, and _LW translation markers to simplify logging and to
avoid confusion with new contributions.
Change-Id: I9ddb6595fc52e46ed8844e39d2fa71029c90b65c
Closes-Bug: #1674567
Without this fix, the simulator splits the "topic" value in different
strings, in notify-client case
Instead of having one queue there are multi-queues.
See [1] for more details
Closes-Bug: #1824339
1- https://bugs.launchpad.net/oslo.messaging/+bug/1824339
Change-Id: I438c7279d584bdb00da18477a83d5e2a53c588c4
The use of ABC classes directly from collections has been deprecated in
3.x versions of Python. The direction is to use the classes defined in
collections.abc. Python 2.7 does not have this, but Python 3.8 will be
dropping the backwards compatibility to use the old location.
Six also does not have support for this yet, so in the mean time to make
sure we don't run into issues as folks try to move to 3.8, and to get
rid of deprecation warnings in logs, this handles importing from the
preferred location and falls back if it not available.
Change-Id: If67133813634f41d89ccdf0f6d6d5ffa66c97dd8
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
Without this commit, the client can lose the messages, because the
client does not handler the 'AMQP internal error 541',
read here [2] for details.
The fix retries to create the queue after a delay.
When the virtual-host is ready the declare does not fail.
This is a rare condiction, please read the bug [1] for details.
Closes-Bug: #1822778
[1] https://bugs.launchpad.net/oslo.messaging/+bug/1822778
[2] https://www.rabbitmq.com/amqp-0-9-1-reference.html
Change-Id: I7ab1f9d21ebb807285bf1422bc14cc6e07dcd32a
Commit 287a4f56f45ed9cd40116a9e7b6e529f3382a925 marked RabbitMQ
heartbeat as experimental in 2015, due to issues with the underlying
amqp library. Later, commit 168f6cc2bb72591012d54b15e1c2e4a0332fb5c9
enabled heartbeat by default, but the EXPERIMENTAL string was left.
Change-Id: I78838df2a0db9836f98d8f79624d688c3ed351ae
