Go to file

Hervé Beraud 22f240b82f Allow users run the rabbitmq heartbeat inside a standard pthread.

This is an experimental feature.

The proposed changes will fix related issues when we run
heartbeat under apache/httpd enviornment with the apache MPM `prefork`
[1] engine and mod_wsgi or uwsgi in a monkey patched environment.

Propose changes to allow user to choose to run the rabbitmq health check
heartbeat in a standard python thread.

Issue
=====

We facing an issue with the rabbitmq driver heartbeat
under apache MPM `prefork` module and mod_wsgi when nova_api monkey
patched the stdlib by using eventlet.

nova_api calling eventlet.monkey_patch() [2] when it runs under mod_wsgi.

This impacts the AMQP heartbeat thread,
which is meant to be a native thread. Instead of checking AMQP sockets
every 15s, It is now suspended and resumed by eventlet. However,
resuming greenthreads can take a very long time if mod_wsgi isn't
processing traffic regularly, which can cause rabbitmq to close the AMQP
connection.

Root Cause
==========

The oslo.messaging RabbitMQ driver and especially the heartbeat
suffer to inherit the execution model of the service which consume him.

In this scenario nova_api need green threads to manage cells and edge
features so nova_api monkey patch the stdlib to obtain async features,
and the oslo.messaging rabbitmq driver endure these changes.

I think the main issue here is that nova_api want async and use eventlet green
threads to obtain it.

Solution
========

We want to allow user to isolate the heartbeat execution model
from the parent process inherited execution model by passing the
`heartbeat_in_pthread` option through the driver config.

While we use MPM `prefork` we want to avoid to use libevent and epoll.

If the `heartbeat_in_pthread` option is given we want to force to use the
python stdlib threading module to run the
rabbitmq heartbeat to avoid issue related to a non "standard"
environment. I mean "standard" because async features isn't the default
config in mostly case, starting by apache which define `prefork` is the
default engine.

This is an experimental feature, we can help us to ensure to run heartbeat
through a classical python thread

Specifications
==============

- https://review.opendev.org/661314

[1] https://httpd.apache.org/docs/2.4/fr/mod/prefork.html
[2] 3c5e2b0e9f

Change-Id: If8846599efc48fe18ecfb99c04e2c38f9a45b9ed

2019-08-07 17:17:02 +02:00

doc

Cap Bandit below 1.6.0 and update Sphinx requirement

2019-05-14 10:46:32 +02:00

etc

Fix spelling typo in output

2015-09-24 18:11:22 +08:00

oslo_messaging

Allow users run the rabbitmq heartbeat inside a standard pthread.

2019-08-07 17:17:02 +02:00

playbooks

OpenDev Migration Patch