2600 Commits

Author SHA1 Message Date
Zuul
04b2b5d451 Merge "Adding debug logs on AMQPListener poll" 2020-01-15 13:15:32 +00:00
Hervé Beraud
e4a8900aa7 Adding debug logs on AMQPListener poll
Adding some debug logs to track execution on the AMQPListener poll
function to observe if workers incoming messages are dispatched to
workers and if listener is stuck.

Change-Id: Ia9b9ef62b0b59c78283c6a8cc5b7ce42282e73b5
2020-01-14 11:57:10 +00:00
Hervé Beraud
cc6cf12c3d tox: Trivial cleanup
Move 'basepython' to the top-level 'testenv'.

Change-Id: I13e72ec6cbd006ca6eccfe21f2a0584e86a52669
2020-01-08 19:59:34 +01:00
Zuul
41a4423043 Merge "Ignore releasenote cache within git untracked files" 2019-12-19 19:19:22 +00:00
Zuul
4fc0e6241b Merge "Remove telemetry checks" 2019-12-19 12:40:28 +00:00
Zuul
4e49a286e7 Merge "Do not use threading.Event" 2019-12-18 23:58:11 +00:00
Hervé Beraud
2a4cea9e85 Ignore releasenote cache within git untracked files
Change-Id: If616ac023f578f073d9ae8596bc8ce870096ef0b
2019-12-18 17:03:10 +01:00
Zuul
524f72261c Merge "Removed unused variable pools" 2019-12-18 15:33:35 +00:00
John Eckersberg
d873c0d8f5 Do not use threading.Event
Waiting on a threading.Event with eventlet can cause busy looping via
epoll_wait, see related bug for more details.

Change-Id: I007613058a2d21d1712c02fa6d1602b63705c1ab
Related-bug: #1518430
2019-12-18 13:11:41 +00:00
Chenjun Shen
2b0613b208 Removed unused variable pools
Change-Id: I8f3a22d2af3f34b368bae3465bf8433fe2d3e22b
2019-12-18 11:06:17 +00:00
Kenneth Giusti
859e0d4eaa Revert "Add RPC incoming and reply log"
This broke non-rabbitmq message bus drivers. See:

https://bugs.launchpad.net/oslo.messaging/+bug/1855775

This reverts commit b104f254ab43d5e2589e845eff84b1bd096cb42d.

Change-Id: I17a448a768b544482b375b0076889db989e03e8c
2019-12-09 16:22:06 -05:00
Andy Smith
1057cfc60f Remove telemetry checks
The telemetry tests are not actively maintained and have been failing
checks for quite some time. Remove the non-voting tests for now and
plan to introduce new tests in the future for better coverage of
notifications.

Change-Id: I6c30c28ee4c8be4677ccecf8822af15b060cf804
2019-12-02 07:54:34 -05:00
Ghanshyam Mann
37fabdd895 Migrate grenade jobs to py3
As part of community goal of dropping py27 support[1], we are
moving the devstack to py3 by default[2]. That will make grenade job
to perform upgrade from py2 to py3 which will not work (we have seen
the failure in neutron-grenade job).

To avoid existing grenade py2 job break, this commit moves grenade
jobs to py3 which is what we planned as part dropping the py2 support.

[1] https://governance.openstack.org/tc/goals/selected/ussuri/drop-py27.html
[2] https://review.opendev.org/#/c/649097/12
    http://lists.openstack.org/pipermail/openstack-discuss/2019-November/010938.html

Change-Id: I128827b77ed791ea0084008cdecbd7ac37fa9079
2019-11-19 22:21:41 +00:00
Eyal
c8d6fed762 Make sure minimum amqp is 2.5.2
amqp fixed a bug in 2.5.2 that is needed
also update kombu to support amqp 2.5.2

see 
https://review.opendev.org/#/c/693704/
86cb254dce


Change-Id: I4b72d8feb85c2b9b4657510c356cd21e22fe40c2
Closes-bug: #1852058
2019-11-18 15:05:33 +00:00
Zuul
da3c3e7d1d Merge "Switch to Ussuri jobs" 2019-11-12 12:59:27 +00:00
Zuul
b2658bafb6 Merge "Add RPC incoming and reply log" 2019-11-12 03:59:43 +00:00
Zuul
f1ca5ee6b7 Merge "tox: Keeping going with docs" 2019-11-11 17:03:36 +00:00
Zuul
19cba65785 Merge "Modify some comments to make them clickable" 2019-11-04 15:39:29 +00:00
Chenjun Shen
6bca848f5b Remove unused variable WAKE_UP
Change-Id: I65f57973b66f353a1e6fef3d0bf7793f821c1c5c
Closes-Bug: 1850259
2019-10-29 09:09:17 +01:00
pengyuesheng
4529981d8c Switch to Ussuri jobs
Change-Id: I4297685cbe5c5ec71b68467a54746c2f4814f104
2019-10-24 14:29:00 +08:00
caoyuan
1e7a0c6fbb tox: Keeping going with docs
Sphinx 1.8 introduced [1] the '--keep-going' argument which, as its name
suggests, keeps the build running when it encounters non-fatal errors.
This is exceptionally useful in avoiding a continuous edit-build loop
when undertaking large doc reworks where multiple errors may be
introduced.

[1] https://github.com/sphinx-doc/sphinx/commit/e3483e9b045

Change-Id: If9885a1f064226909181d8b69241eb814deb2105
2019-10-24 14:21:22 +08:00
LIU Yulong
b104f254ab Add RPC incoming and reply log
Typically a simple log will not narrow down the
performance, but give us more information about
the service status.

Change-Id: I51c8f2743dd39cccd3d1d021d3c50dc09f70cd97
Closes-Bug: #1847747
2019-10-21 22:32:07 +08:00
Hervé Beraud
40e5be488e Modify some comments to make them clickable
Change-Id: Ib085fa551950f1c1461a11bb80f4909fb8425484
2019-10-11 18:29:46 +02:00
Zuul
c01b03e87c Merge "Update master for stable/train" 2019-10-10 12:09:24 +00:00
Pierre Riteau
b36c1529f1 Fix spacing in help message
Change-Id: Ie622b8a39514f87b60ff2e801843c5e3c06106e7
2019-10-03 14:10:28 +02:00
Zuul
7734ac1376 Merge "Align message serialization between drivers" 2019-09-30 09:26:11 +00:00
pengyuesheng
b72922670f Update the constraints url
For more detail, see http://lists.openstack.org/pipermail/openstack-discuss/2019-May/006478.html

Change-Id: Ie0b54914f5ca45f4888eb1cf8faac36f87e45535
2019-09-23 16:20:21 +08:00
Balazs Gibizer
d8980bfed8 Align message serialization between drivers
Every in tree driver that implements RPC send uses jsonutils.dumps to
serialize the message, except FakeDriver. FakeDriver uses json.dumps.
However json.dumps is a lot more strict than jsonutils. This caused
nova to introduce test specific changes in the rpc handling [1].

This patch makes sure that each driver uses the same json serialization.

I've tried to dig in to the history of the strictness of the
FakeDriver. That driver with the json.dumps() call was added back in
2013 with e2b74cc9e6605156dfd6e36cdfd1b5136161d526. (I cannot link to
that commit in any online way but it is in my local git clone.)
Checking out that commit I don't see any other drivers present in the
repo but the code does mention drivers like RabbitDriver and ZmqDriver
in oslo.messaging/openstack/common/messaging/drivers.py but only there.

Today the oslo_messaging._drivers.common.serialize_msg() call is used
to do the final serialization of the message. It uses jsonutils.dumps
since Icd54ee8e3f5c976dfd50b4b62c7f51288649e112 which is a revert of
I0e0f6b715ffc4a9ad82be52e55696d032b6d0976 that changed from
jsonutils.dumps to jsonutils.dump_as_bytes by mistake. And before this
back and forth it was jsonutils.dumps since the code was imported from
oslo-incubator by I38507382b1ce68c7f8f697522f9a1bf00e76532d. Here
I lost the trail. Honestly I don't know the reason why the fake driver
was made stricter than the real drivers. Still I think today the
strictness is unnecessary as every driver uses jsonutils and even
counterproductive as in [1].

[1] 09bf71407f/nova/compute/rpcapi.py (L820)

Change-Id: I186305b7897a2a4ce033c11ab9e6bc028854381b
Closes-Bug: #1529084
2019-09-21 14:53:29 +02:00
92ec6ec93e Update master for stable/train
Add file to the reno documentation build to show release notes for
stable/train.

Use pbr instruction to increment the minor version number
automatically so that master versions are higher than the versions on
stable/train.

Change-Id: I99e73a88a654b27cbd9334974413daf9e4a30e5d
Sem-Ver: feature
2019-09-20 16:34:26 +00:00
Matt Riedemann
440357d740 Fix help text for heartbeat_in_pthread option
Change-Id: Ibc40ad1ddedbe04d6ba614467a78e43b3aa6348d
2019-09-18 10:30:14 -04:00
Gabriele
b7e9faf659
Add the mandatory flag for direct send
With this feature, the server will raise and log a Message Undeliverable
exception. So it is possible to log immediately an error in case the
reply queue does not exist for some reason.

This is part of blueprint transport-options
The blueprint link is [1]
Please follow the link [2] to use and test the feature.

1-
https://blueprints.launchpad.net/oslo.messaging/+spec/transport-options
2- https://github.com/Gsantomaggio/rabbitmq-utils/
tree/master/openstack/mandatory_test

Change-Id: Iac7474c06ef425a2afe5bcd912e51510ba1c8fb3
2019-08-21 14:46:59 +02:00
Hervé Beraud
b56380654a Introduce RabbitMQ driver documentation
Introduce a RabbitMQ driver documentation for admin.

Describing:
- some RabbitMQ and AMQP specifications (exchanges, queues, routing-key)
- the heartbeat specification and the type of used threads
- the driver options

Change-Id: I8fd1624834510f8dee81ab9342c708d726b8f827
2019-08-08 17:59:56 +02:00
Zuul
1541b0c7f9 Merge "Allow users run the rabbitmq heartbeat inside a standard pthread." 2019-08-08 15:50:07 +00:00
Hervé Beraud
22f240b82f Allow users run the rabbitmq heartbeat inside a standard pthread.
This is an experimental feature.

The proposed changes will fix related issues when we run
heartbeat under apache/httpd enviornment with the apache MPM `prefork`
[1] engine and mod_wsgi or uwsgi in a monkey patched environment.

Propose changes to allow user to choose to run the rabbitmq health check
heartbeat in a standard python thread.

Issue
=====

We facing an issue with the rabbitmq driver heartbeat
under apache MPM `prefork` module and mod_wsgi when nova_api monkey
patched the stdlib by using eventlet.

nova_api calling eventlet.monkey_patch() [2] when it runs under mod_wsgi.

This impacts the AMQP heartbeat thread,
which is meant to be a native thread. Instead of checking AMQP sockets
every 15s, It is now suspended and resumed by eventlet. However,
resuming greenthreads can take a very long time if mod_wsgi isn't
processing traffic regularly, which can cause rabbitmq to close the AMQP
connection.

Root Cause
==========

The oslo.messaging RabbitMQ driver and especially the heartbeat
suffer to inherit the execution model of the service which consume him.

In this scenario nova_api need green threads to manage cells and edge
features so nova_api monkey patch the stdlib to obtain async features,
and the oslo.messaging rabbitmq driver endure these changes.

I think the main issue here is that nova_api want async and use eventlet green
threads to obtain it.

Solution
========

We want to allow user to isolate the heartbeat execution model
from the parent process inherited execution model by passing the
`heartbeat_in_pthread` option through the driver config.

While we use MPM `prefork` we want to avoid to use libevent and epoll.

If the `heartbeat_in_pthread` option is given we want to force to use the
python stdlib threading module to run the
rabbitmq heartbeat to avoid issue related to a non "standard"
environment. I mean "standard" because async features isn't the default
config in mostly case, starting by apache which define `prefork` is the
default engine.

This is an experimental feature, we can help us to ensure to run heartbeat
through a classical python thread

Specifications
==============

- https://review.opendev.org/661314

[1] https://httpd.apache.org/docs/2.4/fr/mod/prefork.html
[2] 3c5e2b0e9f

Change-Id: If8846599efc48fe18ecfb99c04e2c38f9a45b9ed
2019-08-07 17:17:02 +02:00
Ben Nemec
39196a4e90 Fix nits on kafka compression help text
There is no need to explicitly list the choices in the help text.
The oslo.config sample generator will include the choices automatically[0]

Also tweaks the wording of text to make it clear that it is the allowed
values which vary based on kafka version.

Change-Id: I4116e8871436097dea650f56e7b187358367d92e
0: 2488c1e1ce/oslo_config/generator.py (L263)
2019-08-06 19:00:24 +00:00
Zuul
dfc8fe9a8f Merge "Support kafka message compression" 2019-08-05 15:53:35 +00:00
root
042caf8a53 Correcting typo in acknowledge spelling
There is a typographical errors in amqpdriver.py. Correcting spelling
from acknowlege to acknowledge.

Change-Id: I4a80d8c6b162a99176eadb052f6201dc38dbc5f9
2019-07-30 13:17:07 +05:30
Zuul
002ac1453b Merge "Use default exchange for direct messaging" 2019-07-26 09:35:06 +00:00
Zuul
a5276be3dc Merge "Bump the openstackdocstheme extension to 1.20" 2019-07-22 15:12:22 +00:00
Zuul
46cdd88358 Merge "doc: Cleanup admin docs" 2019-07-19 08:26:04 +00:00
pengyuesheng
33dc529c37 Bump the openstackdocstheme extension to 1.20
Some options are now automatically configured by the version 1.20:
- project
- html_last_updated_fmt
- latex_engine
- latex_elements
- version
- release.

Change-Id: Ib5e22f6a5374f05e576bbc00a209209fdb09acad
2019-07-19 11:24:08 +08:00
Zuul
cbdc5df3eb Merge "Blacklist sphinx 2.1.0 (autodoc bug)" 2019-07-18 09:35:03 +00:00
Zuul
f7eb82a1e4 Merge "Implement mandatory flag for RabbitMQ driver" 9.8.0 2019-07-11 16:45:55 +00:00
Corey Bryant
1c31abc7bc Add Python 3 Train unit tests
This is a mechanically generated patch to ensure unit testing is in place
for all of the Tested Runtimes for Train.

See the Train python3-updates goal document for details:
https://governance.openstack.org/tc/goals/train/python3-updates.html

Change-Id: I699a3ee210c4b45018566b520eaf641e0e582362
Story: #2005924
Task: #34234
2019-07-05 14:47:53 -04:00
pengyuesheng
515a2d79ef Blacklist sphinx 2.1.0 (autodoc bug)
See https://github.com/sphinx-doc/sphinx/issues/6440 for upstream details
Depend-On: https://review.opendev.org/#/c/663060/

Change-Id: I648dd4d4a98a3d423c98d28c193737cffe1a2c49
2019-07-05 10:16:48 +08:00
Oleg Bondarev
6fe1aec1c7 Use default exchange for direct messaging
Lots of exchanges create problems during failover under high
load. Please see bug report for details.

This is a step 2 patch.

Step 1 was: only using default exchange
when publishing.
Step 2 is to update consumers to only listen on default exchange,
happening now in T release.

Change-Id: Ib2ba62a642e6ce45c23568daeef9703a647707f3
Closes-Bug: #1789177
2019-07-04 19:47:25 +04:00
Stephen Finucane
f0eee1e16b doc: Cleanup admin docs
Use a sensible header style, fix some syntax highlighting, and generally
tidy things up.

Change-Id: I0b141b968ed8db10ff41a626569dd185edbdc641
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
2019-07-04 14:50:57 +01:00
Gabriele
c50076b4ef
Implement mandatory flag for RabbitMQ driver
With this feature it is possible to use the mandatory RabbitMQ
mandatory flag.

Implements: blueprint transport-options (point 3)

The blueprint link is [1]

Please follow the link [2] to use and test the feature.

1- https://blueprints.launchpad.net/oslo.messaging/+spec/transport-options
2- https://github.com/Gsantomaggio/rabbitmq-utils/
tree/master/openstack/mandatory_test

Change-Id: Ie269fc08ba80c4b94a24a8207c1e86c19c3b3fcb
2019-07-01 21:38:32 +02:00
Zuul
6cdd4cb007 Merge "Implement the transport options" 2019-06-26 13:09:53 +00:00
Gabriele
e804874c50
Implement the transport options
With this feature, it is possible to specialize the parameters to send.
`options = oslo_messaging.TransportOptions(at_least_once=True)`

TransportOptions is used in every single driver,
for example in RabbitMQ driver is used to handle the mandatory flag.

Notes:
  - The idea of creating a new  class TransportOptions is because I'd like
    to have an abstract class not related only to the RPCClient
  - at_least_once is the first parameter, when needed we can add the
    others.

Implements: blueprint transport-options (second point)

The blueprint link is [1]
To test it you can use [2]

1- https://blueprints.launchpad.net/oslo.messaging/+spec/transport-options
2- https://github.com/Gsantomaggio/rabbitmq-utils/
   tree/master/openstack/mandatory_test

Change-Id: I1858e4a990507d3c2bac2ef7fbef75d8c2dbfce2
2019-06-24 16:50:35 +02:00