oslo.messaging: remove double reply
Currently, when we are waiting for a RPC reply, for each call we receive two AMQP messages - first one with the payload, a second one to ensure the other have finish to send the payload. We are going to remove the second message for RPC reply. Co-Authored-By: Victor Sergeyev <vsergeyev@mirantis.com> Change-Id: I42106f4948279984b4dc037ca17eb9bc1cbc8207
This commit is contained in:
Mehdi Abaakouk
Victor Sergeyev
parent
d1d30679a2
commit
12014f2e5f
214
specs/liberty/oslo.messaging-remove-double-reply.rst
Normal file
214
specs/liberty/oslo.messaging-remove-double-reply.rst
Normal file
@ -0,0 +1,214 @@
|
|||||||
|
=====================================================
|
||||||
|
oslo.messaging: remove ending message for rpc reply
|
||||||
|
=====================================================
|
||||||
|
|
||||||
|
https://blueprints.launchpad.net/oslo.messaging?searchtext=remove-double-reply
|
||||||
|
|
||||||
|
We are going to send a single message for RPC reply.
|
||||||
|
|
||||||
|
Problem description
|
||||||
|
===================
|
||||||
|
|
||||||
|
Currently, when we wait for a RPC reply, for each msg_id we receive two AMQP
|
||||||
|
messages - first one with the payload, a second one to ensure the other have
|
||||||
|
finish to send the payload. This was made because a long time ago 'reply'
|
||||||
|
allowed generator as payload to send multiple messages on one 'rpc.call' -
|
||||||
|
see [1]_.
|
||||||
|
|
||||||
|
Oslo.messaging do not support providing a generator as the payload - it send
|
||||||
|
reply with data and then reply with ending - so it becomes useless to double
|
||||||
|
RPC messages for each call. Based on this suggestions, so we are going to
|
||||||
|
remove the second AMQP message sending.
|
||||||
|
|
||||||
|
This change will be not backward compatible, so we have to choice how we handle
|
||||||
|
this backward compatibility and the deprecation. This spec is the proposed
|
||||||
|
change about that.
|
||||||
|
|
||||||
|
Proposed change
|
||||||
|
===============
|
||||||
|
|
||||||
|
This change is not backward compatible, so we need to make sure that there is
|
||||||
|
some way to run mixed versions of services, where version N can always talk to
|
||||||
|
N+1 and the reverse, so that we can do rolling upgrades.
|
||||||
|
|
||||||
|
|
||||||
|
Plan for the rolling upgrade:
|
||||||
|
-----------------------------
|
||||||
|
|
||||||
|
In Kilo:
|
||||||
|
````````
|
||||||
|
The old behavior, fully compatible with Liberty and older versions.
|
||||||
|
|
||||||
|
RPC-server sends reply in two messages, can reply to clients, versions <= L+1.
|
||||||
|
RPC-client expects replies in two messages, can receive replies from servers
|
||||||
|
with versions <= L.
|
||||||
|
|
||||||
|
In Liberty:
|
||||||
|
```````````
|
||||||
|
We are going to implement a new behavior, but keep the old one by default.
|
||||||
|
Fully compatible with Kilo and older versions.
|
||||||
|
|
||||||
|
We change the ReplyWaiters to handle reply in one message and two messages,
|
||||||
|
like it does [2]_.
|
||||||
|
|
||||||
|
We create a boolean configuration option, defaulted to False. If this one is
|
||||||
|
True, we sent the reply in one message otherwise we keep the current behavior
|
||||||
|
of two messages. This config option will allow us to test the both - old and
|
||||||
|
new - behavior into our tests suite and allow to enable this new behavior
|
||||||
|
earlier. So this is only dedicated for early adopter and testing.
|
||||||
|
This options is not need in normal upgrade workflow and must be removed as soon
|
||||||
|
as we enable the new behavior by default to be sure that deployed cloud with
|
||||||
|
L+1 will not break when they are upgraded to L+2 because cloud operator
|
||||||
|
enforced the old behavior.
|
||||||
|
|
||||||
|
RPC-server sends reply in two messages, can reply to <= L+1
|
||||||
|
RPC-client expects replies in two messages or one message, can receive replies
|
||||||
|
from all versions
|
||||||
|
|
||||||
|
In M release cycle:
|
||||||
|
```````````````````
|
||||||
|
We are going to enable the new behavior by default, remove the config option
|
||||||
|
and support only the reply in one message. This will break backward
|
||||||
|
compatibility with oslo.messaging <= kilo and oslo-incubator RPC legacy code.
|
||||||
|
|
||||||
|
rpc-server sends reply in one messages, can reply to >= L
|
||||||
|
rpc-client expects replies in two messages or one messages from all versions
|
||||||
|
|
||||||
|
In N release cycle:
|
||||||
|
```````````````````
|
||||||
|
We are going to remove legacy code that allow to receive replies from <= L.
|
||||||
|
|
||||||
|
rpc-server sends reply in one messages, can reply to >= L+1
|
||||||
|
rpc-client expects replies in one messages, can receive replies from >= L+1
|
||||||
|
|
||||||
|
|
||||||
|
Alternatives
|
||||||
|
------------
|
||||||
|
|
||||||
|
Using the oslo.messaging payload version.
|
||||||
|
But this have been designed for the content of the message itself.
|
||||||
|
Not really for this purpose. And the deserialization occurs in the lower
|
||||||
|
layer of oslo.messaging. When we handle the reply the version fields have
|
||||||
|
already been removed.
|
||||||
|
|
||||||
|
This breaks backward compatibility too.
|
||||||
|
|
||||||
|
We can already track the old and the new format because the old format has
|
||||||
|
the attribute "result" OR "ending" the new one will have "result" AND
|
||||||
|
"ending".
|
||||||
|
|
||||||
|
Note that issue is in the RPC call replies code. In case of rolling upgrade, we
|
||||||
|
have to think about the fact that the client will wait message that can
|
||||||
|
come from same or upper version of oslo.messaging. We already do not support
|
||||||
|
lower versions from the application PoV.
|
||||||
|
|
||||||
|
Or from the server point of view, we will send reply that must be
|
||||||
|
understandable by a wide panel of versions.
|
||||||
|
|
||||||
|
This is the first time (I guess), we encounter this kind of issue, some other
|
||||||
|
bugs need to break the backward compatibility to be fixed, too.
|
||||||
|
(Because we need to change RabbitMQ queue attributes or move a queue to
|
||||||
|
another exchange)
|
||||||
|
|
||||||
|
The main goal is to choose the backward compatible versions count and use
|
||||||
|
the same kind of deprecation in other changes like this one.
|
||||||
|
|
||||||
|
Impact on Existing APIs
|
||||||
|
-----------------------
|
||||||
|
|
||||||
|
NA
|
||||||
|
|
||||||
|
Security impact
|
||||||
|
---------------
|
||||||
|
|
||||||
|
NA
|
||||||
|
|
||||||
|
Performance Impact
|
||||||
|
------------------
|
||||||
|
|
||||||
|
This will reduce by 2 the number of reply messages that will transit on a
|
||||||
|
RabbitMQ/QPID cluster.
|
||||||
|
Local performance tests shows, that this change brings nearly 30 percent
|
||||||
|
increase in the number of RPC `call` messages per second.
|
||||||
|
|
||||||
|
Configuration Impact
|
||||||
|
--------------------
|
||||||
|
|
||||||
|
A hidden configuration option will allow to switch to the future behavior
|
||||||
|
for early adopter and for testing purpose.
|
||||||
|
|
||||||
|
Developer Impact
|
||||||
|
----------------
|
||||||
|
|
||||||
|
NA
|
||||||
|
|
||||||
|
Testing Impact
|
||||||
|
--------------
|
||||||
|
|
||||||
|
We must test that the new ReplyWaiters code can handle reply that come from
|
||||||
|
a oslo.messaging version that sent reply in one message and from the one
|
||||||
|
that sent the reply into two messages. This is driver specific change, so we
|
||||||
|
should implement this in unittests.
|
||||||
|
|
||||||
|
Implementation
|
||||||
|
==============
|
||||||
|
|
||||||
|
Assignee(s)
|
||||||
|
-----------
|
||||||
|
|
||||||
|
Primary assignee:
|
||||||
|
vsergeyev
|
||||||
|
|
||||||
|
Other contributors:
|
||||||
|
sileht
|
||||||
|
|
||||||
|
Milestones
|
||||||
|
----------
|
||||||
|
|
||||||
|
Target Milestone for completion:
|
||||||
|
Liberty for the step 1
|
||||||
|
M or N for the step 2
|
||||||
|
|
||||||
|
Work Items
|
||||||
|
----------
|
||||||
|
|
||||||
|
1. Change the ReplyWaiters to handle reply in one message and two messages.
|
||||||
|
2. Add a config option and change the _send_reply() method to allow sent reply
|
||||||
|
and `ending` in a single message, based on config option.
|
||||||
|
3. Remove the config option and enable the new behavior by default.
|
||||||
|
4. Remove the `ending` parameter procession from the ReplyWaiters.
|
||||||
|
|
||||||
|
Incubation
|
||||||
|
==========
|
||||||
|
|
||||||
|
NA
|
||||||
|
|
||||||
|
Documentation Impact
|
||||||
|
====================
|
||||||
|
|
||||||
|
Inform deployer about future incompatibility with a too old oslo.messaging
|
||||||
|
version
|
||||||
|
|
||||||
|
Dependencies
|
||||||
|
============
|
||||||
|
|
||||||
|
NA
|
||||||
|
|
||||||
|
References
|
||||||
|
==========
|
||||||
|
|
||||||
|
.. [1] Legacy oslo rpc behavior: https://github.com/openstack/oslo-incubator/blob/stable/icehouse/openstack/common/rpc/amqp.py#L461-L470
|
||||||
|
.. [2] Refactor processing reply in ReplyWaiter: https://review.openstack.org/#/c/180583/
|
||||||
|
|
||||||
|
WIP reviews:
|
||||||
|
|
||||||
|
* https://review.openstack.org/#/c/180542/
|
||||||
|
* https://review.openstack.org/#/c/180583/
|
||||||
|
|
||||||
|
|
||||||
|
.. note::
|
||||||
|
|
||||||
|
This work is licensed under a Creative Commons Attribution 3.0
|
||||||
|
Unported License.
|
||||||
|
http://creativecommons.org/licenses/by/3.0/legalcode
|
||||||
|
|
||||||
Loading…
Reference in New Issue
Block a user