22 Commits

Author SHA1 Message Date
Gabriele
83266cc6ea
Add transport_options parameter
With this new parameter is possible to pass other parameters
from the client to the drivers.
So it is possible to tune the driver behavior.

For example can be used to send the mandatory flag in RabbitMQ

Note:
  - The transport_options parameter is not actually used (yet).
  - This part of blueprint transport-options  (first part)

Implements: blueprint transport-options
The blueprint link is
https://blueprints.launchpad.net/oslo.messaging/+spec/transport-options

Change-Id: Iff23a9344c2c87259cf36b0d14c0a9fc075a2a72
2019-06-14 15:47:39 +02:00
Dan Smith
b34ab8b1cc [rabbitmq] Implement active call monitoring
This adds an optional call_monitor_timeout parameter to the RPC client,
which if specified, will enable heartbeating of long-running calls by
the server. This enables the user to increase the regular timeout to
a much larger value, allowing calls to take a very long time, but
with heartbeating to indicate that they are still running on the server
side. If the server stops heartbeating, then the call_monitor_timeout
takes over and we fail with the usual MessagingTimeout instead of waiting
for the longer overall timeout to expire.

Change-Id: I60334aaf019f177a984583528b71d00859d31f84
2018-05-09 10:08:37 -07:00
Dan Smith
930e6189e2 Add heartbeat() method to RpcIncomingMessage
This adds a heartbeat() method to RpcIncomingMessage to be used by a
subsequent patch implementation of active-call heartbeating. This is
unimplemented in all drivers for the moment.

Change-Id: If8ab0dc16e3bef69d5a826c31c0fe35e403ac6a1
2018-04-24 07:35:56 -07:00
gord chung
7fe91cc01a fix batch handling
for some reason there are two timeouts. in the batch scenario,
all the time wasted waiting on initial 'get' is never accounted
for so the batch timeout is always longer than it is declared.

Change-Id: I6132c770cccdf0ffad9f178f7463288cf954d672
2017-11-12 13:43:48 -05:00
Jenkins
01098406df Merge "Remove envelope argument from driver send() interface" 2017-08-27 15:19:35 +00:00
Rajath Agasthya
d15a6deb17 Remove envelope argument from driver send() interface
It is ignored by most drivers and hardcoded to True in the rabbit
driver

Closes-Bug: #1712877
Change-Id: Ic6b0d07bff3c544aca4cccdfcb4c59b8705e9128
2017-08-25 08:17:47 -07:00
lidong
b3b1ae1926 Fix a typo
Change-Id: Ie5cb0366804163309f573ca5a2bf5b129c3a8e9b
2017-08-10 01:10:47 +00:00
Kenneth Giusti
adf4cd9d05 Document the transport backend driver interface
Add detailed documentation to the driver API to help driver developers
create drivers that behave consistently.  Specifically prescribes a
set of operational characteristics that a driver must conform to in
order to provide consistent behavior across different implementations.

Change-Id: Icb251ee724f9a0ac4fede702a367910de4ba95e3
2016-11-22 08:11:22 -05:00
Kirill Bespalov
162f6e987b Introduce TTL for idle connections
We can reduce a workload of rabbitmq through implementation
of expiration mechanism for idle connections in the pool with
next properties:

 conn_pool_ttl (default 20 min)
 conn_pool_min_size: the pool size limit for expire() (default 2)

The problem is timeless idle connections in the pool, which can be created
via some single huge workload of RPCServer. One SEND connection is heartbeat
thread + some network activity every n second. So, we can reduce it.

Here is two ways to implement an expiration:

 [1] Create a separated thread for checking expire date of connections
 [2] Make call expire() on pool.get() or pool.put()

The [1] has some threading overhead, but probably insignificant
because the thread can sleep 99% time and wake up every 20 mins (by default).
Anyway current implementation is [2].

Change-Id: Ie8781d10549a044656824ceb78b2fe2e4f7f8b43
2016-07-22 13:03:04 +03:00
Gevorg Davoian
39749c77a8 Remove logging from serialize_remote_exception
This patch removes log_failure argument from the function
serialize_remote_exception and from driver implementations
using it (because it is never used and always defaults to True)
and prevents error logging in this function (because these errors
are already logged by servers while processing incoming messages).

Change-Id: Ic01bb11d6c4f018a17f3219cdbd07ef4d30fa434
Closes-Bug: 1580352
2016-05-19 15:18:28 +03:00
Dmitriy Ukhlov
6db00c77b0 Refactor base interfaces
1) Add MessageHandler base interface for on_incoming_callback replacement
2) Move message_handler parameter form Listener's __init__() to start()
3) Remove wait method from listener

Change-Id: Id414446817e3d2ff67b815074d042a9ce637ec24
2016-04-20 20:46:28 +00:00
Dmitriy Ukhlov
5d7d7253d1 Refactor driver's listener interface
Current Listener interface has poll() method which return messages

To use it we need have poller thread which is located in MessageHandlerServer
But my investigations of existing driver's code shows that some implemetations have
its own thread inside for processing connection event loop. This event loop received
messages and store in queue object. And then our poller's thread reads this queue
This situation can be improved. we can remove poller's thread, remove queue object
and just call on_message server's callback from connection eventloop thread

This path provide posibility to do this for one of drivers and leave as is other drivers

Change-Id: I3e3d4369d8fdadcecf079d10af58b1e4f5616047
2016-04-05 18:08:08 +00:00
Davanum Srinivas
fb73297112 Fix Notification listener blocking behavior
Add a sleep() to allow other threads (like the one collecting
the stats) to run.

Closes-Bug: #1555632
Change-Id: I6fcb63c10acd76f2815e23fbd303f08974feb993
2016-03-11 12:15:52 -05:00
dukhlov
3cb893433f Refactors base classes
Base classes should define interface between driver and
upper oslo messaging layer and shouldn't have fields and
methods used for internal driver's purposes.

1) base listener: + prefetch_size attribute;
                  -driver attribute
2) base incomingMessage: - reply method
                         - listener attribute
3) base RpcIncomingMessage added - it is incomingMessage + reply method

Change-Id: Ic2c0fce830763eb4e00f2cca789e9c1c6b5420ed
2016-02-08 10:34:25 +02:00
John Eckersberg
bb8f950edc rabbit: Add option to configure QoS prefetch count
Change-Id: I585cd4636e62cdd0c9f1595a6c029cb56f845be9
Closes-Bug: #1531222
2016-01-19 23:03:01 -05:00
Mehdi Abaakouk
4dd644ac20 batch notification listener
Gnocchi performs better if measurements are write in batch
When Ceilometer is used with Gnocchi, this is not possible.

This change introduce a new notification listener that allows that.

On the driver side, a default batch implementation is provided.
It's just call the legacy poll method many times.

Driver can override it to provide a better implementation.
For example, kafka handles batch natively and take benefit of this.

Change-Id: I16184da24b8661aff7f4fba6196ecf33165f1a77
2015-12-08 09:14:20 +01:00
Oleksii Zamiatin
141f59bd9b Notifier implementation
Notifier implementation for zmq driver (ROUTER/DEALER variant).
Publishers/consumers refactoring in order to make them pluggable.

Change-Id: I2dd42cc805aa72b929a4dfa17498cd8b9c0ed7af
2015-08-05 13:35:24 +03:00
Oleksii Zamiatin
ebcadf3d5e Fix threading zmq poller and proxy
- Fixed universal proxy to not get stuck with multiple backends
- Fixed threading pollers/executors (proxy side)
    - Driver option to switch green/no-green impl.
    - Swtiched to no-green in real-world proxy (green left for unit tests)
- Minor names fixes in serializer

Change-Id: Id6508101521d8914228c639ed58ecd29db0ef456
2015-07-15 16:46:04 +03:00
Ben Nemec
94bb8adfd0 Remove leftover oslo.config reference
There was still a reference to the oslo.config namespace package
in _drivers/base.py.  This converts it to the new package name.

Change-Id: I9c3878094bcf8015c30d87f693f51e0d48b31a33
2015-06-04 00:06:40 +00:00
Mehdi Abaakouk
18c32bc30c Fix deprecated_group of rpc_conn_pool_size
When rpc_conn_pool_size have been moved from amqp.py to base.py:

  87137e7af05f12a99bd04566036fbf71824f45cf

We loose the deprecated_group, this change reintroduces it.

Change-Id: I8cdea7f042afebcc162bafef881ebe61a1cac989
2015-05-27 13:06:40 +02:00
James Page
de015d5c83 zmq: Add support for ZmqClient pooling
To avoid creating a new ZMQ connection for every message sent
to a remote broker, implement pooling and re-use of ZmqClient
objects and associated ZMQ context.

A pool is created for each remote endpoint (keyed by address);
the size of each pool is configured using rpc_conn_pool_size.

All outbound message client connections are pooled.

Closes-Bug: 1384113
Change-Id: Ia55d5c310a56e51df5e2f5d39e561a4da3fe4d83
2015-04-26 11:01:04 +01:00
Doug Hellmann
e55a83e832 Move files out of the namespace package
Move the public API out of oslo.messaging to oslo_messaging. Retain
the ability to import from the old namespace package for backwards
compatibility for this release cycle.

bp/drop-namespace-packages

Co-authored-by: Mehdi Abaakouk <mehdi.abaakouk@enovance.com>
Change-Id: Ia562010c152a214f1c0fed767c82022c7c2c52e7
2015-01-12 12:50:41 -05:00