When a client is gone (died/restart) and somes replies cannot be sent because
the the exchange of this client will never comeback. We log one message per
reply every 0.25 messages during 60 seconds. When the only useful log
is the one where we decide to drop this replies.
This change moves the less important message to debug level.
Change-Id: I508787c0db4dcec2c0027b89eb4e65c4f98022b9
Related-bug: #1524418
When AMQPDestinationNotFound is raised, we must not
call the error_callback method. The exception is logged
only if needed in upper layer (amqpdriver.py).
Related-bug: #1524418
Change-Id: Ic1ddec2d13172532dbaa572d04a4c22c97ac4fe7
When a heartbeat is missing we call ensure_connection()
that runs a dummy method to trigger the reconnection
code in kombu. But also the code is triggered only if the
channel is None.
In case of the heartbeat threads we didn't reset the channel
before reconnecting, so the dummy method doesn't do anything.
This change sets the channel to None to ensure the connection
is reestablished before the dummy method is run.
Also it replaces the dummy method by checking the kombu connection
object. So we are sure the connection is reestablished.
Change-Id: I39f8cd23c5a5498e6f4c1aa3236ed27f3b5d7c9a
Closes-bug: #1493890
The transport/driver features check is done into the get listener
methods.
So when these methods are not used the driver features checks is not
done.
This change moves it into the dispatcher layer to ensure the
requirements are always checked.
This changes a bit the behavior of when the check occurs. Before
it was during the listener object initialisation. Now this
when the listener server start.
Change-Id: I4d81a4e8496f04d62e48317829d5dd8b942d501c
Gnocchi performs better if measurements are write in batch
When Ceilometer is used with Gnocchi, this is not possible.
This change introduce a new notification listener that allows that.
On the driver side, a default batch implementation is provided.
It's just call the legacy poll method many times.
Driver can override it to provide a better implementation.
For example, kafka handles batch natively and take benefit of this.
Change-Id: I16184da24b8661aff7f4fba6196ecf33165f1a77
This change creates a dispatcher abstraction
to document the interface of a dispatcher.
And also allows in the futur to have attributes with default values.
Change-Id: I9a7e5e03f89635a3790b3851f492a1a7aab58feb
In this change import_zmq() doesn't raise ImportError any more
for the benefit of skipping tests.
Alarm about zmq unavailability moved to driver's init.
Change-Id: I6e6acc39f42c979333510064d9e845228400d233
Closes-Bug: #1522920
This change removes codepath where _reply_q is not
present in the message dict.
This kind of messages have been deprecated in grizzly and cannot
be emitted since havana.
70891c271e
Change-Id: I20558d9fae8f56970c967aa0def77cfb2a1ca3ec
This change the default of kombu_missing_consumer_retry_timeout
The initial value of 60 seconds, have been chosen because the default
rpc_response_timeout is 60. That means, the client doesn't wait for
its reply after rpc_response_timeout is reach, so we don't need
to retry it send it its reply more than rpc_response_timeout.
But the real intent of kombu_missing_consumer_retry_timeout is
to mitigate the side effect when the rabbitmq server(s) died/failover/restart.
So the question is more how long we expect the server(s) to come back
and all the oslo.messaging applications to reconnect.
In that case 60 seconds looks a bit high.
Also this 60 seconds have a sad side effect when we can't send the reply
when the rpc client is really gone (like nova-compute restart).
The rabbitmq connection to send the reply is hold during 60 seconds.
I propose 5 seconds because,i n case of failover or restart I expect
everything because normal in less that 5 seconds.
Change-Id: I2ec174e440eb91e950d9453a9de8b97ed5888968
This change renames kombu_reconnect_timeout to missing_consumer_retry_timeout.
And improves its documentation.
Change-Id: I961cf96108db2f392b13d159f516baac9ff4e989
Problem with recursion shows up only in full runs
of Nova for example. So split the code that sets
up the decorator and add a method to cleanup
the decorated set_override during teardown.
Also add a decorator for clear_override similar to
the one for set_override.
Added more tests for all the above.
Change-Id: Ib16af2e770e96d971aef7f5c5d48ffd781477cfe
Neutron and Ceilometer use set_override to set
the older deprecated key. We should support them
using the ConfFixture
Closes-Bug: #1521776
Change-Id: I2bd77284f80bc4525f062f313b1ec74f2b54b395
This change moves the reply retry code to upper layer
to be able to release the connection while we wait between
two retries.
In the worse scenario, a client waits for more than 30 replies
and died/restart, the server tries to send this 30 replies to this
this client and can wait too 60s per replies. During this
replies for other clients are just stuck.
This change fixes that.
Related-bug: #1477914
Closes-bug: #1521958
Change-Id: I0d3c16ea6d2c1da143de4924b3be41d1cea159bd
Adding a driver for Apache Kafka connection, supporting
notification via Kafka. This driver is experimental
until having functional and integration tests
Change-Id: I7a5d8e3259b21d5e29ed3b795d04952e1d13ad08
Implements: blueprint adding-kafka-support
In change Ief6f95ea906bfd95b3218a930c9db5d8a764beb9, we
decoupled RPC and Notifications a bit. We should take another
step and separate out the options for notifications into
its own group.
Change-Id: Ib51e2839f9035d0cc0e3f459939d9f9003a8c810
ConnectionPool and ConnectionContext can be used by other
drivers (like Kafka) and hence should be outside of amqp.py.
* Moving ConnectionPool to pool.py
* Moving ConnectionContext to common.py
* Moving a couple of global variables to common.py
No other logic changes, just refactoring
Change-Id: I85154509a361690426772ef116590d38a965ca8d
Shuffle strategy we use right now leads to increased reconnection time
and provides no benefit. Sometimes it might lead to RPC operations
timeout because the strategy provides no guarantee on how long the
reconnection process will take. See the referenced bug for details.
On the other side, round-robin strategy provides least achievable
reconnection time. It also provides guarantee that if K of N RabbitMQ
hosts are alive, it will take at most N - K + 1 attempts to
successfully reconnect to RabbitMQ cluster.
With shuffle strategy during failover clients connect to random hosts
and so the load is distributed evenly between alive RabbitMQs.
But since we shuffle list of hosts before providing it to Kombu, load
will be distributed evenly with round-robin strategy as well.
DocImpact
A new configuration option kombu_failover_strategy for Kombu driver is
added. It determines how the next RabbitMQ node is chosen in case the
one we are currently connected to becomes unavailable. It takes effect
only if more than one RabbitMQ node is provided in config. Available
options are:
* round-robin: each RabbitMQ host in the list is tried in cycle until
oslo.messaging successfully connects. Since oslo.messaging
shuffles list of RabbitMQ hosts, the order of hosts in the cycle
will be random and will not depend on order provided in config.
* shuffle: oslo.messaging selects a random host from the list and
tries to connect to it. If connection fails, oslo.messaging repeats
attempt to connect to another random host. Oslo.messaging stops
once it successfully connects to a host. Note that in each
iteration a host to connect is selected independently of previous
iterations, i.e. it might happen that oslo.messaging will try to
connect to the same host several times in a row.
The option's default value is round-robin. Before the option was
introduced, the default strategy was shuffle. For the reasoning,
see the main body of the commit message and the referenced bug.
Closes-Bug: #1519851
Change-Id: I9a510c86bd5a6ce8b707734385af1a83de82804e
