693 Commits

Author SHA1 Message Date
Kenneth Giusti
26b0be585a Issue blocking ACK for RPC requests from the consumer thread
The patch for https://review.openstack.org/#/c/436958/ fixed a
threading problem by moving the ack back to the polling
thread. However the RPC server expects to catch any failures of the
ACK and abort the request.  This patch adds the ACK error handling
back to the polling thread.

This patch is based heavily off the original work done by Mehdi
Abaakouk (sileht).

Change-Id: I708c3d6676b974d8daac6817c15f596cdf35817b
Closes-Bug: #1695746
2018-07-11 08:59:35 -04:00
Zuul
1f0682bc59 Merge "py37: drop use of 'async' as parameter name" 2018-07-09 17:38:37 +00:00
Zuul
9be8c1a6e2 Merge "Do not access the connection's socket during error callback" 2018-07-06 21:21:00 +00:00
Kenneth Giusti
361669764f Do not access the connection's socket during error callback
The _get_connection_info() method attempts to gather debug information
from the connection, and will reach into the amqp channel to get the
local (client's) TCP port number via the 'sock' property.

If _get_connection_info() is called from autoretry's on_error handler
the 'sock' property notices that the transport is not set and attempts
to re-connect.  amqp has deprecated this reconnect behavior, and in
any case the client's socket is irrelevant since the connection may
not be valid at this point.

Closes-Bug: #1745166
Change-Id: I3c42f8463605927f6f94d6c3a7f05e584476abc1
2018-07-06 12:15:53 -04:00
Kenneth Giusti
679572272e Fix debug log message - missing argument
Change-Id: I811c6051808a34fa5235b47e56ee3b41cb35adad
2018-07-06 11:41:07 -04:00
James Page
5626782fac py37: drop use of 'async' as parameter name
Python 3.7 does not allow the use of 'async' as
a parameter name or object attribute; update
occurrences to use a different name.

This is inline with PEP-492, where await and async
are keywords.

Change-Id: I73efcafab1e0832a0ada95f6c12cb6a659dfcf27
2018-07-05 06:09:37 -05:00
Zuul
9e61ad5405 Merge "Fix the bandit security linter test" 2018-07-04 18:04:50 +00:00
Zuul
8a5da2c28f Merge "Enable RPC call monitoring in AMQP 1.0 driver" 2018-07-04 00:04:17 +00:00
Kenneth Giusti
ae9e673887 Fix the bandit security linter test
Add the bandit security linter test to the pep8 target

Change-Id: I9324b9d9df77fd05c2e10bb48d7ccf8b4c634525
2018-07-02 13:16:11 -04:00
Dougal Matthews
044e6f20e6 Replace 'raise StopIteration' with 'return'
With PEP 479, the behaviour of StopIteration is changing. Raising it to
stop a generator is considered incorrect and from Python 3.7 this will
cause a RuntimeError. The PEP recommends using the return statement.

More details: https://www.python.org/dev/peps/pep-0479/#examples-of-breakage

Change-Id: Ib27581fccbbf14c082fb919d8b6edea1ac83e3c0
2018-06-29 14:39:46 +01:00
Steve Kowalik
5de0494739 Remove fake_rabbit configuration option
The fake_rabbit configuration option has been deprecated since the
release of 1.5.0 in late 2014. Finally remove it, and its test.

Change-Id: I014c2012cca0f289de0d95b9bb35bbde7f61d2ee
2018-06-26 15:49:27 +10:00
Kenneth Giusti
684e3f0e41 Enable RPC call monitoring in AMQP 1.0 driver
The call monitoring feature was introduced in commit
b34ab8b1cc9f4d513a2927c102dbbe82031d9c2a for RabbitMQ.  This patch
enables the feature on the AMQP 1.0 driver - currently the only other
driver that supports RPC.

Change-Id: Ic787696852690b59779fb4716aec1e78c48bbe6a
2018-06-13 11:01:38 -04:00
Kenneth Giusti
42f0358d5d Mark the ZeroMQ driver deprecated
As per the Dublin 2018 PTG decision:

http://lists.openstack.org/pipermail/openstack-dev/2018-March/128055.html

Change-Id: I1cda5dffbc29aad5fea001a79562db7c144a339b
2018-06-11 15:11:31 -04:00
TommyLike
7188835890 Add warning output if failed to rebuild exception when deserialize
In some specific cases, the middleware would fail to rebuild
the original exception, see bug [1] below. Adding this output
may help locate the root cause quickly.

[1]: https://bugs.launchpad.net/cinder/+bug/1728826

Change-Id: Ia9304bda4e515812b146885f830e70f28a285f2d
2018-06-06 14:04:40 +08:00
Dan Smith
b34ab8b1cc [rabbitmq] Implement active call monitoring
This adds an optional call_monitor_timeout parameter to the RPC client,
which if specified, will enable heartbeating of long-running calls by
the server. This enables the user to increase the regular timeout to
a much larger value, allowing calls to take a very long time, but
with heartbeating to indicate that they are still running on the server
side. If the server stops heartbeating, then the call_monitor_timeout
takes over and we fail with the usual MessagingTimeout instead of waiting
for the longer overall timeout to expire.

Change-Id: I60334aaf019f177a984583528b71d00859d31f84
2018-05-09 10:08:37 -07:00
Zuul
4d03b16334 Merge "Add heartbeat() method to RpcIncomingMessage" 2018-05-01 16:49:07 +00:00
Dan Smith
930e6189e2 Add heartbeat() method to RpcIncomingMessage
This adds a heartbeat() method to RpcIncomingMessage to be used by a
subsequent patch implementation of active-call heartbeating. This is
unimplemented in all drivers for the moment.

Change-Id: If8ab0dc16e3bef69d5a826c31c0fe35e403ac6a1
2018-04-24 07:35:56 -07:00
Andrew Smith
5f4755b92b Add kafka for python 3 functional test
Change-Id: I743cd09e3450fac215ff65db37c3fe53e2e43601
2018-04-17 14:05:20 -04:00
Zuul
b0776c515d Merge "Add restart() method to DecayingTimer" 2018-04-04 00:12:43 +00:00
Wonil Choi
e0e8a1ea68 Revert "rabbit: Don't prefetch when batch_size is set"
This reverts commit c881baed29db49c5710795496cb07907e35ceaba.

Eventhough batch_size is set, rabbitMQ prefetch count is not changed
at the first time of connection creation. It works only connection reset
cases(i.e. rabbitMQ restart). So this patch can not fix original issue
  https://bugs.launchpad.net/ceilometer/+bug/1551667

Moreover, it makes another bug that messages of sameple queues are not
consumed properly. So revert it.

Change-Id: Ia8ebee8e2a670e46b6a68859bc30e717bd56ed7e
Signed-off-by: Wonil Choi <wonil22.choi@samsung.com>
Closes-bug: 1759755
2018-04-02 10:36:33 +00:00
Kenneth Giusti
222a939361 Remove the deprecated Pika driver
It is recommended that all users of the Pika driver transition to
using the Rabbit driver instead.  Typically this is done by changing
the prefix of the transport_url configuration option from "pika://..."
to "rabbit://...".  There are no changes required to the RabbitMQ
server configuration.

Change-Id: I52ea5ccb7e7c247abd95e2d8d50dac4c4ad11246
Closes-Bug: #1744741
2018-03-21 10:58:23 -04:00
Dan Smith
68c48ad0bb Add restart() method to DecayingTimer
This merely provides a restart() method that passes through to the existing
restart() method on the StopWatch used as the internal for DecayingTimer so
that we can reset it.

Change-Id: Ie6b607ec588db94e2c768bd22ae736a05ab484c1
2018-03-07 10:24:52 -08:00
Andrew Smith
90f7610f9d Add kafka config options for security (ssl/sasl)
Change-Id: Ia1be1b67a7151d449185e2ad52eff1787e8b0933
2018-02-13 09:17:06 -05:00
Andrew Smith
d09bf21897 Add support for synchronous commit
This patch changes the default driver behavior to synchronously
commit messages following consumer poll. A configuration option
will enable the auto commit for asynchronous commit if desired.

Depends-On: I5b4f01c928373cac530aa6877a34c684577bc64e
Change-Id: I92a3dc95c5d424aa722138195fef5a855a66b31d
2018-01-18 14:57:54 -05:00
Andrew Smith
1ccdccddaa Add kafka driver vhost emulation
Emulate vhost support by adding the virtual host name to the
topic created on the kafka server. Also, update connection
management for producer/consumer.

This patch:
* updates target to topic generation
* add consumer and producer connection classes
* remove connection pool
* update driver test

Change-Id: Idd164444c04e9f465a43ee909af840a41bb090c0
2017-12-27 12:57:39 -05:00
Andrew Smith
3afc3a0a1d Update kafka functional test
This patch addresses a number of issues that prevented the functional
tests from running. The functional tests now execute and can complete
succesfully. At times, the test will fail (noticiably in CI) indicating
an underlying issue with consumer interaction with the kafka server.

It would be beneficial to merge this patch as it provides repeatability
and visibility for driver-kafka server integration to facilitate
additional debugging and testing.

This patch:

* removes use of deprecated get_transport
* override consumer_group for each test
* changed to synchronous send
* update to kafka 1.0.0 server

Depends-On: Ib552152e841a9fc0bffdcb7c3f7bc75613d0ed62
Change-Id: I7009a3b96ee250c177c10f5121eb73d908747a52
2017-12-16 14:41:03 -05:00
Mehdi Abaakouk
c38857e110 rabbitmq: don't wait for message ack/requeue
I don't see any obvious reason why we should wait ack/requeue is done.

This waiter have already be removed from amqp1.

https://git.openstack.org/cgit/openstack/oslo.messaging/tree/oslo_messaging/_drivers/amqp1_driver/controller.py#n242

So, this change remove it from rabbitmq driver too.

Closes-bug: #1734788

Change-Id: I5ecedc762596181be19410b863851a0054fd6579
2017-11-29 21:13:50 +01:00
Vincent Untz
8bfc3637a2 Catch socket.timeout when doing heartbeat_check
heartbeat_check in kombu.connection is not reraising exceptions as
exceptions.OperationalError, and the socket timeout during the heartbeat
check is really an issue seen in the field when a node goes down; the
heartbeat thread just tries again and again to deal with it, without
success.

Change-Id: I26dbdb18a7e64946db2cba676764ff2d428c7897
Closes-Bug: #1657444
2017-11-14 18:00:42 +01:00
gord chung
7fe91cc01a fix batch handling
for some reason there are two timeouts. in the batch scenario,
all the time wasted waiting on initial 'get' is never accounted
for so the batch timeout is always longer than it is declared.

Change-Id: I6132c770cccdf0ffad9f178f7463288cf954d672
2017-11-12 13:43:48 -05:00
Jenkins
6bbb3bc09d Merge "Class-level _exchanges in FakeExchangeManager" 2017-09-08 07:10:45 +00:00
Jenkins
809d4a54f1 Merge "Fix some reST field lists in docstrings" 2017-09-08 05:07:28 +00:00
Jenkins
338b85eb4e Merge "Suppress excessive debug logs when consume rabbit" 2017-09-07 23:24:51 +00:00
lingyongxu
d7229a124f Fix some reST field lists in docstrings
Probably the most common format for documenting arguments is reST field
lists [1]. This change updates some docstrings to comply with the field
lists syntax.
[1] http://sphinx-doc.org/domains.html#info-field-lists

Change-Id: Ifa8c0db3efc03eac3b034ef642aaa8fce514a66e
2017-09-07 05:42:45 +00:00
Jenkins
ff905655e9 Merge "Add support for virtual hosts" 2017-09-07 03:27:50 +00:00
Zhen Qin
147186c7b4 Suppress excessive debug logs when consume rabbit
If using rabbitmq as rpc backend, oslo.messaging generates large amount
of redundant timeout debug logs (several logs per second on multiple
openstack services, such as nova, heat, cinder), in format of 'Timed out
waiting for RPC response: Timeout while waiting on RPC response - topic:
"<unknown>", RPC method: "<unknown>" info: "<unknown>'. It's because
each socket timeout exception is raised to multiple levels of error
recovery callback functions then logged repeatedly.

However, the accompanying value of socket.timeout exception is currently
always “timed out”. Besides, oslo.messaging has implemented retry
mechanism to recover socket timeout failure. Therefore, IMO those logs
should be suppressed, even if at debug level, to save disk space and
make debugging more convenient.

Change-Id: Iafc360f8d18871cff93e7fd721d793ecdef5f4a1
Closes-Bug: #1714558
2017-09-06 21:35:33 -04:00
Jenkins
01098406df Merge "Remove envelope argument from driver send() interface" 2017-08-27 15:19:35 +00:00
Rajath Agasthya
d15a6deb17 Remove envelope argument from driver send() interface
It is ignored by most drivers and hardcoded to True in the rabbit
driver

Closes-Bug: #1712877
Change-Id: Ic6b0d07bff3c544aca4cccdfcb4c59b8705e9128
2017-08-25 08:17:47 -07:00
Jenkins
6f992dfc7f Merge "Prevent rabbit from raising unexpected exceptions" 2017-08-22 16:44:22 +00:00
Kenneth Giusti
f059bba6ae Prevent rabbit from raising unexpected exceptions
Publishing a message using the kombu connection autoretry method may
allow exceptions from the py-amqp library to be raised up to the
application. This does not conform to the documented oslo.messaging
API.

Enhance the try except block to capture any exception and translate it
into a MessageDeliveryFailure.

There are a few cases where exceptions will be raised during autoretry
publishing: recoverable connection or channel errors, and
non-recoverable connection or channel errors.

autoretry will only retry if the error is recoverable. Non recoverable
errors are re-raised immediately regardless of the retry count.

In the case of a recoverable error it seems unlikely that retrying
either the connection or the channel yet again is going to get us
anywhere, so in this case we simply clean up the channel state, log an
error and fail the operation.

In the case of non-recoverable error we are out of luck (think
authentication failure) - further retrying will not help.  Best we can
do is clean up state and log the heck out of it.

Change-Id: I2f65d2ee19a8c3e9a323b30404abbf0cbb45a216
Closes-Bug: #1705351
Closes-Bug: #1707160
2017-08-21 09:36:39 -04:00
lidong
b3b1ae1926 Fix a typo
Change-Id: Ie5cb0366804163309f573ca5a2bf5b129c3a8e9b
2017-08-10 01:10:47 +00:00
dparalen
d1dac1c11d Class-level _exchanges in FakeExchangeManager
The FakeExchangeManager uses an instance-level storage for FakeExchanges
mapping[1].  When a client--server pair is created, each keeps their own
instance of FakeDriver -> FakeExchangeManager -> FakeExchange, each of
which has their own (instance-level) copy of e.g _server_queues[2], making
it impossible for them to communicate.

This patch makes the _exchanges mapping a class-level attribute in order
to keep the registered exchanges shared between all Manager instances,
allowing client and server communication (within a single process).

The test_server unit-tests had to be refactored to explicitly pass an exchange
name when building a target. This is required for an exchange name change to
have any effect during a test case run time when compared to passing the
exchange name through the URL. This issue was revealed with this patch.

[1] https://github.com/openstack/oslo.messaging/blob/master/oslo_messaging/_drivers/impl_fake.py#L145,#L148
[2] https://github.com/openstack/oslo.messaging/blob/master/oslo_messaging/_drivers/impl_fake.py#L88,#L92

Change-Id: I8dff66f4cafeb1f4c57dbfbfaba5d49e50f55fee
Closes-Bug: #1714055
2017-08-02 08:23:34 +02:00
Kenneth Giusti
263dce9ea8 Add support for virtual hosts
Adds the 'pseudo_vhost' option which when enabled will incorporate the
virtual host into the address semantics.  This creates a 'subnet' like
address space for each virtual host.  Use this when the messaging bus
does not provide virtual hosting support.  It is enabled by default as
to date none of the supported AMQP 1.0 message buses natively support
virtual hosting.

It also updates SSL support: SSL can either use the connection
hostname or the vhost name when validating a server's
certificate. This is controlled by the 'ssl_verify_vhost' option.
This option is disabled by default as it requires both vhost and SNI
support from the server.  By default SSL will use the DNS name from
the TransportURL.

Change-Id: I49bb99d1b19e8e7e6fded76198da92ca5f7d65ab
Closes-Bug: #1700835
Partial-Bug: #1706987
2017-07-28 16:37:26 -04:00
ChangBo Guo(gcb)
60f9989920 Update URLs in documents according to document migration
Change-Id: I139d4d38e49590d50e51495b6e3b84836975c6ed
2017-07-12 22:54:02 +08:00
Andrew Smith
eccdea5ceb Add kafka_driver directory
Make uniform with other drivers, kafka driver files, etc.

Change-Id: I6c6e201c304a6005ef191f96e5ac39ffaf4ab8f7
2017-06-13 08:46:50 -04:00
Mehdi Abaakouk
8ee5ae135a Fix rabbitmq driver with blocking executor
We recently move ack/requeue of messages in main/polling thread
of rabbitmq drivers. And break the blocking executor.

This one is not tested by any tests and now deprecated.

This change workaround the issue until we completely remove the
blocking executor.

Change-Id: Id479100f6ff364cf67a199e9b70f9f0c7bf7e1a9
Closes-bug: #1694728
2017-06-08 14:15:15 +02:00
Kenneth Giusti
c35fb276ce Mark the Pika driver as deprecated
The Pika driver was intended to be a more stable and performant
replacement for the default rabbit driver.  However due to lack of
both maintainers and compelling evidence that pika is superior to the
existing rabbit driver in either performance or stability it has been
deprecated for removal.

See:
http://lists.openstack.org/pipermail/openstack-dev/2017-May/116679.html

Change-Id: I98e0123edd3248be665325833283689fc3a897f7
2017-05-30 13:21:23 -04:00
Jenkins
46f9648261 Merge "rabbit: restore synchronous ack/requeue" 2017-05-22 15:28:34 +00:00
Mehdi Abaakouk
da02bc2169 rabbit: restore synchronous ack/requeue
In https://review.openstack.org/#/c/436958, we fix a thread safety
issue. But we make the ack/requeue of message asynchronous. In nominal
case, it works, but if network/rabbit connection issue occurs this
can result to rpc call handle twice. By chance we double check already
processed message ids, and drop duplicates, but that if the message
goes to another node, the mitigation won't work.

This restore the previous behavior, to ensure we run application
callback of rpc.call/rpc.cast only when the message have been
successfully ack.

Change-Id: I62b9e09513e3ebfebc64a941d4b21b6c053b511d
2017-05-21 11:42:41 +02:00
Jenkins
db01a0eec0 Merge "[AMQP 1.0] if RPC call is configured as presettled ignore acks" 2017-05-17 05:02:52 +00:00
Kenneth Giusti
3b53605bad [AMQP 1.0] Add default SASL realm setting
Change-Id: I2ac680bb9b594f0493dfb6d52c3f40bf5282dec7
Closes-Bug: 1681927
2017-04-13 13:46:11 -04:00