39 Commits

Author SHA1 Message Date
John Eckersberg
d873c0d8f5 Do not use threading.Event
Waiting on a threading.Event with eventlet can cause busy looping via
epoll_wait, see related bug for more details.

Change-Id: I007613058a2d21d1712c02fa6d1602b63705c1ab
Related-bug: #1518430
2019-12-18 13:11:41 +00:00
Hervé Beraud
4f385720d7 Remove log translation and i18n
Log messages are no longer being translated. This removes all use of
the _LE, _LI, and _LW translation markers to simplify logging and to
avoid confusion with new contributions.

Change-Id: I9ddb6595fc52e46ed8844e39d2fa71029c90b65c
Closes-Bug: #1674567
2019-04-24 11:55:38 +02:00
ZhijunWei
cb902e5aff Update hacking version
update the hacking to latest, duo to[1]

[1]: https://github.com/openstack/oslo.messaging/blob/master/HACKING.rst

Change-Id: I1ee7387272690f0de0d7b1937d7c2b0ee404063c
2019-01-21 19:45:42 -05:00
Andy Smith
252844879d Don't use monotonic with Python >=3.3
A change to the global-requirements has limited use of monotonic
library to Python versions earlier than 3.3 (later versions have
built-in support for a monotonic clock), so limit it in
requirements.txt.

Note: this patch updates kafka driver (due to deprecated exception
in library) in order to pass unit tests

Change-Id: Id6b0814e05a0e548a8c2a5359daf1a6878cf6859
2018-12-03 14:52:49 -05:00
Steve Kowalik
8881ed4378 Remove deprecated amqp1 options
The amqp1 options allow_insecure_options, username and password are all
deprecated, remove them. allow_insecure_options was deprecated during
Ocata, and the other two were deprecated in 5.22.0, released during
Pike.

Change-Id: I321266c3dba83a38fe6ebd775db2e74c14cc96cd
2018-09-05 16:59:13 +10:00
Zuul
9e61ad5405 Merge "Fix the bandit security linter test" 2018-07-04 18:04:50 +00:00
Kenneth Giusti
ae9e673887 Fix the bandit security linter test
Add the bandit security linter test to the pep8 target

Change-Id: I9324b9d9df77fd05c2e10bb48d7ccf8b4c634525
2018-07-02 13:16:11 -04:00
Kenneth Giusti
684e3f0e41 Enable RPC call monitoring in AMQP 1.0 driver
The call monitoring feature was introduced in commit
b34ab8b1cc9f4d513a2927c102dbbe82031d9c2a for RabbitMQ.  This patch
enables the feature on the AMQP 1.0 driver - currently the only other
driver that supports RPC.

Change-Id: Ic787696852690b59779fb4716aec1e78c48bbe6a
2018-06-13 11:01:38 -04:00
Kenneth Giusti
263dce9ea8 Add support for virtual hosts
Adds the 'pseudo_vhost' option which when enabled will incorporate the
virtual host into the address semantics.  This creates a 'subnet' like
address space for each virtual host.  Use this when the messaging bus
does not provide virtual hosting support.  It is enabled by default as
to date none of the supported AMQP 1.0 message buses natively support
virtual hosting.

It also updates SSL support: SSL can either use the connection
hostname or the vhost name when validating a server's
certificate. This is controlled by the 'ssl_verify_vhost' option.
This option is disabled by default as it requires both vhost and SNI
support from the server.  By default SSL will use the DNS name from
the TransportURL.

Change-Id: I49bb99d1b19e8e7e6fded76198da92ca5f7d65ab
Closes-Bug: #1700835
Partial-Bug: #1706987
2017-07-28 16:37:26 -04:00
Jenkins
db01a0eec0 Merge "[AMQP 1.0] if RPC call is configured as presettled ignore acks" 2017-05-17 05:02:52 +00:00
Kenneth Giusti
3b53605bad [AMQP 1.0] Add default SASL realm setting
Change-Id: I2ac680bb9b594f0493dfb6d52c3f40bf5282dec7
Closes-Bug: 1681927
2017-04-13 13:46:11 -04:00
Kenneth Giusti
d778aa5a44 [AMQP 1.0] if RPC call is configured as presettled ignore acks
The pre-settled configuration option was being ignored in the case of
RPC call transfers.

Change-Id: I35989c8a653bac0d9a7c3e33a8af2027574f4bd9
Closes-Bug: 1680905
2017-04-07 13:23:55 -04:00
Juan Antonio Osorio Robles
bf1cdafd03 Add "ssl" option for amqp driver
This uses the underlying x-ssl option that will attempt an SSL
connection to the server using the system's CA bundle.

Change-Id: I73f3b43b4ebc0c4ce4c0ba43b5a84361fb308686
2017-04-03 17:23:17 +03:00
Kenneth Giusti
53ead5c11d [AMQP 1.0] Fix SSL client authentication
The driver incorrectly set up SSL client authentication.  This patch
fixes the configuration. Unit tests added.

Change-Id: I64b1736f5a708c70013d9fedba73da4fa8d9ccfb
Closes-Bug: #1606985
2017-01-13 11:52:06 -05:00
Kenneth Giusti
9fab0bdbc8 [AMQP 1.0] Resend messages that are released or modified
A message ack status of 'RELEASED' or 'MODIFIED' indicates that the
message was not accepted by the destination due to some temporary
issue.  These status are used to indicate to the sender that the
message can be safely re-transmitted without risk of duplication
(i.e. the delivery is not 'in-doubt').  For example this may happen
during a message bus topology update if a message is sent before the
topology stabilizes.

This change implements re-send for these cases.

Closes-Bug: #1646586
Change-Id: I419e23b59e3eb90fda3f1c0e7ddf54ef98870e4b
2017-01-09 09:30:57 -05:00
Kenneth Giusti
9972158baa [AMQP 1.0] Propagate authentication errors to caller
When the connection fails due to an authentication issue raise a
MessageDeliveryFailure on all pending send requests. Include the
authentication error message in the exception.

Change-Id: I06b40c6c480a4d082dce64801736b3d12f696c26
Closes-Bug: 1508512
2017-01-03 09:48:55 -05:00
Kenneth Giusti
41e23c2436 [AMQP 1.0] Simplify the I/O event loop code
Leverage the fact that there is never more than one socket active to
simplify the event loop's I/O processing.  This removes the need to
look up active connections.  Also removes the need to query the return
values from select since there is only one socket to process. This
change improves RPC throughput under the simulator by up to 20%

As part of this change I've clarified the code's use of the pyngus
connection by renaming the connection property to something a bit more
specific.

Change-Id: If7c020bb0bd96490af78bc06659db10073b02417
2016-12-05 19:49:06 -05:00
Andrew Smith
3c9c97603f Periodically purge sender link cache
Closes-Bug: 1640266
Change-Id: Id38b756792f8fb14803bc9ead69e1fc903b9eea5
2016-12-01 10:31:56 -05:00
Kenneth Giusti
bc46e64711 [AMQP 1.0] Avoid unnecessary thread switch on ack
Avoid blocking the RPC Server thread when it attempts to acknowledge
the RPC request message.  Blocking is unnecessary as the acknowledge
command does not return a status and can be processed asynchronously
from the server thread.

Avoiding this context switch improves overall RPC call throughput
according to the simulator tool by approximately ten percent on my
test systems.

Change-Id: I71548eb6f9f7dcaf74cb426d4e9b369b54856419
2016-10-11 10:11:35 -04:00
Kenneth Giusti
03ef584218 cast() and RPC replies should not block waiting for endpoint to ack
This patch fixes cast() and RPC replies so they do not block the
caller once the messages have been written to the link.  In both of
these cases messages are sent "at most once" with best effort.

If a negative acknowledgment is received a warning is logged.

Change-Id: I84671c62544ec388421ecd0ccafc267c3c3d6087
Closes-Bug: #1630637
2016-10-06 13:11:14 -04:00
Kenneth Giusti
9775ed3cce Remove debug logs from fast path
Decrease the size of the debug log output drastically by removing
debug messages issued for each message.  This patch only removes those
log statements that log correct behavior - errant behavior is still
logged for debug.

Change-Id: I1c67e6ed7e503ef3b9543fdfce31b91c46bd851a
2016-10-03 16:31:35 -04:00
Kenneth Giusti
6c973dae22 [AMQP 1.0] Make the default settlement behavior configurable
This patch allows the ack processing behavior to be configured based
on the type of message sent.  This allows the proper ack behavior for
a given backend to be customized. By default Cast messages are now
sent pre-acked (best effort).

Change-Id: I0d4e44a7a87f3c917a3eb44f6e02af74a9af10a5
Closes-Bug: #1616612
2016-08-24 15:56:51 -04:00
Jenkins
ba387dfd1c Merge "[AMQP 1.0] small fixes to improve timer scalability" into feature/amqp-dispatch-router 2016-08-17 05:49:40 +00:00
Jenkins
5d5596a43d Merge "[AMQP 1.0] Add link credit configuration options" into feature/amqp-dispatch-router 2016-08-16 01:01:01 +00:00
Kenneth Giusti
b7717e1616 [AMQP 1.0] small fixes to improve timer scalability
This patch introduces the following tweaks to the timer
implementation:

Reduce the number of timers that need to be tracked by reducing the
timer granularity to units of seconds.

Decrease the default timeout values to further reduce the total number
of tracked timers.

Batch multiple expiring events that share the same deadline.

Inline the timer comparison code in the main event loop.

Avoid using an expensive comparison method in the heap sort by using
an integer primitive instead.

Use monotonic time instead of time.time()

Change-Id: I83e86bf203e6a641085e482c7ccf0e01f4fb4d86
2016-08-15 13:30:37 -04:00
Jenkins
cdd279b516 Merge "Remove the max_send_retries option" into feature/amqp-dispatch-router 2016-08-06 20:44:39 +00:00
Kenneth Giusti
480cd78157 [AMQP 1.0] Add link credit configuration options
Change-Id: Ifd483dfc67a4ff18f2047dbdca0aab311124627f
Closes-Bug: 1367911
2016-08-03 14:28:24 -04:00
Jenkins
a2a69c7205 Merge "Set the default link property to allow message acks" into feature/amqp-dispatch-router 2016-08-02 16:19:27 +00:00
Kenneth Giusti
5236e99155 Remove the max_send_retries option
This option was added for feature parity with the rabbitmq driver.
Since then the option was removed from rabbitmq.  Since this option
was never released we can simply remove it rather than deprecating it.

Change-Id: Ic10dd374c865eaf7c7563f379426e171996184a0
Closes-Bug: #1599518
2016-08-01 11:19:06 -04:00
Jenkins
7754698cf7 Merge "Fix a timer leak in the AMQP 1.0 driver" into feature/amqp-dispatch-router 2016-07-29 18:38:33 +00:00
Andrew Smith
b7a6a0715c [AMQP 1.0] Cancel response treatment for detached link
This patch removes an exception that occurs when an rpc reply is canceled
following the detachment of the link.

Change-Id: Ia743f571b293a75d851c0b19549f26673d9a734c
Closes-Bug: #1606557
2016-07-28 08:51:39 -04:00
Kenneth Giusti
3f38e6b03a Set the default link property to allow message acks
Previously the default snd-settle-mode caused the sender to assume the
message would not be acknowledged so the sender never waited for an
acknowledgement.  The 'mixed' setting allows the sender to request
that the receiver acknowledges the message.  The term 'mixed' comes
from the AMQP 1.0 standard.

Change-Id: Ida4e04f0827fff902d692f13ec972b099703adf0
2016-07-25 09:01:44 -04:00
Kenneth Giusti
7fe799d2e0 Fix a timer leak in the AMQP 1.0 driver
Change-Id: Ib89ac2f9e866bab7a62fbdd2b1e8cddce030e944
2016-07-22 14:48:06 -04:00
Kenneth Giusti
4c0674d194 Re-factor the AMQP 1.0 addressing semantics
This patch introduces a new routable addressing format.  This new
address format not only works with traditional broker-based messaging
backends but can be used to optimally route messages across a mesh
messaging topology.  This new addressing format may be enabled either
via configuration or by message bus identification.

This patch moves all of the AMQP 1.0 addressing logic into a new
Addresser class.  This class is used to map an oslo.messaging Target
address and notification flag into a corresponding AMQP 1.0
message/subscription address based on the addressing mode.  This hides
the addressing details from the rest of the driver.

For backward compatibility with previous releases the Addresser can be
configured to automatically detect a non-routable messaging back end
and fall back to using the legacy addressing scheme.

The intention is to use routable addressing as the only form of
addressing supported by the driver regardless of messaging back end.
However before that can happen the legacy addressing must follow the
standard deprecation process.  Until then legacy addressing will
remain available for backwards compatibility.

Change-Id: I6b60b6944a4c44da9b2e41ab2a83ab59f15e396e
2016-07-06 10:28:19 -04:00
Andrew Smith
35ea442a0e [AMQP 1.0] Add acknowledge and requeue handling for incoming message
This patch schedules a controller task to handle the acknowledge or
requeue disposition of the incoming message

Change-Id: I9a53959ce205417481f6442f712a8f238394bdae
Depends-On: Icf4dfbb0be1336e81a2815fe17dbe8428d3df619
Closes-Bug: #1593759
2016-07-01 09:58:36 -04:00
Kenneth Giusti
b1b4cfce8c Refactor AMQP 1.0 command task to support timers
Refactors the Task() classes to support per-send timers and
consolidate message sending logic.

Change-Id: Icf4dfbb0be1336e81a2815fe17dbe8428d3df619
2016-06-14 10:12:51 -04:00
Kenneth Giusti
890ce0e543 Refactor link management to support link recovery
This refactors the driver to process send/subscribe requests as they
arrive from the application rather than blocking these requests until
the reply link is activated.  This allows recoverable failures
occurring at the connection or link level to be applied to the sender
retry count.  This patch also implements link re-connect for
subscriptions.

These changes are required to support a brokerless routed mesh
backend.

Change-Id: I14606de55e2e94d03d865fefe8c8c2327207ba5c
Closes-Bug: #1434540
2016-06-13 11:15:13 -04:00
Andrew Smith
9d51fa4a2f [AMQP 1.0] Randomize host list connection attempts
Randomize the initial selection of the host list for amqp 1.0 connection
attempts.

This patch:
* randomizes initial index
* updates failover unit tests

Change-Id: Iab5ae684a2cd8ec0c6d0c59ffffde5eb5baf883b
2016-05-26 11:54:29 -04:00
Kenneth Giusti
e72a8d5bc4 Reorganize the AMQP 1.0 driver source files
This patch simply moves the existing AMQP 1.0 driver files into a
directory layout that is consistent with the other drivers.  The
driver unit tests are also moved to the proper test directory. There
are no changes in driver functionality or API.

Change-Id: I83a5d5433be8c3b317597100af69192ec5be81f1
Closes-Bug: #1579823
2016-05-13 14:04:37 -04:00