2500 Commits

Author SHA1 Message Date
Kenneth Giusti
0a784d2604 Bump amqp requirement version to >= 2.4.0
See http://lists.openstack.org/pipermail/openstack-discuss/2019-January/001998.html

Closes-Bug: #1813029

Change-Id: I4ef3d1184fd593f4acd090395579d91e093dd82c
2019-01-24 16:28:43 -05:00
Dirk Mueller
344ec5e8bb Avoid unnecessary use of items()
The key, value pair returned by items() wasn't used here,
so rewrite the logic to just iterate over the keys, filtered
by the type.

Change-Id: I7512ea93ac794d161a4cd9944546d6ca035a12cf
2019-01-07 14:07:24 +00:00
Zuul
13fa4f5ab1 Merge "Switch driver to confluent-kafka client library" 2018-12-07 19:59:50 +00:00
ZhongShengping
c047f431ea Update mailinglist from dev to discuss
openstack-dev was decomissioned this night in https://review.openstack.org/621258
Update openstack-dev to openstack-discuss

Change-Id: I7d889d3f9a6c95e707ed57dbf5a9cffb1d41ddf5
2018-12-05 09:39:30 +08:00
Andy Smith
5a842ae155 Switch driver to confluent-kafka client library
This patch switches the kafka python client from kafka-python to
confluent-kafka due to documented threading issues with the
kafka-python consumer and the recommendation to use multiplrocessing.
The confluent-kafka client leverages the high performance librdkafka
C client and is safe for multiple thread use.

This patch:
* switches to confluent-kafka library
* revises consumer and producer message operations
* utilizes event.tpool method for confluent-kafka blocking calls
* updates unit tests
* adds kafka specific timeouts for functional tests
* adds release note

Depends-On: Ice374dca539b8ed1b1965b75379bad5140121483
Change-Id: Idfb9fe3700d882c8285c6dc56b0620951178eba2
2018-12-04 11:25:07 -05:00
Andy Smith
252844879d Don't use monotonic with Python >=3.3
A change to the global-requirements has limited use of monotonic
library to Python versions earlier than 3.3 (later versions have
built-in support for a monotonic clock), so limit it in
requirements.txt.

Note: this patch updates kafka driver (due to deprecated exception
in library) in order to pass unit tests

Change-Id: Id6b0814e05a0e548a8c2a5359daf1a6878cf6859
2018-12-03 14:52:49 -05:00
Zuul
274b7c3eb4 Merge "Add a test for rabbit URLs lacking terminating '/'" 2018-11-19 21:55:24 +00:00
Kenneth Giusti
fb3ab00d1d Use ensure_connection to prevent loss of connection error logs
Change-Id: I31e3f83304c0780cfd380aa63ec2962c3c28ec78
Closes-Bug: #1801011
2018-11-05 16:02:06 -05:00
Zuul
1a781bd6dc Merge "Use default exchange for direct messaging" 2018-11-01 20:03:07 +00:00
Zuul
7b6e2e49e3 Merge "doc: Remove crud from conf.py file" 2018-10-23 10:23:04 +00:00
Kenneth Giusti
d913c179c7 Add a test for rabbit URLs lacking terminating '/'
Change-Id: I062ca84b6b8a5d1eeccafef87608c783ad5ffb03
2018-10-19 09:22:48 -04:00
Kenneth Giusti
92b4080654 Use '/' for the vhost if the transport_url has no trailing '/'
This change https://review.openstack.org/#/c/599912/ accidentally
removed setting the virtual host to '/' if none were provided in the
transport_url configuration option.

Closes-Bug: #1798241
Change-Id: I2dbcec3f77eb7cc513350b00ea020cc8faaecec1
2018-10-18 13:08:13 -04:00
Zuul
e0de93c793 Merge "Clean up .gitignore references to personal tools" 2018-10-16 21:24:57 +00:00
Hervé Beraud
0410bcbf54 Using pip as a python module
Since python 2.7.9 and python 3.4
pip is a stdin module

Change-Id: I17c8c93337fa4f0a33f9131d4117739179133ea9
2018-10-15 10:56:55 +02:00
Hervé Beraud
ad0d99c812 doc: Remove crud from conf.py file
Change-Id: I35c6221f78cac9fe4ab3e194e8a94ca4f0b24ca2
2018-10-15 09:42:18 +02:00
ZhongShengping
b868aebf45 Clean up .gitignore references to personal tools
Developers run all sorts of different tools within Git repositories,
any of which can leave their own special trashfiles all over the
place. We can't every hope to catalog them all, so better to
recommend developers simply configure a global core.excludesfile to
filter the irrelevant files which tend to get created by their
personal choice of tools.

To this end, remove the long-standing sections for "Mr Developer"
and "Editors" since their mere existence here sends the signal that
we welcome (and have time to review) additions for any old tool
someone ever might happen to try. Also add a comment block
explaining this, for clarity.

We can, and should of course, continue to list files created by the
tools recommended by our workflow (test frameworks called from tox,
documentation and packaging builds, et cetera).

This change is a port of I1b41efac219fca44e2548fc36633724d0ecfc0cb
from the openstack-dev/oslo-cookiecutter repository.

Change-Id: I4078179f3d55847ea23f8790572ba911e52fce53
2018-10-15 11:49:38 +08:00
Kenneth Giusti
9ca7359bfd Allow transport_url initialization in ConfFixture constructor
Change-Id: Ibe2a5277cdcf89cd006507f13eddbbec4b384681
2018-10-09 13:46:00 -04:00
Zuul
16fea5b368 Merge "Fix oslo.messaging default transport" 2018-10-05 22:48:55 +00:00
Mehdi Abaakouk
172cfb33f3 Fix oslo.messaging default transport
Change d3f6ca0b47e2a5a3413b27617d9655f02b191c6a have broken
oslo.messaging defaults, no more transport are selected by default and
oslo.messaging just crash:

    Traceback (most recent call last):
      File "/home/sileht/workspace/openstack/oslo.messaging/.tox/py27/lib/python2.7/site-packages/mock/mock.py", line 1305, in patched
        return func(*args, **keywargs)
      File "oslo_messaging/tests/test_transport.py", line 111, in test_get_transport
        transport_ = oslo_messaging.get_transport(self.conf, **kwargs)
      File "/home/sileht/workspace/openstack/oslo.messaging/.tox/py27/lib/python2.7/site-packages/debtcollector/removals.py", line 242, in wrapper
        return f(*args, **kwargs)
      File "oslo_messaging/transport.py", line 247, in get_transport
        transport_cls=RPCTransport)
      File "oslo_messaging/transport.py", line 205, in _get_transport
        url.transport.split('+')[0],
    AttributeError: 'NoneType' object has no attribute 'split'

This change restores rabbit:// as default.

Change-Id: I945a2aca60347dad672cc406cd33e80e93c588a7
2018-10-05 15:21:45 +02:00
Hervé Beraud
e9ed47f416 always build universal wheels
All of our projects are pure python, so we can build universal wheels by default.

Change-Id: I3369303d278af6154cdcf06b727e8ff941dc410a
2018-10-04 19:56:43 +02:00
Oleg Bondarev
3a5de89dd6 Use default exchange for direct messaging
Lots of exchanges create problems during failover under high
load. Please see bug report for details.

This is step 1 in the process: only using default exchange
when publishing. Consumers will still consume on separate
exchanges (and on default exchange by default) so this
should be (and tested to be) a non-breaking and
upgrade-friendly change.

Step 2 is to update consumers to only listen on default exchange,
to happen in T release.

Change-Id: Id3603f4b7e1274b616d76e1c0c009d2ab7f6efb6
Closes-Bug: #1789177
2018-09-25 07:59:28 +00:00
Steve Kowalik
95b487ba6b Refactor GetTransportSadPathTestCase
GetTransportSadPathTestCase did an awful lot of mocking for things that
were no longer called. Since we only have one callable now, make use of
self.assertRaises.

Change-Id: Ie93ff74de84ff4d6cc75a3c89dd9a03fdcba5d35
2018-09-25 14:18:57 +10:00
Zuul
815e25e55b Merge "Add release note about deprecated option removals" 2018-09-24 16:49:12 +00:00
Zuul
34519efd2d Merge "Remove rpc_backend and ConfFixture.transport_driver" 2018-09-24 16:49:11 +00:00
Zuul
4eca2a29b0 Merge "Use templates for cover and lower-constraints" 2018-09-24 16:28:50 +00:00
Steve Kowalik
58fb5b0675 Add release note about deprecated option removals
Since we have removed a large amount of deprecated options, add a
release note documenting which options have been removed.

Depends-On: https://review.openstack.org/580910
Change-Id: Idb45f691dab77e5bfae841ea45fa167c224c3322
2018-09-14 11:48:28 -06:00
Zuul
ede61c360d Merge "Remove deprecated amqp1 options" 2018-09-14 17:46:26 +00:00
Zuul
14c5bed497 Merge "Remove default_{host,port} deprecated options" 2018-09-14 16:00:45 +00:00
Steve Kowalik
d3f6ca0b47 Remove rpc_backend and ConfFixture.transport_driver
Now that most projects have transitioned from using
ConfFixture.transport_driver, we can remove it and rpc_backend as a
whole. rpc_backend was first deprecated in 5.10.0, released during
Ocata.

Closes-Bug: #1712399
Change-Id: I193cc0e613459a6dbbfd54ed0901a54ded78d712
2018-09-13 15:01:27 -06:00
Steve Kowalik
b0d3bfceb8 Remove deprecated rabbit options
Remove the deprecated options rabbit_host, rabbit_port, rabbit_hosts,
rabbit_userid, rabbit_password, rabbit_virtual_host and
rabbit_max_retries which were deprecated in 5.10.0, released during
Ocata.

Change-Id: I39dec568e5de0b653e5af1f196537e09ef126a36
Closes-Bug: #1712394
2018-09-10 15:11:19 -06:00
Andreas Jaeger
3d7ad9cab4 Use templates for cover and lower-constraints
Use openstack-tox-cover template, this runs the cover job as
non-voting in the check queue only.

Use openstack-lower-constraints-jobs template

Remove jobs that are part of the templates.

Sort list of templates, move templates first in project stanza as is
common practice.

Change-Id: Idb976b160479efcf9f3726c1716bf9d2c39ab6cc
2018-09-07 12:18:29 +02:00
Steve Kowalik
8881ed4378 Remove deprecated amqp1 options
The amqp1 options allow_insecure_options, username and password are all
deprecated, remove them. allow_insecure_options was deprecated during
Ocata, and the other two were deprecated in 5.22.0, released during
Pike.

Change-Id: I321266c3dba83a38fe6ebd775db2e74c14cc96cd
2018-09-05 16:59:13 +10:00
Steve Kowalik
3b1b08b74a Remove rabbit_durable_queues deprecated option
The rabbit_durable_queues option has been deprecated since amqp code was
moved from olso-incubator to this project, so it's high time it was
removed.

Change-Id: If2450696a43c05c32d35bff26d3dc38423f4330e
2018-09-05 15:35:20 +10:00
Steve Kowalik
4cc1264a18 Remove default_{host,port} deprecated options
The Kafka driver deprecated the kafka_default_host and
kafka_default_port options in 5.10.0, released in Ocata. Remove them.

Change-Id: I206e68ec1624bb6d5d6ba320572530352bbd4378
2018-09-05 14:44:54 +10:00
Zuul
90a5bd5077 Merge "Remove the deprecated ZeroMQ driver" 2018-08-29 14:08:20 +00:00
Zuul
9ab5a2e30e Merge "Fix the coverage tox tests" 2018-08-29 11:57:24 +00:00
Andy Smith
f2fd8d4375 Remove the deprecated ZeroMQ driver
Users of the oslo.messaging RPC communications service must use
the rabbit ("rabbit://...") or AMQP 1.0 ("amqp://...") drivers.

Change-Id: If3474142f1fe99d41d7b4466061ed0e23ca38549
Closes-Bug: 1789259
2018-08-28 16:00:50 -04:00
Kenneth Giusti
d8b0d48055 Fix the coverage tox tests
Change-Id: I45f7269c95807e7a804f116c2d1aed6baf9f46e2
2018-08-20 15:20:47 -04:00
Kenneth Giusti
d9866029a2 Avoid logging passwords on connection events
Change-Id: I15c8c4a1177c363283281d2fed63545658eda5de
Closes-Bug: #1787214
2018-08-20 12:31:16 -04:00
Doug Hellmann
a4a8eef010 add lib-forward-testing-python3 test job
This is a mechanically generated patch to add a functional test job
running under Python 3 as part of the python3-first goal.

See the python3-first goal document for details:
https://governance.openstack.org/tc/goals/stein/python3-first.html

Change-Id: Ib99a323861c0fdb548287446ab25e0516a7c182e
Story: #2002586
Task: #24322
2018-08-11 18:20:14 -04:00
Doug Hellmann
deec8c1f13 add python 3.6 unit test job
This is a mechanically generated patch to add a unit test job running
under Python 3.6 as part of the python3-first goal.

See the python3-first goal document for details:
https://governance.openstack.org/tc/goals/stein/python3-first.html

Change-Id: I1da69d0b77a3970c727b60de4c341552970fbb47
Story: #2002586
Task: #24322
2018-08-07 16:17:34 -04:00
Doug Hellmann
818fd688a2 import zuul job settings from project-config
This is a mechanically generated patch to complete step 1 of moving
the zuul job settings out of project-config and into each project
repository.

Because there will be a separate patch on each branch, the branch
specifiers for branch-specific jobs have been removed.

See the python3-first goal document for details:
https://governance.openstack.org/tc/goals/stein/python3-first.html

Change-Id: I0401a53945ad87e1dc046904054b12e2b106a3e8
Story: #2003250
2018-08-03 17:45:00 -04:00
Andras Kovi
d8fa6d5edf Call listener stop only if listener is initialized
Change-Id: I1303d74af7a242edefe774d8f5e12e63614c0158
Closes-Bug: #1785048
2018-08-02 13:28:42 +02:00
Zuul
3a725f957b Merge "Update reno for stable/rocky" 2018-08-01 20:57:30 +00:00
Zuul
01a37733eb Merge "Issue blocking ACK for RPC requests from the consumer thread" 2018-07-30 15:48:21 +00:00
59c47233f5 Update reno for stable/rocky
Change-Id: I1a8834b95487805805395e919e2c6cdc435522b0
2018-07-26 12:38:43 +00:00
Dirk Mueller
e233334a9e Remove setting of DEVSTACK_GATE_EXERCISES
This defaults to 0 anyway, and is being removed as the devstack
excercises are being removed. see https://review.openstack.org/#/c/581377/

Change-Id: Ifb147f9fb2071724a9eaf306273276db65125cf7
2018-07-17 10:46:55 +02:00
Daniel Alvarez
7dc7684311 Bump py-amqp to >= 2.3.0
Due to the bug below we want to have py-amqp>=2.3.0 so that connecting
to rabbit endpoints don't timeout when relying on /etc/hosts entries.
Closes-Bug: 1780992

Change-Id: Ia2c1114cb0f8d480695197714f7c6293751430dd
2018-07-11 15:22:21 +02:00
Kenneth Giusti
26b0be585a Issue blocking ACK for RPC requests from the consumer thread
The patch for https://review.openstack.org/#/c/436958/ fixed a
threading problem by moving the ack back to the polling
thread. However the RPC server expects to catch any failures of the
ACK and abort the request.  This patch adds the ACK error handling
back to the polling thread.

This patch is based heavily off the original work done by Mehdi
Abaakouk (sileht).

Change-Id: I708c3d6676b974d8daac6817c15f596cdf35817b
Closes-Bug: #1695746
2018-07-11 08:59:35 -04:00
Zuul
b97481ce19 Merge "No longer allow redundant calls to server start()" 2018-07-10 10:24:58 +00:00