44 Commits

Author SHA1 Message Date
Oleksii Zamiatin
f61f0c1c1b [zmq] Redis unavailability is not critical
Each time driver couldn't update from Redis
warning message is being logged and work continues
with cached targets state and on existing connections.

Change-Id: I459532f1f60fab1d4c9926e02cdc3c3c6a162047
Closes-Bug: #1610958
2016-08-16 11:29:59 +03:00
Julien Danjou
ff9b4bb938 notify: add a CLI tool to manually send notifications
Change-Id: I01ebd8402e322c5c27dc2fc4c61fb5beb2dddae8
2016-07-13 17:30:56 +02:00
ChangBo Guo(gcb)
8e778658e5 Add Python 3.5 classifier and venv
Now that there is a passing gate job, we can claim support for
Python 3.5 in the classifier. This patch also adds the convenience
py35 venv.

Change-Id: I8871f4e17ab806c00661d0c960a216d0a3fc5449
2016-07-12 16:35:36 +08:00
Kenneth Giusti
e72a8d5bc4 Reorganize the AMQP 1.0 driver source files
This patch simply moves the existing AMQP 1.0 driver files into a
directory layout that is consistent with the other drivers.  The
driver unit tests are also moved to the proper test directory. There
are no changes in driver functionality or API.

Change-Id: I83a5d5433be8c3b317597100af69192ec5be81f1
Closes-Bug: #1579823
2016-05-13 14:04:37 -04:00
Dmitriy Ukhlov
63de855fef Implements configurable connection factory
new - create new connection each times
single - use single connection for whole transport
read_write - use two connections for whole transport
                    (one for listening and one for sending)

Change-Id: I464c83beb498453b6df2237e7b8022d47ca3fa14
2016-05-12 11:51:26 +03:00
Ronald Bradford
0a7d7f86e2 Remove Beta development status from classifiers
Remove the Beta trove classifier to be in line with other Oslo
projects. While there is a Production/Stable status none of the other
Oslo projects use it.

https://pypi.python.org/pypi?%3Aaction=list_classifiers

Change-Id: I3689c6faf60be49ce2bf19f45e977dd64798954b
2016-05-02 13:43:49 -04:00
Oleksii Zamiatin
cc1cb30321 [zmq] Reduce number of connections
In this change 'use_router_proxy' option was added
to switch between direct connections and proxy.

Proxy was reimplemented and splitted onto two
types of proxies:

    * PUBLISHER proxy for fanout pattern
    * ROUTER proxy for direct messaging

Each type of proxy is configured over command line
argument --type.

Deployment guide was updated accordingly to the change.

Change-Id: If36e9c26e2a8ebe622cfa7e9f2a07b1a69aabe34
Closes-Bug: #1555007
2016-03-31 15:40:36 +03:00
Davanum Srinivas
5e0bda3a46 Claim python3 compatability for Newton onwards
The following oslo.messaging changes make sure python3 works:
Id630cb32dc5fe29774ae9e0d2766535906f605bf
I5cd7475618f7a7532db770c2c5c61576a657fbb4
I1f8bb964aef23867a651e192dc355635e36f78a1

This one adds a functional test for py34:
Ibc695762fb12b60a8094be187001dd2bd42f402e

Which is enabled in the gate:
Ifdbd4ab1eae6c5e5364a99906995dff3bb1ff4f0

The following re-enables voting for py34 unit tests:
Ibbcd20d297fdf90ce846f0162329b9410cd6f482

With all the above, we are ready to claim python3 support

Change-Id: Ie89f96cee0eadcad5f84dd0423645b894eec9116
2016-03-24 10:22:59 +00:00
dukhlov
1482687ff7 Move server's logic from executors
Now we have situation when openstack projects like Mistral needs
extra oslo.messaging functionality.

 But it is too complicated now to to implement something new and
 integrate it with current code because there is a little bit mess.
 1) Executor should be responsible for how to run jobs
      (but now also has code with server logic)
 2) Dispatcher should be responsible for routing message to the
     target endpoint for processing (but it also has serialisation, sending replies,
     executing some executor's callbacks etc)
 3) Server should do all server specific logic, we need to have different
      implementation of servers for RPC and notification, not different implementations
      of dispatchers

 This patch fixes 1-st point

Change-Id: Ib6408f408889bb7b7056722be636a5547b1a780d
2016-02-23 15:00:58 +00:00
Victor Stinner
d260744773 Remove aioeventlet executor
I wrote the aioeventlet executor as a first step to replace eventlet
with trollius in OpenStack. Sadly, the project of replacing eventlet
with something else (trollius, threads or whatever) looks to be stuck
because of various reasons. See the spec:

    "Replace eventlet + monkey-patching with ??"
    https://review.openstack.org/#/c/164035/

Recently, I deprecated the trollius project in favor of asyncio:

    http://trollius.readthedocs.org/deprecated.html

Since I abandonned my project to replace eventlet with trollius, I
now propose to remove the unused aioeventlet executor from Oslo
Messaging to simplify the code and remove unused dependencies
(especially trollius).

Since the aioeventlet executor is not used by any application, it's
safe to remove it. Removing the executor removes trollius and
aioeventlet dependencies.

... The executor may come back "later" when the first OpenStack
service will start to drop Python 2 support.

For more information on asyncio in OpenStack, see:
http://aioeventlet.readthedocs.org/openstack.html

Note: the executor was added by the change
I7a78ed998719a703077232726f66d882463b1297.

Change-Id: I10d8d7d4134b32e4fc0badfa15e9ffb005910139
2016-02-17 11:27:15 +01:00
Andreas Jaeger
e32560e9eb Update translation setup
Follow new infra setup for translations, see spec
http://specs.openstack.org/openstack-infra/infra-specs/specs/translation_setup.html
for full details.

This basically renames
oslo.messaging/locale/oslo.messaging.pot to
oslo_messaging/locale/oslo_messaging.pot. For this we need to update
setup.cfg.

Update also domain name in i18n.py.

Change-Id: Idf5300224f572d29953261ac998e7bf32a12822d
2016-01-29 10:39:35 +00:00
Davanum Srinivas
cc0f8cc8a9 Merge remote-tracking branch 'origin/master' into merge-branch
Change-Id: Ice6dcc3867dff6b7fea5647a3b79d1c765bf9d73
2015-12-14 16:21:42 +02:00
Dmitriy Ukhlov
bee303cf6f Adds comment for pika_pooler.py
Also this patch:
1) fixed import's order
2) removes PikaDriverCompatibleWithOldRabbit
     (tests, show that after retry implementation all works fine)
Change-Id: Ib34f6db569cadb5c27d8865f13ba32ef9a6c73e9
2015-12-09 19:32:22 +02:00
Komei Shimamura
67c63031f5 Add a driver for Apache Kafka
Adding a driver for Apache Kafka connection, supporting
notification via Kafka. This driver is experimental
until having functional and integration tests

Change-Id: I7a5d8e3259b21d5e29ed3b795d04952e1d13ad08
Implements: blueprint adding-kafka-support
2015-12-01 14:20:33 +00:00
Flavio Percoco
925eb734a9 Remove qpidd's driver from the tree
Back in liberty we marked this driver as deprecated. This patch removes
it from the tree. The patch also removes tests, options and other
references in the documentation. Note that one script is being kept
because it's required by the amqp driver.

Depends-On: If4b1773334e424d1f4a4e112bd1f10aca62682a9
Change-Id: I4a9cba314c4a2f24307504fa7b5427424268b114
2015-11-20 18:43:28 +00:00
Dmitriy Ukhlov
ad2f475955 Implements rabbit-pika driver
In this patch new driver implementation added and registered
in setup.cfg, integrated ith tox functional tests.

Implements: bp rabbit-pika

Depends-On: I7bda78820e657b1e97bf888d4065a917eb317cfb
Change-Id: I40842a03ce73d171644c362e3abfca2990aca58a
2015-10-20 15:04:36 +03:00
Jenkins
3a5db723aa Merge "Non-blocking outgoing queue was implemented" 2015-09-30 16:11:29 +00:00
Oleksii Zamiatin
3067dbd198 Non-blocking outgoing queue was implemented
The main issue with eventlet.green.zmq is that libzmq as a C-library
is completely monkey-patch unfriendly. So any blocking call inside
the native library makes calling process stuck. We can't avoid this
actually in an absolutely normal situation when a client appears
earlier than listener we have all client process get stuck until listener
raised. If the listener for example is also blocked awaiting for some
other service to appear we have a chain of locks which may occasionally
result in a dead-lock. The other situation with Notifier is quite similar.

For that reason zmq-broker was restored, but now it serves as an outgoing
queue on a client side. Servers remained the same dynamically port-binded.
Now all clients can still use green-zmq, but presence of the broker-queue
on a host guarantees that green threads will never blocked in a client
because all messages will wait their listeners inside the broker queue.
The broker process's modules are not monkey-patched, they make use of native
threading and native zmq.

Possibility to run without broker also remains. The option zmq_use_broker
introduced for that reason.

Closes-Bug: #1497315

Change-Id: I786b100fd6ee1cf4b99139db0ca044d358d36345
2015-09-28 14:14:53 +03:00
Jenkins
8b52367c53 Merge "Fix the home-page value with Oslo wikipage" 2015-09-25 22:59:29 +00:00
Davanum Srinivas
1893c495f6 Allow custom notification drivers
Our class hierarchy hides classes and modules that so its hard
for folks to write a custom Notification driver. We should
make these public and document them

Closes-Bug: #1426046
Change-Id: Ifb96c2ae9868426cac2700bf4917c27c02c90b15
2015-09-25 10:07:05 -04:00
venkatamahesh
0a4c321ee0 Fix the home-page value with Oslo wikipage
Change-Id: I4b72377329e055096d8636f4734c9272ca8ec3a8
2015-09-25 14:44:50 +05:30
Davanum Srinivas
97892e656a Merge remote-tracking branch 'origin/feature/zmq' into merge-branch
Change-Id: If189d03131efc02045955508cef06fdd2ed590ee
2015-09-15 11:07:44 -04:00
Doug Hellmann
c90525bfea Remove oslo namespace package
Blueprint remove-namespace-packages

Cherry-picked from: 03265410e0294e176d18dd42b57268a8056eb8fc

Change-Id: Ibaba19ef10b4902c4f4f9fbdf7078e66b75f2035
2015-07-29 11:17:57 +03:00
Doug Hellmann
03265410e0 Remove oslo namespace package
Blueprint remove-namespace-packages

Depends-on: I2eeef93ee2e61a721c69f62add819f93f62f077d
for openstack/ceilometer
Depends-on: I26390dd908769be5f1a5b70be22d3b98e3a45563
for openstack/ceilometermiddleware
Depends-on: Ifa8baab33cdb3e606cf175a8c29c3a4ef6c44480
for openstack/glance
Depends-on: I029c3260051aa48cfaae428c096c1ac7b43b2bd2
for openstack/ceilometermiddleware

Change-Id: I8c5595bbafa82db33f62fa47d214f5cb756a2639
2015-07-16 17:47:16 +00:00
Victor Sergeyev
9e4831c022 ZMQ: Initial matchmaker implementation
This patch replaces the old outdated matchmakers and replace it into the
new ones.

Call/Cast test_specific_server() functional tests passes now.

Change-Id: I8635396110d30d26812f39b242fbbabd1a0feaaa
2015-07-16 09:23:46 +00:00
Joshua Harlow
7c2aa0ced0 Remove 2.6 classifier
This removes the 2.6 classifier so that support for 2.6
is not noted (when building packages, on pypi...) as happening
anymore for oslo.messaging (since check/gating support for
it has been turned off).

Depends-On: I4013bf38a197c9a9d82b3956ddbd25450d913df9

Change-Id: Ic4150332fff0724e5178e14f91ac54d5a80d408b
2015-07-06 12:02:54 +03:00
Li Ma
53fde06c44 Add pluggability for matchmakers
Closes-Bug: #1291701

Change-Id: I2a10eca0760ef84f34e5dcdb8132845b88cd4b98
2015-03-19 03:03:53 -07:00
Victor Stinner
d8e68c365b Add a new aioeventlet executor
The new executor supports trollius coroutines, explicit asynchronous
programming, in addition to eventlet greenthreads, implicit asynchronous
programming.

The new AsyncioEventletExecutor class is based on the EventletExecutor
class and so it is compatible with it. The aioeventlet executor can be
used to replace the eventlet executor, but it requires an aioeventlet
event loop running in the thread running the executor (usually the main
thread). See AsyncioEventletExecutor docstring for an example how to
setup such event loop.

The aioeventlet module implements the asyncio API (PEP 3156) on top of
eventlet, see aioeventlet documentation:
http://aioeventlet.readthedocs.org/

The change adds an unit test with an endpoint implemented as a trollius
coroutine.

The executor is not supported on Python 3 yet because of eventlet issues
with monkey patching.

Implements: blueprint greenio-executor
Change-Id: I7a78ed998719a703077232726f66d882463b1297
2015-02-05 12:11:37 +01:00
Joshua Harlow
14752468c6 Add the threading executor setup.cfg entrypoint
It seems like this was forgotten about; add it so
that people can more easily use this executor and
use it.

Change-Id: I4590bb7f224569d73683c640dd46bde48f8003c3
2015-02-02 19:29:15 -08:00
Doug Hellmann
bf4ab5a9e4 Treat sphinx warnings as errors
Closes-Bug: #1287415

Change-Id: I43f406a8608724ac55c9ae6aea847d75594ba6f6
2015-01-29 14:24:55 -05:00
Dan Prince
0ca1b1e6a1 Make setup.cfg packages include oslo.messaging
This properly deploys the oslo.messaging package may resolve
sphinx build errors for projects which have not yet
upgraded to oslo_messaging.

Change-Id: I4db750fb2356ebf44a8fccf7c422b474fefec0ee
2015-01-15 08:47:02 -05:00
Doug Hellmann
e55a83e832 Move files out of the namespace package
Move the public API out of oslo.messaging to oslo_messaging. Retain
the ability to import from the old namespace package for backwards
compatibility for this release cycle.

bp/drop-namespace-packages

Co-authored-by: Mehdi Abaakouk <mehdi.abaakouk@enovance.com>
Change-Id: Ia562010c152a214f1c0fed767c82022c7c2c52e7
2015-01-12 12:50:41 -05:00
Mehdi Abaakouk
d753b030c6 Ensure the amqp options are present in config file
This change ensures that the amqp1 will be present into the
configfile even the required python module are not present and
also put them into the correct section, [oslo_messaging_amqp1].

Change-Id: I1005405d7ed51090495688eadbe400dbff7c3cc9
2014-09-09 11:57:12 +02:00
Gordon Sim
fbee94170f An initial implementation of an AMQP 1.0 based messaging driver
The key driver interfaces are implemented in the ProtonDriver class in
driver.py.  The logic for interfacing with Pyngus in order to
send/receive messages, manage AMQP connections and links, and handle
protocol events is in controller.py.  eventloop.py is a fairly generic
socket connection and I/O processor which runs in its own thread.
controller.py uses the eventloop.py thread to schedule subscription
and message send requests from the driver, as well as handle all
protocol event callbacks coming from Pyngus.

Included in this patch are a set of functional tests that can be run
under tox (tox -eamqp1).  These tests fully exercise the new driver,
from the driver API down to the 'wire' - nothing in the driver is
mocked out.  The functional tests implement a simple loopback test
broker, which allows the driver to send and receive messages via the
local network.  All RPC call patterns, RPC timeouts, and even broker
failover are verified by the included functional tests.

This driver uses the Pyngus module, which is a pure-python client API
built on the Proton AMQP 1.0 protocol engine library from the Apache
Qpid project.  Pyngus is available via pypi.python.org.

This driver introduces a dependency on the Proton AMQP 1.0 protocol
library, which is a platform-dependent library that must be installed
in order to use this driver and run the functional tests.

Change-Id: I871703e4cdc04cee3e6c214e911c9df464ede2ed
Implements: blueprint amqp10-driver-implementation
2014-09-04 15:12:43 -04:00
Andreas Jaeger
0630c0577b Setup for translation
To start translation, we need to initially import the
translation file - and place it at the proper place so that
the usual CI scripts can handle it.

The proper place is for all python projects
$PROJECT/locale/$PROJECT.pot - see setup.cfg.

Further imports will be done by the OpenStack Proposal bot.

Setup also setup.cfg with the usual babel commands and add the default
babel.cfg file.

Change-Id: I583e98fac2f9b84cd1115640b936057ad95957e8
2014-06-05 22:48:44 +02:00
Jenkins
f81cde600b Merge "Expose an entry point to list all config options" 2014-01-28 06:17:56 +00:00
Mark McLoughlin
0eb5a6f6c7 Expose an entry point to list all config options
Register a oslo.messaging entry point in the oslo.config.opts namespace
which, when called, returns a list of the configuration options which
may be registered by the library at runtime.

The idea here is that the sample config file generator can query this
and include the returned options in the sample config file of any
applications which use the library.

Some sample code for fetching available options:

  import pkg_resources

  for ep in pkg_resources.iter_entry_points('oslo.config.opts'):
      if ep.name == "oslo.messaging":
          list_opts = ep.load()
          for g, opts in list_opts():
              opts.sort(key=lambda x: x.name)
              print "[%s]" % g
              for o in opts:
                  print "%s = %s" % (o.name, o.default)
              print

Related-Bug: #1241566
Change-Id: Ic28351258652d2ea38974e2f4d1aa6b1d3dd7192
2014-01-27 11:24:52 +00:00
Sandy Walsh
30c9334306 Routing notifier
Takes a yaml-based config file
(see etc/oslo/routing_notifier.yaml.sample) via
routing_notifier_config option.

Events may be routed by priority or event_type.

Implements: blueprint configurable-notification
Change-Id: I437dfac348f387044e6da3d6a0bbb208323c1741
2014-01-15 04:08:18 +00:00
Monty Taylor
a1b57a5452 Remove cruft from setup.cfg
packages is recursive, so there is no need to list subpackages
versions are set by tags, so the version in the file is not needed
non-d2to1-based pbr does not need the hook specification

Change-Id: Id5e6c19dfe81c630862e9b87b7f9e5f67a965945
2013-10-04 10:07:58 -04:00
Mark McLoughlin
a2d113198c Add zmq-receiver
This is the ZeroMQ server which acts as a proxy for all messages
destined to a particular host. Again, there are a bunch of FIXMEs
here. This still needs work.

Change-Id: I9384f486e44b0b0cbca028e219ad66f1990d5181
2013-08-28 12:25:10 +01:00
Mark McLoughlin
5aa7c37144 Ensure namespace package is installed
This is apparently fixed in pbr since I3972b3132619e8e2dd7e362ca5fe9d1e3add43b8
but I'm seeing the issue with pbr 0.5.21 on Fedora.

Related-Bug: #1194742
Change-Id: I136b8493c8d8d48a0116facf5f23c2a1479c070f
2013-08-12 14:35:33 +01:00
Mark McLoughlin
d0bb030d59 Fix notify drivers namespace 2013-06-24 12:52:41 +01:00
Mark McLoughlin
e03c2d4e9a Remove backwards compat entry point aliases
There's no point in supporting

  oslo.rpc.impl_kombu

since what we need to support is

  nova.openstack.common.rpc.impl_kombu
2013-06-24 12:52:31 +01:00
Mark McLoughlin
b5c41bba00 Add oslo.messaging project infrastructure 2013-06-15 08:43:50 +01:00