This producer singleton works only if one transport use kafka.
If multiple transports with kafka are used, their overrides each other
the producer on each send.
This change creates only one producer per connection.
A later optimisation can be one producer per driver instance.
Change-Id: I429f7c5efcb41690dd1b17f856bda2425c788e53
Currently Kafka driver for an oslo.messaging uses kafka-python==0.9.5
and mostly broken. This package version supports only low level Kafka
producer and consumer API which are marked as deprecated now [1]. Using
of these interfaces bring a big concern to the message processing,
because current KafkaConsumer has not any consuming coordination. This
fact causes a message duplication for the several consumers of one
topic. This behavior is specific to Ceilometer and services which read
and process notifications from other services.
New version of kafka-python allows to use async thread safe message
producers and coordinated consumers [1].
[1] http://kafka-python.readthedocs.io/en/master/changelog.html#feb-15-2016
The driver is currently experimental, python-kafka<1.0.0 API have major
issue described above that can't make the oslo.messaging driver works,
so we prefer having a working driver with a non-synced dependencies, that the
reverse.
Co-Authored-By: Mehdi Abaakouk <sileht@redhat.com>
Change-Id: I29862ed7bf56b9d8878fa8e9fb1cbd9d643908a4
The current zeromq target name are too long and reach the limitation
of 127 chars of the sheband of the virtualenv binary.
A target name must not be more than 22 chars to run in gate with
the tox job template.
This renames them to a shorter one.
It also creates aliases for the legacy jobs that we will remove
when the tox jobs are ready.
Change-Id: I5cbc368e7ea252f8c0c949c8e7ee3a70a560db22
Restore static direct connections which may be useful
for services running RPC on controllers only and talking
to agents via other means (ssh for example).
Change-Id: Icbe45978cb4a8ba5db74e5593ea7be23cba3f44e
The kafka functional tests don't use our setup framework but hardcode
the kafka url.
The kafka driver is broken and currently try to be loaded with other
driver tests (and break py3 gate job).
This change remove the hardcoded stuff and uses the normal functional
tests setup to check kafka.
This avoids kafka tests to be ran on other driver tests job.
It also creates the tox target for functional tests and use
pifpaf to setup kafka.
As today, the functional test suite doesn't work all.
Change-Id: Ie6af35321905a04b38e549bf7268dd40543d9e3a
Use zmq_target_expire rather than zmq_target_update
as it supposed to represent longer period of time so
all services will definitely be updated.
Change-Id: I0b9a1c8cbc7fea9b2c0c543a270dad318d135c1f
Leverage the fact that there is never more than one socket active to
simplify the event loop's I/O processing. This removes the need to
look up active connections. Also removes the need to query the return
values from select since there is only one socket to process. This
change improves RPC throughput under the simulator by up to 20%
As part of this change I've clarified the code's use of the pyngus
connection by renaming the connection property to something a bit more
specific.
Change-Id: If7c020bb0bd96490af78bc06659db10073b02417
When transport url is set to something like "amqp://hostname:/"
it currently raise ValueError while any url parsing library
(like urlparse) just ignore the port, and don't fail since this is a
value url.
This change make the transport url ignore empty port instead of raising
ValueError.
Change-Id: I04df76012e3b133eddbcf365363cd261d277e369
oslo.messaging doesn't ensure ordering of notification. By chance
that works for some driver/setup. But for example zmq job often
fail this one.
This changes removes this assumption.
That fixes the zmq gate job
Change-Id: I2087f0a219a4a5b8fb30bb8ed84e1a72f8d9e0ab
