70 Commits

Author SHA1 Message Date
Kenneth Giusti
0608fe2d5b update configuration for qdrouter v1.0.0
Change-Id: I20637c51d5ae4a96f1de9ea6e2a50891adaca52a
2018-03-20 16:42:32 -04:00
Andrew Smith
3afc3a0a1d Update kafka functional test
This patch addresses a number of issues that prevented the functional
tests from running. The functional tests now execute and can complete
succesfully. At times, the test will fail (noticiably in CI) indicating
an underlying issue with consumer interaction with the kafka server.

It would be beneficial to merge this patch as it provides repeatability
and visibility for driver-kafka server integration to facilitate
additional debugging and testing.

This patch:

* removes use of deprecated get_transport
* override consumer_group for each test
* changed to synchronous send
* update to kafka 1.0.0 server

Depends-On: Ib552152e841a9fc0bffdcb7c3f7bc75613d0ed62
Change-Id: I7009a3b96ee250c177c10f5121eb73d908747a52
2017-12-16 14:41:03 -05:00
Zuul
63775d9e97 Merge "Provide bindep_profile in openstack-tox job setup" 2017-12-11 09:39:31 +00:00
Andreas Jaeger
d68b53f056 Avoid tox_install.sh for constraints support
We do not need tox_install.sh, pip can handle constraints itself
and install the project correctly. Thus update tox.ini and remove
the now obsolete tools/tox_install.sh file.

This follows https://review.openstack.org/#/c/508061 to remove
tools/tox_install.sh.

Change-Id: Ifaaf656effff20ef08214f111645a3b5fc8b4d28
2017-12-02 17:01:55 +00:00
Andrew Smith
0ca9567c4a Provide bindep_profile in openstack-tox job setup
Oslo.messaging uses a single bindep.txt to define common
and driver dependencies. The bindep_profile is used by the
bindep role to select the driver specific packages.

This patch:
* Add bindep_profile var to jobs
* Add kafka dependencies
* update test-setup for use by amqp1

Depends-On: I6beb7b07a7dbc345e63e758ebbc86400e0ee16d3
Change-Id: Ib552152e841a9fc0bffdcb7c3f7bc75613d0ed62
2017-11-29 11:10:35 -05:00
Kenneth Giusti
e1a3da9443 Move legacy zuulv3 tests into oslo.messaging repo
Next step will be to re-write these for zuulv3

Change-Id: I2f6a8432e037f75d6abee3bc977933a310a4a2c7
2017-10-17 16:01:29 -04:00
gord chung
631ade59bc remove all kombu<4.0.0 workarounds
we lower cap at kombu>=4.0.0 so no need to support kombu3 stuff

Change-Id: I4c3bc4de6026b089fffae7ae4d34f718d45cb972
2017-03-30 12:32:58 +00:00
Kenneth Giusti
7ebf500ec5 Fix simulator's use of Notifier - use 'topics' not 'topic'
Change-Id: Id315418fcf517734b8136f8177c82951658fe794
Closes-Bug: 1673890
2017-03-20 11:25:17 -04:00
gord chung
5bacea1f42 support kombu4
- kombu4 wraps recoverable errors as OperationalErrors rather than
raising amqp errors
- also, raise a recoverable error and redeclare if for some reason a
message is double acknowledged... previously, this was hidden.
- ensure socket is not none
- use connect method to ensure connection

Depends-On: I9f980b51901ac31599b9651633956ad2eea6a1ac
Change-Id: I73958c8057353a2eefe1baaa7a41148193d507f7
2017-02-10 13:21:22 +00:00
Mehdi Abaakouk
61c9e03b6b kafka: Remove testing hack for kafka
Change-Id: Ie304f1ed34156bfb60b9bc9acd4ad0aa5bad4657
2017-01-26 08:00:39 +01:00
Ilya Tyaptin
f139eb258d Moving driver to new kafka-python version
Currently Kafka driver for an oslo.messaging uses kafka-python==0.9.5
and mostly broken. This package version supports only low level Kafka
producer and consumer API which are marked as deprecated now [1]. Using
of these interfaces bring a big concern to the message processing,
because current KafkaConsumer has not any consuming coordination. This
fact causes a message duplication for the several consumers of one
topic. This behavior is specific to Ceilometer and services which read
and process notifications from other services.

New version of kafka-python allows to use async thread safe message
producers and coordinated consumers [1].

[1] http://kafka-python.readthedocs.io/en/master/changelog.html#feb-15-2016

The driver is currently experimental, python-kafka<1.0.0 API have major
issue described above that can't make the oslo.messaging driver works,
so we prefer having a working driver with a non-synced dependencies, that the
reverse.

Co-Authored-By: Mehdi Abaakouk <sileht@redhat.com>
Change-Id: I29862ed7bf56b9d8878fa8e9fb1cbd9d643908a4
2017-01-02 11:46:48 +01:00
Kenneth Giusti
2a9094af5d [AMQP 1.0] Setup the amqp1 test environment on ubuntu
Fixes the amqp1 functional tests so they can be run on Xenial.

Closes-Bug: 1643644
Change-Id: I3e3ee6779a60774ebffc1f45ae5564db934f8eaf
2016-12-19 14:47:04 -05:00
Mehdi Abaakouk
04f9be69e5 tests: fix test-setup.sh
tox doesn't have quiet option, so we have to create the virtualenv first
and run bindep manually.

Change-Id: I3ae76278391baa5cd80202f32ad88b1368b0f16d
2016-12-15 09:41:29 +01:00
Mehdi Abaakouk
5fc28ec9cf Add bindep.txt/test-setup.sh to prepare the system
Change-Id: I9ac3132c082a31020dc2bf2a778fb5c4c0836c4c
2016-12-07 09:29:29 +01:00
Tony Breeds
78f1137805 Add Constraints support
Adding constraints support to libraries is slightly more complex than
services as the libraries themselves are listed in upper-constraints.txt
which leads to errors that you can't install a specific version and a
constrained version.

This change adds constraints support by also adding a helper script to
edit the constraints to remove oslo.messaging.

Change-Id: I8be883215f27abb58d15b85e8542cbdf32000bac
2016-11-23 20:32:32 +11:00
Jenkins
bb740f5a58 Merge "[simulator] Fix transport_url usage" 2016-11-03 16:59:29 +00:00
Kirill Bespalov
401102daab [simulator] Fix transport_url usage
1) Remove hardcoding 'redis' matchmaker from any zmq:// transport because it's
   fails in cases zmq+etcd:// or zmq+sentinel:// urls.
2) Allow to use transport_url from config file by removing default value for
   args.url option. In this case by default args.url is None and
   transport_url will be used from config file.

Change-Id: I1edcabe46b1e92dd4e6eccc9151b50d03f394186
2016-10-19 14:13:29 +03:00
Kirill Bespalov
6eb57b4d5f [simulator] Fix a message length generator usage
Do initialization of the generator only for rpc/notify clients.
No needs to load messages_length.yaml and calculate distribution for servers.

Change-Id: Ib1ac30181bec20eacf6d29ccd5f76c0b6b1e96f8
2016-10-18 14:51:19 +03:00
Kirill Bespalov
dd8e9fb5e3 [simulator] Automatic stopping of rpc-servers
This patch provide the sync flag:

  simulator.py rpc-client --sync call
  simulator.py rpc-client --sync fanout

The --sync values means next:
- call:   to send sync msg via rpc.call
- fanout: to broadcast sync msg via rpc.fanout to all servers on topic

When clients has sent all messages, the rpc-server will be stopped
automatically and dump statistics to a file (if --json-file is used).

This is much usefull than rough killing the process of the server in
benchmark frameworks.

Change-Id: I06fd8dbbcdc8b2b9f13029029f730b417ff128ce
2016-10-06 16:14:01 +03:00
Gevorg Davoian
e491573654 Fix simulator bool command line args
--debug, --is-cast and --is-fanout are defined as args of type=bool.
This means that, for example, if we want to enable debug logging
level, we have to type '--debug True'. But we can also use
'--debug False' in order to do the same, which is very misleading
(in fact, any non-empty string will evaluate to True). This patch
tries to solve this problem by replacing type=bool and
default=False with action='store_true' for these args, so that
we will be able to enable them (they will remain False by default
as before) simply as '--debug' etc.

Change-Id: I8ee04c35427df446966161491da8d264b44975bf
2016-09-30 12:46:02 +03:00
kbespalov
6a41a81c9e Fix calculating of duration in simulator.py
When we calculate metrics like msg/sec, latency, etc
we expect the start and end time as time of
the first and last processed message:

RPC Server life timeline:

[----0..5 sec----][---5..10 sec---][---10..15 sec--]
 waiting clients    10 msg recved     wait sigint

expected: duration 5 sec, 2 msg/sec
actual (incorrect): duration 15 sec, 0.6 msg/sec

no reason to set the boundaries if server was idle few seconds
before running of clients and after.

Change-Id: I33e0a605b54ea7b89977504892528c41c3b00a68
2016-08-16 14:32:19 +00:00
Oleksii Zamiatin
7c5d039fd3 Move zmq driver options into its own group
ZeroMQ driver options are current stored into the DEFAULT group.
This change makes the zmq configuration clearer by putting its
options into oslo_messaging_zmq group.

Change-Id: Ia00fda005b1664750d2646f8c82ebdf295b156fb
Closes-bug: #1417040
Co-Authored-By: Oleksii Zamiatin <ozamiatin@mirantis.com>
2016-08-05 11:36:50 +03:00
Gevorg Davoian
564e423d24 Properly cleanup listener and driver on simulator exit
Change-Id: Id04d4d1ce131bf7a4681273c438cbe6e58b44e78
Closes-Bug: #1584743
Co-Authored-By: Oleksii Zamiatin <ozamiatin@mirantis.com>
2016-07-24 19:23:39 +03:00
Jenkins
80d8b18ebf Merge "Fix simulator stat printing" 2016-05-08 16:24:16 +00:00
Yulia Portnova
d522fcd981 Fix simulator stat printing
Change-Id: I97f05f64475541ec9269c17644cb27b46ef4f1cf
2016-05-06 17:34:24 +00:00
Jenkins
e2b57bb590 Merge "Simulator: align stats to whole seconds" 2016-05-05 03:22:06 +00:00
Dmitriy Ukhlov
a98fa8f52f Fixes sumulator.py signal_handler logic
Change-Id: I95ee737a6c12572e7b3d27d8ea117c9f2909ef28
Closes-bug: #1576614
2016-04-29 07:24:25 -05:00
Jenkins
501659bc66 Merge "Simulator: handle SIGINT and SIGTERM signals" 2016-04-20 07:25:44 +00:00
Ilya Shakhat
439d8ca4d8 Simulator: handle SIGINT and SIGTERM signals
Now it is possible to stop simulator client and server
by sending SIGINT or SIGTERM signals. Note that both stop
gracefully and it takes some time to do this.

Change-Id: Ie3fd1ea8b146070d61a247fd8ccc124df8d34848
2016-04-19 15:41:57 +03:00
Dmitry Mescheryakov
09d5669565 Allow simulator to be launched from arbitrary directory
Right now simulator.py expects messages_length.yaml to be in the
current directory and as a result, it is impossible to launch
simulator while you are at the repository root.

The change makes simulator to search for the file in the same
directory as simulator itself is in.

Change-Id: Ic2ca21d87e2d888432dc6204c7b4b995a06aecbd
2016-04-07 15:08:03 +03:00
Jenkins
e728383c45 Merge "Simulator: collect error stats" 2016-03-28 10:35:23 +00:00
Ilya Shakhat
6957173cb3 Simulator: collect error stats
Collect client-side error statistics.

Change-Id: I8fb41e3a5e58a9215961775be3c70d7c1a822285
2016-03-23 13:09:56 +00:00
Ilya Shakhat
e2e59a508b Simulator: make parameter wait_after_msg float
Introduce more precise control on message sending delay,

Change-Id: I7bc2e575d5fe29c4940fef9b28e2e25df68b3095
2016-03-23 13:09:11 +00:00
Ilya Shakhat
782ab770cf Simulator: align stats to whole seconds
Aligning message sending and stats collection to whole seconds.
This allows to merge data from client and server - useful
for visualization of message flow.

Change-Id: I507c021d851254d4d84e8922de687931f8545864
2016-03-22 13:38:00 +03:00
Davanum Srinivas
38b907a130 Support python3 in simulator.py
Tested with:
python3 simulator.py --url \
    rabbit://stackrabbit:flopsymopsy@localhost:5672/ notify-server
python3 simulator.py --url \
    rabbit://stackrabbit:flopsymopsy@localhost:5672/ notify-client -m 1000

and:
python3 simulator.py --url \
    rabbit://stackrabbit:flopsymopsy@localhost:5672/ rpc-server
python3 simulator.py --url \
    rabbit://stackrabbit:flopsymopsy@localhost:5672/ rpc-client -m 1000

Change-Id: I1f8bb964aef23867a651e192dc355635e36f78a1
2016-03-21 21:03:14 -04:00
Ilya Shakhat
55882795f0 Simulator: store results in JSON format
This patch introduces new parameter --json <file name>.
When it is specified the simulator stores stats in JSON file.

Change-Id: I0b7a87ee614f44baa920569fe0c7046269c90925
2016-03-10 19:23:35 +03:00
Ilya Shakhat
84109fe8fc Simulator: calculate message latency statistics
Calculate message latency: for RPC calls on both server-side and
on client-side (round-trip); for RPC cast and NOTIFY on server-side
only.

The message is extended with metadata: sequence id, creation time,
reception time and return time (for RPC calls). For every message
the metadata is stored into MessageStatCollector. To reduce memory
consumption the collector aggregates stats on every second and can
print the ongoing stats to stdout. At the finish the overall stats
are printed.

NOTIFY tests are changed to be similar to RPC:
 * use pre-generated messages
 * parameter "-w WAIT_BEFORE_ANSWER" for processing delay
 * parameter "--requeue" to re-queue new messages (only those that
   miss the cache)

Change-Id: I4c2fc11bfaee17b6825cdc7d6edf07b8e91ef83a
2016-03-10 13:52:58 +03:00
Ilya Shakhat
37c0db5769 Simulator: always use random messages for time-bound tests
When the test is time-bound the simulator should use as many random
messages as possible.

Change-Id: I1e112d9ff0c3f1281b544f05d25eff2d3912fd6c
2016-03-03 13:57:41 +03:00
Ilya Shakhat
97385ef5b5 Simulator: implement own random generator instead of scipy
Dependency on scipy results in extra requirements to the system
(include the need of Fortran compiler). With this patch the
weighted random choice algorithm is implemented directly.

Change-Id: I60cbc6c452945000add5a65263d1ab4e42dd91f9
2016-03-02 19:06:16 +03:00
Ilya Shakhat
d2496c34d9 Simulator: fix batch-notify-server command
This patch fixes the name of parameters used to construct
batch notification listener

Change-Id: I8eae9494943b2592c9cf4b92fccca93eab3b8e93
2016-03-02 18:28:11 +03:00
ozamiatin
681f631123 Make simulator more asynchronous
We need to use eventlet.sleep() in green thread loops
in order to have more chances to switch between them.

Change-Id: I7c08e82182b68b95c36265d58df0644c5ce2c171
2016-02-26 13:15:00 +02:00
Yulia Portnova
d70dfc2f7e Added duration to notify server/client
Change-Id: I4feeeec0c69305d92dce5baf60502a39ebe6b247
2016-02-24 11:58:12 +03:00
Davanum Srinivas
11f78de5f0 Avoid hardcoding the notification topic and specify driver
"n-t1" as notification topic not really useful. Let's allow
it to be specified from the command line. Also make sure
we set the notification driver as "messaging" by default

Change-Id: I5daf09b9a01e4d642b0f53d7634d5b305d5a82cb
2016-02-22 16:07:30 -08:00
Yulia Portnova
c10ee15fe0 Log format change in simulator.py
Change-Id: I046717f9ea7f203f7e5837c7b62a6a7d7cc7bb4d
2016-02-18 13:43:38 +02:00
Jenkins
b80fdc850d Merge "simulator.py improvements" 2016-02-15 13:32:49 +00:00
Yulia Portnova
3c0a48aace simulator.py improvements
Added fanout option to target
Fixed timeout option
Removed setting config values as they are
reverted to defaults on driver load

Change-Id: Ia2d8a7bbcb04706deb3d701c1a0f93a0dd835a19
2016-02-15 10:38:47 +02:00
Javeme
668062ed07 Use more secure yaml.safe_load() instead of yaml.load()
The function yaml.load() provides the ability to construct an arbitrary
Python object. For security, we use yaml.safe_load() instead which
limits this ability to simple Python objects(like integers or lists).

ref: https://en.wikipedia.org/wiki/YAML#Security

Change-Id: Iea2a2d79c764d635b02c5d6d36c9a5652010d716
2016-02-11 14:35:24 +08:00
Yulia Portnova
062fedccd0 Improve simulator.py
* passing targets to rpc-client using -tg arg
so clients can send messages to different targets
* removed redis.flush_db() so we can start
several simulator processes for zmq.

Change-Id: I0d36c5d8cbec938c8fd85d1041d2a3bff02b49e5
2016-01-27 13:08:24 +02:00
Javeme
6e2f4efb19 replace string format arguments with function parameters
There are files containing string format arguments inside
logging messages. Using logging function parameters should
be preferred.

Change-Id: Iaa15bc52d1d8832032cc46205effb49630cea494
Closes-Bug: #1321274
2016-01-22 20:36:43 +08:00
Jenkins
2ae5196747 Merge "py3: Replaces xrange() with six.moves.range()" 2016-01-21 07:57:28 +00:00