2795 Commits

Author SHA1 Message Date
b9797f3385 reno: Update master for unmaintained/wallaby
Update the wallaby release notes configuration to build from
unmaintained/wallaby.

Change-Id: I77d6152203fda5c8acc4dff534562b39b5d7d23b
2024-03-07 14:21:38 +00:00
Zuul
ebdc7db19e Merge "Display the reply queue's name in timeout logs" 14.7.0 2024-02-09 14:39:10 +00:00
Zuul
a0f3a7ddaf Merge "Display coverage report" 2024-02-07 16:59:44 +00:00
Takashi Kajinami
795185d43c Display coverage report
... for easy reference. Also make sure old data is purged.

Change-Id: Ic017107ae15fa33cdfd4866f812af9a583a32f14
2024-02-07 12:17:40 +09:00
94a3ebcb4d reno: Update master for unmaintained/yoga
Update the yoga release notes configuration to build from
unmaintained/yoga.

Change-Id: I90b84b3d4995c54e70db680fde83c83034f478f6
2024-02-06 14:28:20 +00:00
Hervé Beraud
97d457f0af
Display the reply queue's name in timeout logs
It would be helpful if "Timed out waiting for <service>" log messages at least
specified on which `reply_q` it was waited for.

Example without the reply_q:

```
12228 2020-09-14 14:56:37.187 7 WARNING nova.conductor.api
[req-1e081db6-808b-4af1-afc1-b87db7839394 - - - - -] Timed out waiting for
nova-conductor.  Is it running? Or did this service start before
nova-conductor?  Reattempting establishment of nova-conductor connection...:
oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to
message ID 1640e7ef6f314451ba9a75d9ff6136ad
```

Example after adding the reply_q:

```
12228 2020-09-14 14:56:37.187 7 WARNING nova.conductor.api
[req-1e081db6-808b-4af1-afc1-b87db7839394 - - - - -] Timed out waiting for
nova-conductor.  Is it running? Or did this service start before
nova-conductor?  Reattempting establishment of nova-conductor connection...:
oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply
(reply_2882766a63b540dabaf7d019cf0c0cda)
to message ID 1640e7ef6f314451ba9a75d9ff6136ad
```

It could help us to more merely debug and observe if something went
wrong with a reply queue.

Change-Id: Ied2c881c71930dc631919113adc00112648f9d72
Closes-Bug: #1896925
2024-02-06 15:17:46 +01:00
Zuul
b5244bd05a Merge "Bump hacking (again)" 2024-01-27 11:46:42 +00:00
Zuul
ef381d2172 Merge "Remove scenario 03/04 tests from CI" 2024-01-26 11:24:05 +00:00
Takashi Kajinami
4f4c2772da Bump hacking (again)
The previous attempt did not update the version in pre commit config
so the old version is still used by pep8 target.

Change-Id: Idf8c7d99f7c6aeb0244d58e85524ba1f039195d8
2024-01-26 01:10:57 +09:00
Takashi Kajinami
22634362a8 Bump hacking
hacking 4.0.x is too old.

Change-Id: Ide85649ba04efcc295f4dae6f5147d351666daa3
2024-01-25 12:20:39 +09:00
Zuul
98f4f81186 Merge "Bump bandit" 2024-01-23 13:23:08 +00:00
Zuul
6a1f4618c2 Merge "Drop unused function from scenario test script" 2024-01-23 05:14:22 +00:00
Takashi Kajinami
a196e7751a Remove scenario 03/04 tests from CI
We should have removed these when we removed the tox targets[1].

[1] 6e7a5725fa90e66c060a06a8ffe5e5454fd7a7b6

Change-Id: I4bdf987ea52479ef7790fc506158a57d8d060dc5
2024-01-23 13:01:18 +09:00
Takashi Kajinami
c9202b01e2 Drop unused function from scenario test script
This function is no longer used since we removed ampq1 tests, when we
deprecated the amqp1 driver[1]

[1] 0f63c227f5425995ae8c61f1d40ec85e7728528a

Change-Id: I47fe04d6a39ed2b5f33b02fa6736d588d0383f5a
2024-01-23 00:44:03 +09:00
Zuul
d43696c284 Merge "Update the python search path for extra qdrouter modules" 2024-01-22 15:42:51 +00:00
Zuul
a417b425a0 Merge "Add an option to use rabbitmq stream for fanout queues" 2024-01-19 15:24:44 +00:00
Zuul
5e91847af4 Merge "cleanup amqp1 scenarios remnants setups" 2024-01-18 16:34:27 +00:00
Jay Faulkner
800c58826e Utilize the new RequestContext redacted_copy method
We now expect context objects to support returning a redacted copy of
themselves.

As a related cleanup, removed the practice entirely of using
dictionaries to represent contexts in unit tests and the logging driver.

As part of developing this change, I discovered code in Glance (and
potentially other services) which explicitly pass {} in lieu of a
context when notifying; so we now properly handle dictionaries as
contexts.

To ensure we have the method required; require oslo.context 5.3.0 or
newer.

Change-Id: I894f38cc83c98d3e8d48b59864c0c7c2d27e7dcd
2024-01-16 12:08:20 -08:00
Hervé Beraud
6e7a5725fa cleanup amqp1 scenarios remnants setups
Change-Id: I9e042732b1d512bb4bebff7a72ea2063a25d2926
2024-01-16 22:17:57 +09:00
julien.cosmao
e95f334459 Add an option to use rabbitmq stream for fanout queues
This is introducing the "stream" queues for fanout so all components
relying on fanout can use the same stream, lowering the number of queues
needed and leveraging the new "stream" type of queues from rabbitmq.

Closes-Bug: #2031497

Change-Id: I5056a19aada9143bcd80aaf064ced8cad441e6eb
Signed-off-by: Arnaud Morin <arnaud.morin@ovhcloud.com>
2024-01-15 09:23:36 +01:00
Zuul
6ad1ccf89c Merge "Add QManager to amqp driver" 2024-01-12 19:42:37 +00:00
Kenneth Giusti
df0e4acdd4 Update the python search path for extra qdrouter modules
The latest release of qdrouterd on focal has changed where the
internal python modules are installed.  This patch updates the tox
tests python path configuration.

Change-Id: Icb53ee17af01580d899f388f69be9560e23675e0
2024-01-12 17:33:06 +01:00
Ghanshyam Mann
ee5b351742 Update python classifier in setup.cfg
As per the current release tested runtime, we test
python version from 3.8 to 3.11 so updating the
same in python classifier in setup.cfg

Change-Id: I303912894d12be87355f83a1a53be071db94cf84
2024-01-11 16:18:53 -08:00
Zuul
875506fff0 Merge "Enable use of quorum queues for transient messages" 2024-01-11 22:19:53 +00:00
Zuul
37888462d1 Merge "Fix clearing of the fake RPC Exchange" 14.6.0 2023-12-19 19:07:43 +00:00
Takashi Kajinami
e2b2008ed9 Remove translation sections from setup.cfg
These translation sections are not needed anymore, Babel can generate
translation files without them.

Change-Id: Ib60671941371aa22fbdeeb9d42fc619f60aa15e5
2023-12-19 22:45:57 +09:00
Gorka Eguileor
f65607fa48 Fix clearing of the fake RPC Exchange
The current fake driver does not properly clean up the fake RPC exchange
between tests.

This means that if a test invokes code that makes an RPC request, using
the fake driver, without consuming the RPC message, then another test
may receive this request making it fail.

This issues has been found while working on a Cinder patch and has been
worked-arounded there with Change-Id
I52ee4b345b0a4b262e330a9a89552cd216eafdbe.

This patch fixes the source of the problem by clearing the exchange
class dictionary in the FakeExchangeManager during the FakeDriver
cleanup.

Change-Id: If82c2175cf7242b80509d180cdf92323c0f4c43b
2023-11-15 12:26:47 +01:00
Arnaud Morin
4614132ad0 Add QManager to amqp driver
The purpose of this change is to introduce an optional mechanism to keep
the queues name consistent between service restart.
Oslo messaging is already re-using the queues while running, but the
queues are created using a random name at the beginning.

This change propose an option named use_queue_manager (default to False
- so the behavior is not changed) that can be set to True to switch to a
consistent naming based on hostname and processname.

Related-bug: #2031497

Signed-off-by: Arnaud Morin <arnaud.morin@ovhcloud.com>
Change-Id: I2acdef4e03164fdabcb50fb98a4ac14b1aefda00
2023-11-12 00:08:20 +01:00
Arnaud Morin
989dbb8aad Enable use of quorum queues for transient messages
Add a new flag rabbit_transient_quorum_queue to enable the use of quorum
for transient queues (reply_ and _fanout_)

This is helping a lot OpenStack services to not fail (and recover) from
a rabbit node issue.

Related-bug: #2031497

Signed-off-by: Arnaud Morin <arnaud.morin@ovhcloud.com>
Change-Id: Icee5ee6938ca7c9651f281fb835708fc88b8464f
2023-11-12 00:08:20 +01:00
Arnaud Morin
8e3c523fd7 Auto-delete the failed quorum rabbit queues
When rabbit is failing for a specific quorum queue, the only thing to
do is to delete the queue (as per rabbit doc, see [1]).

So, to avoid the RPC service to be broken until an operator eventually
do a manual fix on it, catch any INTERNAL ERROR (code 541) and trigger
the deletion of the failed queues under those conditions.
So on next queue declare (triggered from various retries), the queue
will be created again and the service will recover by itself.

Closes-Bug: #2028384
Related-bug: #2031497

[1] https://www.rabbitmq.com/quorum-queues.html#availability

Signed-off-by: Arnaud Morin <arnaud.morin@ovhcloud.com>
Change-Id: Ib8dba833542973091a4e0bf23bb593aca89c5905
2023-11-12 00:08:20 +01:00
Arnaud Morin
f23f3276c4 Allow creating transient queues with no expire
When an operator rely on rabbitmq policies, there is no point to set the
queue TTL in config.
Moreover, using policies is much more simpler as you dont need to
delete/recreate the queues to apply the new parameter (see [1]).
So, adding the possibility to set the transient queue TTL to 0 will
allow the creation of the queue without the x-expire parameter and only
the policy will apply.

[1] https://www.rabbitmq.com/parameters.html#policies

Related-bug: #2031497

Signed-off-by: Arnaud Morin <arnaud.morin@ovhcloud.com>
Change-Id: I34bad0f6d8ace475c48839adc68a023dd0c380de
2023-11-12 00:08:20 +01:00
Arnaud Morin
3438726dd0 Add some logs when sending RPC messages
Related-bug: #2031497

Signed-off-by: Arnaud Morin <arnaud.morin@ovhcloud.com>
Change-Id: I7d0c318624d3d02182392ca3f06eed04d4133728
14.5.0
2023-11-12 00:08:11 +01:00
Zuul
f455edd601 Merge "Bump bandit and make oslo.messaging compatible with latest rules" 2023-10-20 13:47:33 +00:00
Zuul
b45733ad92 Merge "Move integration jobs to Ubuntu 22.04 (Jammy)" 2023-10-19 00:09:09 +00:00
Stephen Finucane
d16cddd822 Move integration jobs to Ubuntu 22.04 (Jammy)
I noticed the confluent-kafka package isn't compatible with the version
of librdkafka-dev provided on Jammy [1][2]. Thankfully they provide a
binary wheel though.

[1] https://github.com/confluentinc/confluent-kafka-python/blob/v1.9.2/src/confluent_kafka/src/confluent_kafka.h#L59
[2] https://packages.ubuntu.com/jammy/librdkafka-dev

Change-Id: If55f246d9d2a990f2389d14d7bda22a82d9bb43c
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
2023-10-13 17:31:11 +09:00
Zuul
38c86a93ad Merge "Set default heartbeat_rate to 3" 2023-10-11 13:29:33 +00:00
OpenStack Proposal Bot
8759cd7d9a Imported Translations from Zanata
For more information about this automatic import see:
https://docs.openstack.org/i18n/latest/reviewing-translation-import.html

Change-Id: I59074f7789476cdfb861be97c60b720bc3ccba54
2023-09-30 04:10:18 +00:00
Sairam Vengala
a3715386de test py311 job
this patch fixes the py3.11 job by updating bindep.txt

Change-Id: I8eeb1b345033269a98989ef4a563ca65fe04e57f
2023-09-28 11:04:30 +00:00
Zuul
3c8dd7d290 Merge "Add is_admin to safe fields list for notifications" 2023-09-26 09:37:17 +00:00
Jay Faulkner
c1b606f77e Add is_admin to safe fields list for notifications
We encountered bug 2037312 in unit tests when attempting to get this
change rolled out. Heat apparently will attempt to set is_admin using
policy logic if it's not passed in for a new context; this breaks as the
context we are requested doesn't have all the needed information to
exercise the policy logic.

is_admin is just a bool; it's not sensitive; easiest route forward is to
add it to the safe list

Closes-bug: 2037312
Change-Id: I78b08edfcb8115cddd7de9c6c788c0a57c8218a8
2023-09-25 17:51:32 +00:00
Zuul
84d489eead Merge "Add Python3 antelope unit tests" 2023-09-19 06:17:19 +00:00
7705b4f302 Update master for stable/2023.2
Add file to the reno documentation build to show release notes for
stable/2023.2.

Use pbr instruction to increment the minor version number
automatically so that master versions are higher than the versions on
stable/2023.2.

Sem-Ver: feature
Change-Id: I8e9c35ebe41e0283309d64db97a4d9ffebcf9626
2023-09-07 09:37:14 +00:00
Zuul
3485301b18 Merge "Only allow safe context fields in notifications" 14.4.0 2023-08-17 12:43:26 +00:00
7da68f269b Add Python3 antelope unit tests
This is an automatically generated patch to ensure unit testing
is in place for all the of the tested runtimes for antelope.

See also the PTI in governance [1].

[1]: https://governance.openstack.org/tc/reference/project-testing-interface.html

Change-Id: Iad97644ff56b721fda6a16d51e373709d6f8ee9d
2023-08-15 15:03:12 +00:00
Zuul
fa15630041 Merge "Deprecate the amqp1 driver and Remove qpid functional tests" 2023-08-13 10:36:35 +00:00
Jay Faulkner
1b315615e7 Only allow safe context fields in notifications
Publishing a fully hydrated context object in a notification would give
someone with access to that notification the ability to impersonate the
original actor through inclusion of sensitive fields.

Now, instead, we pare down the context object to the bare minimum before
passing it for serialization in notification workflows.

Related-bug: 2030976
Change-Id: Ic94323658c89df1c1ff32f511ca23502317d0f00
2023-08-11 13:07:54 -07:00
Arnaud Morin
36fb5bceab Set default heartbeat_rate to 3
Kombu recommend to run heartbeat_check every seconds but we use a lock
around the kombu connection so, to not lock to much this lock to most of
the time do nothing except waiting the events drain, we start
heartbeat_check and retrieve the server heartbeat packet only two times
more than the minimum required for the heartbeat works:
    heartbeat_timeout / heartbeat_rate / 2.0

Because of this, we are not sending the heartbeat frames at correct
intervals. E.G.

If heartbeat_timeout=60 and rate=2, AMQP protocol expects to send a
frame
every 30sec.

With the current heartbeat_check implementation, heartbeat_check will be
called every:
    heartbeat_timeout / heartbeat_rate / 2.0 = 60 / 2 / 2.0 = 15
Which will result in the following frame flow:
    T+0  --> do nothing (60/2 > 0)
    T+15 --> do nothing (60/2 > 15)
    T+30 --> do nothing (60/2 > 30)
    T+45 --> send a frame (60/2 < 45)
    ...

With heartbeat_rate=3, the heartbeat_check will be executed more often:
    heartbeat_timeout / heartbeat_rate / 2.0 = 60 / 3 / 2.0 = 10
Frame flow:
    T+0  --> do nothing (60/3 > 0)
    T+10 --> do nothing (60/3 > 10)
    T+20 --> do nothing (60/3 > 20)
    T+30 --> send a frame (60/3 < 30)
    ...

Now we are sending the frame with correct intervals

Closes-bug: #2008734

Signed-off-by: Arnaud Morin <arnaud.morin@ovhcloud.com>
Change-Id: Ie646d254faf5e45ba46948212f4c9baf1ba7a1a8
2023-08-08 15:23:59 +02:00
OpenStack Proposal Bot
ec1c99d562 Imported Translations from Zanata
For more information about this automatic import see:
https://docs.openstack.org/i18n/latest/reviewing-translation-import.html

Change-Id: If124c09bfa878585b89dee7578a97c12c42497f8
2023-06-22 03:45:12 +00:00
Hervé Beraud
ee13e53614 Bump bandit and make oslo.messaging compatible with latest rules
- Apply a timeout to requests calls to avoid uncontrolled
  resource consumption (CWE-400) [1].
- Ignore CWE 377

[1] https://cwe.mitre.org/data/definitions/400.html
[2] https://cwe.mitre.org/data/definitions/377.html

Change-Id: Ic558ad392424a25b5fd9a10749163d8427159eda
2023-05-17 11:06:34 +02:00
Andrew Bogott
0602d1a10a Increase ACK_REQUEUE_EVERY_SECONDS_MAX to exceed default kombu_reconnect_delay
Previously the two values were the same; this caused us
to always exceed the timeout limit ACK_REQUEUE_EVERY_SECONDS_MAX
which results in various code paths never being traversed
due to premature timeout exceptions.

Also apply min/max values to kombu_reconnect_delay so it doesn't
exceed ACK_REQUEUE_EVERY_SECONDS_MAX and break things again.

Closes-Bug: #1993149
Change-Id: I103d2aa79b4bd2c331810583aeca53e22ee27a49
14.3.1
2023-04-20 15:27:58 -05:00