11 Commits

Author SHA1 Message Date
Radosław Piliszek
564b405a91 [CI] Restore token critical error filter
It was removed as part of improvements in [1]. But it seems it's
a different type/class of errors and should still be filtered
out here.
This time for all the services to avoid extra copies.

[1] Ie6a95c3ca84df82ca8463cf76054e125cb1ffff1

Change-Id: If7c5b938dc7417831a90f360d6e3655d1ad4084e
2022-05-21 13:27:15 +00:00
Radosław Piliszek
7ccea27827 [CI] Raise [keystone_authtoken]http_request_max_retries
This is in the hope to avoid random "Unable to validate token" in
AIO upgrade jobs which restart the Keystone container while
other services are trying to use it (most notably Placement but
also Nova and Neutron).

Change-Id: Ie6a95c3ca84df82ca8463cf76054e125cb1ffff1
2022-05-10 17:16:30 +00:00
Radosław Piliszek
58dd72583b [CI] Filter fluentd errors more
Make it less likely to fail on false positives.

Change-Id: I493430e255c76bf2a30818b63913ecabb40c5afc
2022-02-09 18:24:19 +01:00
Zuul
e4bfedd731 Merge "[CI] Check fluentd errors" 2022-01-27 11:13:55 +00:00
Radosław Piliszek
581544a037 [CI] Check fluentd errors
We consider them critical and prefer to fail the jobs.

Change-Id: I3c2475522f8f37e3b608d3c321aee27277fc9dee
2021-12-29 10:15:55 +00:00
Michal Nasiadka
661ca4dc06 CI: check-logs - add another exception
It was seen in [1] and some other occassions, possibly neutron was trying
to fetch token while keystone was being upgraded.

[1]: https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_ec2/665117/12/check/kolla-ansible-ubuntu-source-upgrade/ec2fc04/primary/logs/kolla/all-CRITICAL.txt

Change-Id: If8c569e1f7815520c84627671e3cbf275bed8ff9
2021-11-26 11:28:24 +01:00
Radosław Piliszek
16ade28d5f CI: Ignore neutron-server CRITICAL about WSREP
We are getting this randomly on neutron-server shutdown
for upgrade.
As it does not affect real operations (and if it did,
we would definitely see it now), let's ignore it.

Change-Id: Ibe561517d44a1108e8223442a71fab36b69c2258
Related-bug: #1863579
2020-03-16 10:00:46 +00:00
Mark Goddard
2f7640d396 CI: Don't fail on expected critical log messages
There is a number of critical log messages that we see in CI from time
to time. While these should be fixed, let's not fail jobs unnecessarily.

This change introduces one expected critical message in
placement-api.log:

    Failed to fetch token data from identity server

Co-authored-by: Radosław Piliszek <radoslaw.piliszek@gmail.com>
Related-bug: #1847727
Change-Id: I92ad0be70ed05925612f0c709907ab62280326b8
2019-12-08 08:36:33 +00:00
Radosław Piliszek
22cea7f08c CI: run check-logs in post too
Also makes check-logs clear the aggregate files
to prevent duplicates due to upgrade and post.

Change-Id: I72377b6ac48e29dc5e24d1e3bd343e87b74a7a71
2019-10-23 12:58:36 +00:00
Mark Goddard
bfd8ee1978 CI: Fix check-logs.sh
This script has a few issues:

* It catches false positives, due to log levels in config options.

* It doesn't fail on CRITICAL logs, due to variable reset issue.

This change fixes these.

Change-Id: I50c859eb2991e498eeb64bca45daf1e6f237761f
2019-09-20 15:36:50 +00:00
Mark Goddard
a14eee24d1 Check for CRITICAL, WARNING and ERROR log messages in CI
At the end of a CI run, check all log files.

Change-Id: I99afc1c5207757e35beabf7daebd86c56151c96d
2019-08-16 15:33:54 +00:00