146 Commits

Author SHA1 Message Date
Tim Burke
fb477d17ae CI: Remove nodeset pin for DSVM jobs
These should be fine on jammy/py310 now.

Change-Id: Iae3daadc9d880b4980aa2305df626929c93b475d
Closes-Bug: #1996627
Related-Bug: #1999278
Related-Change: Iae28097668213aa0734837ff21aef83251167d19
2023-01-09 11:13:39 -08:00
Tim Burke
f77172e2ad CI: pin tox at the project level
No sooner do we fix the gate than tox has a new release that breaks
it again. Let's give them a bit to settle down; in the mean time, stick
with 3.x.

See https://github.com/tox-dev/tox/issues/2811

Also simplify our warning suppressions. The message filter is a regex,
so any prefix of the message will suffice. This allows us to also drop a
new message seen on CentOS 8:

   CryptographyDeprecationWarning: Python 3.6 is no longer
   supported by the Python core team. Therefore, support for
   it is deprecated in cryptography. The next release of
   cryptography (40.0) will be the last to support Python 3.6.

As we've previously seen with cryptography warnings, this can slow down
our probe tests to the point that they time out.

Change-Id: I316170442c67c1b4a5b87f9a1168cc04ca2417b8
2023-01-04 10:50:01 -08:00
Tim Burke
2d7c1dc6dd CI: Fix our usage of tox
For tox 3.x and earlier, passenv was a space-separated list; as of tox
4.0.0, it's comma-separated. For a while, our spaces would be silently
included in the now-one-and-only passenv value parsed (which wasn't
great, but mostly just caused confusion) -- as of tox 4.0.6, however, it
became a hard error, and all tests would fail like

  pass_env values cannot contain whitespace, use comma to have multiple
  values in a single line, invalid values found 'SWIFT_* *_proxy'

Unfortunately, we don't really know what versions of tox all our various
stakeholders might want/need to use (though we previously set a
minversion of 2.3.2). We might be able to spread values over multiple
lines to make it compatible with both tox 3 *and* tox 4, but I'm fairly
certain *_proxy was only included for some variables that are recent
versions of tox include by default anyway, so just increase our
minversion (which was too low, anyway -- allowlist_externals which we
already configure was added in 3.18.0) and get rid of *_proxy.

FWIW, python-swiftclient was already specifying 3.18.0 as a minversion,
so I expect the new minversion to not be a problem.

Also, add ./.functests to a bunch of allowlist_externals, as newer tox
is more strict about that sort of thing.

Drop skipsdist in a bunch of places so we can import swift from func
tests and docs. (Still not sure why I don't see us hitting a similar
problem for unit tests...)

Change-Id: I4be1e86e3291ad1619c695fb93d7cadf053b556d
2022-12-29 13:36:06 -08:00
Tim Burke
2d0fea6a57 CI: Add py310 unit test job
Change-Id: Ib90ebfefb387779c376249aa834f843dfc3d71a1
2022-12-13 11:43:39 -08:00
Tim Burke
ef155bd74a Switch to pytest
nose has not seen active development for many years now. With py310, we
can no longer use it due to import errors.

Also update lower contraints

Closes-Bug: #1993531
Change-Id: I215ba0d4654c9c637c3b97953d8659ac80892db8
2022-12-09 11:38:02 -08:00
Ghanshyam Mann
5a9f5f521a Temporary pin the swift-dsvm-functional nodeset to Focal
As per the community wide goal to migrate the CI/CD from
Ubuntu Focal to Ubuntu jammy, we need to merge the devstack, tox base
jobs to jammy on Nov 18. But swift-dsvm-functional job is
failing on Ubuntu Jammy.

To move ahead to merge the base job patches we need to pin the
swift-dsvm-functional job nodeset to Focal until this is fixed
for Jammy.

Needed-By: https://review.opendev.org/c/openstack/devstack/+/860795

Related-bug: #1996627
Change-Id: Ia35e55782592175a6a9aaafd59d72a0ff8362d61
2022-11-17 15:22:24 -06:00
Tim Burke
d1b2bbdcf0 Mark rolling-upgrade job non-voting
Upstream CPython broke our HTTP parsing; while we can fix our own
HttpProtocol, previous tags won't have the fix (naturally).

See also: 4abab6b603

Change-Id: Ibe67b1a485350967e37809ba8575a33eba56ee97
Related-Change: https://review.opendev.org/c/openstack/swift/+/863441
2022-11-07 12:24:40 -08:00
Tim Burke
052abed755 Move base CI job to jammy
We'll want that by the time we can add a py310 job. It'll also make us
be explicit about the OS to use for py27 and py36-py39 jobs.

Also, update bindep so we don't try to install py2 stuff on jammy.

Change-Id: If70cd2fc03d10e55cd782ec8ffc87bd2b2b321ea
2022-11-03 15:39:05 -07:00
Tim Burke
3571cdf2f7 CI: Install pip from pinned upstream for py2 probe tests
As part of that, invert the inheritance so the py2 job derives from the
py3 job.

Change-Id: If8efd9a0fd18d17444a58119c525ff54cfd55dd9
2022-10-26 21:25:02 -07:00
Tim Burke
94fb7a0e78 Set irrelevant-files for openstacksdk-functional-devstack job
Change-Id: I8510c3f6ec6cdf658a918cd761fa17cf791bfd00
2022-09-13 13:09:37 -07:00
Zuul
ddeba903d0 Merge "CI: Add rolling upgrade job coming from stable/yoga" 2022-08-30 22:13:44 +00:00
Tim Burke
c4ff49c325 CI: Add rolling upgrade job coming from stable/yoga
Change-Id: Ifd86ef011951f0f8c2a142ad3a060af347e9b8c9
2022-08-29 16:20:17 -07:00
afariasa
da52e56751 Migrate CentOS Stream 8 FIPS job to CentOS Stream 9
Change-Id: I8b0bd7ee1c649294ac3795cba801c9e07bc3d51b
2022-08-25 16:14:41 -07:00
Tim Burke
59508de0c3 CI: Add nslookup_target to FIPS jobs
At some point, this became a required parameter.

Change-Id: I4d807f3b3649a45dbb2166bb8e911c45fb9cb701
Related-Change: Iebf474c9351e4246d7ab2072b48a50e93dbf0b94
2022-07-28 14:58:09 -07:00
Tim Burke
27db5213dc CI: Run s3api test suite
Change-Id: I70dec389c1ac608a9d2767d162bd5edc315e564b
2022-05-27 23:46:20 -07:00
Tim Burke
0708edecdf Drop arm64 probe test job
Most of the time, we're so resource constrained that we trip the
three-hour (!!) job timeout, while the x86_64 job can typically complete
within an hour. When the arm64 probe tests *do* fail, they've often
failed due to eventual-consistency problems; things like object-updaters
popping 10s timeouts and not getting asyncs off disk.

The job last passed back in January.

Change-Id: I31efa3fef673a5541505e4855e52bfd0c9668ffd
2022-04-28 08:53:11 -07:00
Tim Burke
d29cbc3996 CI: Run ceph and rolling upgrade tests under py3
As part of that, the ceph test runner needed up-rev'ing to run under
py3. As a result, the known-failures shifted.

Trim the on-demand rolling upgrade jobs list -- now that it's running
py3, we only expect it to pass for train and beyond.

Also, pin smmap version on py2 -- otherwise, the remaining experimental
jobs running on centos-7 fail.

Change-Id: Ibe46aecf0f4461be59eb206bfe9063cc1bfff706
2022-04-04 17:17:06 -07:00
Tim Burke
ffb173f8ad CI: Run CORS tests under py3
Change-Id: I49d8480fd0c775249d8b586f9dea026448b8910d
2022-03-22 23:12:01 -07:00
Tim Burke
3358345682 CI: remove swift-tox-func-encryption-py36-centos-8 job
This was added while debugging issues with the FIPS jobs, and
we already have a swift-tox-func-encryption-py36-centos-8-stream
job in the experimental pipeline.

Change-Id: I77178aa7def0e02e7b7add662135d62ed7f8c697
2022-02-16 11:56:50 -08:00
Ade Lee
03be71c44a Add FIPS CI jobs
Added jobs to run the swift functional tests when FIPS is enabled.
We'll set these jobs to be non-voting initially, so that we can
ensure that they are stable.

Change-Id: If0b4cfc1ac9a8c66085eb4dc95366d43806d5ae2
2022-01-25 12:07:09 -08:00
Zuul
720874dfa3 Merge "CI: Add rolling upgrade job coming from stable/xena" 2022-01-08 07:31:21 +00:00
Zuul
1b8708d9c5 Merge "CI: Move py37 and py38 jobs to experimental pipeline" 2021-12-21 19:49:16 +00:00
Tim Burke
ac92b3d8d8 CI: Add rolling upgrade job coming from stable/xena
Change-Id: I17638d469f704e106ce691bed51e879c736ce153
2021-12-10 13:50:29 -08:00
Tim Burke
d6206e6fe5 CI: Move py37 and py38 jobs to experimental pipeline
The confirmation that tests pass will still be nice come release time,
but the py36 and py39 jobs probably suffice to say the intermediary
versions work.

Change-Id: I3de9fb217b402df214430a6cbf920d7bd7694b32
2021-12-10 13:48:50 -08:00
Tim Burke
546c9629eb Move CI from CentOS 8 to CentOS 8 Stream
See also: http://lists.opendev.org/pipermail/service-announce/2021-December/000029.html

Change-Id: Id34eea3f20621c2b25cc7753b1323d4ffb88f11b
2021-12-10 13:25:49 -08:00
Alistair Coles
346f518d6c Make arm jobs voting (but not the pipeline)
Make the non-voting pipeline indicate failure when any of the jobs
fail.

Previously zuul would report "Build succeeded (ARM64 pipeline)."
even when individual job(s) had failed. Now zuul will report
"Build failed (ARM64 pipeline)." but the pipeline still does
vote on the patchset.

Change-Id: I84f59256df3aa41338df9829a565ae830ee8e847
2021-11-24 12:51:32 +00:00
Dr. Jens Harbott
3d190dba5d
Clean up devstack job definitions
Keystone can only provide v3 now, remove variables that no longer have
any effect within devstack.

Signed-off-by: Dr. Jens Harbott <harbott@osism.tech>
Change-Id: I66f6fd712cd14f60be39e2ee7ddd0a089cedabab
2021-11-10 06:39:52 +01:00
Tim Burke
4f84e77a62 Bump up timeout on swift-probetests-centos-8-arm64 job
We have seen a fair number of timeouts on this lately.

Change-Id: I8c77c3c7e724768bf40d6d33bfbe72604fd4f6bd
2021-10-13 16:04:17 +00:00
Matthew Oliver
510fe6962b Add some more arm64 non-voting tests
Adds:
 - a centos-8 probe test on py36
 - an ubuntu-bionic functest on py38
 - an ubuntu-bionic functest with encryption on py38

Change-Id: Id835b49312ec04d7ce9094718b98d8e23ddba0b7
2021-07-14 22:17:42 +00:00
Zuul
bbcd8e90f2 Merge "Run in-process func tests under py38" 2021-06-17 01:39:29 +00:00
Tim Burke
8e75faff80 Run in-process func tests under py38
Change-Id: Ic59f2f0a1890a496398a0d30c9ed5ab7c6fa7d87
2021-06-08 15:25:23 -07:00
ricolin
41a3e1ff51 Add non-voting swift python3 unit test jobs
Add non-voting swift-tox-jobs-arm64 project template.
It include two non-voting project jobs:
* swift-tox-py38-arm64
* swift-tox-py39-arm64

And running on seperate pipeline `check-arm64`.

Story: 2007938
Task: 42511

Change-Id: I92e5b159779422b064b2d6d034cfb6dd0d4afa59
2021-05-25 12:10:45 +08:00
Tim Burke
2a3a0fb75e Add rolling upgrade job coming from stable/wallaby
Change-Id: I8fd5d11e4b33a87b5fcb1a2d59c84986ca770fd8
2021-03-25 14:53:16 -07:00
Tim Burke
9ab9acb03f gate: Make swift-multinode-rolling-upgrade voting again
Change-Id: I04dc249365bbdb511b036c7f433778ff30285591
2021-03-18 16:47:57 -07:00
Tim Burke
81db980690 s3api: Pass through CORS headers
This adds support for presigned GET URLs, at least.

Note that there is no support yet for preflight requests, so a whole
bunch of other CORS stuff *doesn't* work (yet). This was just an easy
first step.

Change-Id: I43150a630a2a7620099e6bfecaed3bbe958ba423
2021-03-01 10:55:15 -08:00
Tim Burke
c5152ed4d3 Add some functional CORS tests
If you've got selenium installed (and working), the whole thing can be
automated pretty well; run main.py, wait while some windows pop up (or
use xvfb-run to run things on a virtual display), then check out what
tests were run on which browsers and whether any of them failed. Exit
code is the number of failed tests.

Includes tests against:
- Account
- Containers, with various ACLs/CORS settings
- Objects
- /info
- SLOs
- DLOs
- Symlinks

Include a gate job that runs the tests in firefox.

Areas for future work:

- Install chromium and chromedriver in the gate; tests should
  automatically pick up on the fact that it's available
- Capture the web browser's console logs, too, so we can get
  more info when things go wrong

Change-Id: Ic1d3a062419f1133c6e2f00a598867d567358c9f
2021-03-01 10:09:03 -08:00
Tim Burke
44390d1ec1 Test under py39
Depends-On: https://review.opendev.org/c/openstack/project-config/+/774906
Change-Id: I4a16370646a1a684240d29950ba197baccae91bb
2021-02-10 18:42:21 +00:00
Tim Burke
be908186be zuul: Clean up/de-dupe irrelevant-files
Change-Id: I397834fe00011df5fc161395012ed45afd859c44
2021-01-28 01:17:48 +00:00
Tim Burke
0c0d9d73de gate: Add centos8/py36 experimental jobs
Change-Id: I47a88afe9f71bce783cf8bd05d4cb7495decadf6
2021-01-09 08:19:46 +00:00
Tim Burke
d186af14fa Add py3 probe tests on CentOS 8
Change-Id: Iae86d8838854023010686d3d4bed3befe6160ca5
2020-12-17 11:25:42 -08:00
Zuul
6e9e84a5c2 Merge "Run DSVM func tests under py3" 2020-12-16 15:34:10 +00:00
Zuul
36107b4a69 Merge "Give probe tests a second chance to pass" 2020-12-11 12:57:39 +00:00
Tim Burke
b87bc7d18f Increase gate timeouts from 1hr to 1.5hr
Change-Id: I0a77025237549b192b6b695b6f18512dff28af6c
2020-12-08 15:31:20 -08:00
Tim Burke
d4c0a7d3b3 Give probe tests a second chance to pass
...and bump up their timeout, since that seems more likely to happen if
we have to retry.

Change-Id: Ie05521f6cd146234dc5615c96ad19681b43e9110
2020-12-08 15:24:20 -08:00
Tim Burke
2b4f581080 Run DSVM func tests under py3
The services themselves were already runing under py3. I think we've
done about all of the py2-client-can-talk-to-py3-services testing we
need to.

Change-Id: I459d6065a3e4290768c499e7973917f7cf4ce2e7
2020-12-07 15:18:59 -08:00
Tim Burke
232cb85cb9 Pin Bandit on py2
...and, since the previous tag didn't have the Bandit pin, make the
rolling upgrade job non-voting. We should plan on backporting this so we
can check that upgrades from stable branches are still OK.

See also: https://github.com/PyCQA/bandit/issues/654

Change-Id: If7f3ad8b275271d748426133232ed06c2a1cd1de
2020-12-07 15:18:50 -08:00
Tim Burke
711fb302b8 gate: Add victoria rolling upgrade job
Change-Id: I6ca407b86658191b2db1e806d284cdd6348c86e1
2020-11-09 13:25:19 -08:00
Tim Burke
41335f7b47 Revert "Make rolling-upgrade job non-voting"
This reverts commit 63e6c91ab86f3a832d7af9dc081ee90b73d8fa04.

Change-Id: I1af54679c48474d3aa375c5d1192871e9bb16c7d
2020-09-19 09:05:25 -07:00
Tim Burke
318c1a5660 Run swift-tox-func-encryption-py37 job in the gate
This used to be the name of the py2 job, but now even that's gone;
I'm not sure how Zuul let us get away with this.

Change-Id: I1a2f22d592fc7245e40c645000026e4ba1fca528
Related-Change: Ia9ae0fc226dfc9b40157faebac100c10a9180c62
2020-09-17 15:46:52 -07:00
Ghanshyam Mann
829a0d9a70 [goal] Migrate testing to ubuntu focal
As per victoria cycle testing runtime and community goal[1]
we need to migrate upstream CI/CD to Ubuntu Focal(20.04).

Moving py38 job to focal.

Story: #2007865
Task: #40221

[1] https://governance.openstack.org/tc/goals/selected/victoria/migrate-ci-cd-jobs-to-ubuntu-focal.h>

Change-Id: I2a1ce0bda37a4764f7e56a9c15a283eefa46e17b
2020-08-17 14:17:00 +00:00