Commit Graph

9739 Commits

Author SHA1 Message Date
Tim Burke
118cf2ba8a tempurl: Deprecate sha1 signatures
We've known this would eventually be necessary for a while [1], and
way back in 2017 we started seeing SHA-1 collisions [2].

[1] https://www.schneier.com/blog/archives/2012/10/when_will_we_se.html
[2] https://security.googleblog.com/2017/02/announcing-first-sha1-collision.html

UpgradeImpact:
==============
"sha1" has been removed from the default set of `allowed_digests` in the
tempurl middleware config. If your cluster still has clients requiring
the use of SHA-1,

- explicitly configure `allowed_digests` to include "sha1" and
- encourage your clients to move to more-secure algorithms.

Depends-On: https://review.opendev.org/c/openstack/tempest/+/832771
Change-Id: I6e6fa76671c860191a2ce921cb6caddc859b1066
Related-Change: Ia9dd1a91cc3c9c946f5f029cdefc9e66bcf01046
Closes-Bug: #1733634
2022-04-22 20:43:01 +10:00
Zuul
b621a6f932 Merge "tests: Fix swiftclient/requests log level adjustment" 2022-04-22 02:31:42 +00:00
Zuul
e2beaefd7b Merge "replicator: Tolerate ENOENT when calling listdir" 2022-04-21 06:11:45 +00:00
Zuul
cd84e4ead8 Merge "DB: Encode the device to the DB id" 2022-04-21 06:11:42 +00:00
Zuul
41475d20b7 Merge "Stop partial()ing hashlib.new" 2022-04-21 05:54:23 +00:00
Zuul
177f60e318 Merge "cors: Include Vary: Origin when using the request's Origin" 2022-04-21 05:53:56 +00:00
Tim Burke
0bf5474bfa ceph tests: Register output/ceph-s3-summary.log as a job output
Change-Id: I7eed34ab19a97b2c561e4aac8c95a8a5e1689266
2022-04-19 12:16:33 -07:00
Takashi Kajinami
bb220f6f41 Doc: Update links in associated projects
Replace github by opendev because currently opendev is the source and
github is its mirror.

Also, update links for repositories managed by SwiftStack organization.
Unfortunately some repositories are no longer available so are removed
from the list.

Change-Id: Ic223650eaf7a1934f489c8b713c6d8da1239f3c5
2022-04-19 20:03:06 +09:00
Takashi Kajinami
5a272421d0 Swauth is retired
The swauth project is already retired[1]. The documentation is updated
to reflect status of the project.

Also, this change removes reference to this middleware in unit tests.

[1] https://opendev.org/x/swauth/
Change-Id: I3d8e46d85ccd965f9b51006c330e391dcdc24a34
2022-04-19 19:50:23 +09:00
Aymeric Ducroquetz
179fc43eb5 s3api: Improve error message when bucket is not empty
When the versioning is enabled (or suspended), AWS specifies
in the error message that all versions should be deleted.

Change-Id: I3da9469a5cfed031a2cee85e1dfcd78bbe54695a
2022-04-15 15:36:56 +02:00
Tim Burke
6142ce88cc s3api: Use constant-time string comparisons in check_signature
Change-Id: Ibe514a7ab22d475517b1efc50de676f47d741a4c
2022-04-13 15:42:30 -07:00
Matthew Oliver
1cee51d526 doc: also add reverse option to pagination doc
Change-Id: I4ee5a52ec9fb5f1920cd6869f6b1245c3787391c
2022-04-08 12:59:27 +10:00
Tim Burke
5c3bf6d26a replicator: Tolerate ENOENT when calling listdir
Change-Id: Iac30e0bc383544809c7b761023924dbf55c62940
2022-04-07 17:10:34 -07:00
Tim Burke
a5a98d7e3b tests: Fix swiftclient/requests log level adjustment
Gotta update now that swiftclient changed how it imports and uses
requests.

Related-Change: https://review.opendev.org/c/openstack/python-swiftclient/+/828821
Change-Id: I0e0f802fa355060f43f9e63f52897fbcf66816d2
2022-04-07 14:47:14 -07:00
Tim Burke
d496d03b7f api-ref: Document reverse param
Change-Id: I731ff275e14fbb0271ddb50f91550197d6d0a4a4
2022-04-07 12:45:07 -07:00
Zuul
3d7ff12064 Merge "Obj Auditor: Quarantine ENODATA" 2022-04-05 21:16:08 +00:00
Zuul
91a8e2c89d Merge "Ring: Change py2 only tests to py3" 2022-04-05 20:27:13 +00:00
Tim Burke
d29cbc3996 CI: Run ceph and rolling upgrade tests under py3
As part of that, the ceph test runner needed up-rev'ing to run under
py3. As a result, the known-failures shifted.

Trim the on-demand rolling upgrade jobs list -- now that it's running
py3, we only expect it to pass for train and beyond.

Also, pin smmap version on py2 -- otherwise, the remaining experimental
jobs running on centos-7 fail.

Change-Id: Ibe46aecf0f4461be59eb206bfe9063cc1bfff706
2022-04-04 17:17:06 -07:00
Matthew Oliver
fce7ad5f18 Ring: Change py2 only tests to py3
We have some purposely brittle ring tests that test rebalances that are
PY2 only. This was because the random seeding between PY2 and PY3 is
different. Now that we're stepping towards py3 only, these tests needed
to be migrated so we don't loose important test coverage.

This patch flips these PY2 only tests to be PY3 only, for the same
reason as before. It took quite a bit of effort to find seeds that would
behave like the ones in the PY2 birttle test.. but found them.

Change-Id: I0978041830ed3e231476719efed66bf9b337ff8a
2022-03-31 15:10:32 +11:00
Matthew Oliver
f92be1bdd0 Obj Auditor: Quarantine ENODATA
We've seen a disk corruption take place that throws -ENODATA when a diskfile
it opened. We currently don't quarantine on this IOError.

This patch catches and quarantines the datafile. The catch happens in
diskfile.open in order to group similar quarantines in the same place
for easier maintenance.

Change-Id: I729e3ba5f19160aa09dd8eb983d9594cb102e85b
2022-03-31 08:14:52 +11:00
Zuul
ec964b23bb Merge "s3api: Copy more headers from MPU marker to final object" 2022-03-27 21:16:46 +00:00
Tim Burke
1c4acf2d8f s3api: Copy more headers from MPU marker to final object
Closes-Bug: 1966396
Change-Id: I253d8e3e8678fad3fde43259ed3225df4048a458
2022-03-25 09:49:11 +00:00
Aymeric Ducroquetz
fd2dd11562 s3api: Make the 'Quiet' key value case insensitive
When deleting multiple objects, S3 allows to enable
a quiet mode with the 'Quiet' key.

At AWS S3, the value of this key is case-insensitive.
- Quiet mode is enabled if the value is 'true'
  (regardless of case).
- Otherwise, in all other cases (even a non-boolean value),
  this mode will be disabled.

Also, some tools (like Minio's python API) send the value 'True'
(and not 'true').

Change-Id: Id9d1da2017b8d13242ae1f410347febb013e9ce1
2022-03-24 20:22:21 +01:00
Zuul
0651d8175d Merge "trivial: Replace assertRegexpMatches with assertRegex" 2022-03-24 00:33:27 +00:00
Zuul
7ec2036db9 Merge "trivial: Replace assertRaisesRegexp with assertRaisesRegex" 2022-03-24 00:28:36 +00:00
Zuul
d82f8af6b3 Merge "CI: Run CORS tests under py3" 2022-03-23 19:15:18 +00:00
Tim Burke
ffb173f8ad CI: Run CORS tests under py3
Change-Id: I49d8480fd0c775249d8b586f9dea026448b8910d
2022-03-22 23:12:01 -07:00
f83bfe1df9 Update master for stable/yoga
Add file to the reno documentation build to show release notes for
stable/yoga.

Use pbr instruction to increment the minor version number
automatically so that master versions are higher than the versions on
stable/yoga.

Sem-Ver: feature
Change-Id: I267907b99364ec0450460012a3c1b343269db33e
2022-03-21 13:20:37 +01:00
Tim Burke
471a559a4d Stop partial()ing hashlib.new
Seems to be causing issues on (some?) FIPS nodes.

Change-Id: I4fcacc76ab52b45510aee82156a7cbf6ad72467d
2022-03-16 13:53:17 -07:00
Tim Burke
954032d5d2 CHANGELOG for 2.29.1
Change-Id: I37316d8e40e1532d5189535fced4fb95f7323014
2022-03-16 12:46:42 -07:00
Thibault Person
cb8b3cdab2 Comply with AWS signature calculation (s3v4)
The current implementation of s3 signature calculation
rely on WSGI Url encoding which is discouraged by AWS:
https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-header-based-auth.html.
This leads to reject requests with valid signature.

This update encode only characters specified by AWS except
'A'-'Z', 'a'-'z', '0'-'9', '-', '.', '_', and '~' to comply
AWS signature calculation.

Fixes LP Bug #1961841

Change-Id: Ifa8f94544224c3379e7f2805f6f86d0b0a47279a
2022-03-15 15:25:02 -07:00
Tim Burke
bab7f93223 cors: Include Vary: Origin when using the request's Origin
Otherwise, multiple frontends attempting to use the same data may get
denials because the browser served a cached response from when it used a
different origin.

Change-Id: I6ec8b8ceb8c6a58e74772e57e6fe5700f6ff8db1
2022-03-09 22:18:55 -08:00
Zuul
014c98e853 Merge "s3api: Fix multi_delete with object names using non-ASCII characters" 2022-03-10 03:46:16 +00:00
Zuul
7ac2b2eb76 Merge "s3api: Delete all parts when aborting MPU with non-ASCII characters" 2022-03-09 23:54:05 +00:00
Aymeric Ducroquetz
82ca37517d s3api: Delete all parts when aborting MPU with non-ASCII characters
Change-Id: Idcda76f7a880a18c3bac699e0fb2435e4a54abbd
2022-03-09 11:57:29 -08:00
Aymeric Ducroquetz
dd64a81e65 s3api: Fix multi_delete with object names using non-ASCII characters
Co-Authored-By: Florent Vennetier <florent.vennetier@ovhcloud.com>
Change-Id: I635bc91faa7709f9df9cdf3aec157a21c08923ca
2022-03-01 13:47:35 +01:00
Aymeric Ducroquetz
5b3ec5aa64 s3api: Properly decode MPU request parameters before using them
Specifically, parameters that may contain non-ASCII characters,
such as the prefix and marker to list current uploads.

Change-Id: Icfae68825f94ddf2412c0274c3d500e265117e8e
2022-03-01 09:13:21 +01:00
Zuul
3ff3076ce6 Merge "db: Attempt to clean up part dir post replication" 2022-03-01 00:03:51 +00:00
Zuul
7fc522743b Merge "expirer: Only try to delete empty containers" 2022-02-26 08:28:38 +00:00
Tim Burke
cd7159c69c db: Attempt to clean up part dir post replication
Change-Id: Id844f47ead7fab47915b6db76c06155b22586efe
2022-02-22 16:05:28 -08:00
Zuul
c7774d960c Merge "object-updater: defer ratelimited updates" 2022-02-22 07:38:37 +00:00
Alistair Coles
51da2543ca object-updater: defer ratelimited updates
Previously, objects updates that could not be sent immediately due to
per-container/bucket ratelimiting [1] would be skipped and re-tried
during the next updater cycle. There could potentially be a period of
time at the end of a cycle when the updater slept, having completed a
sweep of the on-disk async pending files, despite having skipped
updates during the cycle. Skipped updates would then be read from disk
again during the next cycle.

With this change the updater will defer skipped updates to an
in-memory queue (up to a configurable maximum number) until the sweep
of async pending files has completed, and then trickle out deferred
updates until the cycle's interval expires. This increases the useful
work done in the current cycle and reduces the amount of repeated disk
IO during the next cycle.

The deferrals queue is bounded in size and will evict least recently
read updates in order to accept more recently read updates. This
reduces the probablility that a deferred update has been made obsolete
by newer on-disk async pending files while waiting in the deferrals
queue.

The deferrals queue is implemented as a collection of per-bucket
queues so that updates can be drained from the queues in the order
that buckets cease to be ratelimited.

[1] Related-Change: Idef25cd6026b02c1b5c10a9816c8c6cbe505e7ed

Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Co-Authored-By: Matthew Oliver <matt@oliver.net.au>
Change-Id: I95e58df9f15c5f9d552b8f4c4989a474f52262f4
2022-02-21 10:56:23 +00:00
Alistair Coles
16a3067746 sharder: refactor _cleave_shard_range
Move most of the method to a helper so that there is a place that
receives the multiple return points, before finally returning the
result.

Change-Id: I48e18beeb32372a287e8297f0870e8627147141f
2022-02-17 14:36:46 +00:00
Alistair Coles
6609fcd9a1 sharder: don't cleave DB if own_shard_range missing
The sharder aborts cleaving shards if the root DB does not have an
own_shard_range (see Related-Change). Previously, the cleaving was
aborted *after* a shard broker had been created, which potentially
leaves an unused shard DB in a handoff location. This shard DB would
eventually be replicated and cleaned up by the replicator, but its
creation can be avoided by checking the root DB own_shard_range
earlier.

Change-Id: I05ec542faa452db2053302910d34b93f0abd10a7
Related-Change: I68c3e7558d04e8dcd5874f80b4763ec0b820331d
2022-02-17 13:06:19 +00:00
Tim Burke
3358345682 CI: remove swift-tox-func-encryption-py36-centos-8 job
This was added while debugging issues with the FIPS jobs, and
we already have a swift-tox-func-encryption-py36-centos-8-stream
job in the experimental pipeline.

Change-Id: I77178aa7def0e02e7b7add662135d62ed7f8c697
2022-02-16 11:56:50 -08:00
Zuul
713b944eb4 Merge "Proxy: override user_agent with backend_user_agent" 2022-02-16 02:13:33 +00:00
Zuul
f15c4cc6b5 Merge "swift-ring-builder: exit ERROR (2) on uncaught exceptions" 2022-02-15 21:41:44 +00:00
Zuul
36d32b907c Merge "memcache: Add an item_size_warning_threshold option" 2022-02-15 21:41:40 +00:00
Matthew Oliver
05d83b0a47 memcache: Add an item_size_warning_threshold option
Whenever an item is set which is larger than item_size_warning_threshold
then a warning is logged in the form:

  'Item size larger than warning threshold: 2048576 (2Mi) >= 1000000 (977Ki)'

Setting the value to -1 (default) will turn off the warning.

Change-Id: I1fb50844d6b9571efaab8ac67705b2fc1fe93e25
2022-02-15 16:54:17 +00:00
Matthew Vernon
89ee843080 swift-ring-builder: exit ERROR (2) on uncaught exceptions
swift-ring-builder has three exit statuses: 0 (OK), 1 (WARNING),
2 (ERROR). Uncaught exceptions in python result in an exit code of 1,
so for example problems writing a builder file to disk will result in
an exit of 1 (warning) rather than 2 (error).

This addresses that by overriding sys.excepthook to produce the usual
backtrace and then exit 2 (error); excepthook is called when an
exception is unhandled, unless that is SystemExit.

Closes-Bug: 1960657
Change-Id: I7cfeff4f436ade319cf21d0d29853931aef6d20f
2022-02-15 14:29:14 +00:00