Replace github by opendev because currently opendev is the source and
github is its mirror.
Also, update links for repositories managed by SwiftStack organization.
Unfortunately some repositories are no longer available so are removed
from the list.
Change-Id: Ic223650eaf7a1934f489c8b713c6d8da1239f3c5
The swauth project is already retired[1]. The documentation is updated
to reflect status of the project.
Also, this change removes reference to this middleware in unit tests.
[1] https://opendev.org/x/swauth/
Change-Id: I3d8e46d85ccd965f9b51006c330e391dcdc24a34
When the versioning is enabled (or suspended), AWS specifies
in the error message that all versions should be deleted.
Change-Id: I3da9469a5cfed031a2cee85e1dfcd78bbe54695a
As part of that, the ceph test runner needed up-rev'ing to run under
py3. As a result, the known-failures shifted.
Trim the on-demand rolling upgrade jobs list -- now that it's running
py3, we only expect it to pass for train and beyond.
Also, pin smmap version on py2 -- otherwise, the remaining experimental
jobs running on centos-7 fail.
Change-Id: Ibe46aecf0f4461be59eb206bfe9063cc1bfff706
We have some purposely brittle ring tests that test rebalances that are
PY2 only. This was because the random seeding between PY2 and PY3 is
different. Now that we're stepping towards py3 only, these tests needed
to be migrated so we don't loose important test coverage.
This patch flips these PY2 only tests to be PY3 only, for the same
reason as before. It took quite a bit of effort to find seeds that would
behave like the ones in the PY2 birttle test.. but found them.
Change-Id: I0978041830ed3e231476719efed66bf9b337ff8a
We've seen a disk corruption take place that throws -ENODATA when a diskfile
it opened. We currently don't quarantine on this IOError.
This patch catches and quarantines the datafile. The catch happens in
diskfile.open in order to group similar quarantines in the same place
for easier maintenance.
Change-Id: I729e3ba5f19160aa09dd8eb983d9594cb102e85b
When deleting multiple objects, S3 allows to enable
a quiet mode with the 'Quiet' key.
At AWS S3, the value of this key is case-insensitive.
- Quiet mode is enabled if the value is 'true'
(regardless of case).
- Otherwise, in all other cases (even a non-boolean value),
this mode will be disabled.
Also, some tools (like Minio's python API) send the value 'True'
(and not 'true').
Change-Id: Id9d1da2017b8d13242ae1f410347febb013e9ce1
Add file to the reno documentation build to show release notes for
stable/yoga.
Use pbr instruction to increment the minor version number
automatically so that master versions are higher than the versions on
stable/yoga.
Sem-Ver: feature
Change-Id: I267907b99364ec0450460012a3c1b343269db33e
The current implementation of s3 signature calculation
rely on WSGI Url encoding which is discouraged by AWS:
https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-header-based-auth.html.
This leads to reject requests with valid signature.
This update encode only characters specified by AWS except
'A'-'Z', 'a'-'z', '0'-'9', '-', '.', '_', and '~' to comply
AWS signature calculation.
Fixes LP Bug #1961841
Change-Id: Ifa8f94544224c3379e7f2805f6f86d0b0a47279a
Otherwise, multiple frontends attempting to use the same data may get
denials because the browser served a cached response from when it used a
different origin.
Change-Id: I6ec8b8ceb8c6a58e74772e57e6fe5700f6ff8db1
Specifically, parameters that may contain non-ASCII characters,
such as the prefix and marker to list current uploads.
Change-Id: Icfae68825f94ddf2412c0274c3d500e265117e8e
Previously, objects updates that could not be sent immediately due to
per-container/bucket ratelimiting [1] would be skipped and re-tried
during the next updater cycle. There could potentially be a period of
time at the end of a cycle when the updater slept, having completed a
sweep of the on-disk async pending files, despite having skipped
updates during the cycle. Skipped updates would then be read from disk
again during the next cycle.
With this change the updater will defer skipped updates to an
in-memory queue (up to a configurable maximum number) until the sweep
of async pending files has completed, and then trickle out deferred
updates until the cycle's interval expires. This increases the useful
work done in the current cycle and reduces the amount of repeated disk
IO during the next cycle.
The deferrals queue is bounded in size and will evict least recently
read updates in order to accept more recently read updates. This
reduces the probablility that a deferred update has been made obsolete
by newer on-disk async pending files while waiting in the deferrals
queue.
The deferrals queue is implemented as a collection of per-bucket
queues so that updates can be drained from the queues in the order
that buckets cease to be ratelimited.
[1] Related-Change: Idef25cd6026b02c1b5c10a9816c8c6cbe505e7ed
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Co-Authored-By: Matthew Oliver <matt@oliver.net.au>
Change-Id: I95e58df9f15c5f9d552b8f4c4989a474f52262f4
Move most of the method to a helper so that there is a place that
receives the multiple return points, before finally returning the
result.
Change-Id: I48e18beeb32372a287e8297f0870e8627147141f
The sharder aborts cleaving shards if the root DB does not have an
own_shard_range (see Related-Change). Previously, the cleaving was
aborted *after* a shard broker had been created, which potentially
leaves an unused shard DB in a handoff location. This shard DB would
eventually be replicated and cleaned up by the replicator, but its
creation can be avoided by checking the root DB own_shard_range
earlier.
Change-Id: I05ec542faa452db2053302910d34b93f0abd10a7
Related-Change: I68c3e7558d04e8dcd5874f80b4763ec0b820331d
This was added while debugging issues with the FIPS jobs, and
we already have a swift-tox-func-encryption-py36-centos-8-stream
job in the experimental pipeline.
Change-Id: I77178aa7def0e02e7b7add662135d62ed7f8c697
Whenever an item is set which is larger than item_size_warning_threshold
then a warning is logged in the form:
'Item size larger than warning threshold: 2048576 (2Mi) >= 1000000 (977Ki)'
Setting the value to -1 (default) will turn off the warning.
Change-Id: I1fb50844d6b9571efaab8ac67705b2fc1fe93e25
swift-ring-builder has three exit statuses: 0 (OK), 1 (WARNING),
2 (ERROR). Uncaught exceptions in python result in an exit code of 1,
so for example problems writing a builder file to disk will result in
an exit of 1 (warning) rather than 2 (error).
This addresses that by overriding sys.excepthook to produce the usual
backtrace and then exit 2 (error); excepthook is called when an
exception is unhandled, unless that is SystemExit.
Closes-Bug: 1960657
Change-Id: I7cfeff4f436ade319cf21d0d29853931aef6d20f