I wanna see how far I can get *without* mucking around in swob and
request_helpers. Maybe eventually we can get some helpers out there to
make working with UTF-8-pretending-to-be-Latin-1 strings better, but for
the time being, I feel more at ease *embracing* the crazy.
Change-Id: I0b9983a182daedd9dbec483b805d263238fcfac7
Co-Authored-By: Pete Zaitcev <zaitcev@kotori.zaitcev.us>
User can cofigure KEEPIDLE time for sockets in TCP connection.
The default value is the old value which is 600.
Change-Id: Ib7fb166deb8a87ae4e97ba0671048b1ec079a2ef
Closes-Bug:1759606
This is useful for deallocating disk blocks as part of an alternate disk
file implementation.
Additionally, add an offset argument to the existing fallocate utility
function; this allows you to grow an existing file.
Sam always had the best descriptions:
utils.fallocate(fd, size) allocates <size> bytes for the file referred
to by <fd>. It allows for keeping a reserve of an additional N bytes
or X% of the filesystem free. If neither fallocate() or
posix_fallocate() C functions are avaialble, utils.fallocate() will
log a warning (once only) and not actually allocate space.
utils.punch_hole(fd, offset, length) deallocates <length> bytes
starting at <offset> from the file referred to by <fd>. It uses the C
function fallocate(). If fallocate() is not available, calls to
utils.punch_hole() will raise an exception.
Since these both use the fallocate syscall, refactor that a bit and get
rid of FallocateWrapper. We add a new _LibcWrapper to do some
lazy-loading of a C function and expose whether the function is actually
available in Python, though. This allows utils.fallocate and
utils.punch_hole to keep their fancy logic pretty well-contained.
Modernized the tests for utils.fallocate() and utils.punch_hole().
Co-Authored-By: Samuel Merritt <sam@swiftstack.com>
Change-Id: Ieac30a477d784905c94742ee3d0898d7e0194b39
Multipart uploads in AWS (seem to) have ETags like:
'"' + MD5_hex(MD5(part1) + ... + MD5(partN)) + '-' + N + '"'
On the other hand, Swift SLOs have Etags like:
MD5_hex(MD5_hex(part1) + ... + MD5_hex(partN))
(In both examples, MD5 gets the raw 16-byte digest while MD5_hex
gets the 32-byte hex-encoded digest.)
Some clients (such as aws-sdk-java) use the presence of a dash
to decide whether to perform client-side validation of downloads.
Other clients (like s3cmd) use the presence of a dash *in bucket
listings* to decide whether or not to perform additional HEAD requests
to look for MD5 metadata that can be used to compare against the MD5s
of local files.
Now we include a dash as well, to prevent spurious errors like
> Unable to verify integrity of data download. Client calculated
> content hash didn't match hash calculated by Amazon S3. The data
> may be corrupt.
or unnecessary uploads/downloads because the client assumes data has
changed that hasn't.
For new multipart-uploads via the S3 API, the ETag that is stored will
be calculated in the same way that AWS uses. This ETag will be used in
GET/HEAD responses, bucket listings, and conditional requests via the S3
API. Accessing the same object via the Swift API will use the SLO Etag;
however, in JSON container listings the multipart upload etag will be
exposed in a new "s3_etag" key.
New SLOs and pre-existing multipart-uploads will continue to behave as
before; there is no data migration or mitigation as part of this patch.
Change-Id: Ibe68c44bef6c17605863e9084503e8f5dc577fab
Closes-Bug: 1522578
Almost one year has passed since all post_as_copy related code
removed by [1], we don't have to keep the warning message for
post_as_copy setting anymore in the code tree.
1: 1e79f828ad10918bd76ae8df6fe4c4dbf7bbf3c5
Change-Id: Id9eea22ed688574d84ca582584c0c207d5f01383
While s3api is the preferred method going forward, it would be nice to
allow for backwards compatibility with swift3 in the tempauth
middleware.
Change-Id: I4fd4772b1d7b173ee0faf72d1c1f1531646dde8b
I've at least tried it out with a py3 proxy, and it seems to work out
OK. I haven't tried killing the socket and verifying that it's actualy
dead, but getting a hold of _real_close *seems like* what we want?
At least the three (!!) tests pass.
Change-Id: Ic08c26185d63a36a5422793d81f621e0698fa572
This is comparable to what AWS returns, and should greatly simplify
debugging when diagnosing 403s.
Change-Id: Iabfcbaae919598e22f39b2dfddac36b75653fc10
Now that the trivial keymaster supports multiple keys, let's do
something similar for the KMIP keymaster. Additional keys are
configured as:
key_id_<secret_id> = <KMIP unique identifier>
While it might be tempting to use the unique identifier directly as the
secret_id, the added indirection allows operators to move keys between
different backends, which may cause different identifiers to be issued.
As with the trivial keymaster, the key to use for PUTs and POSTs is
specified with:
active_root_secret_id = <secret_id>
Change-Id: Ie52508e47d15ec5c4e96902d3c9f5f282d275683
For some use cases operators would like to periodically introduce a
new encryption root secret that would be used when new object data is
written. However, existing encrypted data does not need to be
re-encrypted with keys derived from the new root secret. Older root
secret(s) would still be used as necessary to decrypt older object
data.
This patch modifies the KeyMaster class to support multiple root
secrets indexed via unique secret_id's, and to store the id of the
root secret used for an encryption operation in the crypto meta. The
decrypter is modified to fetch appropriate keys based on the secret id
in retrieved crypto meta.
The changes are backwards compatible with previous crypto middleware
configurations and existing encrypted object data.
Change-Id: I40307acf39b6c1cc9921f711a8da55d03924d232
On vanilla Swift, deleting an object that doesn't exist will 404.
On AWS, deleting a key that doesn't exist will either 404 if the bucket
doesn't exist (with a NoSuchBucket code) or 204 (because yep, that's not
accessible).
Change-Id: Ied2a78b56522316bb374f23961621641af3adc83
Related-Change: I6e154594dfda6c3065774c23b24f728625a842bc
We don't support it yet, so return 501 Not Implemented. Previously, we'd
store the aws-chunked content (!) and most clients would see it as data
corruption.
See https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html
for more information.
Change-Id: I697962039667980ef89212bc480f8b1d3fbd718c
The object server can be configured to leave a certain amount of disk
space free; default is 1%. This is useful in avoiding 100%-full
filesystems, as those can get Swift in a state where the filesystem is
too full to write tombstones, so you can't delete objects to free up
space.
When a cluster has accounts/containers and objects on the same disks,
then you can wind up with a 100%-full disk since account and container
servers don't respect fallocate_reserve. This commit makes account and
container servers respect fallocate_reserve so that disks shared
between account/container and object rings won't get 100% full.
When a disk's free space falls below the configured reserve, account
and container PUT, POST, and REPLICATE requests will fail with a 507
status code. These are the operations that can significantly increase
the disk space used by a given database.
I called the parameter "fallocate_reserve" for consistency with the
object server. No actual fallocate() call happens under Swift's
control in the account or container servers (sqlite3 might make such a
call, but it's out of our hands).
Change-Id: I083442eef14bf83c0ea717b1decb3e6b56dbf1d0
Container servers will store an etag like
<MD5 of manifest on disk>; slo_etag=<MD5 on concatenated ETags>
which the SLO middleware will break out into separate
"hash": "<MD5 of manifest on disk",
"slo_etag": "\"<MD5 of concatenated ETags\"",
keys for JSON listings. Text and XML listings are unaffected.
If a middleware left of SLO already specified a container update
override, the slo_etag parameter will be appended. If the base header
value was blank, the MD5 of the manifest will be inserted.
SLOs that were created on previous versions of Swift will continue to
just have the MD5 of the manifest in container listings.
Closes-Bug: 1618573
Change-Id: I67478923619b00ec1a37d56b6fec6a218453dafc
This patch adds an additional optional parameter to tempurl
which restricts the ip's from which a temp url can be used from.
Change-Id: I23fe998a980960d4a32df042b3f6a21f096c36af
Add a new middleware that can be used to fetch an encryption root
secret from a KMIP service. The middleware uses a PyKMIP client
to interact with a KMIP endpoint. The middleware is configured with
a unique identifier for the key to be fetched and options required
for the PyKMIP client.
Co-Authored-By: Tim Burke <tim.burke@gmail.com>
Change-Id: Ib0943fb934b347060fc66c091673a33bcfac0a6d
Instead, require that callers provide an encoding.
Related-Change: I31408f525ba9836f634a35581d4aee6fa2c9428f
Change-Id: I3e5ed9e4401eea76c375bb43ad4afc58b1d8006a
If we're going to drop the quotes during initialization, we should
probably drop them when making comparisons, too.
Change-Id: I90c6e8a7c6faf8fafb0cd64fabbc9629d6c4c48a