3657 Commits

Author SHA1 Message Date
Zuul
ec66c6cbbe Merge "py3: Port listing_formats middleware" 2018-09-25 20:13:28 +00:00
Zuul
fc9ab28927 Merge "py3: port request_helpers" 2018-09-25 20:01:10 +00:00
Tim Burke
1b0172f5d3 py3: Port listing_formats middleware
Change-Id: I4e2712a4a38b5cab3e31cc75bb162e73b4f5dafb
2018-09-17 13:08:12 -07:00
Alistair Coles
904e7c97f1 Add more doc and test for cors_expose_headers option
In follow-up to the related change, mention the new
cors_expose_headers option (and other proxy-server.conf
options) in the CORS doc.

Add a test for the cors options being loaded into the
proxy server.

Improve CORS comments in docs.

Change-Id: I647d8f9e9cbd98de05443638628414b1e87d1a76
Related-Change: I5ca90a052f27c98a514a96ee2299bfa1b6d46334
2018-09-17 12:35:25 -07:00
Tim Burke
2ef21ac05d py3: port request_helpers
Change-Id: I6be1a1c618e4b4fa03b34dad96f378aca01e8e08
2018-09-15 01:33:34 -06:00
Zuul
5d46c0d8b3 Merge "Adding keep_idle config value to socket" 2018-09-15 00:43:52 +00:00
Zuul
a10a8611e6 Merge "Add punch_hole utility function" 2018-09-15 00:15:38 +00:00
FatemaKhalid
cfeb32c66b Adding keep_idle config value to socket
User can cofigure KEEPIDLE time for sockets in TCP connection.
The default value is the old value which is 600.

Change-Id: Ib7fb166deb8a87ae4e97ba0671048b1ec079a2ef
Closes-Bug:1759606
2018-09-15 01:30:53 +02:00
Alexandre Lécuyer
dbacdcf01c Add punch_hole utility function
This is useful for deallocating disk blocks as part of an alternate disk
file implementation.

Additionally, add an offset argument to the existing fallocate utility
function; this allows you to grow an existing file.

Sam always had the best descriptions:

  utils.fallocate(fd, size) allocates <size> bytes for the file referred
  to by <fd>. It allows for keeping a reserve of an additional N bytes
  or X% of the filesystem free. If neither fallocate() or
  posix_fallocate() C functions are avaialble, utils.fallocate() will
  log a warning (once only) and not actually allocate space.

  utils.punch_hole(fd, offset, length) deallocates <length> bytes
  starting at <offset> from the file referred to by <fd>. It uses the C
  function fallocate(). If fallocate() is not available, calls to
  utils.punch_hole() will raise an exception.

Since these both use the fallocate syscall, refactor that a bit and get
rid of FallocateWrapper. We add a new _LibcWrapper to do some
lazy-loading of a C function and expose whether the function is actually
available in Python, though. This allows utils.fallocate and
utils.punch_hole to keep their fancy logic pretty well-contained.

Modernized the tests for utils.fallocate() and utils.punch_hole().

Co-Authored-By: Samuel Merritt <sam@swiftstack.com>
Change-Id: Ieac30a477d784905c94742ee3d0898d7e0194b39
2018-09-14 13:55:42 -06:00
Samuel Merritt
114220279b py3: port list_endpoints
Change-Id: Ic6d0532f9f42b4d57a12aafeec5d34baffab6e23
2018-09-14 13:53:06 -06:00
Clay Gerrard
52ecbf9539 Add a chunks_finished to BaseDiskFileWriter
BaseDiskFileWriter will track md5 and expose upload_size, etag via the
chunks_finished method.

The BaseDiskFileReader already tracks the md5/etag via _iter_etag, for
parity we add a _chunks_etag to the BaseDiskFileReader.

Instead of returning the upload_size and hexdigest every call to write,
we return the tuple from chunks_finished.

Change-Id: I26c58719cff5fde941d0248c250a0204e0379ae5
2018-09-13 12:28:57 -05:00
Zuul
29f71c9119 Merge "s3api: Include '-' in multipart ETags" 2018-09-13 11:30:32 +00:00
Tim Burke
84b85f03b4 s3api: Include '-' in multipart ETags
Multipart uploads in AWS (seem to) have ETags like:

   '"' + MD5_hex(MD5(part1) + ... + MD5(partN)) + '-' + N + '"'

On the other hand, Swift SLOs have Etags like:

   MD5_hex(MD5_hex(part1) + ... + MD5_hex(partN))

(In both examples, MD5 gets the raw 16-byte digest while MD5_hex
gets the 32-byte hex-encoded digest.)

Some clients (such as aws-sdk-java) use the presence of a dash
to decide whether to perform client-side validation of downloads.

Other clients (like s3cmd) use the presence of a dash *in bucket
listings* to decide whether or not to perform additional HEAD requests
to look for MD5 metadata that can be used to compare against the MD5s
of local files.

Now we include a dash as well, to prevent spurious errors like

> Unable to verify integrity of data download.  Client calculated
> content hash didn't match hash calculated by Amazon S3.  The data
> may be corrupt.

or unnecessary uploads/downloads because the client assumes data has
changed that hasn't.

For new multipart-uploads via the S3 API, the ETag that is stored will
be calculated in the same way that AWS uses. This ETag will be used in
GET/HEAD responses, bucket listings, and conditional requests via the S3
API. Accessing the same object via the Swift API will use the SLO Etag;
however, in JSON container listings the multipart upload etag will be
exposed in a new "s3_etag" key.

New SLOs and pre-existing multipart-uploads will continue to behave as
before; there is no data migration or mitigation as part of this patch.

Change-Id: Ibe68c44bef6c17605863e9084503e8f5dc577fab
Closes-Bug: 1522578
2018-09-13 19:28:59 +09:00
Tim Burke
ce257b3d15 DiskFile(Writer) refactor cleanups
Change-Id: I5b0bcc6028dbe6248e0e09baf2cbb72deb011c80
2018-09-12 19:09:12 +00:00
Clay Gerrard
33c7650753 Add writer method to DiskFile
DiskFile already exposes a reader method that creates the DiskFileReader
instance. Add a writer method for parity.

DiskFile currently only provides a context manager create - that will
open and close the DiskFileWriter.  Add explicit open and close methods
to support more flexibility in how callers manage life-cycle on their
DiskFileWriter instances.

Diskfile confusingly manages some state for DiskFileWriter (e.g. fd,
tmppath, use_linkat).  Encapsulate the DiskFileWriter state to improve
readability and reduce coupling (e.g. put_succeeced).

Change-Id: If18e0041680470a9c57a08e9ea9327acba8593df
2018-09-12 10:37:48 -05:00
Kota Tsuyuzaki
b4be37cc25 Remove post_as_copy deprecated warning from copy middleware
Almost one year has passed since all post_as_copy related code
removed by [1], we don't have to keep the warning message for
post_as_copy setting anymore in the code tree.

1: 1e79f828ad10918bd76ae8df6fe4c4dbf7bbf3c5

Change-Id: Id9eea22ed688574d84ca582584c0c207d5f01383
2018-09-10 17:58:26 -06:00
Timur Alperovich
1f29508327 Allow for backwards compatibility with swift3.
While s3api is the preferred method going forward, it would be nice to
allow for backwards compatibility with swift3 in the tempauth
middleware.

Change-Id: I4fd4772b1d7b173ee0faf72d1c1f1531646dde8b
2018-09-05 17:42:34 -07:00
Tim Burke
df954e2709 py3: port bufferedhttp (hopefully)
I've at least tried it out with a py3 proxy, and it seems to work out
OK. I haven't tried killing the socket and verifying that it's actualy
dead, but getting a hold of _real_close *seems like* what we want?

At least the three (!!) tests pass.

Change-Id: Ic08c26185d63a36a5422793d81f621e0698fa572
2018-08-24 11:30:26 +00:00
Zuul
b32578b5d4 Merge "Multi-key KMIP keymaster" 2018-08-21 15:15:05 +00:00
Zuul
ed18495faa Merge "Add debugging info to SignatureDoesNotMatch responses" 2018-08-18 00:54:21 +00:00
Tim Burke
7ca1a67d70 Add debugging info to SignatureDoesNotMatch responses
This is comparable to what AWS returns, and should greatly simplify
debugging when diagnosing 403s.

Change-Id: Iabfcbaae919598e22f39b2dfddac36b75653fc10
2018-08-17 18:03:47 +00:00
Tim Burke
0dc1b6250e Multi-key KMIP keymaster
Now that the trivial keymaster supports multiple keys, let's do
something similar for the KMIP keymaster. Additional keys are
configured as:

    key_id_<secret_id> = <KMIP unique identifier>

While it might be tempting to use the unique identifier directly as the
secret_id, the added indirection allows operators to move keys between
different backends, which may cause different identifiers to be issued.

As with the trivial keymaster, the key to use for PUTs and POSTs is
specified with:

    active_root_secret_id = <secret_id>

Change-Id: Ie52508e47d15ec5c4e96902d3c9f5f282d275683
2018-08-17 17:55:09 +00:00
Alistair Coles
2722e49a8c Add support for multiple root encryption secrets
For some use cases operators would like to periodically introduce a
new encryption root secret that would be used when new object data is
written. However, existing encrypted data does not need to be
re-encrypted with keys derived from the new root secret. Older root
secret(s) would still be used as necessary to decrypt older object
data.

This patch modifies the KeyMaster class to support multiple root
secrets indexed via unique secret_id's, and to store the id of the
root secret used for an encryption operation in the crypto meta. The
decrypter is modified to fetch appropriate keys based on the secret id
in retrieved crypto meta.

The changes are backwards compatible with previous crypto middleware
configurations and existing encrypted object data.

Change-Id: I40307acf39b6c1cc9921f711a8da55d03924d232
2018-08-17 17:54:30 +00:00
Zuul
cfc4f30d63 Merge "s3_acl: Require swift_owner authz to create buckets" 2018-08-16 17:03:27 +00:00
Zuul
a41ca22f46 Merge "Fix the deletion of non-existent keys" 2018-08-15 21:55:08 +00:00
Tim Burke
51b885b3b5 s3_acl: Require swift_owner authz to create buckets
Otherwise, users can create buckets in accounts they don't own.

Change-Id: I13d557c32b12529ef1087c52f7af302a33d33acb
2018-08-15 12:56:33 +00:00
Tim Burke
bd640cdbae Fix the deletion of non-existent keys
On vanilla Swift, deleting an object that doesn't exist will 404.
On AWS, deleting a key that doesn't exist will either 404 if the bucket
doesn't exist (with a NoSuchBucket code) or 204 (because yep, that's not
accessible).

Change-Id: Ied2a78b56522316bb374f23961621641af3adc83
Related-Change: I6e154594dfda6c3065774c23b24f728625a842bc
2018-08-13 14:19:23 -07:00
Tim Burke
5dca610e94 Fix up test_static_web_pretend_to_be_giant_json
It was actually testing the invalid-JSON handling before...

Change-Id: Ia8b5eaeb42fea5136525c80e67e8d33548c2a8df
2018-08-13 20:58:13 +00:00
Zuul
9b19df362f Merge "Disallow uploads with x-amz-tagging headers set" 2018-08-11 01:42:06 +00:00
Zuul
18f6ce92f5 Merge "Disallow aws-chunked uploads" 2018-08-10 19:59:35 +00:00
Tim Burke
7895718ce9 Fix keymmaster_conf_section typo
Change-Id: I3bce1c4efeb3a3a7319020de76ba7f06015a5a36
2018-08-08 17:06:17 +00:00
Christian Schwede
dcbdcb8f1c Disallow uploads with x-amz-tagging headers set
We don't support it yet, so return 501 Not Implemented.

Change-Id: Ie2f4bd1bfdb1bcbdf1a0f0db9d542b6057e9d2ec
2018-08-08 04:57:32 -04:00
Tim Burke
79539eabf7 Disallow aws-chunked uploads
We don't support it yet, so return 501 Not Implemented. Previously, we'd
store the aws-chunked content (!) and most clients would see it as data
corruption.

See https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html
for more information.

Change-Id: I697962039667980ef89212bc480f8b1d3fbd718c
2018-08-03 19:30:01 +00:00
Tim Burke
3c92e3ce87 Move keymaster_config_path parsing out of _get_root_secret
Change-Id: Iddc0f333861b6c1f81e181f006cd592b5eb6ea17
2018-07-30 16:51:24 +00:00
Zuul
00373dad61 Merge "Add keymaster to fetch root secret from KMIP service" 2018-07-25 03:49:50 +00:00
Zuul
89854250c3 Merge "Add fallocate_reserve to account and container servers." 2018-07-20 08:42:51 +00:00
Samuel Merritt
8e651a2d3d Add fallocate_reserve to account and container servers.
The object server can be configured to leave a certain amount of disk
space free; default is 1%. This is useful in avoiding 100%-full
filesystems, as those can get Swift in a state where the filesystem is
too full to write tombstones, so you can't delete objects to free up
space.

When a cluster has accounts/containers and objects on the same disks,
then you can wind up with a 100%-full disk since account and container
servers don't respect fallocate_reserve. This commit makes account and
container servers respect fallocate_reserve so that disks shared
between account/container and object rings won't get 100% full.

When a disk's free space falls below the configured reserve, account
and container PUT, POST, and REPLICATE requests will fail with a 507
status code. These are the operations that can significantly increase
the disk space used by a given database.

I called the parameter "fallocate_reserve" for consistency with the
object server. No actual fallocate() call happens under Swift's
control in the account or container servers (sqlite3 might make such a
call, but it's out of our hands).

Change-Id: I083442eef14bf83c0ea717b1decb3e6b56dbf1d0
2018-07-18 17:27:11 +10:00
Zuul
791009a6ff Merge "PUT+POST: break out putter specific test classes" 2018-07-17 02:38:12 +00:00
Alistair Coles
0cd42a2d33 Check other params preserved when slo_etag is extracted
Change-Id: Ie34ce2a33f2a642b97986fa28cf9db9e6da964d5
Related-Change: I67478923619b00ec1a37d56b6fec6a218453dafc
Related-Change: Ibaa630b5b4251cc4f821c01d3c09a8b8a6be342c
2018-07-12 10:01:58 -05:00
Clay Gerrard
f8b9c24a1c Add unittest for slo_etag
Related-Change-Id: I67478923619b00ec1a37d56b6fec6a218453dafc

Change-Id: Ibaa630b5b4251cc4f821c01d3c09a8b8a6be342c
2018-07-12 10:01:54 -05:00
Tim Burke
c4c98eb64d Include SLO ETag in container updates
Container servers will store an etag like

   <MD5 of manifest on disk>; slo_etag=<MD5 on concatenated ETags>

which the SLO middleware will break out into separate

   "hash": "<MD5 of manifest on disk",
   "slo_etag": "\"<MD5 of concatenated ETags\"",

keys for JSON listings. Text and XML listings are unaffected.

If a middleware left of SLO already specified a container update
override, the slo_etag parameter will be appended. If the base header
value was blank, the MD5 of the manifest will be inserted.

SLOs that were created on previous versions of Swift will continue to
just have the MD5 of the manifest in container listings.

Closes-Bug: 1618573
Change-Id: I67478923619b00ec1a37d56b6fec6a218453dafc
2018-07-10 15:41:29 -07:00
Zuul
1ab691f637 Merge "IP Range restrictions in temp urls" 2018-07-10 04:28:16 +00:00
Alistair Coles
9a7b46e1e3 swift-ring-builder shows hint about composite builder file
If swift-ring-builder is erroneously given a composite builder
file, which it will fail to load, it will now print a hint
that the file is a composite builder file.

Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Change-Id: If4517f3b61977a7f6ca3e08ed5deb182aa87a366
2018-07-05 15:57:05 +01:00
Zuul
44f60d9245 Merge "Address some review comments" 2018-07-05 09:40:47 +00:00
Zuul
cb3692f8db Merge "swob.Match: remove quotes when checking __contains__" 2018-07-04 06:34:17 +00:00
mmcardle
26b20ee729 IP Range restrictions in temp urls
This patch adds an additional optional parameter to tempurl
which restricts the ip's from which a temp url can be used from.

Change-Id: I23fe998a980960d4a32df042b3f6a21f096c36af
2018-07-03 12:25:28 +01:00
Alistair Coles
1951dc7e9a Add keymaster to fetch root secret from KMIP service
Add a new middleware that can be used to fetch an encryption root
secret from a KMIP service. The middleware uses a PyKMIP client
to interact with a KMIP endpoint. The middleware is configured with
a unique identifier for the key to be fetched and options required
for the PyKMIP client.

Co-Authored-By: Tim Burke <tim.burke@gmail.com>
Change-Id: Ib0943fb934b347060fc66c091673a33bcfac0a6d
2018-07-03 09:00:21 +01:00
Pete Zaitcev
91a8cd2952 PUT+POST: break out putter specific test classes
In test_obj.py there are PUT tests that are specific to the Putter
type and others that apply to any Putter type. This patch refactors
them into separate classes to provide greater clarity and to allow
common tests to be applied to each Putter type.

Taking this infrastracture out and ahead of PUT+POST itself allows
it to be reviewed much easier, especially when you look at the diff.

Related-Change-Id: I64b0d8fdb2ffce786f56665a74ed7eb2603abfda
Change-Id: Ibb09b5a28098fb51e25ab5a7134b518cc68eaf89
2018-07-02 23:02:13 -05:00
Zuul
3378a48733 Merge "py3: port proxy/controllers/info.py" 2018-07-02 23:11:05 +00:00
Zuul
e93c6187f6 Merge "swob: Stop auto-encoding unicode bodies" 2018-07-02 02:41:05 +00:00