1973 Commits

Author SHA1 Message Date
Tim Burke
ddd5cc1592 py3: port direct_client
I wanna see how far I can get *without* mucking around in swob and
request_helpers. Maybe eventually we can get some helpers out there to
make working with UTF-8-pretending-to-be-Latin-1 strings better, but for
the time being, I feel more at ease *embracing* the crazy.

Change-Id: I0b9983a182daedd9dbec483b805d263238fcfac7
Co-Authored-By: Pete Zaitcev <zaitcev@kotori.zaitcev.us>
2018-09-25 13:46:59 -07:00
Zuul
ec66c6cbbe Merge "py3: Port listing_formats middleware" 2018-09-25 20:13:28 +00:00
Zuul
fc9ab28927 Merge "py3: port request_helpers" 2018-09-25 20:01:10 +00:00
Tim Burke
1b0172f5d3 py3: Port listing_formats middleware
Change-Id: I4e2712a4a38b5cab3e31cc75bb162e73b4f5dafb
2018-09-17 13:08:12 -07:00
Tim Burke
2ef21ac05d py3: port request_helpers
Change-Id: I6be1a1c618e4b4fa03b34dad96f378aca01e8e08
2018-09-15 01:33:34 -06:00
Zuul
5d46c0d8b3 Merge "Adding keep_idle config value to socket" 2018-09-15 00:43:52 +00:00
Zuul
a10a8611e6 Merge "Add punch_hole utility function" 2018-09-15 00:15:38 +00:00
FatemaKhalid
cfeb32c66b Adding keep_idle config value to socket
User can cofigure KEEPIDLE time for sockets in TCP connection.
The default value is the old value which is 600.

Change-Id: Ib7fb166deb8a87ae4e97ba0671048b1ec079a2ef
Closes-Bug:1759606
2018-09-15 01:30:53 +02:00
Alexandre Lécuyer
dbacdcf01c Add punch_hole utility function
This is useful for deallocating disk blocks as part of an alternate disk
file implementation.

Additionally, add an offset argument to the existing fallocate utility
function; this allows you to grow an existing file.

Sam always had the best descriptions:

  utils.fallocate(fd, size) allocates <size> bytes for the file referred
  to by <fd>. It allows for keeping a reserve of an additional N bytes
  or X% of the filesystem free. If neither fallocate() or
  posix_fallocate() C functions are avaialble, utils.fallocate() will
  log a warning (once only) and not actually allocate space.

  utils.punch_hole(fd, offset, length) deallocates <length> bytes
  starting at <offset> from the file referred to by <fd>. It uses the C
  function fallocate(). If fallocate() is not available, calls to
  utils.punch_hole() will raise an exception.

Since these both use the fallocate syscall, refactor that a bit and get
rid of FallocateWrapper. We add a new _LibcWrapper to do some
lazy-loading of a C function and expose whether the function is actually
available in Python, though. This allows utils.fallocate and
utils.punch_hole to keep their fancy logic pretty well-contained.

Modernized the tests for utils.fallocate() and utils.punch_hole().

Co-Authored-By: Samuel Merritt <sam@swiftstack.com>
Change-Id: Ieac30a477d784905c94742ee3d0898d7e0194b39
2018-09-14 13:55:42 -06:00
Samuel Merritt
114220279b py3: port list_endpoints
Change-Id: Ic6d0532f9f42b4d57a12aafeec5d34baffab6e23
2018-09-14 13:53:06 -06:00
Zuul
29f71c9119 Merge "s3api: Include '-' in multipart ETags" 2018-09-13 11:30:32 +00:00
Tim Burke
84b85f03b4 s3api: Include '-' in multipart ETags
Multipart uploads in AWS (seem to) have ETags like:

   '"' + MD5_hex(MD5(part1) + ... + MD5(partN)) + '-' + N + '"'

On the other hand, Swift SLOs have Etags like:

   MD5_hex(MD5_hex(part1) + ... + MD5_hex(partN))

(In both examples, MD5 gets the raw 16-byte digest while MD5_hex
gets the 32-byte hex-encoded digest.)

Some clients (such as aws-sdk-java) use the presence of a dash
to decide whether to perform client-side validation of downloads.

Other clients (like s3cmd) use the presence of a dash *in bucket
listings* to decide whether or not to perform additional HEAD requests
to look for MD5 metadata that can be used to compare against the MD5s
of local files.

Now we include a dash as well, to prevent spurious errors like

> Unable to verify integrity of data download.  Client calculated
> content hash didn't match hash calculated by Amazon S3.  The data
> may be corrupt.

or unnecessary uploads/downloads because the client assumes data has
changed that hasn't.

For new multipart-uploads via the S3 API, the ETag that is stored will
be calculated in the same way that AWS uses. This ETag will be used in
GET/HEAD responses, bucket listings, and conditional requests via the S3
API. Accessing the same object via the Swift API will use the SLO Etag;
however, in JSON container listings the multipart upload etag will be
exposed in a new "s3_etag" key.

New SLOs and pre-existing multipart-uploads will continue to behave as
before; there is no data migration or mitigation as part of this patch.

Change-Id: Ibe68c44bef6c17605863e9084503e8f5dc577fab
Closes-Bug: 1522578
2018-09-13 19:28:59 +09:00
Kota Tsuyuzaki
b4be37cc25 Remove post_as_copy deprecated warning from copy middleware
Almost one year has passed since all post_as_copy related code
removed by [1], we don't have to keep the warning message for
post_as_copy setting anymore in the code tree.

1: 1e79f828ad10918bd76ae8df6fe4c4dbf7bbf3c5

Change-Id: Id9eea22ed688574d84ca582584c0c207d5f01383
2018-09-10 17:58:26 -06:00
Timur Alperovich
1f29508327 Allow for backwards compatibility with swift3.
While s3api is the preferred method going forward, it would be nice to
allow for backwards compatibility with swift3 in the tempauth
middleware.

Change-Id: I4fd4772b1d7b173ee0faf72d1c1f1531646dde8b
2018-09-05 17:42:34 -07:00
Tim Burke
df954e2709 py3: port bufferedhttp (hopefully)
I've at least tried it out with a py3 proxy, and it seems to work out
OK. I haven't tried killing the socket and verifying that it's actualy
dead, but getting a hold of _real_close *seems like* what we want?

At least the three (!!) tests pass.

Change-Id: Ic08c26185d63a36a5422793d81f621e0698fa572
2018-08-24 11:30:26 +00:00
Zuul
b32578b5d4 Merge "Multi-key KMIP keymaster" 2018-08-21 15:15:05 +00:00
Zuul
ed18495faa Merge "Add debugging info to SignatureDoesNotMatch responses" 2018-08-18 00:54:21 +00:00
Tim Burke
7ca1a67d70 Add debugging info to SignatureDoesNotMatch responses
This is comparable to what AWS returns, and should greatly simplify
debugging when diagnosing 403s.

Change-Id: Iabfcbaae919598e22f39b2dfddac36b75653fc10
2018-08-17 18:03:47 +00:00
Tim Burke
0dc1b6250e Multi-key KMIP keymaster
Now that the trivial keymaster supports multiple keys, let's do
something similar for the KMIP keymaster. Additional keys are
configured as:

    key_id_<secret_id> = <KMIP unique identifier>

While it might be tempting to use the unique identifier directly as the
secret_id, the added indirection allows operators to move keys between
different backends, which may cause different identifiers to be issued.

As with the trivial keymaster, the key to use for PUTs and POSTs is
specified with:

    active_root_secret_id = <secret_id>

Change-Id: Ie52508e47d15ec5c4e96902d3c9f5f282d275683
2018-08-17 17:55:09 +00:00
Alistair Coles
2722e49a8c Add support for multiple root encryption secrets
For some use cases operators would like to periodically introduce a
new encryption root secret that would be used when new object data is
written. However, existing encrypted data does not need to be
re-encrypted with keys derived from the new root secret. Older root
secret(s) would still be used as necessary to decrypt older object
data.

This patch modifies the KeyMaster class to support multiple root
secrets indexed via unique secret_id's, and to store the id of the
root secret used for an encryption operation in the crypto meta. The
decrypter is modified to fetch appropriate keys based on the secret id
in retrieved crypto meta.

The changes are backwards compatible with previous crypto middleware
configurations and existing encrypted object data.

Change-Id: I40307acf39b6c1cc9921f711a8da55d03924d232
2018-08-17 17:54:30 +00:00
Zuul
cfc4f30d63 Merge "s3_acl: Require swift_owner authz to create buckets" 2018-08-16 17:03:27 +00:00
Zuul
a41ca22f46 Merge "Fix the deletion of non-existent keys" 2018-08-15 21:55:08 +00:00
Tim Burke
51b885b3b5 s3_acl: Require swift_owner authz to create buckets
Otherwise, users can create buckets in accounts they don't own.

Change-Id: I13d557c32b12529ef1087c52f7af302a33d33acb
2018-08-15 12:56:33 +00:00
Tim Burke
bd640cdbae Fix the deletion of non-existent keys
On vanilla Swift, deleting an object that doesn't exist will 404.
On AWS, deleting a key that doesn't exist will either 404 if the bucket
doesn't exist (with a NoSuchBucket code) or 204 (because yep, that's not
accessible).

Change-Id: Ied2a78b56522316bb374f23961621641af3adc83
Related-Change: I6e154594dfda6c3065774c23b24f728625a842bc
2018-08-13 14:19:23 -07:00
Tim Burke
5dca610e94 Fix up test_static_web_pretend_to_be_giant_json
It was actually testing the invalid-JSON handling before...

Change-Id: Ia8b5eaeb42fea5136525c80e67e8d33548c2a8df
2018-08-13 20:58:13 +00:00
Zuul
9b19df362f Merge "Disallow uploads with x-amz-tagging headers set" 2018-08-11 01:42:06 +00:00
Zuul
18f6ce92f5 Merge "Disallow aws-chunked uploads" 2018-08-10 19:59:35 +00:00
Tim Burke
7895718ce9 Fix keymmaster_conf_section typo
Change-Id: I3bce1c4efeb3a3a7319020de76ba7f06015a5a36
2018-08-08 17:06:17 +00:00
Christian Schwede
dcbdcb8f1c Disallow uploads with x-amz-tagging headers set
We don't support it yet, so return 501 Not Implemented.

Change-Id: Ie2f4bd1bfdb1bcbdf1a0f0db9d542b6057e9d2ec
2018-08-08 04:57:32 -04:00
Tim Burke
79539eabf7 Disallow aws-chunked uploads
We don't support it yet, so return 501 Not Implemented. Previously, we'd
store the aws-chunked content (!) and most clients would see it as data
corruption.

See https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html
for more information.

Change-Id: I697962039667980ef89212bc480f8b1d3fbd718c
2018-08-03 19:30:01 +00:00
Tim Burke
3c92e3ce87 Move keymaster_config_path parsing out of _get_root_secret
Change-Id: Iddc0f333861b6c1f81e181f006cd592b5eb6ea17
2018-07-30 16:51:24 +00:00
Zuul
00373dad61 Merge "Add keymaster to fetch root secret from KMIP service" 2018-07-25 03:49:50 +00:00
Zuul
89854250c3 Merge "Add fallocate_reserve to account and container servers." 2018-07-20 08:42:51 +00:00
Samuel Merritt
8e651a2d3d Add fallocate_reserve to account and container servers.
The object server can be configured to leave a certain amount of disk
space free; default is 1%. This is useful in avoiding 100%-full
filesystems, as those can get Swift in a state where the filesystem is
too full to write tombstones, so you can't delete objects to free up
space.

When a cluster has accounts/containers and objects on the same disks,
then you can wind up with a 100%-full disk since account and container
servers don't respect fallocate_reserve. This commit makes account and
container servers respect fallocate_reserve so that disks shared
between account/container and object rings won't get 100% full.

When a disk's free space falls below the configured reserve, account
and container PUT, POST, and REPLICATE requests will fail with a 507
status code. These are the operations that can significantly increase
the disk space used by a given database.

I called the parameter "fallocate_reserve" for consistency with the
object server. No actual fallocate() call happens under Swift's
control in the account or container servers (sqlite3 might make such a
call, but it's out of our hands).

Change-Id: I083442eef14bf83c0ea717b1decb3e6b56dbf1d0
2018-07-18 17:27:11 +10:00
Alistair Coles
0cd42a2d33 Check other params preserved when slo_etag is extracted
Change-Id: Ie34ce2a33f2a642b97986fa28cf9db9e6da964d5
Related-Change: I67478923619b00ec1a37d56b6fec6a218453dafc
Related-Change: Ibaa630b5b4251cc4f821c01d3c09a8b8a6be342c
2018-07-12 10:01:58 -05:00
Clay Gerrard
f8b9c24a1c Add unittest for slo_etag
Related-Change-Id: I67478923619b00ec1a37d56b6fec6a218453dafc

Change-Id: Ibaa630b5b4251cc4f821c01d3c09a8b8a6be342c
2018-07-12 10:01:54 -05:00
Tim Burke
c4c98eb64d Include SLO ETag in container updates
Container servers will store an etag like

   <MD5 of manifest on disk>; slo_etag=<MD5 on concatenated ETags>

which the SLO middleware will break out into separate

   "hash": "<MD5 of manifest on disk",
   "slo_etag": "\"<MD5 of concatenated ETags\"",

keys for JSON listings. Text and XML listings are unaffected.

If a middleware left of SLO already specified a container update
override, the slo_etag parameter will be appended. If the base header
value was blank, the MD5 of the manifest will be inserted.

SLOs that were created on previous versions of Swift will continue to
just have the MD5 of the manifest in container listings.

Closes-Bug: 1618573
Change-Id: I67478923619b00ec1a37d56b6fec6a218453dafc
2018-07-10 15:41:29 -07:00
Zuul
1ab691f637 Merge "IP Range restrictions in temp urls" 2018-07-10 04:28:16 +00:00
Zuul
cb3692f8db Merge "swob.Match: remove quotes when checking __contains__" 2018-07-04 06:34:17 +00:00
mmcardle
26b20ee729 IP Range restrictions in temp urls
This patch adds an additional optional parameter to tempurl
which restricts the ip's from which a temp url can be used from.

Change-Id: I23fe998a980960d4a32df042b3f6a21f096c36af
2018-07-03 12:25:28 +01:00
Alistair Coles
1951dc7e9a Add keymaster to fetch root secret from KMIP service
Add a new middleware that can be used to fetch an encryption root
secret from a KMIP service. The middleware uses a PyKMIP client
to interact with a KMIP endpoint. The middleware is configured with
a unique identifier for the key to be fetched and options required
for the PyKMIP client.

Co-Authored-By: Tim Burke <tim.burke@gmail.com>
Change-Id: Ib0943fb934b347060fc66c091673a33bcfac0a6d
2018-07-03 09:00:21 +01:00
Zuul
e93c6187f6 Merge "swob: Stop auto-encoding unicode bodies" 2018-07-02 02:41:05 +00:00
Zuul
a4b986fd78 Merge "tempauth: Send back url-encoded account names" 2018-06-30 04:56:11 +00:00
Tim Burke
d03fc9bc54 swob: Stop auto-encoding unicode bodies
Instead, require that callers provide an encoding.

Related-Change: I31408f525ba9836f634a35581d4aee6fa2c9428f
Change-Id: I3e5ed9e4401eea76c375bb43ad4afc58b1d8006a
2018-06-28 09:58:44 -07:00
Zuul
729fe6f79d Merge "swob: Fix up some WSGI string business" 2018-06-28 12:23:58 +00:00
Zuul
eb43d30320 Merge "Enforce Content-Length in catch_errors" 2018-06-28 12:23:45 +00:00
Tim Burke
158d4c28b2 swob.Match: remove quotes when checking __contains__
If we're going to drop the quotes during initialization, we should
probably drop them when making comparisons, too.

Change-Id: I90c6e8a7c6faf8fafb0cd64fabbc9629d6c4c48a
2018-06-27 16:52:38 -07:00
Tim Burke
efcf7e6a95 swob: Fix up some WSGI string business
Change-Id: Iee1bab5775b243c318aa22ee4a548d793e6684ca
2018-06-27 15:55:30 -07:00
Samuel Merritt
306f9a150f py3: port domain_remap tests
The domain_remap middleware didn't need to change, only its tests.

Change-Id: I5edf05b609ce263708f851219c38c1bd1b645a6f
2018-06-26 13:20:49 -07:00
Samuel Merritt
50ea4e1e08 py3: port healthcheck
Typed "b" in a half-dozen places and fixed a typo in a test name.

Change-Id: Idcb8b2eeef5b545eb0944cc22b7c7961e8c2f8f0
2018-06-26 13:20:49 -07:00