74 Commits

Author SHA1 Message Date
Tim Burke
ce9e56a6d1 lint: Consistently use assertIsInstance
This has been available since py32 and was backported to py27; there
is no point in us continuing to carry the old idiom forward.

Change-Id: I21f64b8b2970e2dd5f56836f7f513e7895a5dc88
2024-02-07 15:48:39 -08:00
indianwhocodes
0893cedc35 Include accept-ranges header in s3api response
Change-Id: Ib3fa895ea13a6703b0f146bc8833c4e635976fdd
2023-10-13 16:39:09 +00:00
Matthew Oliver
0996433fe5 s3api: Add basic GET object-lock support
Some tooling out there, like Ansible, will always call to see if
object-lock is enabled on a bucket/container. This fails as Swift doesn't
understand the object-lock or the get object lock api[0].

When you use the get-object-lock-configuration to a bucket in s3 that
doesn't have it applied it returns a specific 404:

  GET /?object-lock HTTP/1.1" 404 None
  ...

  <?xml version="1.0" encoding="UTF-8"?>
  <Error>
    <Code>ObjectLockConfigurationNotFoundError</Code>
    <Message>Object Lock configuration does not exist for this bucket</Message>
    <BucketName>bucket_name</BucketName>
    <RequestId>83VQBYP0SENV3VP4</RequestId>
  </Error>'

This patch doesn't add support for get_object lock, instead it always
returns a similar 404 as supplied by s3, so clients know it's not
enabled.

Also add a object-lock PUT 501 response.

[0] https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObjectLockConfiguration.html

Change-Id: Icff8cf57474dfad975a4f45bf2d500c2682c1129
2023-10-05 16:38:35 +11:00
Tim Burke
3f3f5be9bb tests: boto is always <3.0
Otherwise, it'd be boto3.

Change-Id: I2961740fd4f3e914675083331f2465591d63b755
2023-09-25 18:34:16 -07:00
Tim Burke
5392a2057b tests: Add test(s) for MPU part copy from range
When using the copy-part API it is expected for s3api to write down an
empty value for X-Object-Sysmeta-S3Api-Etag on segments.  This was
ostensibly to prevent writing down an unrelated S3Api-Etag when copying
a part from another MPU the copy transfers object sysmeta.  We should
assume a S3Api-Etag w/o X-Static-Large-Object is non-sense, and SLO
should forever expect empty values for it's sysmeta.

Drive-By: consolidate handling of boto2 sigv4 skips

Related-Bug: #2035158
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Change-Id: Ic6f04a5a6af8a3e65b226cff2ed6c9fce8ce1fa2
2023-09-18 15:25:46 -05:00
Tim Burke
052bcadb27 tests: Skip s3api functional tests when no s3api user configured
Change-Id: I61f141a71eddcac600058d66ddf802306df455c1
2023-03-10 14:07:21 -08:00
Zuul
bba3a3145d Merge "tests: Get rid of test.unit.SkipTest" 2023-02-28 02:20:04 +00:00
Tim Burke
8dd2d010ac Skip S3 versioning test when versioning is not enabled
Change-Id: I36e42f459a74ed71a1cc57570a564e5562abbae3
2023-02-24 11:48:13 -08:00
Tim Burke
be16d6c4fd tests: Get rid of test.unit.SkipTest
unittest.SkipTest suffices.

Change-Id: I11eb73f7dc4a8598fae85d1efca721f69067fb4f
2023-02-16 23:59:53 -08:00
Tim Burke
3550e00dd9 tests: Ensure XXE injection tests have config loaded
Depending on test order (and possibly whether there were earlier
failures?) the new tests may trip KeyErrors when trying to get
s3_access_key values. Solution seems to be defining
setUpModule() / tearDownModule() like other functional tests.

Also fix up some Content-MD5 handling; if we're using pre-signed URLs,
we can't provide a Content-MD5.

Change-Id: Ifce72ec255b1b618b9914ce5785d04ee0ebd3b8c
Related-Change: I84494123cfc85e234098c554ecd3e77981f8a096
2023-01-19 11:24:34 -08:00
Aymeric Ducroquetz
b8467e190f s3api: Prevent XXE injections
Previously, clients could use XML external entities (XXEs) to read
arbitrary files from proxy-servers and inject the content into the
request. Since many S3 APIs reflect request content back to the user,
this could be used to extract any secrets that the swift user could
read, such as tempauth credentials, keymaster secrets, etc.

Now, disable entity resolution -- any unknown entities will be replaced
with an empty string. Without resolving the entities, the request is
still processed.

[CVE-2022-47950]

Closes-Bug: #1998625
Co-Authored-By: Romain de Joux <romain.de-joux@ovhcloud.com>
Change-Id: I84494123cfc85e234098c554ecd3e77981f8a096
2023-01-17 15:03:41 +00:00
Tim Burke
f6ac7d4491 Tolerate absolute-form request targets
We've seen S3 clients expecting to be able to send request lines like

    GET https://cluster.domain/bucket/key HTTP/1.1

instead of the expected

    GET /bucket/key HTTP/1.1

Testing against other, independent servers with something like

    ( echo -n $'GET https://www.google.com/ HTTP/1.1\r\nHost: www.google.com\r\nConnection: close\r\n\r\n' ; sleep 1 ) | openssl s_client -connect www.google.com:443

suggests that it may be reasonable to accept them; the RFC even goes so
far as to say

> To allow for transition to the absolute-form for all requests in some
> future version of HTTP, a server MUST accept the absolute-form in
> requests, even though HTTP/1.1 clients will only send them in
> requests to proxies.

(See https://datatracker.ietf.org/doc/html/rfc7230#section-5.3.2)

Fix it at the protocol level, so everywhere else we can mostly continue
to assume that PATH_INFO starts with a / like we always have.

Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Change-Id: I04012e523f01e910f41d5a41cdd86d3d2a1b9c59
2023-01-03 12:49:30 -08:00
indianwhocodes
d363236a24 s3api errors for unsupported headers x-delete-at, x-delete-after
We need to support the aforementioned headers in our s3 apis
and raise an InvalidArgumentError if a s3 client makes a request

Change-Id: I2c5b18e52da7f33b31ba386cdbd042f90b69ef97
2022-12-01 13:21:01 -08:00
Tim Burke
bc3625142c py310: Fix formatdate() call
Previously, this would trip TypeErrors on py310:

    TypeError: 'S3Timestamp' object cannot be interpreted as an integer

Change-Id: I124c1957264c80d28a6b3e852d042cbc8468939c
2022-07-25 17:14:46 -07:00
Alistair Coles
2f607cd319 Round s3api listing LastModified to integer resolution
s3api bucket listing elements currently have LastModified values with
millisecond precision. This is inconsistent with the value of the
Last-Modified header returned with an object GET or HEAD response
which has second precision. This patch reduces the precision to
seconds in bucket listings and upload part listings. This is also
consistent with observation of an aws listing response.

The last modified values in the swift native listing *up* to
the nearest second to be consistent with the seconds-precision
Last-Modified time header that is returned with an object GET or HEAD.
However, we continue to include millisecond digits set to 0 in the
last-modified string, e.g.: '2014-06-10T22:47:32.000Z'.

Also, fix the last modified time returned in an object copy response
to be consistent with the last modified time of the object that was
created. Previously it was rounded down, but it should be rounded up.

Change-Id: I8c98791a920eeedfc79e8a9d83e5032c07ae86d3
2022-05-10 11:26:27 +01:00
Zuul
ec964b23bb Merge "s3api: Copy more headers from MPU marker to final object" 2022-03-27 21:16:46 +00:00
Tim Burke
1c4acf2d8f s3api: Copy more headers from MPU marker to final object
Closes-Bug: 1966396
Change-Id: I253d8e3e8678fad3fde43259ed3225df4048a458
2022-03-25 09:49:11 +00:00
Zuul
0651d8175d Merge "trivial: Replace assertRegexpMatches with assertRegex" 2022-03-24 00:33:27 +00:00
Zuul
014c98e853 Merge "s3api: Fix multi_delete with object names using non-ASCII characters" 2022-03-10 03:46:16 +00:00
Zuul
7ac2b2eb76 Merge "s3api: Delete all parts when aborting MPU with non-ASCII characters" 2022-03-09 23:54:05 +00:00
Aymeric Ducroquetz
82ca37517d s3api: Delete all parts when aborting MPU with non-ASCII characters
Change-Id: Idcda76f7a880a18c3bac699e0fb2435e4a54abbd
2022-03-09 11:57:29 -08:00
Aymeric Ducroquetz
dd64a81e65 s3api: Fix multi_delete with object names using non-ASCII characters
Co-Authored-By: Florent Vennetier <florent.vennetier@ovhcloud.com>
Change-Id: I635bc91faa7709f9df9cdf3aec157a21c08923ca
2022-03-01 13:47:35 +01:00
Aymeric Ducroquetz
5b3ec5aa64 s3api: Properly decode MPU request parameters before using them
Specifically, parameters that may contain non-ASCII characters,
such as the prefix and marker to list current uploads.

Change-Id: Icfae68825f94ddf2412c0274c3d500e265117e8e
2022-03-01 09:13:21 +01:00
Tim Burke
5f25e1cc77 s3api: Fix non-ascii MPUs
Previous problems included:
  - returning wsgi strings quoted assuming UTF-8 on py3 when initiating
    or completing multipart uploads
  - trying to str() some unicode on py2 when listing parts, leading to
    UnicodeEncodeErrors

Change-Id: Ibc1d42c8deffe41c557350a574ae80751e9bd565
2022-01-28 21:01:17 -08:00
Florent Vennetier
c15818f1e6 s3api: fix the copy of non-ASCII objects
Trying to copy an object with non-ASCII characters in its name results
in, depending on the pipeline:
- an error code 412 because of a badly urlencoded path
- an error code 500 "TypeError: Expected a WSGI string"

This commit fixes the problem by calling str_to_wsgi on the object name
after it has been urldecoded. We do not need to call this on the
container name because it is supposed to contain only ASCII characters.

Change-Id: If837d4e55735b10a783c85d91f37fbea5e3baf1d
2021-09-29 19:05:14 -07:00
Tim Burke
b4e532a46f func test improvements
Not all v1 auth systems use an acct:user format; s3api tests should not require it.

Be a little more tolerant of listing consistency issues when resetting.

Tolerate s3api /info results returning strings instead of ints.

Related-Change: I4a46bd650a53f88c642d402e697869df28bd2fd3
Change-Id: I8f2f247dd113ad637b17d241133b14c35cadecae
2021-09-14 21:49:32 -07:00
Tim Burke
57e41685ba trivial: Replace assertRegexpMatches with assertRegex
Change-Id: Id1ee51fea6b754042ebe5204517dd91865ffbc28
2021-07-16 10:58:00 -07:00
Tim Burke
ca0c344456 s3api: Return KeyTooLongError when upload name exceeds constraints
Change-Id: I8ce73e2e21e9216484130ba3bd1e77b45eb1d77c
2021-06-09 13:10:48 -07:00
Zuul
076b7828ea Merge "s3api: Fix prefix/delimiter/marker quoting" 2021-05-25 18:15:31 +00:00
Zuul
80ccd1e7bf Merge "s3api: Fix blank delimiter handling" 2021-05-25 18:15:19 +00:00
Tim Burke
d022781bc6 s3api: Transfer REMOTE_USER when using s3_acl
Some middlewares (notably staticweb) use the absence of a REMOTE_USER to
determine that a request was unauthenticated and as a result should be
handled differently. This could cause problems for S3 requests that
were authenticated via s3api's custom auth logic, including

* server errors when a container listing request gets handled by
  staticweb or
* losing storage policy information because staticweb copied the request
  environment.

Change-Id: Idf29c6866fec7b413c4369dce13c4788666c0934
Closes-Bug: #1833287
Related-Change: I5fe5ab31d6b2d9f7b6ecb3bfa246433a78e54808
2021-01-08 16:11:02 -08:00
Ade Lee
5320ecbaf2 replace md5 with swift utils version
md5 is not an approved algorithm in FIPS mode, and trying to
instantiate a hashlib.md5() will fail when the system is running in
FIPS mode.

md5 is allowed when in a non-security context.  There is a plan to
add a keyword parameter (usedforsecurity) to hashlib.md5() to annotate
whether or not the instance is being used in a security context.

In the case where it is not, the instantiation of md5 will be allowed.
See https://bugs.python.org/issue9216 for more details.

Some downstream python versions already support this parameter.  To
support these versions, a new encapsulation of md5() is added to
swift/common/utils.py.  This encapsulation is identical to the one being
added to oslo.utils, but is recreated here to avoid adding a dependency.

This patch is to replace the instances of hashlib.md5() with this new
encapsulation, adding an annotation indicating whether the usage is
a security context or not.

While this patch seems large, it is really just the same change over and
again.  Reviewers need to pay particular attention as to whether the
keyword parameter (usedforsecurity) is set correctly.   Right now, all
of them appear to be not used in a security context.

Now that all the instances have been converted, we can update the bandit
run to look for these instances and ensure that new invocations do not
creep in.

With this latest patch, the functional and unit tests all pass
on a FIPS enabled system.

Co-Authored-By: Pete Zaitcev
Change-Id: Ibb4917da4c083e1e094156d748708b87387f2d87
2020-12-15 09:52:55 -05:00
Zuul
ae85ee2474 Merge "s3api: Allow CompleteMultipartUpload requests to be retried" 2020-06-25 20:47:29 +00:00
Tim Burke
02548717ac s3api: Allow CompleteMultipartUpload requests to be retried
When completing a multipart-upload, include the upload-id in sysmeta.
If we can't find the upload marker, check the final object name; if it
has an upload-id in sysmeta and it matches the upload-id that we're
trying to complete, allow the complete to continue.

Also add an early return if the already-completed upload's ETag matches
the computed ETag for the user's request. This should help clients that
can't take advantage of how we dribble out whitespace to try to keep the
conneciton alive: The client times out, retries, and if the upload
actually completed, it gets a fast 200 response.

Change-Id: I38958839be5b250c9d268ec7c50a56cdb56c2fa2
2020-06-24 10:09:26 -07:00
Hervé Beraud
e91a3d73ad Remove lxml deprecated methods
All our supported runtimes [1] are compatible with the recommended
alternatives.

The `getchildren` method is deprecated [2] since lxml 2.0 and will be removed
in future versions, these changes switch usages to `list(elem)` as
recommended in the official documentation [2].

[1] https://governance.openstack.org/tc/reference/runtimes/victoria.html#python-runtimes-for-train
[2] https://lxml.de/api/lxml.etree._Element-class.html#getchildren

Change-Id: I5264edb93532d90703ae7b59cd71520a2cd3518a
2020-06-23 13:32:22 +02:00
Zuul
3b651d945c Merge "tests: Skip s3api copy-version test when OV not enabled" 2020-06-08 21:53:06 +00:00
Zuul
8f1a40eaae Merge "swift-dsvm: Enable s3api" 2020-05-28 00:27:36 +00:00
Tim Burke
984b57a873 tests: Skip s3api copy-version test when OV not enabled
Change-Id: I175b9e1464494454b1193b9b9d5311cb6fd6503f
2020-05-27 13:29:08 -07:00
Tim Burke
bb9b0326fd swift-dsvm: Enable s3api
Depends-On: https://review.opendev.org/#/c/571021/
Change-Id: I3ac3288cd61b745ce7dbf2bded8eade026d0418f
2020-05-22 16:04:52 -07:00
Tim Burke
10c24e951c s3api: Fix prefix/delimiter/marker quoting
And stop sending WSGI strings on py3.

Change-Id: I9b769e496aa7c8ed5862c2d7310f643838328084
Closes-Bug: #1853654
2020-05-15 22:47:25 -07:00
Thiago da Silva
f4bc951508 fix s3api functional tests
Connection was hard coded to use saio config

Change-Id: I9c11162de89fa3aa2a78aea093b187d0309860f5
Signed-off-by: Thiago da Silva <thiago@redhat.com>
2020-05-15 22:38:04 -07:00
Tim Burke
f0b8790c12 s3api: Fix blank delimiter handling
Real AWS only includes an empty delimiter element when doing a
version-aware listing.

Change-Id: Id246a157c576eac93375be084ada3740f1e09793
Closes-Bug: #1853663
2020-05-15 18:43:07 -07:00
Tim Burke
668242c422 pep8: Turn on E305
Change-Id: Ia968ec7375ab346a2155769a46e74ce694a57fc2
2020-04-03 21:22:38 +02:00
Andreas Jaeger
96b56519bf Update hacking for Python3
The repo is Python using both Python 2 and 3 now, so update hacking to
version 2.0 which supports Python 2 and 3. Note that latest hacking
release 3.0 only supports version 3.

Fix problems found.

Remove hacking and friends from lower-constraints, they are not needed
for installation.

Change-Id: I9bd913ee1b32ba1566c420973723296766d1812f
2020-04-03 21:21:07 +02:00
karen chan
6097660f0c s3api: Implement object versioning API
Translate AWS S3 Object Versioning API requests to native Swift Object
Versioning API, speficially:

 * bucket versioning status
 * bucket versioned objects listing params
 * object GETorHEAD & DELETE versionId
 * multi_delete versionId

Change-Id: I8296681b61996e073b3ba12ad46f99042dc15c37
Co-Authored-By: Tim Burke <tim.burke@gmail.com>
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
2020-01-28 14:00:08 -08:00
Sean McGinnis
5b26b749b5
Drop use of unittest2
unittest2 was needed for Python version <= 2.6, so it hasn't been needed
for quite some time. See unittest2 note one:

https://docs.python.org/2.7/library/unittest.html

This drops unittest2 in favor of the standard unittest module.

Change-Id: I2e787cfbf1709b7f9c889230a10c03689e032957
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
2020-01-12 03:13:41 -06:00
Tim Burke
1ded0d6c87 Allow arbitrary UTF-8 strings as delimiters in listings
AWS seems to support this, so let's allow s3api to do it, too.

Previously, S3 clients trying to use multi-character delimiters would
get 500s back, because s3api didn't know how to handle the 412s that the
container server would send.

As long as we're adding support for container listings, may as well do
it for accounts, too.

Change-Id: I62032ddd50a3493b8b99a40fb48d840ac763d0e7
Co-Authored-By: Thiago da Silva <thiagodasilva@gmail.com>
Closes-Bug: #1797305
2019-09-12 10:44:00 -07:00
Tim Burke
f05119c16f py3: mostly port s3 func tests
test_bucket.py is proving somewhat problematic.

Change-Id: I5b337ef66a23fc989762801dd6a5ba1ed903f57b
2019-08-09 16:09:55 -07:00
Tim Burke
e22960fd71 s3api: Fix ETag when copying a MU part from another MU
Previously, we'd preserve the sysmeta that we wrote down with the
original multipart-upload to track its S3-style etag on the new part,
causing it to have an ETag like `<MD5>-<N>`. Later, when the client
tried to complete the new multipart-upload, it would send that etag back
to the server, which would reject the request because the ETag didn't
look like a normal MD5.

Now, have s3api include blank values in the copy request to overwrite
the source sysmeta, and treat a blank etag override the same as a
missing one.

Change-Id: Id33a7ab9d0b8f33fede73eae540d6137708e1218
Closes-Bug: #1829959
2019-05-22 13:36:41 -07:00
Timur Alperovich
d4e7940457 Start transition to boto3 in s3api tests.
Adds the scaffolding required for tests to use boto3 and converts the
test_bucket.py tests to the new interface. Follow on patches will
convert the other tests to use the boto3 library.

Notable changes: we no longer try to reach for the equivalent of
`boto.make_request()` and instead rely on the boto3/botocore event
system to mutate requests as necessary (or to disable pre-flight
validators).

Partial-Bug: 1557260
Change-Id: I3d77ef4a6b878c49ebfa0c8b8647d7199d87601e
2019-05-21 22:10:20 +00:00