Previously, if object versioning was enabled via the old-style
allow_versions container-server setting rather than the new-style
allow_versioned_writes proxy-server setting,
TestCrossPolicyObjectVersioning would skip tests while
TestObjectVersioning and TestObjectVersioningUTF8 would run them.
Additionally, if versioned_writes was explicitly included in the
proxy-server's pipeline and allow_versioned_writes was disabled,
the functional tests would fail with a 412.
Now, all three will use the same logic to check whether versioning is
enabled. Specifically, they will all try to set an X-Versions-Location
header and skip if it doesn't stick.
Additionally, the TestCrossPolicyObjectVersioningEnv will now properly
clean up after itself.
Change-Id: I4c788a0e18587ff17d3c6e346fd22b881495f06d
Previously, attempting to PUT a SLO manifest with `If-None-Match: *`
would include the header when validating the segments, causing the
upload to fail.
Now when SLO validates segments, no conditional headers will be
included in the HEAD request.
Change-Id: I03ad454092d3caa73d29e6d30d8033b45bc96136
Closes-Bug: #1569253
This change removes the use of the COPY request in the versioned
writes middleware. It changes the COPY verb for GETs and PUTs
requests. The main reasoning for this change is to remove any
dependency that versioning had on copy, which will allow for the COPY
functionality to be moved to middleware and to be to the left of the
versioned writes middleware in the proxy pipeline. In this way,
no COPY request will ever arrive at the versioned writes middleware.
A side benefit of this change is that it removes a HEAD request from
the PUT path. Instead of checking if a current version exists, a
GET request is sent, in case of success, a PUT is sent to the
versions container.
A unit test was removed that tested non-default storage policies.
This test is no longer necessary, since it was used to test
specific policy handling code in the COPY method in the proxy
object controller.
Closes-Bug: #1365862
Change-Id: Idf34fa8d04ff292df7134b6d4aa94ff40887b3a4
Co-Authored-By: Alistair Coles <alistair.coles@hp.com>
Co-Authored-By: Janie Richling <jrichli@us.ibm.com>
Co-Authored-By: Kota Tsuyuzaki <tsuyuzaki.kota@lab.ntt.co.jp>
Signed-off-by: Thiago da Silva <thiago@redhat.com>
Currently, the multipart-manifest=get call returns output in json
format that is inconsistent with the format that is used for the
multipart-manifest=put.
This in turn introduces a new call: ?multipart-manifest=get&format=raw
Change-Id: I2242943a738f667cbda6363bcb6a017f341e834f
Closes-Bug: 1252482
When using an EC policy, 304 responses to conditional GETs
are missing the Accept-Ranges header and have the wrong ETag
value. 412 responses also have the wrong etag.
416 responses to ranged GETs also have the wrong ETag.
This patch ensures behaviour with EC policy is consistent
with replication policy:
- 304 and 416 responses have correct etag and Accept-Ranges
- 412 responses have correct Etag but no Accept-Ranges
Co-Authored-By: Mahati Chamarthy <mahati.chamarthy@gmail.com>
Co-Authored-By: Thiago da Silva <thiago@redhat.com>
Closes-Bug: #1496234
Closes-Bug: #1558197
Closes-Bug: #1558193
Change-Id: Ic21317b9e4f632f0751133a3383eb5487379e11f
With the X-Timestamp validation added in commit e619411, end users
could upload objects with
X-Timestamp: 9999999999.99999_ffffffffffffffff
(the maximum value) and Swift would be unable to delete them.
Now, inbound X-Timestamp headers will be moved to
X-Backend-Inbound-X-Timestamp, effectively rendering them harmless.
The primary reason to allow X-Timestamp before was to prevent
Last-Modified changes for objects coming from either:
* container_sync or
* a migration from another storage system.
To enable the former use-case, the container_sync middleware will now
translate X-Backend-Inbound-X-Timestamp headers back to X-Timestamp
after verifying the request.
Additionally, a new option is added to the gatekeeper filter config:
# shunt_inbound_x_timestamp = true
To enable the latter use-case (or any other use-case not mentioned), set
this to false.
Upgrade Consideration
=====================
If your cluster workload requires that clients be allowed to specify
objects' X-Timestamp values, disable the shunt_inbound_x_timestamp
option before upgrading.
UpgradeImpact
Change-Id: I8799d5eb2ae9d795ba358bb422f69c70ee8ebd2c
This patch makes a number of changes to enable content-type
metadata to be updated when using the fast-POST mode of
operation, as proposed in the associated spec [1].
* the object server and diskfile are modified to allow
content-type to be updated by a POST and the updated value
to be stored in .meta files.
* the object server accepts PUTs and DELETEs with older
timestamps than existing .meta files. This is to be
consistent with replication that will leave a later .meta
file in place when replicating a .data file.
* the diskfile interface is modified to provide accessor
methods for the content-type and its timestamp.
* the naming of .meta files is modified to encode two
timestamps when the .meta file contains a content-type value
that was set prior to the latest metadata update; this
enables consistency to be achieved when rsync is used for
replication.
* ssync is modified to sync meta files when content-type
differs between local and remote copies of objects.
* the object server issues container updates when handling
POST requests, notifying the container server of the current
immutable metadata (etag, size, hash, swift_bytes),
content-type with their respective timestamps, and the
mutable metadata timestamp.
* the container server maintains the most recently reported
values for immutable metadata, content-type and mutable
metadata, each with their respective timestamps, in a single
db row.
* new probe tests verify that replication achieves eventual
consistency of containers and objects after discrete updates
to content-type and mutable metadata, and that container-sync
sync's objects after fast-post updates.
[1] spec change-id: I60688efc3df692d3a39557114dca8c5490f7837e
Change-Id: Ia597cd460bb5fd40aa92e886e3e18a7542603d01
setup_package and teardown_package get called twice when
running functional tests using nose. With in-process mode
the first call to teardown_package clears global config
which causes the test swiftclient to raise a SkipTest error
during the second call to teardown_package.
To reproduce:
SWIFT_TEST_IN_PROCESS=1 nosetests ./test/functional/test_account.py
This patch simply tests for config existence before attempting
to create a test swiftclient in the teardown_package function.
Also fix a related issue whereby in_process flag would be reset to
False during second call to setup_package, thus causing some of the
in process cleanup in teardown_package to never be executed.
Change-Id: I074dcd3d39aa46b262632024b047556ca471e8b8
Previously, if a user could write to (but not read from) a container,
the behavior for object POST would vary depending on whether
object_post_as_copy was enabled (403 response) or disabled (202
response).
Now, POSTs will consistently be allowed, regardless of whether fast-POST
is enabled.
Change-Id: I1d6dcbc4f5034a322a1073850fc3b059ebb1c0fa
test/functional/tests.py:TestObjectVersioningUTF8 does not clean
up the versions files it creates because the class's multiple
inheritance is such that it does not call the tearDown method in
TestObjectVersioning.
As a result, any attempt to clean up account containers wastes
time retrying container delete requests. This occurs either in
the setUp for TestSloEnv, if the TestSlo class is included in a
test run, or in the tests.py package tearDown method.
On the author's dev machine this patch reduces the execution
time of functional tests in tests.py by approx 30% or 1 minute.
Change-Id: I8194672bf2ca82435df5868720b6a55a79b94413
Running functional tests in the in-process mode uses
the default value for proxy-server object_post_as_copy,
which is True. This patch adds support for an environment
variable to change this value to False so that the fast-post
mode is tested when running in-process functional tests.
The patch also adds a new tox environment func-in-process-fast-post
which forces in-process functional testing with
object_post_as_copy=False.
The motivation for this change, apart from enabling configurable
local testing, is to put support in place for an upstream CI job
that will actually functionally test the fast-post mode, which is
otherwise only covered by unit tests.
There are currently two gate jobs that run the functional tests:
- gate-swift-dsvm-functional runs the tests *twice* against a
devstack swift service, once using tempauth and once using
keystoneauth. The devstack swift service uses the default
object_post_as_copy=True.
- gate-swift-tox-func runs the func tests in in-process mode
which also uses tempauth and object_post_as_copy=True. This
duplicates one of the config scenarios above.
With this change either the gate-swift-tox-func job or a new job
could run the functional tests using object_post_as_copy=False.
Change-Id: Ia37f6df1dc38e44ef7404fbf0a52f6fc22fae0c2
It feels silly, and we don't do it for any of the other headers in
headers_to_container_info.
While we're at it, clean up a stray '
Change-Id: I0745038cc3832a77d064e515c37cacbdcb97c4d9
Related-Change: Iea3d06de80210e9e504e296d4572583d7ffabeac
test/functional/tests.py:TestAccount.testAccountHead relies
on the account having a known number of containers. The test
setup attempts to delete all existing containers but this fails
a container has versions (and so is not emptied, and cannot be
deleted). The tests then fails because the expected number of
containers does not match the actual.
'bin/resetswift' before running tests will obviously clear all
state but is not always convenient.
This change removes any x-versions-location header before deleting
containers during test setUp.
Steps to recreate the pre-condition for failure on master:
(based on original work by clayg)
swift post target -r '.r:*, .rlistings'
swift post source -H 'x-versions-location: target'
for i in {1..4}; do
echo "junk${i}" > junk
swift upload source junk
done
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Change-Id: I3efb6c20dc1fb3e979087e8a93d04ba7e346b5b6
This patch adds test cases for PUT, DELETE, GET, HEAD, POST and OPTIONS
requests to accounts, containers and objects using various combinations
of users/projects, roles and/or service tokens.
Change-Id: Iea8141ac74ad949a3ae7fa47fda3135d0f2612f6
Multiple etags can be provided on an if-match or if-none-match
request. This is currently being tested in the unit tests, but not
in the functional tests. Since these etags can be modified by
middleware, we need functional tests to assert multiple-etag
requests are handled correctly.
Change-Id: Idc409c85e8aa82b59dc2bc28af6ca2617de82699
When processing keys where the names start with the delimiter
character, swift should list only the delimiter character. To get the
list of nested keys, the caller should also supply the prefix which is
equal to the delimiter.
Added a functional test and unit tests to verify this behavior.
Fixes Bug: 1475018
Change-Id: I27701a31bfa22842c272b7781738e8c546b82cbc
Currently a HTTP_REFERER (Referer) header isn't passed down to
subrequests. This means *LO subrequests to segment containers
return a 403 on a *LO GET when accessed by requests using referer
ACLs.
Currently the only way around referer access to *LO's is to make the
segments container world readable.
This change makes sure the referer header is passed into subrequests
allowing a segments container to only need to be locked down with
the same referer as the *LO container.
This is a 1 line change to code, but also adds a unit and 2 functional
functional tests (one for DLO and one for SLO).
Change-Id: I1fa5328979302d9c8133aa739787c8dae6084f54
Closes-Bug: #1526575
This patch changes functional test classes to subclass
unittest2.TestCase rather than unittest.TestCase.
This fixes errors when attempting to use
tox -e func -- -n <test_path_including_test_method>
and
tox -e func -- --until-failure
Also migrate from using nose.SkipTest to unittest2.SkipTest
Change-Id: I903033f5e01833550b2f2b945894edca4233c4a2
Closes-Bug: 1526725
Co-Authored-By: Ganesh Maharaj Mahalingam <ganesh.mahalingam@intel.com>
Defcore uses Tempest, which uses Test Repository.
This change makes it easier for Defcore to pull functional
tests from Swift and run them. Additionally, using testr
allows tests to be run in parallel.
Concurrency set to 1 for now, >1 causes failures for
reasons that are still TBD.
With switch to ostestr all the server logs are being sent to stdout
which makes it completely unreadable. Suppressing the logs by default
now with a flag to enable it if desired.
Co-Authored-By: John Dickinson <me@not.mn>
Co-Authored-By: Robert Collins <rbtcollins@hpe.com>
Co-Authored-By: Matthew Oliver <matt@oliver.net.au>
Co-Authored-By: Ganesh Maharaj Mahalingam <ganesh.mahalingam@intel.com>
Change-Id: I53ef4a116996a772cf1f3abc2eb0ad60047322d5
Related-Bug: 1177924
Updates the functional test to verify the fix applied by
change Iff7274aa631a92cd7332212ed8b4378c27da4a1f
Change-Id: Iae63ac027e4f4acfe46a36dc1325888b1f834ea4
This change adds the ability to tell the container or account server to
reverse their listings. This is done by sending a reverse=TRUE_VALUE,
Where TRUE_VALUE is one of the values true can be in common/utils:
TRUE_VALUES = set(('true', '1', 'yes', 'on', 't', 'y'))
For example:
curl -i -X GET -H "X-Auth-Token: $TOKEN" $STORAGE_URL/c/?reverse=on
I borrowed the swapping of the markers code from Kevin's old change,
thanks Kevin. And Tim Burke added some real nuggets of awesomeness.
DocImpact
Co-Authored-By: Kevin McDonald <kmcdonald@softlayer.com>
Co-Authored-By: Tim Burke <tim.burke@gmail.com>
Implements: blueprint reverse-object-listing
Change-Id: I5eb655360ac95042877da26d18707aebc11c02f6
a1c32702, 736cf54a, and 38787d0f remove uses of `simplejson` from
various parts of Swift in favor of the standard libary `json`
module (introduced in Python 2.6). This commit performs the remaining
`simplejson` to `json` replacements, removes two comments highlighting
quirks of simplejson with respect to Unicode, and removes the references
to it in setup documentation and requirements.txt.
There were a lot of places where we were importing json from
swift.common.utils, which is less intuitive than a direct `import json`,
so that replacement is made as well.
(And in two more tiny drive-bys, we add some pretty-indenting to an XML
fragment and use `super` rather than naming a base class explicitly.)
Change-Id: I769e88dda7f76ce15cf7ce930dc1874d24f9498a
This commit tries to give the user a reason that their SLO manifest
was invalid instead of just saying "Invalid SLO Manifest File". It
doesn't get every error condition, but it's better than before.
Examples of things that now have real error messages include:
* bad keys in manifest (e.g. using "name" instead of "path")
* bogus range (e.g. "bytes=123-taco")
* multiple ranges (e.g. "bytes=10-20,30-40")
* bad JSON structure (i.e. not a list of objects)
* non-integer size_bytes
Also fixed an annoyance with unspecified-size segments that are too
small. Previously, if you uploaded a segment reference with
'{"size_bytes": null, ...}' in it and the referenced segment was less
than 1 MiB, you'd get a response that looked like this:
HTTP/1.1 400 Bad Request
Content-Length: 62
Content-Type: text/html; charset=UTF-8
X-Trans-Id: txd9ee3b25896642098e4d9-0055dd095a
Date: Wed, 26 Aug 2015 00:33:30 GMT
Each segment, except the last, must be at least 1048576 bytes.
This is true, but not particularly helpful, since it doesn't tell you
which of your segments violated the rule.
Now you get something more like this:
HTTP/1.1 400 Bad Request
Content-Length: 49
Content-Type: text/plain
X-Trans-Id: tx586e52580bac4956ad8e2-0055dd09c2
Date: Wed, 26 Aug 2015 00:35:14 GMT
Errors:
/segs/small, Too Small; each segment, except the last...
It's not exactly a tutorial on SLO manifests, but at least it names
the problematic segment.
This also changes the status code for a self-referential manifest from
409 to 400. The rest of the error machinery was using 400, and
special-casing self-reference would be really annoying. Besides, now
that we're showing more than one error message at a time, what would
the right status code be for a manifest with a self-referential
segment *and* a segment with a bad range? 400? 409? 404.5? It's much
more consistent to just say invalid manifest --> 400.
Change-Id: I2275683230b36bc273319254e37c16b9e9b9d69c
assertEquals is deprecated in py3, replacing it.
Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8
Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
The urllib, urllib2 and urlparse modules of Python 2 were reorganized
into a new urllib namespace on Python 3. Replace urllib, urllib2 and
urlparse imports with six.moves.urllib to make the modified code
compatible with Python 2 and Python 3.
The initial patch was generated by the urllib operation of the sixer
tool on: bin/* swift/ test/.
Change-Id: I61a8c7fb7972eabc7da8dad3b3d34bceee5c5d93
The unicode type was renamed to str in Python 3. Use six.text_type to
make the modified code compatible with Python 2 and Python 3.
The initial patch was generated by the unicode operation of the sixer
tool on: bin/* swift/ test/.
Change-Id: I9e13748ccde36ee8110756202d55d3ae945d4860
Users can now include an optional 'range' field in segment descriptions
to specify which bytes from the underlying object should be used for the
segment data. Only one range may be specified per segment. Note that the
'etag' and 'size_bytes' fields still describe the backing object as a
whole. So, if a user uploads a manifest like:
[{"path": "/con/obj_seg_1", "etag": null, "size_bytes": 1048576,
"range": "0-1023"},
{"path": "/con/obj_seg_2", "etag": null, "size_bytes": 1048576,
"range": "512-4095"},
{"path": "/con/obj_seg_1", "etag": null, "size_bytes": 1048576,
"range": "-2048"}]
then the segment will consist of the first 1024 bytes of /con/obj_seg_1,
followed by bytes 513 through 4096 (inclusive) of /con/obj_seg_2, and
finally bytes 1046528 through 1048576 (i.e., the last 2048 bytes) of
/con/obj_seg_1.
ETag generation for SLOs had been updated to prevent collisions when
using different ranges for the same set of objects.
Additionally, there are two performance enhancements:
* On download, multiple sequential requests for segments from the same
underlying object will be coalesced into a single ranged request,
provided it still does not meet Swift's "egregious range requests"
critieria.
* On upload, multiple sequential segments referencing the same object
will be validated against the response from a single HEAD request.
Change-Id: Ia21d51c2cef4e2ee5162161dd2c1d3069009b52c
DocImpact
Follow up to [1] to add tests for tempurl POSTs not being allowed
to set a DLO manifest header.
[1] I11e68830009d3f6bff44ae4011a41b67139146f6
Change-Id: I7c0ad5a936f71e56c599b8495a586913d3334422
Related-Bug: 1453948
