I've now got 54 pass, 676 skipped with
pipeline = proxy-logging memcache tempauth proxy-logging proxy-server
Could probably even ditch logging, but that'd get a bit painful to
debug.
Change-Id: I86bcd7a2373cf1e7213cb716888307306b7bb50d
unittest2 was needed for Python version <= 2.6, so it hasn't been needed
for quite some time. See unittest2 note one:
https://docs.python.org/2.7/library/unittest.html
This drops unittest2 in favor of the standard unittest module.
Change-Id: I2e787cfbf1709b7f9c889230a10c03689e032957
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
Users are able to change versioning in a container
from X-Versions-Location to X-History-Location, which affects
how DELETEs are handled. We have some unit tests that check this
behavior, but no functional tests.
This patch adds a functional test that helps us understand and
document how changing modes affects the handling of DELETE
requests.
Change-Id: I5dbe5bdca17e624963cb3a3daba3b240cbb4bec4
Previously, versioned_writes middleware copy an already existing
object using PUT. However, SLO requires the additional query
to properly update the object size when listing.
Propose fix: In _put_versioned_obj - which is called when on
creating version obj and also on restoring obj,
if 'X-Object-Sysmeta-Slo-Size' header is present it will
add needed headers for container to update obj size
Added a new functional test case with size assertion for slo
Change-Id: I47e0663e67daea8f1cf4eaf3c47e7c8429fd81bc
Closes-Bug: #1840322
Also, ensure that the stored symlink headers really *are* url-encoded as
we've been assuming.
Change-Id: I1f300d69bec43f0deb430294da05a4ec04308040
Related-Bug: 1774238
Closes-Bug: #1821240
Some tests make use of multiple accounts without checking of they have
been set up. This commit tries to fix some of these situations.
Change-Id: I461679e78e19ce0866c7618c581a8cb573cca7f5
Fix up function tests to actually *do* that quoting, and fix
_listing_pages_iter to respect that.
Change-Id: I1554042510819ea878b4c70417721944115e17f4
Related-Bug: 1229142
Related-Change: I425440f76b8328f8e119d390bfa4c7022181e89e
Related-Bug: 1755554
Related-Change: Ibcd90cc633c68973929ee5249c6598c22b342e3e
With url encoded object name like '%25ff' that can be url-encoded
value after decoded can cause 412 Precondition Failed. And more,
that can do nothing (no versioned object creation) even it returns
a successful response.
The root causes are in versioned_writes middleware as follows:
A. unnecessary unquote in object_request method
B. incorrect use of make_pre_authed_request that takes 'quoted'
path in the args. That is described at [1] explicitely.
This patch resolved those 2 bugs at once, and then, now we can create
%25ff versioned object reported in the launchpad with this patch.
Perhaps, more tests would be nice to have. This patch added a few
test cases on that.
1: https://github.com/openstack/swift/blob/master/swift/common/wsgi.py#L1174
Note that make_subrequest and its caller should have *quoted* path but
make_env should *NOT*. That might make us confused.
Closes-Bug: #1755554
Change-Id: Ibcd90cc633c68973929ee5249c6598c22b342e3e
Functional tests for symlink and versioned writes run and result in
falure even if symlink is not enabled.
This patch fixes the functional tests to run only if both of
symlink and versioned writes are enabled.
Change-Id: I5ffd0b6436e56a805784baf5ceb722effdf74884
The functional test for versioning symlinks is better located in
test_versioned_writes where it can be added to
TestObjectVersioning. This saves duplicated versioned_writes specific
setup code in test_symlink, and has the benefit of the test being
repeated for each of the versioned writes test subclasses. With a
small refactor this includes the test now running with
x-history-location mode as well as x-versions-location mode.
Related-Change: I838ed71bacb3e33916db8dd42c7880d5bb9f8e18
Change-Id: If215446c558b61c1a8aea37ce6be8fcb5a9ea2f4
Currently, our integrity checking for objects is pretty weak when it
comes to object metadata. If the extended attributes on a .data or
.meta file get corrupted in such a way that we can still unpickle it,
we don't have anything that detects that.
This could be especially bad with encrypted etags; if the encrypted
etag (X-Object-Sysmeta-Crypto-Etag or whatever it is) gets some bits
flipped, then we'll cheerfully decrypt the cipherjunk into plainjunk,
then send it to the client. Net effect is that the client sees a GET
response with an ETag that doesn't match the MD5 of the object *and*
Swift has no way of detecting and quarantining this object.
Note that, with an unencrypted object, if the ETag metadatum gets
mangled, then the object will be quarantined by the object server or
auditor, whichever notices first.
As part of this commit, I also ripped out some mocking of
getxattr/setxattr in tests. It appears to be there to allow unit tests
to run on systems where /tmp doesn't support xattrs. However, since
the mock is keyed off of inode number and inode numbers get re-used,
there's lots of leakage between different test runs. On a real FS,
unlinking a file and then creating a new one of the same name will
also reset the xattrs; this isn't the case with the mock.
The mock was pretty old; Ubuntu 12.04 and up all support xattrs in
/tmp, and recent Red Hat / CentOS releases do too. The xattr mock was
added in 2011; maybe it was to support Ubuntu Lucid Lynx?
Bonus: now you can pause a test with the debugger, inspect its files
in /tmp, and actually see the xattrs along with the data.
Since this patch now uses a real filesystem for testing filesystem
operations, tests are skipped if the underlying filesystem does not
support setting xattrs (eg tmpfs or more than 4k of xattrs on ext4).
References to "/tmp" have been replaced with calls to
tempfile.gettempdir(). This will allow setting the TMPDIR envvar in
test setup and getting an XFS filesystem instead of ext4 or tmpfs.
THIS PATCH SIGNIFICANTLY CHANGES TESTING ENVIRONMENTS
With this patch, every test environment will require TMPDIR to be
using a filesystem that supports at least 4k of extended attributes.
Neither ext4 nor tempfs support this. XFS is recommended.
So why all the SkipTests? Why not simply raise an error? We still need
the tests to run on the base image for OpenStack's CI system. Since
we were previously mocking out xattr, there wasn't a problem, but we
also weren't actually testing anything. This patch adds functionality
to validate xattr data, so we need to drop the mock.
`test.unit.skip_if_no_xattrs()` is also imported into `test.functional`
so that functional tests can import it from the functional test
namespace.
The related OpenStack CI infrastructure changes are made in
https://review.openstack.org/#/c/394600/.
Co-Authored-By: John Dickinson <me@not.mn>
Change-Id: I98a37c0d451f4960b7a12f648e4405c6c6716808
While we're at it, have copy and copy_account raise ResponseErrors
on failure, similar to cluster_info, update_metadata, containers, info,
files, delete, initialize, read, sync_metadata, write, and post.
Related-Change: Ia8b92251718d10b1eb44a456f28d3d2569a30003
Change-Id: I9ef42d922a6b7dbf253f2f8f5df83965d8f47e0f
Previously, requests involving DLOs would bypass versioned_writes:
* Any existing DLOs wouldn't get copied to the archive container during
overwrites (or deletes, with history-mode), so there would be no
evidence they had ever existed.
* Any new DLOs wouldn't copy overwritten objects to the archive
container, potentially leading to data loss.
Now, DLOs will behave like every other type of object under
versioned_writes.
Change-Id: I488e13eead2f33dd272d03f6f898adc52fc7fdad
Related-Change: Ie899290b3312e201979eafefb253d1a60b65b837
Related-Change: Ib5b29a19e1d577026deb50fc9d26064a8da81cd7
Closes-Bug: #1626989
Rather than having a comment declaring that the stack/history modes
only diverge after the delete(), move the common code to a separate
function called by tests for both modes.
Change-Id: I657106b11bf0697338a776513d2a51ac6e562c0a
Addresses a TODO in test/functional/test_account.py where
an account metadata test was having to clean up tempurl keys
in the account metadata that were left by another test in
a different module. This cleanup is necessary because tests
in test_account.py fail if there is any pre-existing
account metadata.
This patch:
* makes the tempurl tests clean up their keys from account
metadata.
* makes the test_account.py:TestAccount class remove any
pre-existing metadata before attempting any tests and
replacing that metadata when all the tests in that class
have completed. This is more robust than the existing code
which only removes any tempurl keys that might be in the
account - now you could have x-account-meta-foo = bar in
the test account and test_account.py will still pass.
* consolidates some common setup code currently repeated for
many of the functional test classes into into a BaseEnv class.
Change-Id: I874a9e23dfcdd1caa934945b46089f11b9f6de65
tests.py is currently at ~5500 lines of code, it's
time to break it down into smaller files.
I started with an easy middleware set of tests
(i.e., versioned writes, ~600 lines of code ) so I can get
some feedback. There are more complicated tests that cover
multiple middlewares for example, it is not so clear where
those should go.
Change-Id: I2aa6c18ee5b68d0aae73cc6add8cac6fbf7f33da
Signed-off-by: Thiago da Silva <thiago@redhat.com>
