We already require swiftclient>=3.2.0, and have for years.
We can stop checking whether it's 1.x
Related-Change: I9842c9975821bda5c7d8bf2fc214480c0c0a5e96
Change-Id: I798904ab66ca10e21b4999ed7f2be74d1b63584c
No version of eventlet that I'm aware of hasany sort of support for
eventlet.wsgi.WRITE_TIMEOUT; I don't know why we've been setting that.
On the other hand, the socket_timeout argument for eventlet.wsgi.Server
has been supported for a while -- since 0.14 in 2013.
Drive-by: Fix up handling of sub-second client_timeouts.
Change-Id: I1dca3c3a51a83c9d5212ee5a0ad2ba1343c68cf9
Related-Change: I1d4d028ac5e864084a9b7537b140229cb235c7a3
Related-Change: I433c97df99193ec31c863038b9b6fd20bb3705b8
This should make us less likely to fail in the gate because of listing
consistency issues.
Drive-by: Remove the allowed_headers override; that hasn't been necessary
since we updated the default value a while back.
Change-Id: Id280bc93ed6e899a62b0115fdf144a564ef0cd8d
Related-Change: Ib82e175096716e42aecdab48f01f079e09da6a1d
This patch removed the separate s3api, staticweb functional tests
gate jobs and added them across all other functional test jobs.
Change-Id: Ie1c606132a054defc2b3cc14a66031090e7b8449
The repo is Python using both Python 2 and 3 now, so update hacking to
version 2.0 which supports Python 2 and 3. Note that latest hacking
release 3.0 only supports version 3.
Fix problems found.
Remove hacking and friends from lower-constraints, they are not needed
for installation.
Change-Id: I9bd913ee1b32ba1566c420973723296766d1812f
This patch adds a new object versioning mode. This new mode provides
a new set of APIs for users to interact with older versions of an
object. It also changes the naming scheme of older versions and adds
a version-id to each object.
This new mode is not backwards compatible or interchangeable with the
other two modes (i.e., stack and history), especially due to the changes
in the namimg scheme of older versions. This new mode will also serve
as a foundation for adding S3 versioning compatibility in the s3api
middleware.
Note that this does not (yet) support using a versioned container as
a source in container-sync. Container sync should be enhanced to sync
previous versions of objects.
Change-Id: Ic7d39ba425ca324eeb4543a2ce8d03428e2225a1
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Co-Authored-By: Tim Burke <tim.burke@gmail.com>
Co-Authored-By: Thiago da Silva <thiagodasilva@gmail.com>
We were (indirectly) importing swiftclient (and therefore requests and
urllib3) before doing our eventlet monkey-patching. This would lead
boto3 (which digs an SSLContext out of urllib3) to trip RecursionErrors
on py3 similar to
>>> from ssl import SSLContext, PROTOCOL_SSLv23
>>> import eventlet
>>> eventlet.monkey_patch(socket=True)
>>> SSLContext(PROTOCOL_SSLv23).options |= 0
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python3.6/ssl.py", line 465, in options
super(SSLContext, SSLContext).options.__set__(self, value)
File "/usr/lib/python3.6/ssl.py", line 465, in options
super(SSLContext, SSLContext).options.__set__(self, value)
File "/usr/lib/python3.6/ssl.py", line 465, in options
super(SSLContext, SSLContext).options.__set__(self, value)
[Previous line repeated 330 more times]
RecursionError: maximum recursion depth exceeded while calling a Python object
Change-Id: I4bb59edd87336597791416c4f2a096efe0e72fe3
Drive-by: Tighten domain_remap assertions on listings, which required
that we fix proxy pipeline placement. Add a note about it to the sample
config.
Change-Id: I41835148051294088a2c0fb4ed4e7a7b61273e5f
When we fixed bulk response heartbeating in https://review.openstack.org/#/c/510715/,
code review raised the issue of moving the xml header down to after the
early-exit clauses. At the time, it didn't seem to break anything, so
it was left in place. However, that insight was correct.
The purpose of the earlier patch was to force eventlet to use chunked
transfer encoding on the response in order to prevent eventlet from
buffering the whole response, thus defeating the purpose of the
heartbeat responses.
Moving the first line of the body lower (ie after the early exit
checks), allows other headers in a chunked transfer encoding response
to be appropriately processed before sending the headers. Sending the
xml declaration early causes it to get intermingled in the 100-continue
protocol, thus breaking the chunked transfer encoding semantics.
Closes-Bug: #1819252
Change-Id: I072f4dab21cd7cdb81b9e41072eb504131411dc8
You could *try* doing something similar to what we were doing
there over in email.message for py3, but you would end up
breaking pkg_resources (and therefor entrypoints) in the
process.
Drive-by: have mem_diskfile implement more of the diskfile API.
Change-Id: I1ece4b4500ce37408799ee634ed6d7832fb7b721
- ConfigParser.set() requires that the value be a string
- The stdlib HTTP client responses don't have a body property
- We might raise a ResponseError with response=None
- Bodies should be bytes
- Headers should be strings
- Make containers()/files() return native strings
- file() isn't a thing on py3
- format should be a parm, not a header
- Switch sorted() to use key instead of cmp
- Use integer division explicitly
Change-Id: I99d3eebc9d7ec4e8b295352294b831492135c568
Previously we'd use two users, one admin and one unprivileged.
Ceph's s3-tests, however, assume that both users should have access to
create buckets. Further, there are different errors that may be returned
depending on whether you are the *bucket* owner or not when using
s3_acl. So now we've got:
test:tester1 (admin)
test:tester2 (also admin)
test:tester3 (unprivileged)
Change-Id: I0b67c53de3bcadc2c656d86131fca5f2c3114f14
Previously some tests in /test/functional/test_staticweb.py assumed
that domain_remap middleware was configured with the default
storage_domain 'example.com'. As a result those tests would fail if
run against a cluster with domain_remap middleware installed but not
having an 'example.com' storage_domain.
This patch allows a storage_domain to be configured in the test.conf
file so that the tests may be successfully run against any cluster. If
the option is not set then the tests will be skipped.
To ensure that the in process test job func-domain-remap-staticweb
still runs all the tests in test_staticweb.py, its config loader is
modified to add the required storage_domain option to the test config
with the same value used by the domain_remap middleware.
Related-Change: I6bc70039c1b99b8a5bf5a30e9c6877d4f0aa638d
Co-Authored-By: Tim Burke <tim.burke@gmail.com>
Change-Id: Ifab84fc3ef66b9790e96ebb8e5c8061a12636045
...to the proxy-server.
The point is to allow the Swift proxy server to log accurate
client IP addresses when there is a proxy or SSL-terminator between the
client and the Swift proxy server. Example servers supporting this
PROXY protocol:
stud (v1 only)
stunnel
haproxy
hitch (v2 only)
varnish
See http://www.haproxy.org/download/1.7/doc/proxy-protocol.txt
The feature is enabled by adding this to your proxy config file:
[app:proxy-server]
use = egg:swift#proxy
...
require_proxy_protocol = true
The protocol specification states:
The receiver MUST be configured to only receive the protocol
described in this specification and MUST not try to guess
whether the protocol header is present or not.
so valid deployments are:
1) require_proxy_protocol = false (or missing; default is false)
and NOT behind a proxy that adds or proxies existing PROXY lines.
2) require_proxy_protocol = true
and IS behind a proxy that adds or proxies existing PROXY lines.
Specifically, in the default configuration, one cannot send the swift
proxy PROXY lines (no change from before this patch). When this
feature is enabled, one _must_ send PROXY lines.
Change-Id: Icb88902f0a89b8d980c860be032d5e822845d03a
This attempts to import openstack/swift3 package into swift upstream
repository, namespace. This is almost simple porting except following items.
1. Rename swift3 namespace to swift.common.middleware.s3api
1.1 Rename also some conflicted class names (e.g. Request/Response)
2. Port unittests to test/unit/s3api dir to be able to run on the gate.
3. Port functests to test/functional/s3api and setup in-process testing
4. Port docs to doc dir, then address the namespace change.
5. Use get_logger() instead of global logger instance
6. Avoid global conf instance
Ex. fix various minor issue on those steps (e.g. packages, dependencies,
deprecated things)
The details and patch references in the work on feature/s3api are listed
at https://trello.com/b/ZloaZ23t/s3api (completed board)
Note that, because this is just a porting, no new feature is developed since
the last swift3 release, and in the future work, Swift upstream may continue
to work on remaining items for further improvements and the best compatibility
of Amazon S3. Please read the new docs for your deployment and keep track to
know what would be changed in the future releases.
Change-Id: Ib803ea89cfee9a53c429606149159dd136c036fd
Co-Authored-By: Thiago da Silva <thiago@redhat.com>
Co-Authored-By: Tim Burke <tim.burke@gmail.com>
Since Python 2.7, unittest in the standard library has included mulitple
facilities for skipping tests by decorators as well as an exception.
Switch to that directly, rather than importing nose.
Change-Id: I4009033473ea24f0d0faed3670db844f40051f30
Currently, our integrity checking for objects is pretty weak when it
comes to object metadata. If the extended attributes on a .data or
.meta file get corrupted in such a way that we can still unpickle it,
we don't have anything that detects that.
This could be especially bad with encrypted etags; if the encrypted
etag (X-Object-Sysmeta-Crypto-Etag or whatever it is) gets some bits
flipped, then we'll cheerfully decrypt the cipherjunk into plainjunk,
then send it to the client. Net effect is that the client sees a GET
response with an ETag that doesn't match the MD5 of the object *and*
Swift has no way of detecting and quarantining this object.
Note that, with an unencrypted object, if the ETag metadatum gets
mangled, then the object will be quarantined by the object server or
auditor, whichever notices first.
As part of this commit, I also ripped out some mocking of
getxattr/setxattr in tests. It appears to be there to allow unit tests
to run on systems where /tmp doesn't support xattrs. However, since
the mock is keyed off of inode number and inode numbers get re-used,
there's lots of leakage between different test runs. On a real FS,
unlinking a file and then creating a new one of the same name will
also reset the xattrs; this isn't the case with the mock.
The mock was pretty old; Ubuntu 12.04 and up all support xattrs in
/tmp, and recent Red Hat / CentOS releases do too. The xattr mock was
added in 2011; maybe it was to support Ubuntu Lucid Lynx?
Bonus: now you can pause a test with the debugger, inspect its files
in /tmp, and actually see the xattrs along with the data.
Since this patch now uses a real filesystem for testing filesystem
operations, tests are skipped if the underlying filesystem does not
support setting xattrs (eg tmpfs or more than 4k of xattrs on ext4).
References to "/tmp" have been replaced with calls to
tempfile.gettempdir(). This will allow setting the TMPDIR envvar in
test setup and getting an XFS filesystem instead of ext4 or tmpfs.
THIS PATCH SIGNIFICANTLY CHANGES TESTING ENVIRONMENTS
With this patch, every test environment will require TMPDIR to be
using a filesystem that supports at least 4k of extended attributes.
Neither ext4 nor tempfs support this. XFS is recommended.
So why all the SkipTests? Why not simply raise an error? We still need
the tests to run on the base image for OpenStack's CI system. Since
we were previously mocking out xattr, there wasn't a problem, but we
also weren't actually testing anything. This patch adds functionality
to validate xattr data, so we need to drop the mock.
`test.unit.skip_if_no_xattrs()` is also imported into `test.functional`
so that functional tests can import it from the functional test
namespace.
The related OpenStack CI infrastructure changes are made in
https://review.openstack.org/#/c/394600/.
Co-Authored-By: John Dickinson <me@not.mn>
Change-Id: I98a37c0d451f4960b7a12f648e4405c6c6716808
It was deprecated and we discussed on this topic in Denver PTG
for Queen cycle. Main motivation for this work is that deprecated
post_as_copy option and its gate blocks future symlink work.
Change-Id: I411893db1565864ed5beb6ae75c38b982a574476
If swift-recon/swift-get-nodes/swift-object-info is used with the
swiftdir option they will read rings from the given directory; however
they are still using /etc/swift/swift.conf to find the policies on the
current node.
This makes it impossible to maintain a local swift.conf copy (if you
don't have write access to /etc/swift) or check multiple clusters from
the same node.
Until now swift-recon was also not usable with storage policy aliases,
this patch fixes this as well.
Closes-Bug: 1577582
Closes-Bug: 1604707
Closes-Bug: 1617951
Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Co-Authored-By: Thiago da Silva <thiago@redhat.com>
Change-Id: I13188d42ec19e32e4420739eacd1e5b454af2ae3
Recently out gate started blowing up intermittently with a strange
case of ports mixed up. Sometimes a functional tests tries to
authorize on a port that's clearly an object server port, and
the like. As it turns out, eventlet developers added an unavoidable
SO_REUSEPORT into listen(), which makes listen(("localhost",0)
to reuse ports.
There's an issue about it:
https://github.com/eventlet/eventlet/issues/411
This patch is working around the problem while eventlet people
consider the issue.
Change-Id: I67522909f96495a6a30e1acdb79835dce2189549
We've seen some failures in the gate like
==============================
Failed 1 tests - output below:
==============================
setUpModule (test.functional.test_account)
------------------------------------------
Captured traceback:
~~~~~~~~~~~~~~~~~~~
Traceback (most recent call last):
File "test/functional/test_account.py", line 33, in setUpModule
tf.setup_package()
File "test/functional/__init__.py", line 716, in setup_package
mem_object_server if in_mem_obj else object_server))
File "test/functional/__init__.py", line 621, in in_process_setup
create_account(AUTH_test)
File "test/functional/__init__.py", line 619, in create_account
assert(resp.status == 201)
AssertionError
...which aren't terribly useful in figuring out what went wrong.
Change-Id: I3cd31bb480dc8508828fe21416bfae33bc0985b7
Add support for a 2+1 EC policy to be optionally used as default
policy when running in process functional tests.
The EC policy may be selected by setting the env var:
SWIFT_TEST_IN_PROCESS_CONF_LOADER=ec tox
when running .functests, or by using the new tox test env:
tox -e func-ec
Change-Id: I02e3553a74a024efdab91dcd609ac1cf4e4f3208
This patch is follow up for [1] and [2] to add new functional
tests for versioned_writes middlware 'history' mode.
(i.e. using X-History-Location header to a container).
The new test class, TestObjectHistoryModeVersioning, will use obvious
setting the mode via new X-History-Location header, since the change [2],
the setting X-Versions-Mode header added since [1] for incomming request has
been deprecated. Hence, since [2], the syntax for stack mode is back to
be same with older Swift than [1] so that the only thing we need now is
just adding a test suite for the new X-History-location.
It means the API has been changing like:
---------------
For stack mode:
---------------
Older than [1]:
X-Versions-Location
[1]~[2]:
X-Vesions-Location (and X-Versions-Mode: 'stack' for obvious)
Newer than [2]:
X-Vesions-Location
-----------------
For history mode:
-----------------
Older than [1]:
(Not supported)
[1]~[2]:
X-Vesions-Location and X-Versions-Mode: 'history'
Newer than [2]:
X-History-Location
Note that this functional tests work on newer swift than [2].
And then, this patch also sets allow_versioned_writes=True
for in-process testing (the container server allow_versions
option was already set, so this is just enabling in the middleware
too). That means that in-process functional tests (such as run by
the tox envs func-in-process-*) because history mode requires the
middleware allow_versioned_writes option to be explicity set to True.
1: https://review.openstack.org/#/c/214922/
2: https://review.openstack.org/#/c/373537/
Co-Authored-By: Alistair Coles <alistair.coles@hpe.com>
Related-Change: I555dc17fefd0aa9ade681aa156da24e018ebe74b
Related-Change: Icfd0f481d4e40dd5375c737190aea7ee8dbc3bf9
Change-Id: Ifebc1c3ce558b1df9e576a58a4100f2219dfc7e7
Running functional tests in the in-process mode uses
the default value for the pipeline. This patch adds support
to specify the SWIFT_TEST_IN_PROCESS_CONF_LOADER variable
to point to a labeled function that changes the proxy
configuration for the functional test.
The patch also adds a new tox environment
func-in-process-encryption
which runs with the environment variable
SWIFT_TEST_IN_PROCESS_CONF_LOADER=encryption
The motivation for this change is to put support in place for an
upstream CI job that will functionally test using encryption
middleware in the pipeline. The gate job is proposed at:
https://review.openstack.org/#/c/348292/
Change-Id: I15c4b20f1d2be57ae21c69c614f6a9579145bee9
setup_package and teardown_package get called twice when
running functional tests using nose. With in-process mode
the first call to teardown_package clears global config
which causes the test swiftclient to raise a SkipTest error
during the second call to teardown_package.
To reproduce:
SWIFT_TEST_IN_PROCESS=1 nosetests ./test/functional/test_account.py
This patch simply tests for config existence before attempting
to create a test swiftclient in the teardown_package function.
Also fix a related issue whereby in_process flag would be reset to
False during second call to setup_package, thus causing some of the
in process cleanup in teardown_package to never be executed.
Change-Id: I074dcd3d39aa46b262632024b047556ca471e8b8
Running functional tests in the in-process mode uses
the default value for proxy-server object_post_as_copy,
which is True. This patch adds support for an environment
variable to change this value to False so that the fast-post
mode is tested when running in-process functional tests.
The patch also adds a new tox environment func-in-process-fast-post
which forces in-process functional testing with
object_post_as_copy=False.
The motivation for this change, apart from enabling configurable
local testing, is to put support in place for an upstream CI job
that will actually functionally test the fast-post mode, which is
otherwise only covered by unit tests.
There are currently two gate jobs that run the functional tests:
- gate-swift-dsvm-functional runs the tests *twice* against a
devstack swift service, once using tempauth and once using
keystoneauth. The devstack swift service uses the default
object_post_as_copy=True.
- gate-swift-tox-func runs the func tests in in-process mode
which also uses tempauth and object_post_as_copy=True. This
duplicates one of the config scenarios above.
With this change either the gate-swift-tox-func job or a new job
could run the functional tests using object_post_as_copy=False.
Change-Id: Ia37f6df1dc38e44ef7404fbf0a52f6fc22fae0c2
This patch adds test cases for PUT, DELETE, GET, HEAD, POST and OPTIONS
requests to accounts, containers and objects using various combinations
of users/projects, roles and/or service tokens.
Change-Id: Iea8141ac74ad949a3ae7fa47fda3135d0f2612f6
This patch changes functional test classes to subclass
unittest2.TestCase rather than unittest.TestCase.
This fixes errors when attempting to use
tox -e func -- -n <test_path_including_test_method>
and
tox -e func -- --until-failure
Also migrate from using nose.SkipTest to unittest2.SkipTest
Change-Id: I903033f5e01833550b2f2b945894edca4233c4a2
Closes-Bug: 1526725
Co-Authored-By: Ganesh Maharaj Mahalingam <ganesh.mahalingam@intel.com>
Defcore uses Tempest, which uses Test Repository.
This change makes it easier for Defcore to pull functional
tests from Swift and run them. Additionally, using testr
allows tests to be run in parallel.
Concurrency set to 1 for now, >1 causes failures for
reasons that are still TBD.
With switch to ostestr all the server logs are being sent to stdout
which makes it completely unreadable. Suppressing the logs by default
now with a flag to enable it if desired.
Co-Authored-By: John Dickinson <me@not.mn>
Co-Authored-By: Robert Collins <rbtcollins@hpe.com>
Co-Authored-By: Matthew Oliver <matt@oliver.net.au>
Co-Authored-By: Ganesh Maharaj Mahalingam <ganesh.mahalingam@intel.com>
Change-Id: I53ef4a116996a772cf1f3abc2eb0ad60047322d5
Related-Bug: 1177924
The urllib, urllib2 and urlparse modules of Python 2 were reorganized
into a new urllib namespace on Python 3. Replace urllib, urllib2 and
urlparse imports with six.moves.urllib to make the modified code
compatible with Python 2 and Python 3.
The initial patch was generated by the urllib operation of the sixer
tool on: bin/* swift/ test/.
Change-Id: I61a8c7fb7972eabc7da8dad3b3d34bceee5c5d93
'print' function is compatible with 2.x and 3.x python versions
Link : https://www.python.org/dev/peps/pep-3105/
Python 2.6 has a __future__ import that removes print as language syntax,
letting you use the functional form instead
Change-Id: I416c6ac21ccbfb91ec328ffb1ed21e492ef52d58
