Previously some tests in /test/functional/test_staticweb.py assumed
that domain_remap middleware was configured with the default
storage_domain 'example.com'. As a result those tests would fail if
run against a cluster with domain_remap middleware installed but not
having an 'example.com' storage_domain.
This patch allows a storage_domain to be configured in the test.conf
file so that the tests may be successfully run against any cluster. If
the option is not set then the tests will be skipped.
To ensure that the in process test job func-domain-remap-staticweb
still runs all the tests in test_staticweb.py, its config loader is
modified to add the required storage_domain option to the test config
with the same value used by the domain_remap middleware.
Related-Change: I6bc70039c1b99b8a5bf5a30e9c6877d4f0aa638d
Co-Authored-By: Tim Burke <tim.burke@gmail.com>
Change-Id: Ifab84fc3ef66b9790e96ebb8e5c8061a12636045
Add test assertions to verify that the related change fixes usync diff
stats being erroneously incremented as a side-effect of syncing shard
ranges when the object tables are in sync.
Related-Change: I2630bb127841837b35e7786b59895fa50090719b
Change-Id: Idffe93c63d16e74ea9ca42b33a636c0c0d9e35b5
Lock timeouts happen, especially on very busy containers. It's not
worth a stack trace, and it's not an error. Now we still log the
lock timeout, but at INFO level and without a stack trace.
Change-Id: Ia202c876fb726f9ae6040d875a83105dd975539b
We probably don't need an extra INFO-level message every time a user
makes a PUT request with X-Copy-From set. It's still in the proxy's
access logs.
Change-Id: I32ce8ff16cb296dd5acef07202a59bc5125111c1
...instead of 10,000,000. The sample configs were already using one
million, all of our testing with non-SAIO containers was done with
one million, and the resulting container DBs were around 100MB which
seems like a comfortable size. Pretty sure this was just a typo during
some code cleanup.
Change-Id: Icd31f9d8efaac2d5dc0f021cad550687859558b9
in process of container update, when locking one container failed,
container updater stops with LockTimeout exception.
it's better for updater server to continue processing
other containers, partitions instead of stop updating.
this path is to catch timeout exception and log it,
then continue processing other containers and partitions.
Closes-bug: #1400939
Change-Id: I42adec07d980be22044a5d4ef6771318a3eed168
The mock call got pulled out in the related change.
Change-Id: Ife1e452fa34026ac934e88a05708cdc5978215e1
Related-Change-Id: Id089e29e7ecfc8cec79c520f604aa01bdae0dcf0
Fix up function tests to actually *do* that quoting, and fix
_listing_pages_iter to respect that.
Change-Id: I1554042510819ea878b4c70417721944115e17f4
Related-Bug: 1229142
Related-Change: I425440f76b8328f8e119d390bfa4c7022181e89e
Related-Bug: 1755554
Related-Change: Ibcd90cc633c68973929ee5249c6598c22b342e3e
For a while, this was conditional because we supported old Eventlet
versions that didn't have this keyword arg. Now, we require new-enough
Eventlet that it's always available, so let's get rid of the
conditional crud.
The flag was introduced in Eventlet 0.15, and we require >= 0.17.4.
Change-Id: Id089e29e7ecfc8cec79c520f604aa01bdae0dcf0
...to the proxy-server.
The point is to allow the Swift proxy server to log accurate
client IP addresses when there is a proxy or SSL-terminator between the
client and the Swift proxy server. Example servers supporting this
PROXY protocol:
stud (v1 only)
stunnel
haproxy
hitch (v2 only)
varnish
See http://www.haproxy.org/download/1.7/doc/proxy-protocol.txt
The feature is enabled by adding this to your proxy config file:
[app:proxy-server]
use = egg:swift#proxy
...
require_proxy_protocol = true
The protocol specification states:
The receiver MUST be configured to only receive the protocol
described in this specification and MUST not try to guess
whether the protocol header is present or not.
so valid deployments are:
1) require_proxy_protocol = false (or missing; default is false)
and NOT behind a proxy that adds or proxies existing PROXY lines.
2) require_proxy_protocol = true
and IS behind a proxy that adds or proxies existing PROXY lines.
Specifically, in the default configuration, one cannot send the swift
proxy PROXY lines (no change from before this patch). When this
feature is enabled, one _must_ send PROXY lines.
Change-Id: Icb88902f0a89b8d980c860be032d5e822845d03a
If an account contains non-ASCII characters, currently SLO delete code
will fail, as get_slo_segments() method receives a unicode object, but
UTF-8 encoded account name. Attempting to concatenate the strings fails
with a UnicodeError, as it tries to use the ASCII codec to decode the
UTF-8 encoded account name.
This patch allows accounts with non-ASCII characters in their names to
delete SLOs.
Change-Id: I619d41e62c16b25bd5f58d300a3dc71aa4dc75c2
In the object server's PUT, POST, and DELETE handlers, we use the
request's X-Timestamp value for checking object expiration. In the GET
and HEAD handlers, we use it if present, but default to the current
time. That way, one can still use curl to make direct object GET or
HEAD requests as before.
If one object server's clock is ahead of the proxy server's clock for
some reason, and a client makes a POST request to update X-Delete-At,
then the skewed-time object server may refuse the new X-Delete-At
value.
In a cluster where two of the three replicas for an object live on the
same time-skewed node, this can result in confusing behavior for
clients. A client can make a POST request to update X-Delete-At,
receive a 400, and then discover later that the X-Delete-At value was
updated anyway, since one object server accepted the POST and
replication spread the new metadata around.
DELETE is somewhat less confusing. The client might get a spurious 404
in the above case, but the object will still be removed.
For PUT, an object server with a slow clock might refuse to overwrite
an object with an "older" one because it believes the on-disk object
is newer than the current time.
Change-Id: I10c28f97d4c6aca1d64bef3b93506cfbb50ade30
Some suggestions in the review of the related change:
* use maybe_get in another obvious place
* make maybe_gets test assertions stronger
Related-Change-Id: Ifd54d76ab1a5a9d82848f3cae89c3e53134aa129
Change-Id: I751f1086d885c18d938f18e8afe1dd0e9c0c57e5
We refactor a bunch of methods to support re-using an open connection
when available. There's some code in the connection manager to support
nesting connections to avoid deadlocks when we call a method that can
open a connection while we already have a connection checked out - but
it might be better in the long run if we just got better at passing open
connections around whenever possible.
Add a helper method on the base db broker class to make it easier to
write methods that can optionally take an existing connection.
Change-Id: Ifd54d76ab1a5a9d82848f3cae89c3e53134aa129
With url encoded object name like '%25ff' that can be url-encoded
value after decoded can cause 412 Precondition Failed. And more,
that can do nothing (no versioned object creation) even it returns
a successful response.
The root causes are in versioned_writes middleware as follows:
A. unnecessary unquote in object_request method
B. incorrect use of make_pre_authed_request that takes 'quoted'
path in the args. That is described at [1] explicitely.
This patch resolved those 2 bugs at once, and then, now we can create
%25ff versioned object reported in the launchpad with this patch.
Perhaps, more tests would be nice to have. This patch added a few
test cases on that.
1: https://github.com/openstack/swift/blob/master/swift/common/wsgi.py#L1174
Note that make_subrequest and its caller should have *quoted* path but
make_env should *NOT*. That might make us confused.
Closes-Bug: #1755554
Change-Id: Ibcd90cc633c68973929ee5249c6598c22b342e3e
Previously test_misplaced_object_movement() deleted objects from both
shards and then relied on override-partitions option to selectively
run the sharder on root or shard containers and thereby control when
each shard range was identified for shrinking. This approach is flawed
when the second shard container lands in the same partition as the
root: running the sharder on the empty second shard's partition would
also cause the sharder to process the root and identify the second
shard for shrinking, resulting in premature shrinking of the second
shard.
Now, objects are only deleted from each shard range as that shard is
wanted to shrink.
Change-Id: I9f51621e8414e446e4d3f3b5027f6c40e01192c3
Drive-by: use the run_sharders() helper more often.
Verify the log line truncation method is actually called.
Related-Change: If063a12cac74b67078b6db1c4f489160a2a69de1
Change-Id: I8dcd0eac1396b251a2c2a31e167598bc1e48c463
When building a listing from shard containers, objects fetched from
each shard range are appended to the existing listing provided their
name is greater than the last entry in the current listing and less
than or equal to the fetched shard range. This allows misplaced
objects below the shard range to possibly be included in the listing
in correct name order. Previously that behaviour only occurred if the
existing listing had entries, but now it occurs even if no objects
have yet been found.
Change-Id: I25cab53b9aa2252c98ebcf70aafb9d39887a11f1
Before, merge_objects() always used storage policy index of 0 when
inserting a fake misplaced object into a shard container. If the shard
broker had a different policy index then the misplaced object would
not show in listings causing test_misplaced_object_movement() to
fail. This test bug might be exposed by having policy index 0 be an EC
policy, since the probe test requires a replication policy and would
therefore choose a non-zero policy index.
The fix is simply to specify the shard's policy index when inserting
the fake object.
Change-Id: Iec3f8ec29950220bb1b2ead9abfdfb1a261517d6
