- Expose cluster-wide read-only status in /info
- Use a set instead of list for write_methods
- Provide a more explicit response when returning 405
- Consider self.read_only in account_read_only
- Use return_values instead of callables for more mocks
Change-Id: I45387067a69919706800df3a8ca12ae8f5d16d90
When running in multiprocess mode, the object reconstructor would
periodically aggregate its workers' recon data into a single recon
measurement. However, at the end of the run, all that was left in
recon was the last periodic measurement; any work that took place
after that point was not recored in the aggregate. However, it was
recorded in the per-disk stats that the worker processes emitted.
This commit adds a final recon aggregation after the worker processes
have finished.
Change-Id: Ia6a3a931e9e7a23824765b2ab111a5492e509be8
Add test assertions to verify that the related change fixes usync diff
stats being erroneously incremented as a side-effect of syncing shard
ranges when the object tables are in sync.
Related-Change: I2630bb127841837b35e7786b59895fa50090719b
Change-Id: Idffe93c63d16e74ea9ca42b33a636c0c0d9e35b5
The object updater emits a log line saying "Object update sweep
completed on <disk> in <time>: <lots-of-stats>" after finishing each
storage policy's async pendings. That's not right; an update sweep is
completed on a device when you've looked at all the async pendings on
the device.
This commit moves the "sweep completed" message so it's once per
device, not per policy.
Change-Id: Ibb5f02a18ac431da230afe30c556a756f60d1680
This patch adds a read_only middleware to swift. It gives the ability
to make an entire cluster or individual accounts read only.
When a cluster or an account is in read only mode, requests that would
result in writes to the cluser are not allowed.
DocImpact
Change-Id: I7e0743aecd60b171bbcefcc8b6e1f3fd4cef2478
Lock timeouts happen, especially on very busy containers. It's not
worth a stack trace, and it's not an error. Now we still log the
lock timeout, but at INFO level and without a stack trace.
Change-Id: Ia202c876fb726f9ae6040d875a83105dd975539b
We probably don't need an extra INFO-level message every time a user
makes a PUT request with X-Copy-From set. It's still in the proxy's
access logs.
Change-Id: I32ce8ff16cb296dd5acef07202a59bc5125111c1
...instead of 10,000,000. The sample configs were already using one
million, all of our testing with non-SAIO containers was done with
one million, and the resulting container DBs were around 100MB which
seems like a comfortable size. Pretty sure this was just a typo during
some code cleanup.
Change-Id: Icd31f9d8efaac2d5dc0f021cad550687859558b9
in process of container update, when locking one container failed,
container updater stops with LockTimeout exception.
it's better for updater server to continue processing
other containers, partitions instead of stop updating.
this path is to catch timeout exception and log it,
then continue processing other containers and partitions.
Closes-bug: #1400939
Change-Id: I42adec07d980be22044a5d4ef6771318a3eed168
The mock call got pulled out in the related change.
Change-Id: Ife1e452fa34026ac934e88a05708cdc5978215e1
Related-Change-Id: Id089e29e7ecfc8cec79c520f604aa01bdae0dcf0
Fix up function tests to actually *do* that quoting, and fix
_listing_pages_iter to respect that.
Change-Id: I1554042510819ea878b4c70417721944115e17f4
Related-Bug: 1229142
Related-Change: I425440f76b8328f8e119d390bfa4c7022181e89e
Related-Bug: 1755554
Related-Change: Ibcd90cc633c68973929ee5249c6598c22b342e3e
For a while, this was conditional because we supported old Eventlet
versions that didn't have this keyword arg. Now, we require new-enough
Eventlet that it's always available, so let's get rid of the
conditional crud.
The flag was introduced in Eventlet 0.15, and we require >= 0.17.4.
Change-Id: Id089e29e7ecfc8cec79c520f604aa01bdae0dcf0
...to the proxy-server.
The point is to allow the Swift proxy server to log accurate
client IP addresses when there is a proxy or SSL-terminator between the
client and the Swift proxy server. Example servers supporting this
PROXY protocol:
stud (v1 only)
stunnel
haproxy
hitch (v2 only)
varnish
See http://www.haproxy.org/download/1.7/doc/proxy-protocol.txt
The feature is enabled by adding this to your proxy config file:
[app:proxy-server]
use = egg:swift#proxy
...
require_proxy_protocol = true
The protocol specification states:
The receiver MUST be configured to only receive the protocol
described in this specification and MUST not try to guess
whether the protocol header is present or not.
so valid deployments are:
1) require_proxy_protocol = false (or missing; default is false)
and NOT behind a proxy that adds or proxies existing PROXY lines.
2) require_proxy_protocol = true
and IS behind a proxy that adds or proxies existing PROXY lines.
Specifically, in the default configuration, one cannot send the swift
proxy PROXY lines (no change from before this patch). When this
feature is enabled, one _must_ send PROXY lines.
Change-Id: Icb88902f0a89b8d980c860be032d5e822845d03a
If an account contains non-ASCII characters, currently SLO delete code
will fail, as get_slo_segments() method receives a unicode object, but
UTF-8 encoded account name. Attempting to concatenate the strings fails
with a UnicodeError, as it tries to use the ASCII codec to decode the
UTF-8 encoded account name.
This patch allows accounts with non-ASCII characters in their names to
delete SLOs.
Change-Id: I619d41e62c16b25bd5f58d300a3dc71aa4dc75c2
In the object server's PUT, POST, and DELETE handlers, we use the
request's X-Timestamp value for checking object expiration. In the GET
and HEAD handlers, we use it if present, but default to the current
time. That way, one can still use curl to make direct object GET or
HEAD requests as before.
If one object server's clock is ahead of the proxy server's clock for
some reason, and a client makes a POST request to update X-Delete-At,
then the skewed-time object server may refuse the new X-Delete-At
value.
In a cluster where two of the three replicas for an object live on the
same time-skewed node, this can result in confusing behavior for
clients. A client can make a POST request to update X-Delete-At,
receive a 400, and then discover later that the X-Delete-At value was
updated anyway, since one object server accepted the POST and
replication spread the new metadata around.
DELETE is somewhat less confusing. The client might get a spurious 404
in the above case, but the object will still be removed.
For PUT, an object server with a slow clock might refuse to overwrite
an object with an "older" one because it believes the on-disk object
is newer than the current time.
Change-Id: I10c28f97d4c6aca1d64bef3b93506cfbb50ade30
Some suggestions in the review of the related change:
* use maybe_get in another obvious place
* make maybe_gets test assertions stronger
Related-Change-Id: Ifd54d76ab1a5a9d82848f3cae89c3e53134aa129
Change-Id: I751f1086d885c18d938f18e8afe1dd0e9c0c57e5
We refactor a bunch of methods to support re-using an open connection
when available. There's some code in the connection manager to support
nesting connections to avoid deadlocks when we call a method that can
open a connection while we already have a connection checked out - but
it might be better in the long run if we just got better at passing open
connections around whenever possible.
Add a helper method on the base db broker class to make it easier to
write methods that can optionally take an existing connection.
Change-Id: Ifd54d76ab1a5a9d82848f3cae89c3e53134aa129
With url encoded object name like '%25ff' that can be url-encoded
value after decoded can cause 412 Precondition Failed. And more,
that can do nothing (no versioned object creation) even it returns
a successful response.
The root causes are in versioned_writes middleware as follows:
A. unnecessary unquote in object_request method
B. incorrect use of make_pre_authed_request that takes 'quoted'
path in the args. That is described at [1] explicitely.
This patch resolved those 2 bugs at once, and then, now we can create
%25ff versioned object reported in the launchpad with this patch.
Perhaps, more tests would be nice to have. This patch added a few
test cases on that.
1: https://github.com/openstack/swift/blob/master/swift/common/wsgi.py#L1174
Note that make_subrequest and its caller should have *quoted* path but
make_env should *NOT*. That might make us confused.
Closes-Bug: #1755554
Change-Id: Ibcd90cc633c68973929ee5249c6598c22b342e3e
