Even if stdlib socket is probably patched by the time StatsdClient
creates a socket, we want to import the green socket module explicitly
for better testing.
Move test_statsd.py to test_statsd_client.py so it matches the naming
convention of the rest of our test files.
Fix some patching of utils in test_statsd_client to patch
statsd_client.
Rename some vars in test_statsd_client that shadowed the statsd_client
module name.
Move some utils tests out of test_statsd_client and back into
test_utils.
Related-Change: I4b5b12a3b0288b696a39903264741bc862a94ad7
Change-Id: I3de22b7f15dd386fa9c873587782f0dfc4c42a27
Adds ClosingMapper class which is like map() but closes the
iterable.
Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Change-Id: Idd0ac21b365a138b065f01d05a257af62ea88177
This shouldn't impact real servers, as those processes were about to
wrap up anyway. It *can* cause some confusing behaviors in tests,
though.
Change-Id: Ifd8a64efcd3fc983596ba7cd9fe28eb9663c93d6
- Move statsd client into it's own module
- Move all logging functions into their own module
- Move all config functions into their own module
- Move all helper functions into their own module
Partial-Bug: #2015274
Change-Id: Ic4b5005e3efffa8dba17d91a41e46d5c68533f9a
If the global configuration option 'enable_open_expired' is set
to true in the config, then the client will be able to make a
request with the header 'x-open-expired' set to true in order
to access an object that has expired, provided it is in its
grace period. If this config flag is set to false, the client
will not be able to access any expired objects, even with the
header, which is the default behavior unless the flag is set.
When a client sets a 'x-open-expired' header to a true value for a
GET/HEAD/POST request the proxy will forward x-backend-open-expired to
storage server. The storage server will allow clients that set
x-backend-open-expired to open and read an object that has not yet
been reaped by the object-expirer, even after the x-delete-at time
has passed.
The header is always ignored when used with temporary URLs.
Co-Authored-By: Anish Kachinthaya <akachinthaya@nvidia.com>
Related-Change: I106103438c4162a561486ac73a09436e998ae1f0
Change-Id: Ibe7dde0e3bf587d77e14808b169c02f8fb3dddb3
The object expirer can be configured to delay the reaping of
objects from disk after their expiration time using account
and container level delay_reaping values. The delay_reaping
value of accounts and containers in seconds is configured in
the object server config. The object expirer references these
configured values to only reap objects from specified accounts
and containers after their corresponding delays.
The goal of the delay_reaping feature is to prevent accidental or
premature data loss if an object marked for deletion with the
'x-delete-at' feature should not be reaped immediately, for
whatever reason.
Configuring the delay_reaping value at a granular account and
container level is beneficial for being able to keep storage
capacity consumption in control while maintaining a desired
data recovery window.
This patch also adds a sample configuration, documentation, and
tests for bad configurations and grace period functionality.
Co-Authored-By: Anish Kachinthaya <akachinthaya@nvidia.com>
Change-Id: I106103438c4162a561486ac73a09436e998ae1f0
Add some test assertions to cover the first-byte timing metrics
introduced in the related change.
Add ttfb param to log_request docstring.
Change-Id: I530652dd672d7d4e5eac351ccbad318773414f7d
Related-Change: I1611e34846e586703e9d3709fa64e8df41f2d685
Previously, we relied on raising a swob.HTTPException from within a
wrapped wsgi.input and just hoping that it either
- comes uncaught all the way back through the pipeline, or at least
- gets translated back to a 422 response.
This was brittle, however; if any middleware between s3api and the
proxy took s3api's approach of turning unexpected responses into 500s,
for example, it all breaks down.
Take a cue from eventlet's Timeout and create a new s3api-specific
BaseException (which does *not* inherit from Exception) to improve
our ability to cut through all the layers.
Change-Id: I9924ff3b8d7d246631fe61b916823e028e2c01f2
We stuff the access key into the request path until we get back a
more-authoritative account name from auth. But it needs to be a WSGI
string when we do!
Closes-Bug: #2058748
Change-Id: I34adb8141cc9e62d17a27f01c63f40d1dd25991c
Any of these directories may get unlinked between when we saw them in
their parent's directory listing and when we go to descend.
Change-Id: I1dfc0ee1d9e70cb0600557cde980bd5880bd40b3
This change allows individual SLO segments to be downloaded by adding
an extra 'part-number' query parameter to the GET request. You can
also retrieve the Content-Length of an individual segment with a HEAD
request.
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Change-Id: I7af0dc9898ca35f042b52dd5db000072f2c7512e
Add support for a backend_ratelimit_conf_path option in the
[filter:backend_ratelimit] config. If specified then the middleware
will give precedence to config options from that file over config
options from the [filter:backend_ratelimit] section.
The path defaults to /etc/swift/backend-ratelimit.conf.
The config file is periodically reloaded and any changed options are
applied. The middleware will log a warning the first time it fails to
load a config file that had previously been successfully loaded. The
middleware also logs at info level when it first successfully loads a
config file that had previously failed to be loaded. Otherwise, the
middleware will log when a config file is loaded that results in the
config being changed.
Change-Id: I6554e37c6ab5b0a260f99b54169cb90ab5718f81
Currently when the memcachering `_get_conns` method runs out of memcached
servers to try and so fails to yield anything we log a:
All memcached servers error-limited
However, this error message isn't entirely accurate. It can also fail
because it failed to connect all it's memcached servers not just because
they're error limited.
You can disable error-limiting of memcached servers. So in this case
this error message is a red-herring.
Downstream we use a mcrouter client on each node which itself talks to a
bunch of memcache servers. Therefore in swift's memcachering client we
only configure the 1 mcrouter client as a single server in the ring.
Because of this we disable memcached error-limiting.
If the node gets too overloaded we've had timeouts talking to the local
mcrouter client. This fires off error-limitted log messages which can
confuse things.
Because it's possible to turn off error-limiting, the log line isn't
quite adequate anymore. So this patch changes it to:
No more memcached servers to try
Change-Id: I97fb4f3ee2ac45831aae14a782b2c6dc73e82d85
Since we fake out all the greenthread stuff to run in the main thread,
we can (sometimes?) find that a transaction ID has already been set,
leading to failures in test_bad_request_app_logging like
AssertionError: b'X-Trans-Id: test-trans-id' not found
in b'X-Trans-Id: tx...'
By resetting the logger's txn_id, we're assured that our mock will be
run and the expected transaction ID will be used.
Change-Id: I465eed5372a2a5e591f80a09676f4b7f091cd444
Currently, when object-server serves GET request and DiskFile
reader iterate over disk file chunks, there is no explicit
eventlet sleep called. When network outpace the slow disk IO,
it's possible one large and slow GET request could cause
eventlet hub not to schedule any other green threads for a
long period of time. To improve this, this patch add a
configurable sleep parameter into DiskFile reader, which
is 'cooperative_period' with a default value of 0 (disabled).
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Change-Id: I80b04bad0601b6cd6caef35498f89d4ba70a4fd4
The existing test fails on macOS because the value of errno.ENODATA is
platform dependent. On macOS ENODATA is 96:
% man 2 intro|grep ENODATA
96 ENODATA No message available.
Change-Id: Ibc760e641d4351ed771f2321dba27dc4e5b367c1
Object GET requests with a truthy X-Newest header are not resumed if a
backend request times out. The GetOrHeadHandler therefore uses the
regular node_timeout when waiting for a backend connection response,
rather than the possibly shorter recoverable_node_timeout. However,
previously while reading data from a backend response the
recoverable_node_timeout would still be used with X-Newest requests.
This patch simplifies GetOrHeadHandler to never use
recoverable_node_timeout when X-Newest is truthy.
Change-Id: I326278ecb21465f519b281c9f6c2dedbcbb5ff14
Both GetOrHeadHandler (used for replicated policy GETs) and
ECFragGetter (used for EC policy GETs) have _get_next_response_part
methods that are very similar. This patch replaces them with a single
method in the common GetterBase superclass.
Both classes are modified to use *only* the Request instance passed to
their constructors. Previously their entry methods
(GetOrHeadHandler.get_working_response and
ECFragGetter.response_parts_iter) accepted a Request instance as an
arg and the class then variably referred to that or the Request
instance passed to the constructor. Both instances must be the same
and it is therefore safer to only allow the Request to be passed to
the constructor.
The 'newest' keyword arg is dropped from the GetOrHeadHandler
constructor because it is never used.
This refactoring patch makes no intentional behavioral changes, apart
from the text of some error log messages which have been changed to
differentiate replicated object GETs from EC fragment GETs.
Change-Id: I148e158ab046929d188289796abfbbce97dc8d90
