When responding to a GET request for a manifest, it was intended that
the proxy server lazily fetch the pieces of the container
listing. That way, a single client request doesn't immediately turn
into a bunch of requests to backends. The additional requests should
only get made if the client is putting in the work of receiving the
object body.
However, commit 156f27c accidentally changed this so that all the
pieces of the container listing are eagerly fetched up-front. Better
yet, if an object has more than CONTAINER_LISTING_LIMIT (default
10,000) segments, the container listing is then fetched a second time,
albeit lazily, while streaming out the response.
This commit restores the laziness and adds tests for it.
Change-Id: I49840a7059e6f999ce19199ecb10cdb77358526b
Most of the test files set the HASH_PATH_SUFFIX so you can run the test
file stand alone. This change made it easier for me to run specific
proxy tests separately.
Change-Id: I87d70367dac7f240a2b6779649f8a02cf324ae0f
Ensures that the Allow header is set for 405 responses from the proxy,
as per RFC 2616.
CORS preflight requests now require both the Origin and Access-
Control-Request-Method headers to get a successful (200) response. The
draft spec defines errors as a response with a non-200 status code.
This patch sets the CORS error response code to 401 (Not Authorized).
A later patch may choose to make this configurable.
There is some ambiguity between RFC 2616 and the CORS draft spec
around what to do when a CORS request is made but the cluster has no
CORS information about the requested resource. This patch chooses to
return an error in this case because it is what would be simplest for
CORS client apps.
Further improvements to the OPTIONS verb not included in this patch
include support of more top-level resources (eg / or /v1/) or
sending the configured constraints in the reponse body.
Change-Id: I40be059e8bbf3737dafc4e6fefa7598d05669c60
Fix bug 1067677
When we delete a versioned object, the last version will be poped out from the
versiones container. When a versioned object is expired and deleted by object-
expirer, the last version is restored but remains in the versions container
instead of getting deleted. The reason is object-expirer will set
'X-If-Delete- At' header when deleteing an object. However this is for the
current version - not for the last version. When the object-server is trying
to delete the last version, the transaction will fail with error:
X-If-Delete-At and X-Delete-At do not match. Delete the 'X-If-Delete-At'
field in the later built delete request would help to solve this issue.
This patch, without the test, was first proposed by
Zhou Yuan <yuan.zhou@intel.com>.
Change-Id: I62c97e6a0f140888497e189129825865fb6f7966
Currently, a container's info can be cached without cors data intact after
a container GET.
I made headers_to_container_info a function instead of a method and I crammed
all container metadata into container_info.
This is so e.g. staticweb can eventually re-use the same container info cache.
Fix pep8 in swift/proxy/controllers/container.py
Change-Id: I4bbb042dde79afac48395efc38bd80f0ff240e1f
This change adds multi range retrieval to OpenStack Swift. For non-
segmented data object, a client can use HTTP Range header to specify
multiple ranges to retrieve sections of the data object.
This implementation currently does not support segmented data object
multi range retrieval. When a client sends a multi range request
against a segmented data object, Swift will return HTTP status code
200. Support for segmented data multi range retrieval will be added
in near future.
This implementation is to bring Swift closer to CDMI multi range
data retrieval standard. Once support for segemented data multi
range is added, Swift will be compliant with CDMI standard in
this area.
DocImpact
Change-Id: I4ed1fb0a0a93c037ddb2f551ea62afe447945107
Swift proxy server, when communicating with
the back-end servers ALWAYS sends 100-continue.
Even if the length headers indicate that there
is no body (content-length: 0).
This behavior is not inline with the standard.
RFC2616:8.2.3 disallows 100-continue
for req.content_length==0
This fix removes 100-continue during PUT requests
without a body while maintining the ability for
handoff in case of faliure.
On branch bp/wsgi-application-interface-5
modified: swift/proxy/controllers/obj.py
Fixes: Bug #1070025
Implements Blueprint: wsgi-application-interface
Change-Id: Ia4eb8b886a74968cca4e8bde208641b37f2c104c
The proxy server's ObjectController was performing multiple GET
requests to the object server when processing requests with Range
headers. This was a workaround for a bug in the proxy server's
Controller.GETorHEAD_base method where a response code of 416 from the
object server was incorrectly treated as a bad response from the
backend, the same way a 404 or a 5xx would be.
A 416 (Requested Range Not Satisfiable) from the object server is now
considered a good response. Since the response headers from the object
server already include the X-Object-Manifest header, there's no need
to make a second request (sans Range header) to see if the object is a
manifest.
Bonus fix: updated message for status 416 to match RFC2616.
Bonus fix 2: removed a leftover debugging print() in
test/unit/proxy/test_server.py.
Fixes bug 1065869.
Change-Id: I156af0b463f02ef19a8cfe37092544a599d89b78
fixes bug 1049017
As per the bug report, the -remove- form of deletion only works with
meta headers and not with x-container-read and x-container-write.
This patch by Yukihiro KAWADA detects the container acls and sends
them through to the backend empty like the other metadata values.
patch2 extends metadata-helper in ContainerController tests so that
the new functionality can be tested.
patch3 changes the k.lower().startswith() commands for read/write to
a single k.lower - thanks David :)
patch4 fixes PEP8
patch5 fixes more than two hundred pep8 errors. Except one,
where the pep8 tool complains of overindent, or underindent
depending on the position on the line. No idea how to fix that one.
patch6 fixes the remaining pep8 error - thanks Darrell
Change-Id: I36c2dd72b4636136c5ce7db414bf2b61501090ad
This change replaces WebOb with a mostly compatible local library,
swift.common.swob. Subtle changes to WebOb's API over the years have been a
huge headache. Swift doesn't even run on the current version.
There are a few incompatibilities to simplify the implementation/interface:
* It only implements the header properties we use. More can be easily added.
* Casts header values to str on assignment.
* Response classes ("HTTPNotFound") are no longer subclasses, but partials
on Response, so things like isinstance no longer work on them.
* Unlike newer webob versions, will never return unicode objects.
Change-Id: I76617a0903ee2286b25a821b3c935c86ff95233f
This fixes an issue where Request objects (and related objects) were
not getting garbage collected when a 499 (client disconnect) occurred
for responses that still would have had more than the proxy server's
client chunk size left to send.
Fixed bug #1055834
Change-Id: Ib7b5fffe970dcdec60b90a9bdcd67c49b68fb914
I think not doing this was setting the SIGPIPE handler to signal.SIG_DFL
instead of the Python default of signal.SIG_IGN. This could cause other
tests which make a client stop reading before all data "chunks" are read
to fail harder than they should (i.e. the SIGPIPE there is benign and
even expected--the other side of the socket really did get closed
early).
Change-Id: I946a1913714a34ec677b701a2b5724bc8b715f70
This change refactors container_info to return a dict instead of a tuple
with 6 (and counting...) values.
Also adds container size to cached info.
Change-Id: If68a06ab170504fbf07323354f3d0e073850493d
Change previously hard-coded constants into config variables. This
allows deployers to tune their cluster more specifically based on
their needs. For example, a deployment that uses direct swift access
for public content may need to set a larger header value constraint to
allow for the full object name to be represented in the Content-
Disposition header (which browsers check to determine the name of a
downloaded object).
The new settings are set in the [swift-constraints] section of
/etc/swift/swift.conf. Comments were also added to this config file.
Cleaned up swift/common/constraints.py to pass pep8 1.3.3
Funtional tests now require constraints to be defined in /etc/test.conf or in
/etc/swift/swift.conf (in the case of running the functional tests against a
local swift cluster). To have any hope of tests passing, the defined
constraints must match the constraints on the tested cluster.
Removed a ton of "magic numbers" in both unit and functional tests.
Change-Id: Ie4588e052fd158314ddca6cd8fca9bc793311465
Removed many StatsD logging calls in proxy-server and added
swift-informant-style catch-all logging in the proxy-logger middleware.
Many errors previously rolled into the "proxy-server.<type>.errors"
counter will now appear broken down by response code and with timing
data at: "proxy-server.<type>.<verb>.<status>.timing". Also, bytes
transferred (sum of in + out) will be at:
"proxy-server.<type>.<verb>.<status>.xfer". The proxy-logging
middleware can get its StatsD config from standard vars in [DEFAULT] or
from access_log_statsd_* config vars in its config section.
Similarly to Swift Informant, request methods ("verbs") are filtered
using the new proxy-logging config var, "log_statsd_valid_http_methods"
which defaults to GET, HEAD, POST, PUT, DELETE, and COPY. Requests with
methods not in this list use "BAD_METHOD" for <verb> in the metric name.
To avoid user error, access_log_statsd_valid_http_methods is also
accepted.
Previously, proxy-server metrics used "Account", "Container", and
"Object" for the <type>, but these are now all lowercase.
Updated the admin guide's StatsD docs to reflect the above changes and
also include the "proxy-server.<type>.handoff_count" and
"proxy-server.<type>.handoff_all_count" metrics.
The proxy server now saves off the original req.method and proxy_logging
will use this if it can (both for request logging and as the "<verb>" in
the statsd timing metric). This fixes bug 1025433.
Removed some stale access_log_* related code in proxy/server.py. Also
removed the BaseApplication/Application distinction as it's no longer
necessary.
Fixed up the sample config files a bit (logging lines, mostly).
Fixed typo in SAIO development guide.
Got proxy_logging.py test coverage to 100%.
Fixed proxy_logging.py for PEP8 v1.3.2.
Enhanced test.unit.FakeLogger to track more calls to enable testing
StatsD metric calls.
Change-Id: I45d94cb76450be96d66fcfab56359bdfdc3a2576
If rate_limit_after_segment was 10 in the proxy config, then after 10
segments were coughed up by _load_next_segment() with no calls to
sleep(), the 11th segment would not trigger a sleep() call. The 12th
segment triggered a sleep(0) call, but it was only after the 13th
segment was loaded that an actual rate-limiting (non-zero) sleep got
called.
With this patch, a rate_limit_after_segment of 10 will start sleeping
the correct amount after the 11th segment.
Updated proxy-server.conf-sample with rate_limit_after_segment and
rate_limit_segments_per_sec.
Change-Id: I937c366996e6d6ab47c614d6db470e3be9657c07
As we use the 'account_autocreate' for auto-create accounts, we send a
request to HEAD an account which is just deleted(the account-reaper not
reclaim it immediately), the proxy-server will return 'HTTP 500 Internal
Server Error'.
In my opinion, this is unreasonable, so I change the code in swift/proxy
/server.py for returning a reasonable response. I modified the code in
POST, GETorHEAD and account_info. At last, I modified some code in the
unittest test/unit/proxy/test_server.py.
Change-Id: Ib057b387c9da073d707ffae49ead54206a8fb7dd
Fixes: Bug #1037889
A warning log line is emitted whenever the proxy has to use a handoff
node. Monitoring these warnings can indicate a problem within your
cluster; however, you can disable these log lines by setting the
proxy conf's log_handoffs to false.
While working on this, I also noticed why many proxy log lines did
not have txn_id and client_ip -- subcoroutines. Now the logger thread
locals are copied to the subcoroutines.
Change-Id: Ibac086e1b985f566c068d083620287509de35da8
Based on PatchSet 3 of https://review.openstack.org/#/c/7569/ , make them to pass all funcional tests with both webob 1.x and 1.2.
The additional following compatibility issues were addressed:
- Until patch for range header issue is merged into official webob release, testRangedGetsWithLWSinHeader() should skip test against webob 1.2
(49c175aec2)
- common.constraints.check_utf8() can accept both utf8 str and unicode.
- To convert unicode to utf-8 str if necessary.
- Making proxy_logging can handle invalid utf-8 str
bug 888371
bug 959881
blueprint webob-support
Change-Id: I00e5fd04cd1653259606a4ffdd4926db3c84c496
The X-Object-Manifest header value should be url encoded to allow for
"special" characters to be used. This fix decodes such encoding. A
corresponding fix to python-swiftclient (and any other clients) needs
to be made to url encode the header value when sending.
I also fixed any pep8 1.3.1 violations I found in the test I modified
to verify this.
Change-Id: I67f23dec1dc95ee37354dfdd9897f34758135ee6
Fixed bug 969411
Ranged GET request checked wheter the object is large objects only when response is 416 Requested Range Not Satisfiable.
All ranged GET requests should check the object is large object or not because size of the manifest file can be larger than 0.
Change-Id: Ia3632252d6879a773be881149c10ce0f5ffbcd82
- It has been to its own gerrit project.
- direct_client should follow next.
- Implements blueprint clientbindings.
Change-Id: I3bb50c95eba81302bfec71cb7ce5288b85a41dc0
This fixes 500 server crashes caused by requests such as:
curl -X__init__ "http://your-swift-object-server:6000/sda1/p/a/c/o"
Fixes bug 1005903
Change-Id: I6c0ad39a29e07ce5f46b0fdbd11a53a9a1010a04
Not sure how this got introduced (which really annoys me)
but here's the fix to make sure the content-length /
transfer-encoding headers are set properly. Specifically,
the proxy was sometimes returning transfer-encoding: chunked
with no content-length on 204 No Content responses where it
used to return content-length: 0 and no transfer-encoding
header at all.
Change-Id: I0927d102bc5e4324e38dbbb44be9033a6cd8ee20
While fixing something else, the strange code in the test suite
presented itself. Looks like a massive copy-paste error and a couple
of random oddities.
Change-Id: I191e8cd9299b9336b0600363780d2930a04d1fd5
Documentation, including a list of metrics reported and their semantics,
is in the Admin Guide in a new section, "Reporting Metrics to StatsD".
An optional "metric prefix" may be configured which will be prepended to
every metric name sent to StatsD.
Here is the rationale for doing a deep integration like this versus only
sending metrics to StatsD in middleware. It's the only way to report
some internal activities of Swift in a real-time manner. So to have one
way of reporting to StatsD and one place/style of configuration, even
some things (like, say, timing of PUT requests into the proxy-server)
which could be logged via middleware are consistently logged the same
way (deep integration via the logger delegate methods).
When log_statsd_host is configured, get_logger() injects a
swift.common.utils.StatsdClient object into the logger as
logger.statsd_client. Then a set of delegate methods on LogAdapter
either pass through to the StatsdClient object or become no-ops. This
allows StatsD logging to look like:
self.logger.increment('some.metric.here')
and do the right thing in all cases and with no messy conditional logic.
I wanted to use the pystatsd module for the StatsD client, but the
version on PyPi is lagging the git repo (and is missing both the prefix
functionality and timing_since() method). So I wrote my
swift.common.utils.StatsdClient. The interface is the same as
pystatsd.Client, but the code was written from scratch. It's pretty
simple, and the tests I added cover it. This also frees Swift from an
optional dependency on the pystatsd module, making this feature easier
to enable.
There's test coverage for the new code and all existing tests continue
to pass.
Refactored out _one_audit_pass() method in swift/account/auditor.py and
swift/container/auditor.py.
Fixed some misc. PEP8 violations.
Misc test cleanups and refactorings (particularly the way "fake logging"
is handled).
Change-Id: Ie968a9ae8771f59ee7591e2ae11999c44bfe33b2
This commit introduces a new algorithm for assigning partition
replicas to devices. Basically, the ring builder organizes the devices
into tiers (first zone, then IP/port, then device ID). When placing a
replica, the ring builder looks for the emptiest device (biggest
parts_wanted) in the furthest-away tier.
In the case where zone-count >= replica-count, the new algorithm will
give the same results as the one it replaces. Thus, no migration is
needed.
In the case where zone-count < replica-count, the new algorithm
behaves differently from the old algorithm. The new algorithm will
distribute things evenly at each tier so that the replication is as
high-quality as possible, given the circumstances. The old algorithm
would just crash, so again, no migration is needed.
Handoffs have also been updated to use the new algorithm. When
generating handoff nodes, first the ring looks for nodes in other
zones, then other ips/ports, then any other drive. The first handoff
nodes (the ones in other zones) will be the same as before; this
commit just extends the list of handoff nodes.
The proxy server and replicators have been altered to avoid looking at
the ring's replica count directly. Previously, with a replica count of
C, RingData.get_nodes() and RingData.get_part_nodes() would return
lists of length C, so some other code used the replica count when it
needed the number of nodes. If two of a partition's replicas are on
the same device (e.g. with 3 replicas, 2 devices), then that
assumption is no longer true. Fortunately, all the proxy server and
replicators really needed was the number of nodes returned, which they
already had. (Bonus: now the only code that mentions replica_count
directly is in the ring and the ring builder.)
Change-Id: Iba2929edfc6ece89791890d0635d4763d821a3aa
The test was failing silently. To catch these errors the response
from controller.PUT() is now checked.
Fixes to the TestObjectController.test_PUT_auto_content_type():
* Check for the correct 'Content-Type' header in the fake connect() method
* Pass in the correct number of responses to fake_http_connect()
* Pass a 'transfer-encoding' header to PUT() so that
constraints.check_object_creation() passes
* Pass the correct number of elements in the 'expected' iter to the
test_content_type() method
Change-Id: I8dced7b2f3c7e1b1d292c6995b1faf46f5e863fe
Object versioning in swift is implemented by setting a flag on the container
to tell swift to version all objects in the container. The flag is the
``X-Versions-Location`` header on the container, and its value is the
container where the versions are stored.
When data is ``PUT`` into a versioned container (a container with the
versioning flag turned on), the existing data in the file is redirected to a
new object and the data in the ``PUT`` request is saved as the data for the
versioned object. The new object name (for the previous version) is
``<versions_container>/<object_name>/<timestamp>``, where the timestamp is
generated by converting the ``Last-Modified`` header value of the current
version to a unix timestamp.
A ``GET`` to a versioned object will return the current version of the object
without having to do any request redirects or metadata lookups.
Change-Id: I4fcd723145e02bbb2ec1d3ad356713f5dea43b8b
Added optional max_containers_per_account restriction. If set to a
positive value and if a client tries to perform a container PUT when
at or above the max_containers_per_acount cap, a 403 Forbidden will
be returned with an explanatory message.
This only restricts the proxy server, not any of the background
processes that might need to create containers (replication, for
instance). Also, the container count is cached for the proxy's
recheck_account_existence number of seconds. For these reasons, a
given account could exceed this cap before the 403 Forbidden
responses kick in and therefore this feature should be considered a
"soft" limit.
You may also add accounts to the proxy's max_containers_whitelist
setting to have accounts that ignore this cap.
Change-Id: I74e8fb152de5e78d070ed30006ad4e53f82c8376
Prior to this patch, you removed header metadata (such as
x-account-meta-name or x-container-meta-name) by sending the header
with no value. However, many tools such as curl will not send headers
with empty values.
This patch adds an alternate method for header metadata removal: Send
an x-remove-account-meta-name (x-remove-container-meta-name) header
with any value and the proxy will translate it to the original header
name with no value, indicating removal.
For safety, if you specify both x-remove-account-meta-name and
x-account-meta-name headers in the same request, the -remove header
will be ignored.
Change-Id: Ic220bec05a0e266db85fc8fa50011146ee886d9c
The None, True, and False values are singletons.
All variable *comparisons* to singletons should use 'is' or 'is not'.
All variable *evaluations* to boolean should use 'if' or 'if not'.
All Object type comparisons should use isinstance()
instead of comparing types directly.
Change-Id: I47863c4862791022670619f19b8bc15d8a93fd81
