This is needed for SOS (along with patch
https://github.com/dpgoetz/sos/pull/37)
to work with swift 1.12 . By spec you should always use the absolute
location but this causes a problem with staticweb over a cdn using a
cname. Basically you want to be able to forward the browser to a
relative location instead of whatever full url the proxy server
thinks you are using.
Change-Id: I3fa1d415bf9b566be069458b838f7e65db0c4f39
The proxy server was calling swob.Request.path_info_pop() prior to
instantiating a controller so that req.path_info was just /a/c/o (sans
/v1). The version got moved over into SCRIPT_NAME.
This lead to some unfortunate behavior when trying to re-use a request
from middleware. Something like this:
# Imagine we're a WSGIContext object here.
#
# To start, SCRIPT_NAME = '' and PATH_INFO='/v1/a/c/o'
resp_iter = self._app_call(env, start_response)
# Now SCRIPT_NAME='/v1' and PATH_INFO ='/a/c/o'
if something_special in self._response_headers:
env['REQUEST_METHOD'] = 'GET'
env.pop('HTTP_RANGE', None)
# 404 SURPRISE! The proxy calls path_info_pop() again,
# and now SCRIPT_NAME='/v1/a' and PATH_INFO='/c/o', so this
# gets treated as a container request. Yikes.
resp_iter = self._app_call(env, start_response)
Now we just leave SCRIPT_NAME and PATH_INFO alone. To make life easy
for everyone who does want just /a/c/o, I defined
swob.Request.swift_entity_path, which just strips off the /v1.
Note that there's still one call to path_info_pop() in tempauth, but
that's only for requests going to /auth, so it won't affect Swift API
requests. It might be a good idea to remove that one later, but let's
do one thing at a time.
Change-Id: I87557a11c01f3f3889b610578cda6ba7d3933e7a
PEP 333 (WSGI) says that if your iterable has a close() method, the
framework must call it.
WSGIContext._app_call pulls the first chunk off the returned iterable
to make sure that it gets status and headers, and then it would
itertools.chain() that first chunk back onto the iterable so the whole
body went out. swob.Response.call_application() does it too.
The problem is that an itertools.chain object doesn't have a close()
method, so your iterable's fancy-pants close() method has no chance of
getting called.
This patch adds a slightly smarter CloseableChain that works like
itertools.chain, but has a close() method that calls the underlying
iterables' close() methods, if any.
Change-Id: If975c93f53c27dfa0c2f52f4bbf599af25202f70
Per http://www.ietf.org/rfc/rfc2616.txt, when a 401 error is returned, the
Www-Authenticate response header MUST also be returned. The format is
described in http://www.ietf.org/rfc/rfc2617.txt.
Swift supports and/or implements a number of authentication schemes
including tempauth, Keystone, tempurl, formpost and container sync. In
this fix, we use a catch-all, "Swift". The realm is the account (where
known) or "unknown" (bad path or where the 401 is returned from code
that does not have the request). Examples:
Www-Authenticate: Swift realm="AUTH_1234567889"
Www-Authenticate: Swift realm="unknown"
Fixes bug #1215491
Change-Id: I03362789318dfa156d3733ef9348795062a9cfc4
except x,y: was deprected and is removed in Python 3.x.
Use "except x as y:" instead which works in any Python
version >= 2.6.
Change-Id: I7008c74b807340f3457d3a0c8bd0b83f23169d14
Fixes a warning triggered by Hacking 0.7.x or newer. There
is no need to use a positional string formatting here, since
this is not going to be localized.
Change-Id: Ie38d620aecb0b48cd113af45cc9ca0d61f8f8ff1
I knew webob.Request.blank could take most of the attributes on the class as
kwargs to blank, so I went and looked how. It seems to work ok and is pretty
nice.
Change-Id: I72fae7c28f81c97768ee98b8ebcd69789a4c2e84
You'd think this would just work, given that HeaderKeyDict inherits
from dict and overrides the usual __thingy__ methods, but it
doesn't. It would work if you title-cased the key, but the whole point
of HeaderKeyDict is to do that for you.
Change-Id: If5c22df0690a245d1dd02fa3a52fa135235fe60d
Have json and xml use common record cleanup code.
Do a somewhat better job of parsing extensions from content-types.
Use a real XML serializer.
Change-Id: I10e14dffd1da590b4fd180b4d33ab5de862e2b55
"Accept: application/xml; charset=UTF-8" is totally valid, and has an
implicit q (quality) value of 1.0, just the same as "Accept: text/xml"
does.
Also, you can say things like:
Accept: text/xml; charset=UTF-8; q=0.9; anglebrackets="are awesome"
with as many arbitrary extensions as you want.
See RFC 2616 sections 14.1 Accept and 2.2 Basic Rules for details.
Fixes bug 1202453.
Change-Id: I18e6d0ee3fd6f9d889275ee8335e711c729b7171
RFC 2616 Sec 4.4 Message Length describes how the content-length and
transfer-encoding headers interact. Basically, if chunked transfer
encoding is used, the content-length header value is ignored and if
the content-length header is present, and the request is not using
chunked transfer-encoding, then the content-length must match the body
length.
The only Transfer-Coding value we support in the Transfer-Encoding
header (to date) is "chunked". RFC 2616 Sec 14.41 specifies that if
"multiple encodings have been applied to an entity, the
transfer-codings MUST be listed in the order in which they were
applied." Since we only supported "chunked". If the Transfer-Encoding
header value has multiple transfer-codings, we return a 501 (Not
Implemented) (see RFC 2616 Sec 3.6) without checking if chunked is the
last one specified. Finally, if transfer-encoding is anything but
"chunked", we return a 400 (Bad Request) to the client.
This patch adds a new method, message_length, to the swob request
object which will apply an algorithm based on RFC 2616 Sec 4.4
leveraging the existing content_length property.
In addition to these changes, the proxy server will now notice when
the message length specified by the content-length header is greater
than the configured object maximum size and fail the request with a
413, "Request Entity Too Large", before reading the entire body.
This work flows from https://review.openstack.org/27152.
Change-Id: I5d2a30b89092680dee9d946e1aafd017eaaef8c0
Signed-off-by: Peter Portante <peter.portante@redhat.com>
Enhance internally logged messages to report referer and user-agent.
Pass the referering URL and METHOD between internal servers (when
known), and set the user-agent to be the server type (obj-server,
container-server, proxy-server, obj-updater, obj-replicator,
container-updater, direct-client, etc.) with the process PID. In
conjunction with the transaction ID, it helps to track down which PID
from a given system was responsible for initiating the request and
what that server was working on to make this request.
This has been helpful in tracking down interactions between object,
container and account servers.
We also take things a bit further performaing a bit of refactoring to
consolidate calls to transfer_headers() now that we have a helper
method for constructing them.
Finally we performed further changes to avoid header key duplication
due to string literal header key values and the various objects
representing headers for requests and responses. See below for more
details.
====
Header Keys
There seems to be a bit of a problem with the case of the various
string literals used for header keys and the interchangable way
standard Python dictionaries, HeaderKeyDict() and HeaderEnvironProxy()
objects are used.
If one is not careful, a header object of some sort (one that does not
normalize its keys, and that is not necessarily a dictionary) can be
constructed containing header keys which differ only by the case of
their string literals. E.g.:
{ 'x-trans-id': '1234', 'X-Trans-Id': '5678' }
Such an object, when passed to http_connect() will result in an
on-the-wire header where the key values are merged together, comma
separated, that looks something like:
HTTP_X_TRANS_ID: 1234,5678
For some headers in some contexts, this is behavior is desirable. For
example, one can also use a list of tuples which enumerate the multiple
values a single header should have.
However, in almost all of the contexts used in the code base, this is
not desirable.
This behavior arises from a combination of factors:
1. Header strings are not constants and different lower-case and
title-case header strings values are used interchangably in the
code at times
It might be worth the effort to make a pass through the code to
stop using string literals and use constants instead, but there
are plusses and minuses to doing that, so this was not attempted
in this effort
2. HeaderEnvironProxy() objects report their keys in ".title()"
case, but normalize all other key references to the form
expected by the Request class's environ field
swob.Request.headers fields are HeaderEnvironProxy() objects.
3. HeaderKeyDict() objects report their keys in ".lower()" case,
and normalize all other key references to ".lower()" case
swob.Response.headers fields are HeaderKeyDict() objects.
Depending on which object is used and how it is used, one can end up
with such a mismatch.
This commit takes the following steps as a (PROPOSED) solution:
1. Change HeaderKeyDict() to normalize using ".title()" case to
match HeaderEnvironProxy()
2. Replace standard python dictionary objects with HeaderKeyDict()
objects where possible
This gives us an object that normalizes key references to avoid
fixing the code to normalize the string literals.
3. Fix up a few places to use title case string literals to match
the new defaults
Change-Id: Ied56a1df83ffac793ee85e796424d7d20f18f469
Signed-off-by: Peter Portante <peter.portante@redhat.com>
If a middleware (swift3, I'm looking at you), doesn't pass a Request
object into the Response constructor, Response._response_iter cannot
know to send zero bytes in the body of the HEAD response.
This patch fixes this usage of swob by making Response.__call__
helpfully reify self.request from env if it wasn't already set by the
Response object's constructor.
This fixes a bug in swift3 + swob-enabled-Swift where HEAD requests to
swift3 resulted in a response with a body in violation of the relevant
RFC and confusing clients.
Thanks to kostecky for finding the bug and describing it accurately.
Change-Id: I2bdb098052b161e1cddf1e4e482ab4dfafeb18c0
The container and account servers should respond with 406 if the Accept header
isn't satisfiable. This behavior is defined in RFC 2616 section 14.1.
Change-Id: I8a67ccafe33dc70ef4f7794686a54fbc8581f4dc
It was discovered that uploading items with ? in their names (encoded
with %3F of course) made Swob fail in that it trimmed off everything
after the ? as if it were a query string.
Change-Id: Ie686db9a2177aafad2e77c307ffc3f446646fbb5
path_info_pop didn't behave as the webob one did with single segment
paths like /one and with root-only paths /
Now it should.
Change-Id: Ib88344de386ab9e8975e7f48c1afc47731992ee2
I know it's just TempAuth, but bug #959953 just caught my eye as
something interesting to solve.
This does a best guess on the storage URL to return for a given
request. It allows $HOST to be used in the storage URL configuration,
where $HOST will resolve to scheme://host:port. It bases the scheme
on how the server is running or on storage_url_scheme if set. The
host:port comes from the request's Host header if it exists, and
falls back to the WSGI SERVER_NAME:SERVER_PORT otherwise.
Fixes: bug #959953
DocImpact
Change-Id: Ia494bcb99a04490911ee8d2cb8b12a94e77820c5
If the Host header is available, use it for making relative URLs absolute.
Otherwise, continue using SERVER_NAME.
Change-Id: Ifc028264ad1b122a2d5dff9d5528cb369090429f
This change adds multi range retrieval to OpenStack Swift. For non-
segmented data object, a client can use HTTP Range header to specify
multiple ranges to retrieve sections of the data object.
This implementation currently does not support segmented data object
multi range retrieval. When a client sends a multi range request
against a segmented data object, Swift will return HTTP status code
200. Support for segmented data multi range retrieval will be added
in near future.
This implementation is to bring Swift closer to CDMI multi range
data retrieval standard. Once support for segemented data multi
range is added, Swift will be compliant with CDMI standard in
this area.
DocImpact
Change-Id: I4ed1fb0a0a93c037ddb2f551ea62afe447945107
The proxy server's ObjectController was performing multiple GET
requests to the object server when processing requests with Range
headers. This was a workaround for a bug in the proxy server's
Controller.GETorHEAD_base method where a response code of 416 from the
object server was incorrectly treated as a bad response from the
backend, the same way a 404 or a 5xx would be.
A 416 (Requested Range Not Satisfiable) from the object server is now
considered a good response. Since the response headers from the object
server already include the X-Object-Manifest header, there's no need
to make a second request (sans Range header) to see if the object is a
manifest.
Bonus fix: updated message for status 416 to match RFC2616.
Bonus fix 2: removed a leftover debugging print() in
test/unit/proxy/test_server.py.
Fixes bug 1065869.
Change-Id: I156af0b463f02ef19a8cfe37092544a599d89b78
Add path_qs property to swob.Request. First of all this property
has webob.Request, also this property can be used in swift3 middleware
for generate canonical string, if webob will be replaced in swift3 with
swob (2b36fbd477).
Change-Id: Idf58096baaf7830dd0d624ea6c72eda1eb91ff0d
This fixes swob to handle "Range: bytes=-X" where X > len(content); ex.
"Range: bytes=-17" when the object has 10 bytes. Based on the RFC, the
range is satisfiable and all bytes should be returned.
It also fixes "Range: bytes=-0" to be, correctly, not satisfiable. In
addition, this case's response has Content-Length: 0 and has a zero-byte
body.
It also fixes an existing regression in swob for the case
"Range: bytes=100-" for a body of length < 100 (Content-Length was
negative and the body was returned).
The relevant RFC is 2616, section 14.35.1.
Change-Id: Ib3dc672e083173eb970c10801283813623f26e0e
This change replaces WebOb with a mostly compatible local library,
swift.common.swob. Subtle changes to WebOb's API over the years have been a
huge headache. Swift doesn't even run on the current version.
There are a few incompatibilities to simplify the implementation/interface:
* It only implements the header properties we use. More can be easily added.
* Casts header values to str on assignment.
* Response classes ("HTTPNotFound") are no longer subclasses, but partials
on Response, so things like isinstance no longer work on them.
* Unlike newer webob versions, will never return unicode objects.
Change-Id: I76617a0903ee2286b25a821b3c935c86ff95233f