44 Commits

Author SHA1 Message Date
David Goetz
89deaf5f3c Add flag to stop swob from always using absolute location.
This is needed for SOS (along with patch
https://github.com/dpgoetz/sos/pull/37)
to work with swift 1.12 . By spec you should always use the absolute
location but this causes a problem with staticweb over a cdn using a
cname. Basically you want to be able to forward the browser to a
relative location instead of whatever full url the proxy server
thinks you are using.

Change-Id: I3fa1d415bf9b566be069458b838f7e65db0c4f39
2014-01-24 13:37:45 -08:00
Jenkins
55b8ac4d53 Merge "Stop mutating PATH_INFO in proxy server" 2013-12-06 22:31:21 +00:00
Samuel Merritt
3530708619 Stop mutating PATH_INFO in proxy server
The proxy server was calling swob.Request.path_info_pop() prior to
instantiating a controller so that req.path_info was just /a/c/o (sans
/v1). The version got moved over into SCRIPT_NAME.

This lead to some unfortunate behavior when trying to re-use a request
from middleware. Something like this:

    # Imagine we're a WSGIContext object here.
    #
    # To start, SCRIPT_NAME = '' and PATH_INFO='/v1/a/c/o'

    resp_iter = self._app_call(env, start_response)
    # Now SCRIPT_NAME='/v1' and PATH_INFO ='/a/c/o'

    if something_special in self._response_headers:
        env['REQUEST_METHOD'] = 'GET'
        env.pop('HTTP_RANGE', None)

	# 404 SURPRISE! The proxy calls path_info_pop() again,
        # and now SCRIPT_NAME='/v1/a' and PATH_INFO='/c/o', so this
        # gets treated as a container request. Yikes.
        resp_iter = self._app_call(env, start_response)

Now we just leave SCRIPT_NAME and PATH_INFO alone. To make life easy
for everyone who does want just /a/c/o, I defined
swob.Request.swift_entity_path, which just strips off the /v1.

Note that there's still one call to path_info_pop() in tempauth, but
that's only for requests going to /auth, so it won't affect Swift API
requests. It might be a good idea to remove that one later, but let's
do one thing at a time.

Change-Id: I87557a11c01f3f3889b610578cda6ba7d3933e7a
2013-12-06 10:57:37 -08:00
Samuel Merritt
2db9453722 Preserve closeability of app iterables
PEP 333 (WSGI) says that if your iterable has a close() method, the
framework must call it.

WSGIContext._app_call pulls the first chunk off the returned iterable
to make sure that it gets status and headers, and then it would
itertools.chain() that first chunk back onto the iterable so the whole
body went out. swob.Response.call_application() does it too.

The problem is that an itertools.chain object doesn't have a close()
method, so your iterable's fancy-pants close() method has no chance of
getting called.

This patch adds a slightly smarter CloseableChain that works like
itertools.chain, but has a close() method that calls the underlying
iterables' close() methods, if any.

Change-Id: If975c93f53c27dfa0c2f52f4bbf599af25202f70
2013-12-03 15:56:47 -08:00
Donagh McCabe
9807a358c6 Add WWW-Authenticate to 401 responses
Per http://www.ietf.org/rfc/rfc2616.txt, when a 401 error is returned, the
Www-Authenticate response header MUST also be returned. The format is
described in http://www.ietf.org/rfc/rfc2617.txt.

Swift supports and/or implements a number of authentication schemes
including tempauth, Keystone, tempurl, formpost and container sync. In
this fix, we use a catch-all, "Swift". The realm is the account (where
known) or "unknown" (bad path or where the 401 is returned from code
that does not have the request). Examples:

     Www-Authenticate: Swift realm="AUTH_1234567889"
     Www-Authenticate: Swift realm="unknown"

Fixes bug #1215491

Change-Id: I03362789318dfa156d3733ef9348795062a9cfc4
2013-10-22 15:49:09 +01:00
ZhiQiang Fan
f72704fc82 Change OpenStack LLC to Foundation
Change-Id: I7c3df47c31759dbeb3105f8883e2688ada848d58
Closes-bug: #1214176
2013-09-20 01:02:31 +08:00
Dirk Mueller
3d36a76156 Use Python 3.x compatible except construct
except x,y: was deprected and is removed in Python 3.x.
Use "except x as y:" instead which works in any Python
version >= 2.6.

Change-Id: I7008c74b807340f3457d3a0c8bd0b83f23169d14
2013-09-07 10:50:54 +02:00
Dirk Mueller
3102ad48d5 Do not use locals() for string formatting (H501)
Fixes a warning triggered by Hacking 0.7.x or newer. There
is no need to use a positional string formatting here, since
this is not going to be localized.

Change-Id: Ie38d620aecb0b48cd113af45cc9ca0d61f8f8ff1
2013-09-07 10:26:57 +02:00
Peter Portante
be1cff4f1f Pep8 unit test modules w/ <= 10 violations (5 of 12)
Change-Id: I8e82c14ada52d44df5a31e08982ac79cd7e5c969
Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-09-01 15:12:48 -04:00
Clay Gerrard
27c7abfb69 Add constructor args to swob.Request.blank
I knew webob.Request.blank could take most of the attributes on the class as
kwargs to blank, so I went and looked how.  It seems to work ok and is pretty
nice.

Change-Id: I72fae7c28f81c97768ee98b8ebcd69789a4c2e84
2013-08-29 07:28:32 -07:00
Samuel Merritt
0c505d4456 Fix setdefault() for swob's HeaderKeyDict.
You'd think this would just work, given that HeaderKeyDict inherits
from dict and overrides the usual __thingy__ methods, but it
doesn't. It would work if you title-cased the key, but the whole point
of HeaderKeyDict is to do that for you.

Change-Id: If5c22df0690a245d1dd02fa3a52fa135235fe60d
2013-08-26 19:00:46 -07:00
Michael Barton
53345da70e some container serialization cleanup
Have json and xml use common record cleanup code.
Do a somewhat better job of parsing extensions from content-types.
Use a real XML serializer.

Change-Id: I10e14dffd1da590b4fd180b4d33ab5de862e2b55
2013-07-29 17:35:08 -07:00
Alex Gaynor
ff5a6d0111 Corrected many style violations in the tests.
I focussed primarily on F-category violations, they are all but all fixed with
this patch.

Change-Id: I343f6945c97984ed1093bc347b6def6994297041
2013-07-24 10:18:47 -07:00
Samuel Merritt
91e7e876b5 Accept valid Accept headers in swob.
"Accept: application/xml; charset=UTF-8" is totally valid, and has an
implicit q (quality) value of 1.0, just the same as "Accept: text/xml"
does.

Also, you can say things like:
Accept: text/xml; charset=UTF-8; q=0.9; anglebrackets="are awesome"
with as many arbitrary extensions as you want.

See RFC 2616 sections 14.1 Accept and 2.2 Basic Rules for details.

Fixes bug 1202453.

Change-Id: I18e6d0ee3fd6f9d889275ee8335e711c729b7171
2013-07-19 11:22:35 -07:00
gholt
98ea844478 Replaced <exc>.message with str(<exc>)
Change-Id: I9a030b84dafe2b2300e8735052f6f15f34bc7aa7
2013-05-31 13:19:24 +00:00
Peter Portante
5174b7f85d Rework to support RFC 2616 Sec 4.4 Message Length
RFC 2616 Sec 4.4 Message Length describes how the content-length and
transfer-encoding headers interact. Basically, if chunked transfer
encoding is used, the content-length header value is ignored and if
the content-length header is present, and the request is not using
chunked transfer-encoding, then the content-length must match the body
length.

The only Transfer-Coding value we support in the Transfer-Encoding
header (to date) is "chunked". RFC 2616 Sec 14.41 specifies that if
"multiple encodings have been applied to an entity, the
transfer-codings MUST be listed in the order in which they were
applied." Since we only supported "chunked". If the Transfer-Encoding
header value has multiple transfer-codings, we return a 501 (Not
Implemented) (see RFC 2616 Sec 3.6) without checking if chunked is the
last one specified. Finally, if transfer-encoding is anything but
"chunked", we return a 400 (Bad Request) to the client.

This patch adds a new method, message_length, to the swob request
object which will apply an algorithm based on RFC 2616 Sec 4.4
leveraging the existing content_length property.

In addition to these changes, the proxy server will now notice when
the message length specified by the content-length header is greater
than the configured object maximum size and fail the request with a
413, "Request Entity Too Large", before reading the entire body.

This work flows from https://review.openstack.org/27152.

Change-Id: I5d2a30b89092680dee9d946e1aafd017eaaef8c0
Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-05-25 21:05:56 -04:00
Peter Portante
8825c9c74a Enhance log msg to report referer and user-agent
Enhance internally logged messages to report referer and user-agent.

Pass the referering URL and METHOD between internal servers (when
known), and set the user-agent to be the server type (obj-server,
container-server, proxy-server, obj-updater, obj-replicator,
container-updater, direct-client, etc.) with the process PID. In
conjunction with the transaction ID, it helps to track down which PID
from a given system was responsible for initiating the request and
what that server was working on to make this request.

This has been helpful in tracking down interactions between object,
container and account servers.

We also take things a bit further performaing a bit of refactoring to
consolidate calls to transfer_headers() now that we have a helper
method for constructing them.

Finally we performed further changes to avoid header key duplication
due to string literal header key values and the various objects
representing headers for requests and responses. See below for more
details.

====

Header Keys

There seems to be a bit of a problem with the case of the various
string literals used for header keys and the interchangable way
standard Python dictionaries, HeaderKeyDict() and HeaderEnvironProxy()
objects are used.

If one is not careful, a header object of some sort (one that does not
normalize its keys, and that is not necessarily a dictionary) can be
constructed containing header keys which differ only by the case of
their string literals. E.g.:

   { 'x-trans-id': '1234', 'X-Trans-Id': '5678' }

Such an object, when passed to http_connect() will result in an
on-the-wire header where the key values are merged together, comma
separated, that looks something like:

   HTTP_X_TRANS_ID: 1234,5678

For some headers in some contexts, this is behavior is desirable. For
example, one can also use a list of tuples which enumerate the multiple
values a single header should have.

However, in almost all of the contexts used in the code base, this is
not desirable.

This behavior arises from a combination of factors:

   1. Header strings are not constants and different lower-case and
      title-case header strings values are used interchangably in the
      code at times

      It might be worth the effort to make a pass through the code to
      stop using string literals and use constants instead, but there
      are plusses and minuses to doing that, so this was not attempted
      in this effort

   2. HeaderEnvironProxy() objects report their keys in ".title()"
      case, but normalize all other key references to the form
      expected by the Request class's environ field

      swob.Request.headers fields are HeaderEnvironProxy() objects.

   3. HeaderKeyDict() objects report their keys in ".lower()" case,
      and normalize all other key references to ".lower()" case

      swob.Response.headers fields are HeaderKeyDict() objects.

Depending on which object is used and how it is used, one can end up
with such a mismatch.

This commit takes the following steps as a (PROPOSED) solution:

   1. Change HeaderKeyDict() to normalize using ".title()" case to
      match HeaderEnvironProxy()

   2. Replace standard python dictionary objects with HeaderKeyDict()
      objects where possible

      This gives us an object that normalizes key references to avoid
      fixing the code to normalize the string literals.

   3. Fix up a few places to use title case string literals to match
      the new defaults

Change-Id: Ied56a1df83ffac793ee85e796424d7d20f18f469
Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-05-13 17:39:02 +00:00
Greg Lange
44f00a23c1 fixed some minor things in tests that pyflakes complained about
Change-Id: Ifeab56a964630bcf941e932fcbe39e6572e62975
2013-03-26 20:42:26 +00:00
David Goetz
5d73da158b Static Large Object Support
DocImpact

Change-Id: I7edaa5e44208ab451f7f7566b64bb571b8eea1f9
2013-03-01 16:46:10 -08:00
Chuck Thier
e88ff34685 Cleanup of file permissions
Mostly removed execute on non-executable files

Change-Id: Ibfbe7e0cf0fbeabef602d70b20f75e1dd3bdf9c9
2013-02-15 11:06:11 -06:00
Michael Barton
c45e435d1f Add wsgify and split_path utilities to swob
And refactor some of the code to use them.

Remove unused imports.

Change-Id: Ica479c10247fa85c740bb99cf7d1db7fbb1b2c80
2013-01-25 00:38:32 -08:00
Jenkins
0fce08a255 Merge "get swob.Request.blank to parse path" 2013-01-16 20:44:36 +00:00
David Goetz
54b7594056 get swob.Request.blank to parse path
Change-Id: I3327c915b3b868bb1829103adb718632e58b1b4a
2013-01-16 10:00:18 -08:00
Jenkins
b6d48c62e6 Merge "Fix HEAD request response when request not given to response." 2013-01-16 00:04:41 +00:00
David Goetz
c8b28ba4f4 swob refactors needed for sos
Change-Id: I839c2af9e524f712e6fbeb8bf74d860af1d243e2
2013-01-14 13:57:16 -08:00
Darrell Bishop
e2929ec58a Fix HEAD request response when request not given to response.
If a middleware (swift3, I'm looking at you), doesn't pass a Request
object into the Response constructor, Response._response_iter cannot
know to send zero bytes in the body of the HEAD response.

This patch fixes this usage of swob by making Response.__call__
helpfully reify self.request from env if it wasn't already set by the
Response object's constructor.

This fixes a bug in swift3 + swob-enabled-Swift where HEAD requests to
swift3 resulted in a response with a body in violation of the relevant
RFC and confusing clients.

Thanks to kostecky for finding the bug and describing it accurately.

Change-Id: I2bdb098052b161e1cddf1e4e482ab4dfafeb18c0
2013-01-10 13:15:31 -08:00
gholt
52a2a65ed4 Made 507s report drive, if known.
This functionality was lost with the swob change, but is back now.

Change-Id: I13b3154080a7c601235711b274e4899efb6adc93
2012-12-18 01:18:57 +00:00
Jenkins
8a6922b73e Merge "406 if we can't satisfy Accept" 2012-12-03 23:20:10 +00:00
Michael Barton
064ee2b583 406 if we can't satisfy Accept
The container and account servers should respond with 406 if the Accept header
isn't satisfiable.  This behavior is defined in RFC 2616 section 14.1.

Change-Id: I8a67ccafe33dc70ef4f7794686a54fbc8581f4dc
2012-12-03 11:42:37 -08:00
gholt
6743e4d57f Swob bugfixes; for ? in names specifically
It was discovered that uploading items with ? in their names (encoded
with %3F of course) made Swob fail in that it trimmed off everything
after the ? as if it were a query string.

Change-Id: Ie686db9a2177aafad2e77c307ffc3f446646fbb5
2012-11-30 21:31:21 +00:00
gholt
4063123e3c Fix bug with swob.Request.path_info_pop
path_info_pop didn't behave as the webob one did with single segment
paths like /one and with root-only paths /

Now it should.

Change-Id: Ib88344de386ab9e8975e7f48c1afc47731992ee2
2012-11-28 00:08:26 +00:00
gholt
47ee1d7e17 Better TempAuth storage URL guessing
I know it's just TempAuth, but bug #959953 just caught my eye as
something interesting to solve.

This does a best guess on the storage URL to return for a given
request. It allows $HOST to be used in the storage URL configuration,
where $HOST will resolve to scheme://host:port. It bases the scheme
on how the server is running or on storage_url_scheme if set. The
host:port comes from the request's Host header if it exists, and
falls back to the WSGI SERVER_NAME:SERVER_PORT otherwise.

Fixes: bug #959953
DocImpact

Change-Id: Ia494bcb99a04490911ee8d2cb8b12a94e77820c5
2012-11-10 16:39:25 +00:00
Jenkins
2194897e37 Merge "use Host: for location rewrites" 2012-11-09 00:09:33 +00:00
Michael Barton
65554ea32f charset for default responses
Set a utf-8 charset on default text/html responses.

Change-Id: Ic5f661bd475dca83763d4b55419ad031279e3ba1
2012-11-08 15:23:04 -08:00
Michael Barton
188f834e0a use Host: for location rewrites
If the Host header is available, use it for making relative URLs absolute.
Otherwise, continue using SERVER_NAME.

Change-Id: Ifc028264ad1b122a2d5dff9d5528cb369090429f
2012-11-08 15:16:19 -08:00
litong01
ce274b3532 blueprint Multi-range support implementation
This change adds multi range retrieval to OpenStack Swift. For non-
segmented data object, a client can use HTTP Range header to specify
multiple ranges to retrieve sections of the data object.

This implementation currently does not support segmented data object
multi range retrieval. When a client sends a multi range request
against a segmented data object, Swift will return HTTP status code
200. Support for segmented data multi range retrieval will be added
in near future.

This implementation is to bring Swift closer to CDMI multi range
data retrieval standard. Once support for segemented data multi
range is added, Swift will be compliant with CDMI standard in
this area.

DocImpact

Change-Id: I4ed1fb0a0a93c037ddb2f551ea62afe447945107
2012-11-01 20:45:30 -04:00
Samuel Merritt
9cb7751fad Remove double GET on range requests.
The proxy server's ObjectController was performing multiple GET
requests to the object server when processing requests with Range
headers. This was a workaround for a bug in the proxy server's
Controller.GETorHEAD_base method where a response code of 416 from the
object server was incorrectly treated as a bad response from the
backend, the same way a 404 or a 5xx would be.

A 416 (Requested Range Not Satisfiable) from the object server is now
considered a good response. Since the response headers from the object
server already include the X-Object-Manifest header, there's no need
to make a second request (sans Range header) to see if the object is a
manifest.

Bonus fix: updated message for status 416 to match RFC2616.

Bonus fix 2: removed a leftover debugging print() in
test/unit/proxy/test_server.py.

Fixes bug 1065869.

Change-Id: I156af0b463f02ef19a8cfe37092544a599d89b78
2012-10-24 17:16:11 -07:00
Victor Rodionov
22a8adfcb9 path_qs for swob.Request
Add path_qs property to swob.Request. First of all this property
has webob.Request, also this property can be used in swift3 middleware
for generate canonical string, if webob will be replaced in swift3 with
swob (2b36fbd477).

Change-Id: Idf58096baaf7830dd0d624ea6c72eda1eb91ff0d
2012-10-22 16:56:30 +04:00
Constantine Peresypkin
b304a15b0b fix empty body getter bug 1067923
Change-Id: Ifd609f305ee878c39ea4e6840ed8fa9369595cfa
2012-10-12 08:17:57 +02:00
Victor Rodionov
583850e866 fix bug with swob.Request accept property
Change-Id: I2c1246b9bbd1d3ab22c2a035b735d937dd90da11
2012-10-12 16:53:10 +04:00
Jenkins
89845bc811 Merge "Fix two edge cases with Range: header" 2012-10-09 20:18:15 +00:00
David Goetz
7f476d7b48 fix swob for make_pre_authed_request
Change-Id: Ic263f4a77a0aa0eb40078772a567eb41a60e40f7
2012-10-09 09:19:29 -07:00
Darrell Bishop
e2b03267fd Fix two edge cases with Range: header
This fixes swob to handle "Range: bytes=-X" where X > len(content); ex.
"Range: bytes=-17" when the object has 10 bytes. Based on the RFC, the
range is satisfiable and all bytes should be returned.

It also fixes "Range: bytes=-0" to be, correctly, not satisfiable.  In
addition, this case's response has Content-Length: 0 and has a zero-byte
body.

It also fixes an existing regression in swob for the case
"Range: bytes=100-" for a body of length < 100 (Content-Length was
negative and the body was returned).

The relevant RFC is 2616, section 14.35.1.

Change-Id: Ib3dc672e083173eb970c10801283813623f26e0e
2012-10-07 15:53:08 -07:00
Michael Barton
5e3e9a882d local WSGI Request and Response classes
This change replaces WebOb with a mostly compatible local library,
swift.common.swob.  Subtle changes to WebOb's API over the years have been a
huge headache.  Swift doesn't even run on the current version.

There are a few incompatibilities to simplify the implementation/interface:
 * It only implements the header properties we use.  More can be easily added.
 * Casts header values to str on assignment.
 * Response classes ("HTTPNotFound") are no longer subclasses, but partials
   on Response, so things like isinstance no longer work on them.
 * Unlike newer webob versions, will never return unicode objects.

Change-Id: I76617a0903ee2286b25a821b3c935c86ff95233f
2012-09-28 14:48:48 -07:00