272 Commits

Author SHA1 Message Date
anc
6164fa246d Generic means for persisting system metadata.
Middleware or core features may need to store metadata
against accounts or containers. This patch adds a
generic mechanism for system metadata to be persisted
in backend databases, without polluting the user
metadata namespace, by using the reserved header
namespace x-<server_type>-sysmeta-*.

Modifications are firstly that backend servers persist
system metadata headers alongside user metadata and
other system state.

For accounts and containers, system metadata in PUT
and POST requests is treated in a similar way to user
metadata. System metadata is not yet supported for
object requests.

Secondly, changes in the proxy controllers ensure that
headers in the system metadata namespace will pass through
in requests to backend servers.

Thirdly, system metadata returned from backend servers
in GET or HEAD responses is added to the cached info
dict, which middleware can access.

Finally, a gatekeeper middleware module is provided
which filters all system metadata headers from requests
and responses by removing headers with names starting
x-account-sysmeta-, x-container-sysmeta-. The gatekeeper
also removes headers starting x-object-sysmeta- in
anticipation of future support for system metadata being
set for objects. This prevents clients from writing or
reading system metadata.

The required_filters list in swift/proxy/server.py is
modified to include the gatekeeper middleware so that
if the gatekeeper has not been configured in the
pipeline then it will be automatically inserted close
to the start of the pipeline.

blueprint cluster-federation

Change-Id: I80b8b14243cc59505f8c584920f8f527646b5f45
2014-01-06 22:29:37 +00:00
Jenkins
7f456ef35f Merge "change the last-modified header value with valid one" 2013-12-20 00:57:36 +00:00
Kiyoung Jung
d69e013519 change the last-modified header value with valid one
the Last-Modified header in Response didn't have a suitable
value - an integer part of object's timestamp.
This leads that the the if-[un]modified-since header with the
value from last-modified is always earlier than timestamp
and results the content is always newer than value of these
conditional headers.
Patched code returns math.ceil() of object's timestamp
in Last-Modified header so the later conditional header works
correctly

Closes-Bug: #1248818
Change-Id: I1ece7d008551bf989da74d23f0ed6307c45c5436
2013-12-19 09:31:17 +00:00
Samuel Merritt
ace2aa33b1 Fix obj versioning w/non-ASCII container name
If you create a container with a non-ASCII name, and then make another
container with X-Versions-Location: first-cøntåîner, *and* you're
serializing stuff in memcache as json (the default), when the proxy
tries to make a versioned object, it will crash.

The fix is to make sure that get_container_info() always returns strs,
not unicodes.

The long-term fix would be to get rid of simplejson entirely, as its
decoder can't make up its mind whether JSON strings should be Python
strs or unicodes, and that makes it really really easy to write bugs
like this.

Change-Id: Ib20ea5fb884484a4246d7a21a9f1e2ffd82eb04f
2013-12-18 10:38:34 -08:00
Jenkins
665cb4a8ec Merge "Only retry GETs for objects." 2013-12-14 15:17:44 +00:00
Jenkins
f0aea5a707 Merge "Expose basic constraints in /info" 2013-12-14 07:58:44 +00:00
David Goetz
03513a02a2 Only retry GETs for objects.
Change-Id: I8b6ceeaa0e5e247e45209deced808b0b78181d53
2013-12-13 08:59:22 -08:00
Samuel Merritt
226e46550f Expose basic constraints in /info
Change-Id: I70649e0669e2f7a1d61742a16ed6dc792d4b2a5a
2013-12-12 17:29:45 -08:00
Zhang Jinnan
bdc296abbc Remove start index 0 in range()
Remove the useless arg ("start index" = 0) in files, since its default
value is 0, to make code cleaner.

Fixes bug #1259750

Change-Id: I52afac28a3248895bb1c012a5934d39e7c2cc5a9
2013-12-10 16:16:44 -08:00
Samuel Merritt
3530708619 Stop mutating PATH_INFO in proxy server
The proxy server was calling swob.Request.path_info_pop() prior to
instantiating a controller so that req.path_info was just /a/c/o (sans
/v1). The version got moved over into SCRIPT_NAME.

This lead to some unfortunate behavior when trying to re-use a request
from middleware. Something like this:

    # Imagine we're a WSGIContext object here.
    #
    # To start, SCRIPT_NAME = '' and PATH_INFO='/v1/a/c/o'

    resp_iter = self._app_call(env, start_response)
    # Now SCRIPT_NAME='/v1' and PATH_INFO ='/a/c/o'

    if something_special in self._response_headers:
        env['REQUEST_METHOD'] = 'GET'
        env.pop('HTTP_RANGE', None)

	# 404 SURPRISE! The proxy calls path_info_pop() again,
        # and now SCRIPT_NAME='/v1/a' and PATH_INFO='/c/o', so this
        # gets treated as a container request. Yikes.
        resp_iter = self._app_call(env, start_response)

Now we just leave SCRIPT_NAME and PATH_INFO alone. To make life easy
for everyone who does want just /a/c/o, I defined
swob.Request.swift_entity_path, which just strips off the /v1.

Note that there's still one call to path_info_pop() in tempauth, but
that's only for requests going to /auth, so it won't affect Swift API
requests. It might be a good idea to remove that one later, but let's
do one thing at a time.

Change-Id: I87557a11c01f3f3889b610578cda6ba7d3933e7a
2013-12-06 10:57:37 -08:00
gholt
b332e3e436 Fixed a failing test on my saio
The added sleep makes this test pass on my saio. I have not heard of
it failing for anyone else, but I figured I'd post this up just in
case someone does have the same problem and this fixes it for them.

Change-Id: Ia0bb09d36d0b531ade7c6a6034bbe31dd6c90a98
2013-12-05 19:39:29 +00:00
Jenkins
34b4bf34d2 Merge "Added discoverable capabilities." 2013-11-28 00:11:35 +00:00
Jenkins
9e25d38611 Merge "Add a DebugLogger for wsgi server tests" 2013-11-26 15:09:14 +00:00
Jenkins
901bd044c3 Merge "Remove obsolete future imports" 2013-11-22 22:25:13 +00:00
Richard (Rick) Hawkins
2c4bf81464 Added discoverable capabilities.
Swift can now optionally be configured to allow requests to '/info',
providing information about the swift cluster.  Additionally a HMAC
signed requests to
'/info?swiftinfo_sig=<sign>&swiftinfo_expires=<expires>' can be
configured allowing privileged access to more sensitive information
not meant to be public.

DocImpact
Change-Id: I2379360fbfe3d9e9e8b25f1dc34517d199574495
Implements: blueprint capabilities
Closes-Bug: #1245694
2013-11-22 15:54:13 -06:00
David Goetz
f5648638ee Get retry.
If a source times out on read try another one of them with a
modified range.  There had to be a lot of moved around code
to get this working but it should all make sense.

Change-Id: Ieaf045690a8823927a6f38098a95b37a4d4adb70
2013-11-22 21:03:11 +00:00
Samuel Merritt
b5b0b78fc7 Remove obsolete future imports
The with statement has been standard since Python 2.5, so we can get
rid of these imports.

Change-Id: I280971c3d8c01e94cc2c17cacaedcbe9d9c8a3c3
2013-11-22 12:23:58 -08:00
Clay Gerrard
9e80fd45a0 Add a DebugLogger for wsgi server tests
Change-Id: Ifd2528be443ba3879bf4921f6c5f4ef31f29044b
2013-11-21 01:35:58 -08:00
Jenkins
c8b6bc7bd5 Merge "early quorum responses" 2013-11-08 00:35:15 +00:00
Jenkins
5b0703a73c Merge "HEAD on account returns 410 if account was deleted and not yet reaped" 2013-11-07 22:55:26 +00:00
Michael Barton
f0c0855ec8 early quorum responses
Allow the proxy to respond to many types of requests as soon as it has a
quorum.  This can help speed up responses (without changing the results),
especially when one node is acting up.

I had to fix a few unit tests that no longer match the backend http requests
made by our proxy.

Change-Id: Ieb070dc3019e217e717b96154a7a809409bf40a5
2013-11-07 01:48:14 +00:00
Peter Portante
023a061587 Tie socket write buffer size to server parameters
By default, Python 2.*'s standard library "socket" module performs 8K
writes. For 10ge networks, with large MTUs (typically 9,000), this is
not optimal. We tie the default buffer size to the client_chunk_size
paramter for the proxy server, and to the network_chunk_size for the
object server.

One might be tempted to ask, isn't there a way to set this value on a
per-request basis? This author was unable to find a reference to the
_fileobject in the context of WSGI. By the time a request pass to a
WSGI object's __call__ method, the "wfile" attribute of the
req.environ['eventlet.input'] (Input) object has been set to None, and
the "rfile" attribute is the object wrapping the socket for reading,
not writing.

One might also be tempted to ask, why not just override the
wsgi.HttpProtocol's "wbufsize" class attribute instead? Until
eventlet/wsgi.py is fixed, we can't set wsgi.HttpProtocol.wbufsize to
anything but zero (the default, see Python's SocketServer.py,
StreamRequestHandler class), since Eventlet does not ensure the socket
_fileobject's flush() method is called after Eventlet invokes a
write() method on the same.  NOTE: wbufsize (a class attribute of
StreamRequestHandler originally, not to be confused with the standard
library's socket._fileobject._wbufsize class attribute) is used for
the bufsize parameter of the connection object's makefile() method. As
a result, the socket's _fileobject code uses that value to set both
_rbufsize and _wbufsize. While that would allow us to transmit in 64KB
chunks, it also means that write() and writeline() method calls on the
socket _fileobject are only transmitted once 64KB have been
accumulated, or a flush() is called.

As for performance improvement:

Run       8KB   64KB
  0     8.101  6.367
  1     7.892  6.216
  2     7.732  6.246
  3     7.594  6.229
  4     7.594  6.292
  5     7.555  6.230
  6     7.575  6.270
  7     7.528  6.278
  8     7.547  6.304
  9     7.550  6.313
Average 7.667  6.275  1.3923  18.16%

Run using the following after adjusting the test value for obj_len to
1 GB:

nosetests -v --nocapture --nologcapture \
test/unit/proxy/test_server.py:TestProxyObjectPerformance.test_GET_debug_large_file

Change-Id: I4dd93acc3376e9960fbdcdcae00c6d002e545894
Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-10-30 11:32:09 -04:00
Jon Snitow
97b5f59ea4 HEAD on account returns 410 if account was deleted and not yet reaped
* updated CONTRIBUTING.md
 * improved unit tests
 * 100% coverage for proxy acct controller tests
 * invoke fake_http_connect correctly

Change-Id: I0826c5a1c52efdd5ae95f7fde8024f2bff0751ba
2013-10-29 13:49:38 -07:00
Peter Portante
5202b0e586 DiskFile API, with reference implementation
Refactor on-disk knowledge out of the object server by pushing the
async update pickle creation to the new DiskFileManager class (name is
not the best, so suggestions welcome), along with the REPLICATOR
method logic. We also move the mount checking and thread pool storage
to the new ondisk.Devices object, which then also becomes the new home
of the audit_location_generator method.

For the object server, a new setup() method is now called at the end
of the controller's construction, and the _diskfile() method has been
renamed to get_diskfile(), to allow implementation specific behavior.

We then hide the need for the REST API layer to know how and where
quarantining needs to be performed. There are now two places it is
checked internally, on open() where we verify the content-length,
name, and x-timestamp metadata, and in the reader on close where the
etag metadata is checked if the entire file was read.

We add a reader class to allow implementations to isolate the WSGI
handling code for that specific environment (it is used no-where else
in the REST APIs). This simplifies the caller's code to just use a
"with" statement once open to avoid multiple points where close needs
to be called.

For a full historical comparison, including the usage patterns see:
https://gist.github.com/portante/5488238

(as of master, 2b639f5, Merge
 "Fix 500 from account-quota     This Commit
 middleware")
--------------------------------+------------------------------------
                                 DiskFileManager(conf)

                                   Methods:
                                     .pickle_async_update()
                                     .get_diskfile()
                                     .get_hashes()

                                   Attributes:
                                     .devices
                                     .logger
                                     .disk_chunk_size
                                     .keep_cache_size
                                     .bytes_per_sync

DiskFile(a,c,o,keep_data_fp=)    DiskFile(a,c,o)

  Methods:                         Methods:
   *.__iter__()
    .close(verify_file=)
    .is_deleted()
    .is_expired()
    .quarantine()
    .get_data_file_size()
                                     .open()
                                     .read_metadata()
    .create()                        .create()
                                     .write_metadata()
    .delete()                        .delete()

  Attributes:                      Attributes:
    .quarantined_dir
    .keep_cache
    .metadata
                                *DiskFileReader()

                                   Methods:
                                     .__iter__()
                                     .close()

                                   Attributes:
                                    +.was_quarantined

DiskWriter()                     DiskFileWriter()

  Methods:                         Methods:
    .write()                         .write()
    .put()                           .put()

* Note that the DiskFile class   * Note that the DiskReader() object
  implements all the methods       returned by the
  necessary for a WSGI app         DiskFileOpened.reader() method
  iterator                         implements all the methods
                                   necessary for a WSGI app iterator

                                 + Note that if the auditor is
                                   refactored to not use the DiskFile
                                   class, see
                                   https://review.openstack.org/44787
                                   then we don't need the
                                   was_quarantined attribute

A reference "in-memory" object server implementation of a backend
DiskFile class in swift/obj/mem_server.py and
swift/obj/mem_diskfile.py.

One can also reference
https://github.com/portante/gluster-swift/commits/diskfile for the
proposed integration with the gluster-swift code based on these
changes.

Change-Id: I44e153fdb405a5743e9c05349008f94136764916
Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-10-17 15:03:31 -04:00
Jenkins
62982779af Merge "Fix utf-8 handling in object versions." 2013-10-09 01:41:05 +00:00
Jenkins
49d24e7e48 Merge "Revert "Refactor common/utils methods to common/ondisk"" 2013-10-09 01:40:55 +00:00
Jenkins
3206132339 Merge "Stop reading from object server when client disconnects." 2013-10-07 22:30:13 +00:00
Peter Portante
9411a24ba7 Revert "Refactor common/utils methods to common/ondisk"
This reverts commit 7760f41c3ce436cb23b4b8425db3749a3da33d32

Change-Id: I95e57a2563784a8cd5e995cc826afeac0eadbe62
Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-10-07 17:18:09 -04:00
Jenkins
0b594bc3af Merge "Change OpenStack LLC to Foundation" 2013-10-07 16:09:37 +00:00
Samuel Merritt
def37fb56a Stop reading from object server when client disconnects.
If a client were in the middle of an object GET request and then
disconnected, the proxy would wait a while (default 60s) and then time
out the connection. As part of the teardown for this, the proxy would
attempt to close the connection to the object server, then drain any
associated buffers. However, this didn't work particularly well,
resulting in the proxy reading the entire remainder of the object for
no gain.

Now, the proxy closes the connection hard, by calling .close() on the
underlying socket._socket object. This is different from calling
.close() on a socket._socketobject object, which is what you get back
from socket.socket() and similar methods. Calling .close() on a
socket._socketobject simply decrements a reference counter on the
socket._socket, which has been observed in the past to result in
socket leaks when something holds onto a reference. However, calling
.close() on a socket._socket actually closes the socket regardless of
who else has a reference to it.

I had to delete a test assertion that said the object server never got
SIGPIPE after a GET w/X-Newest. Well, you get a SIGPIPE when you write
to a closed socket, and now the proxy is actually closing the sockets
early, so now you *do* get a SIGPIPE.

closes-bug: 1174660

Note that this will cause a regression on bug 1037337; unfortunately,
the cure is worse than the disease, so out it goes.

Change-Id: I9c7a2e7fdb8b4232e53ea96f86b50e8d34c27221
2013-10-02 15:57:24 -07:00
Jenkins
d7bab86619 Merge "Use created container in unit test" 2013-10-01 05:48:56 +00:00
Samuel Merritt
d8e0492ea8 Fix internal swift.source tracking.
In 1.8.0 (Grizzly), your proxy logs would indicate which middleware
was responsible for an internal request, e.g. TU for tempurl or BD for
bulk delete. At some point, those all turned into GET_INFO, which does
not give you any idea which specific middleware was responsible, only
that it came from a get_account_info/get_container_info call.

This commit puts it back to how it was in 1.8.0. Also, the
new-since-1.8.0 function get_object_info() got swift_source plumbing
added to it, so source tracking for the quota middlewares'
get_object_info() calls will happen now too.

Note that due to the new-since-1.8.0 in-environment caching of
account/container info, you may not see as many lines in the proxy log
as you would with 1.8.0. This is because there are actually fewer
internal requests being made.

Change-Id: I2b2ff7823c612dc7ed7f268da979c4500bbbe911
2013-09-25 10:21:56 -07:00
Jenkins
bb3f9657f2 Merge "Log x-copy-from when it could be useful" 2013-09-24 20:36:25 +00:00
anc
fa308d60bd Fix utf-8 handling in object versions.
Fixes object versioning when object name and/or version
container name contain multibyte utf-8 characters.

When object names containing non-ASCII characters
are PUT multiple times into a container with an
x-versions-location set, subsequent DELETE of the
object results in a 500 response status.

When the versions container name contains
non-ASCII characters the first delete of an object
succeeds but fails to restore previous version of
object, so second delete incorrectly returns 404.

Fixes bug 1229142

Change-Id: I425440f76b8328f8e119d390bfa4c7022181e89e
2013-09-24 17:18:58 +01:00
Peter Portante
4e8b2ffc2e Use created container in unit test
Change-Id: I2573be1ac14f65b8008611edf940363b31c8d86e
Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-09-24 02:29:54 -04:00
David Goetz
01f58d6826 SLOs broken for range requests
Change-Id: I21175a4be0cda9a8a98c425bff11c80895cd6d3e
2013-09-20 08:51:21 -07:00
gholt
4a5c2fa0c6 Log x-copy-from when it could be useful
Change-Id: Ia28a9b47213f848ab5ea59572e14ac710ed881e3
2013-09-19 21:05:46 +00:00
ZhiQiang Fan
f72704fc82 Change OpenStack LLC to Foundation
Change-Id: I7c3df47c31759dbeb3105f8883e2688ada848d58
Closes-bug: #1214176
2013-09-20 01:02:31 +08:00
Jenkins
60beab79bc Merge "Refactor common/utils methods to common/ondisk" 2013-09-18 21:29:54 +00:00
Peter Portante
7760f41c3c Refactor common/utils methods to common/ondisk
Place all the methods related to on-disk layout and / or configuration
into a new common module that can be shared by the various modules
using the same on-disk layout.

Change-Id: I27ffd4665d5115ffdde649c48a4d18e12017e6a9
Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-09-17 17:32:04 -04:00
Fabien Boucher
fffc95c3cc Handle X-Copy-From header in account_quota mw
Content length of the copied object
is checked before allowing the copy
request according to the account
quota set by Reseller.

Fixes: bug #1200271
Change-Id: Ie4700f23466dd149ea5a497e6c72438cf52940fd
2013-09-10 17:12:46 +02:00
Peter Portante
6ae4e17af1 Pep8 account and proxy server unit tests (10 of 12)
Change-Id: Ib83d164997b0d98be921c8b4857caa2429344aa4
Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-09-01 16:12:51 -04:00
Peter Portante
be1cff4f1f Pep8 unit test modules w/ <= 10 violations (5 of 12)
Change-Id: I8e82c14ada52d44df5a31e08982ac79cd7e5c969
Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-09-01 15:12:48 -04:00
Clay Gerrard
27c7abfb69 Add constructor args to swob.Request.blank
I knew webob.Request.blank could take most of the attributes on the class as
kwargs to blank, so I went and looked how.  It seems to work ok and is pretty
nice.

Change-Id: I72fae7c28f81c97768ee98b8ebcd69789a4c2e84
2013-08-29 07:28:32 -07:00
Jenkins
dd15594157 Merge "Clean up some of the proxy unit tests." 2013-08-29 10:47:16 +00:00
Samuel Merritt
e19cf5c047 Clean up some of the proxy unit tests.
Since we've got machinery to catch HTTPException errors that happens
after the proxy server's __call__ method but before the individual
GET()/POST()/etc. methods, we shouldn't just call
GET()/POST()/etc. Instead, we should use swob.Request's get_response()
method, or otherwise actually call the proxy application. That way we
can have the code raise HTTPBadRequest, but the test just gets a
plain old response with a 400 status code, just like you'd expect.

Earlier changes fixed some of the tests; this commit fixes some
more. In the interest of not having a gigantic diff, this only gets
about half the proxy's unit tests, and the others will be done in a
subsequent commit.

Change-Id: Iaca32e74a8f4499440e655c207a1012e14486c74
2013-08-28 17:40:37 -07:00
Chuck Thier
092ca2d2df Added test for timeout regression
Change-Id: Ia0e5f1595d1b1f58e6c415d7397f33496576c944
2013-08-28 21:55:26 +00:00
Jenkins
396d2c91ed Merge "Refactor how we pick listings' content type." 2013-08-27 05:50:23 +00:00
Jenkins
bc75061740 Merge "Run a more GC iterations to make sure weakrefs are collected" 2013-08-23 12:59:41 +00:00
Jenkins
1f8632a2cb Merge "Fix range GET w/If-None-Match." 2013-08-22 22:41:02 +00:00