821 Commits

Author SHA1 Message Date
anc
6164fa246d Generic means for persisting system metadata.
Middleware or core features may need to store metadata
against accounts or containers. This patch adds a
generic mechanism for system metadata to be persisted
in backend databases, without polluting the user
metadata namespace, by using the reserved header
namespace x-<server_type>-sysmeta-*.

Modifications are firstly that backend servers persist
system metadata headers alongside user metadata and
other system state.

For accounts and containers, system metadata in PUT
and POST requests is treated in a similar way to user
metadata. System metadata is not yet supported for
object requests.

Secondly, changes in the proxy controllers ensure that
headers in the system metadata namespace will pass through
in requests to backend servers.

Thirdly, system metadata returned from backend servers
in GET or HEAD responses is added to the cached info
dict, which middleware can access.

Finally, a gatekeeper middleware module is provided
which filters all system metadata headers from requests
and responses by removing headers with names starting
x-account-sysmeta-, x-container-sysmeta-. The gatekeeper
also removes headers starting x-object-sysmeta- in
anticipation of future support for system metadata being
set for objects. This prevents clients from writing or
reading system metadata.

The required_filters list in swift/proxy/server.py is
modified to include the gatekeeper middleware so that
if the gatekeeper has not been configured in the
pipeline then it will be automatically inserted close
to the start of the pipeline.

blueprint cluster-federation

Change-Id: I80b8b14243cc59505f8c584920f8f527646b5f45
2014-01-06 22:29:37 +00:00
Jenkins
cd378edd2e Merge "Fix checkmount error parsing in swift-recon" 2014-01-02 21:39:23 +00:00
Jenkins
b9dd7c7405 Merge "Up nproc limit on startup." 2013-12-31 10:08:08 +00:00
Caleb Tennis
d1dd143952 Up nproc limit on startup.
Separate out setrlimit calls for specific exception handling.

Closes-Bug: #1264561
Change-Id: I5588f19f8d0393409580d17317727977758d5cb3
2013-12-29 11:35:07 -05:00
Florian Hines
62254e42c4 Fix checkmount error parsing in swift-recon
- swift-recon now handles parsing instances where 'mounted' key (in unmounted
  and disk_usage) is an error message instead of a bool.
- Add's checkmount exception handling to the recon umounted endpoint.
- Updates existing unittest to have ismount throw an error.
- Updates unittests to cover the corner cases

Change-Id: Id51d14a8b98de69faaac84b2b34b7404b7df69e9
2013-12-28 20:58:27 -08:00
Jenkins
c72c10a1fe Merge "Preserve tracebacks from run_in_thread" 2013-12-27 18:42:39 +00:00
Jenkins
c9cb39282b Merge "Expose allowed tempurl methods in /info" 2013-12-23 19:01:31 +00:00
Samuel Merritt
16204c706d Preserve tracebacks from run_in_thread
Now the traceback goes all the way down to where the exception came
from, not just down to run_in_thread. Better for debugging.

Change-Id: Iac6acb843a6ecf51ea2672a563d80fa43d731f23
2013-12-17 16:18:35 -08:00
Samuel Merritt
a49fd3d8de Force catch_errors into pipeline
This commit adds a hook for WSGI applications
(e.g. proxy.server.Application) to modify their WSGI pipelines. This
is currently used by the proxy server to ensure that catch_errors is
present; if it is missing, it is inserted as the first middleware in
the pipeline.

This lets us write new, mandatory middlewares for Swift without
breaking existing deployments on upgrade.

Change-Id: Ibed0f2edb6f80c25be182b3d4544e6a67c5050ad
2013-12-16 16:08:16 -08:00
Jenkins
e7298d1d05 Merge "Remove start index 0 in range()" 2013-12-14 06:03:39 +00:00
Jenkins
4dfe31e69e Merge "Return an exit code for configuration errors" 2013-12-13 11:49:15 +00:00
Samuel Merritt
979033a14e Expose allowed tempurl methods in /info
Clients can construct tempurls for any method, but they only work if
they're in this list, so it's helpful for clients to see the list.

Change-Id: Id852f457d65b62c4fe79db01b1d7029a5fa5aa09
2013-12-12 16:13:42 -08:00
Jenkins
813e8e5272 Merge "Increases the UT coverage of db_replicator.py" 2013-12-11 01:40:39 +00:00
Zhang Jinnan
bdc296abbc Remove start index 0 in range()
Remove the useless arg ("start index" = 0) in files, since its default
value is 0, to make code cleaner.

Fixes bug #1259750

Change-Id: I52afac28a3248895bb1c012a5934d39e7c2cc5a9
2013-12-10 16:16:44 -08:00
Jenkins
55dafa2f07 Merge "Prevent early auth deny in tempauth when using swift3 middleware." 2013-12-07 10:02:00 +00:00
Jenkins
55b8ac4d53 Merge "Stop mutating PATH_INFO in proxy server" 2013-12-06 22:31:21 +00:00
Jenkins
46e7ba0e70 Merge "Update handoff algorithm to use IP/port pairs" 2013-12-06 22:31:19 +00:00
Jenkins
fb1a985ff0 Merge "Preserve closeability of app iterables" 2013-12-06 22:20:20 +00:00
Samuel Merritt
3530708619 Stop mutating PATH_INFO in proxy server
The proxy server was calling swob.Request.path_info_pop() prior to
instantiating a controller so that req.path_info was just /a/c/o (sans
/v1). The version got moved over into SCRIPT_NAME.

This lead to some unfortunate behavior when trying to re-use a request
from middleware. Something like this:

    # Imagine we're a WSGIContext object here.
    #
    # To start, SCRIPT_NAME = '' and PATH_INFO='/v1/a/c/o'

    resp_iter = self._app_call(env, start_response)
    # Now SCRIPT_NAME='/v1' and PATH_INFO ='/a/c/o'

    if something_special in self._response_headers:
        env['REQUEST_METHOD'] = 'GET'
        env.pop('HTTP_RANGE', None)

	# 404 SURPRISE! The proxy calls path_info_pop() again,
        # and now SCRIPT_NAME='/v1/a' and PATH_INFO='/c/o', so this
        # gets treated as a container request. Yikes.
        resp_iter = self._app_call(env, start_response)

Now we just leave SCRIPT_NAME and PATH_INFO alone. To make life easy
for everyone who does want just /a/c/o, I defined
swob.Request.swift_entity_path, which just strips off the /v1.

Note that there's still one call to path_info_pop() in tempauth, but
that's only for requests going to /auth, so it won't affect Swift API
requests. It might be a good idea to remove that one later, but let's
do one thing at a time.

Change-Id: I87557a11c01f3f3889b610578cda6ba7d3933e7a
2013-12-06 10:57:37 -08:00
Jenkins
c36eeb3502 Merge "Fix HEAD tempurls" 2013-12-06 03:33:08 +00:00
Samuel Merritt
d18b6d8d5d Fix HEAD tempurls
HEAD-only tempurls didn't work; tempurl only allowed a HEAD request if
the tempurl was generated for GET or PUT (that is, the method in the
HMAC-signed string was "GET" or "PUT").

The intent of the code was to allow a user with a GET or a PUT tempurl
to also perform a HEAD request; I think the breaking of HEAD tempurls
is just a bug.

Change-Id: I621ddaac03e0d058dd9e7c7c374cb5c4b6386d36
2013-12-05 16:52:25 -08:00
Jenkins
7d8887c7cc Merge "Override spelling of tempauth test name" 2013-12-06 00:46:21 +00:00
Jenkins
204110fa10 Merge "Remove swift-bench" 2013-12-06 00:00:54 +00:00
Peter Portante
60bf2dc675 Override spelling of tempauth test name
Change-Id: Ife571bbeb6dd4ba5c552a2f0725c36e133e64cf4
2013-12-05 15:31:09 -05:00
Samuel Merritt
2db9453722 Preserve closeability of app iterables
PEP 333 (WSGI) says that if your iterable has a close() method, the
framework must call it.

WSGIContext._app_call pulls the first chunk off the returned iterable
to make sure that it gets status and headers, and then it would
itertools.chain() that first chunk back onto the iterable so the whole
body went out. swob.Response.call_application() does it too.

The problem is that an itertools.chain object doesn't have a close()
method, so your iterable's fancy-pants close() method has no chance of
getting called.

This patch adds a slightly smarter CloseableChain that works like
itertools.chain, but has a close() method that calls the underlying
iterables' close() methods, if any.

Change-Id: If975c93f53c27dfa0c2f52f4bbf599af25202f70
2013-12-03 15:56:47 -08:00
Cristian A Sanchez
fdc775d6d5 Increases the UT coverage of db_replicator.py
Adds 20 unit tests to increase the coverage of db_replicator.py
from 71% to 90%

Change-Id: Ia63cb8f2049fb3182bbf7af695087bfe15cede54
Closes-Bug: #948179
2013-12-03 14:00:22 -03:00
Jenkins
645d1c93c5 Merge "Add accurate timestamps in proxy log" 2013-12-03 03:51:42 +00:00
Samuel Merritt
68db481ae5 Update handoff algorithm to use IP/port pairs
The replica placement algorithm works on regions, then zones, then
IP/port, then device ID. The handoff algorithm worked on regions, then
zones, then device ID, completely skipping IP/port. It's now been
updated to take IP/port into consideration.

This means you get one handoff on each machine in the cluster before
you start getting handoffs that share a machine with a previous
one. In small clusters, this can help with durability.

Because this is performance-critical code, here are some quick
benchmark results:

Run time averages over 25000 trials on a 1200-device ring (20 part
power, 3 replicas, 2 regions, 10 zones, 120 nodes):

		   |   master    |   branch
===================+=============+============
get 1 more node	   |  2.727e-05	 |  3.076e-05
get 6 more nodes   |  3.55e-05   |  4.214e-05
get all more nodes |  0.002247   |  0.002691

There's a small slowdown from the additional bookkeeping, but nothing
too awful. The time to get 6 more nodes (for handoff checks on 404,
it's 2x replica count by default, hence 6) went from 35 to 42
microseconds, so it remains small.

Change-Id: Ie7da4dfcb0fcf1a38e2fb13f60c204540fadbf06
2013-12-02 17:41:38 -08:00
Chmouel Boudjnah
63d91386f9 Remove swift-bench
swift-bench has moved to :

http://github.com/openstack/swift-bench

DocImpact

Change-Id: Id09765f53d1e493fb0eeae0dba57879ba9dd0ade
2013-12-02 12:46:41 -08:00
Samuel Merritt
0d45e99ff0 Expose bulk-operation limits in /info.
These will allow clients to perform the minimal number of requests
required to accomplish some bulk tasks. For example, a client with
many objects to delete can learn that the cluster's limit on
deletes-per-request is, say, 128, and then batch up their deletes in
groups of 128. Without this, the client has to either discover the
limit out-of-band somehow (and get notified if it changes), or do some
sort of binary search to figure out the limit.

Similar reasoning applies to the containers-per-request value.

The errors-per-request values are included so that clients may size
their requests such that everything is attempted regardless of
failure.

I split the 'bulk' entry into 'bulk_delete' and 'bulk_upload' because,
from a client's standpoint, they're separate operations. It so happens
that Swift implements both in one piece of middleware, but clients
don't care.

Bonus fix: documented a missing config setting for the bulk middleware.

Change-Id: Ic3549aef79682fd5b798145c3545c1609aa1592b
2013-11-30 11:09:34 -08:00
Jenkins
438596fc0f Merge "in case you lose your builder backups" 2013-11-28 01:24:46 +00:00
Peter Portante
d4e02a2e8b Add accurate timestamps in proxy log
Add accurate timestamps to the proxy-logging middleware log lines for
the start and end of a request. We use 9 digits of precision since on
some systems the clock resolution can be as fine as 1 ns.

The goal is to allow for log processing that can use the more accurate
timestamps to correlate requests, computing number of requests in
flight at a given historical point in time.

Change-Id: I61e8784b1c455d629f1299207fc4fc7e4a134814
Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-11-27 20:21:54 -05:00
Jenkins
7171a8e680 Merge "test: improve db_replicator coverage" 2013-11-28 00:59:01 +00:00
Jenkins
7cc009735a Merge "Add a unit test to verify proxy logging fields" 2013-11-28 00:20:22 +00:00
Jenkins
34b4bf34d2 Merge "Added discoverable capabilities." 2013-11-28 00:11:35 +00:00
Peter Portante
e0147e60d8 Add a unit test to verify proxy logging fields
Also bring unit test coverage to 100% (well, at least every line is
reported as "covered").

Change-Id: I659d0c02008368897b1307a7a5c9aaba73b80588
Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-11-27 16:19:51 -05:00
Jenkins
7ea00f30bc Merge "Reorganize SLO unit tests" 2013-11-27 21:09:41 +00:00
Peter Portante
3c7c355120 Use TCP_NODELAY for created sockets.
Mark Seger at HP has been looking at small objects, 1 and 2 KB size,
and with Rick Jones' help noticed that TCP protocol traces showed
effects from the Nagel algorithm client-to-server and
server-to-client.

This patch just addresses our WSGI server responses, but does not
address out-bound connections from the various servers.

Change-Id: I11f86df1f56fba1c6ab6084dc1f580c395f072dc
Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-11-27 14:07:02 -05:00
Jenkins
d8e46eba47 Merge "slightly less early quorum" 2013-11-27 15:59:37 +00:00
Jenkins
fa8eef4b3f Merge "bare excepts, as is proper" 2013-11-27 01:57:28 +00:00
Jenkins
2ce0e62c31 Merge "add bare except to catch errors" 2013-11-26 22:57:52 +00:00
John Dickinson
71d1805683 bare excepts, as is proper
Change-Id: Ifd28f6f14a781a67644315690491888161a7250c
2013-11-26 14:39:30 -08:00
Jenkins
9248c91f6d Merge "Use upstream patched Pool.get" 2013-11-26 21:47:31 +00:00
John Dickinson
cdb6cd830a add bare except to catch errors
Change-Id: Ibe78912cf923591bddd6a8cf0e683cd028c9c4e8
2013-11-26 14:03:30 -05:00
Peter Portante
6e313e957d Fix for memcache middleware configuration
The documentation rightly said to use "memcache_max_connections", but
the code was looking for "max_connections", and only looking for it in
proxy-server.conf, not in memcache.conf as a fall back.

This commit brings the code coverage for the memcache middleware to
100%.

Closes-Bug: 1252893
Change-Id: I6ea64baa2f961a09d60b977b40d5baf842449ece
Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-11-26 18:03:33 +00:00
Michael Barton
7207926cff slightly less early quorum
The early quorum change has maybe added a little bit too much
eventual to the consistency of requests in Swift, and users can
sometimes get unexpected
results.

This change gives us a knob to turn in finding the right balance,
by adding a timeout where pending requests can finish after quorum
is achieved.

Change-Id: Ife91aaa8653e75b01313bbcf19072181739e932c
2013-11-25 21:25:55 +00:00
John Dickinson
fa1f7d9420 Use upstream patched Pool.get
This works around an eventlet bug in eventlet 0.9.16.
This version properly keeps track of pool size accounting, and
therefore doesn't let the pool grow without bound. This patched
version is the result of commit
f5e5b2bda7b442f0262ee1084deefcc5a1cc0694 in eventlet and is
documented at https://bitbucket.org/eventlet/eventlet/issue/91

This patch includes full test coverage of the back-ported code, even
when the actually-installed eventlet is newer.

This fixes bug #1254119

Change-Id: I075bb5e40e08571d52fe17fcc3fa0e25be5befed
2013-11-25 15:56:38 -05:00
Pete Zaitcev
31b311af57 Return an exit code for configuration errors
Red Hat's QA noticed that in case of the infamous "xattr>=0.4"
error, swift-init exits with a zero error code, which confuses
startup scripts (not Systemd though -- that one knows exactly
if processes run or not).

The easiest fix is to return the error code like Guido's blog
post suggested.

Change-Id: I7badd8742375a7cb2aa5606277316477b8083f8d
Fixes: rhbz#1020480
2013-11-22 19:01:10 -07:00
Jenkins
5989849512 Merge "Per device replication_lock" 2013-11-22 23:04:33 +00:00
Jenkins
20c445cc60 Merge "add an "inline" query parameter to tempurl" 2013-11-22 22:38:22 +00:00