383 Commits

Author SHA1 Message Date
Samuel Merritt
5fca0d07a8 Ensure swift.source is set for DLO/SLO requests
SLO was setting it sometimes (the PUT path), but not others (the GET
path), and DLO just wasn't doing it at all. One could already
distinguish DLO/SLO internal requests by the user agent, but we should
set swift.source too.

Change-Id: I5d5b7dd49bd1522a9c830d0abd21fff92ae79a39
2014-02-11 10:22:13 -08:00
Samuel Merritt
6acea29fa6 Move all DLO functionality to middleware
This is for the same reason that SLO got pulled into middleware, which
includes stuff like automatic retry of GETs on broken connection and
the multi-ring storage policy stuff.

The proxy will automatically insert the dlo middleware at an
appropriate place in the pipeline the same way it does with the
gatekeeper middleware. Clusters will still support DLOs after upgrade
even with an old config file that doesn't mention dlo at all.

Includes support for reading config values from the proxy server's
config section so that upgraded clusters continue to work as before.

Bonus fix: resolve 'after' vs. 'after_fn' in proxy's required filters
list. Having two was confusing, so I kept the more-general one.

DocImpact

blueprint multi-ring-large-objects

Change-Id: Ib3b3830c246816dd549fc74be98b4bc651e7bace
2014-02-03 18:29:48 -08:00
Jenkins
d2885febbf Merge "Fix container quota MW for handling a bad source path" 2014-02-04 01:53:36 +00:00
Jenkins
251b7b8734 Merge "Privileged acct ACL header, new ACL syntax, TempAuth impl." 2014-02-03 22:40:52 +00:00
David Goetz
b89ac55c05 Add a way to ratelimit all writes to an account
This is in case a cluster gets a problem user who has distributed the
writes to a bunch of containers but is just taking too much of the
cluster's resources.

Change-Id: Ibd2ffd0e911463a432117b478585b9f8bc4a2495
2014-01-29 14:08:19 -08:00
Jon Snitow
282fa0c398 Privileged acct ACL header, new ACL syntax, TempAuth impl.
* Introduce a new privileged account header: X-Account-Access-Control
 * Introduce JSON-based version 2 ACL syntax -- see below for discussion
 * Implement account ACL authorization in TempAuth

X-Account-Access-Control Header
-------------------------------

Accounts now have a new privileged header to represent ACLs or any other
form of account-level access control.  The value of the header is an opaque
string to be interpreted by the auth system, but it must be a JSON-encoded
dictionary.  A reference implementation is given in TempAuth, with the
knowledge that historically other auth systems often use TempAuth as a
starting point.

The reference implementation describes three levels of account access:
"admin", "read-write", and "read-only".  Adding new access control
features in a future patch (e.g. "write-only" account access) will
automatically be forward- and backward-compatible, due to the JSON
dictionary header format.

The privileged X-Account-Access-Control header may only be read or written
by a user with "swift_owner" status, traditionally the account owner but
now also any user on the "admin" ACL.

Access Levels:

Read-only access is intended to indicate to the auth system that this
list of identities can read everything (except privileged headers) in
the account.  Specifically, a user with read-only account access can get
a list of containers in the account, list the contents of any container,
retrieve any object, and see the (non-privileged) headers of the
account, any container, or any object.

Read-write access is intended to indicate to the auth system that this
list of identities can read or write (or create) any container.  A user
with read-write account access can create new containers, set any
unprivileged container headers, overwrite objects, delete containers,
etc.  A read-write user can NOT set account headers (or perform any
PUT/POST/DELETE requests on the account).

Admin access is intended to indicate to the auth system that this list of
identities has "swift_owner" privileges.  A user with admin account access
can do anything the account owner can, including setting account headers
and any privileged headers -- and thus changing the value of
X-Account-Access-Control and thereby granting read-only, read-write, or
admin access to other users.

The auth system is responsible for making decisions based on this header,
if it chooses to support its use.  Therefore the above access level
descriptions are necessarily advisory only for other auth systems.

When setting the value of the header, callers are urged to use the new
format_acl() method, described below.

New ACL Format
--------------

The account ACLs introduce a new format for ACLs, rather than reusing the
existing format from X-Container-Read/X-Container-Write.  There are several
reasons for this:
 * Container ACL format does not support Unicode
 * Container ACLs have a different structure than account ACLs
  + account ACLs have no concept of referrers or rlistings
  + accounts have additional "admin" access level
  + account access levels are structured as admin > rw > ro, which seems more
    appropriate for how people access accounts, rather than reusing
    container ACLs' orthogonal read and write access

In addition, the container ACL syntax is a bit arbitrary and highly custom,
so instead of parsing additional custom syntax, I'd rather propose a next
version and introduce a means for migration.  The V2 ACL syntax has the
following benefits:
 * JSON is a well-known standard syntax with parsers in all languages
 * no artificial value restrictions (you can grant access to a user named
    ".rlistings" if you want)
 * forward and backward compatibility: you may have extraneous keys, but
    your attempt to parse the header won't raise an exception

I've introduced hooks in parse_acl and format_acl which currently default
to the old V1 syntax but tolerate the V2 syntax and can easily be flipped
to default to V2.  I'm not changing the default or adding code to rewrite
V1 ACLs to V2, because this patch has suffered a lot of scope creep already,
but this seems like a sensible milestone in the migration.

TempAuth Account ACL Implementation
-----------------------------------

As stated above, core Swift is responsible for privileging the
X-Account-Access-Control header (making it only accessible to swift_owners),
for translating it to -sysmeta-* headers to trigger persistence by the
account server, and for including the header in the responses to requests
by privileged users.  Core Swift puts no expectation on the *content* of
this header.  Auth systems (including TempAuth) are responsible for
defining the content of the header and taking action based on it.

In addition to the changes described above, this patch defines a format
to be used by TempAuth for these headers in the common.middleware.acl
module, in the methods format_v2_acl() and parse_v2_acl().  This patch
also teaches TempAuth to take action based on the header contents.  TempAuth
now sets swift_owner=True if the user is on the Admin ACL, authorizes
GET/HEAD/OPTIONS requests if the user is on any ACL, authorizes
PUT/POST/DELETE requests if the user is on the admin or read-write ACL, etc.

Note that the action of setting swift_owner=True triggers core Swift to
add or strip the privileged headers from the responses.  Core Swift (not
the auth system) is responsible for that.

DocImpact: Documentation for the new ACL usage and format appears in
summary form in doc/source/overview_auth.rst, and in more detail in
swift/common/middleware/tempauth.py in the TempAuth class docstring.
I leave it to the Swift doc team to determine whether more is needed.

Change-Id: I836a99eaaa6bb0e92dc03e1ca46a474522e6e826
2014-01-29 13:02:54 -08:00
Jenkins
5f1d3ee82c Merge "Use a tempfile.mkdtemp() based temporary directory" 2014-01-24 15:23:48 +00:00
Peter Portante
ca87827db9 Use a tempfile.mkdtemp() based temporary directory
Change-Id: Ie0e8137615348b130b67323d0d2913dc5ebfd5fb
2014-01-19 16:29:45 -05:00
Samuel Merritt
a8ba751370 Make sure SLO's bulk-delete logger is configured
The SLO middleware supports deletion of an object and all its
segments; internally, it uses the same code as the bulk middleware to
do this (swift.common.middleware.bulk.Bulk).

If something goes wrong in the bulk delete, the Bulk object logs the
exception; however, SLO's Bulk object's logger always has the default
logging config, so if you've got a non-default logging setup
(e.g. syslog to a remote log host), this one exception message goes
somewhere else (e.g. local syslog).

This patch makes SLO and its bulk deleter share a logger so all the
messages go to the same place.

Change-Id: Idb01b5640257a02dbb9c698851a14ad8fba11a2d
2014-01-19 09:57:12 -05:00
Samuel Merritt
928f5c1964 Expose some SLO constraints via /info
These are the ones that affect what requests a client can make; the
others are just time and speed limits, so they weren't as interesting.

Change-Id: I21c19c950227f02725aafc309a3996fc6749a383
2014-01-14 22:19:58 -08:00
Jenkins
d698c21ab3 Merge "New container sync configuration option" 2014-01-14 02:40:39 +00:00
Samuel Merritt
1901719542 Move all SLO functionality to middleware
This way, with zero additional effort, SLO will support enhancements
to object storage and retrieval, such as:
 * automatic resume of GETs on broken connection (today)
 * storage policies (in the near future)
 * erasure-coded object segments (in the far future)

This also lets SLOs work with other sorts of hypothetical third-party
middleware, for example object compression or encryption.

Getting COPY to work here is sort of a hack; the proxy's object
controller now checks for "swift.copy_response_hook" in the request's
environment and feeds the GET response (the source of the new object's
data) through it. This lets a COPY of a SLO manifest actually combine
the segments instead of merely copying the manifest document.

Updated ObjectController to expect a response's app_iter to be an
iterable, not just an iterator. (PEP 333 says "When called by the
server, the application object must return an iterable yielding zero
or more strings." ObjectController was just being too strict.) This
way, SLO can re-use the same response-generation logic for GET and
COPY requests.

Added a (sort of hokey) mechanism to allow middlewares to close
incompletely-consumed app iterators without triggering a warning. SLO
does this when it realizes it's performed a ranged GET on a manifest;
it closes the iterable, removes the range, and retries the
request. Without this change, the proxy logs would get 'Client
disconnected on read' in them.

DocImpact

blueprint multi-ring-large-objects

Change-Id: Ic11662eb5c7176fbf422a6fc87a569928d6f85a1
2014-01-13 10:52:29 -08:00
Fabien Boucher
8e1e67c02d Fix container quota MW for handling a bad source path
The copy source must be container/object.
This patch avoids the server to return
an internal server error when user provides
a path without a container.

Fixes: bug #1255049
Change-Id: I1a85c98d9b3a78bad40b8ceba9088cf323042412
2014-01-13 13:25:02 +01:00
Jenkins
13563e3d87 Merge "Make POST for bulk delete actually work" 2014-01-12 07:15:11 +00:00
gholt
f60d05686f New container sync configuration option
Summary of the new configuration option:

The cluster operators add the container_sync middleware to their
proxy pipeline and create a container-sync-realms.conf for their
cluster and copy this out to all their proxy and container servers.
This file specifies the available container sync "realms".

A container sync realm is a group of clusters with a shared key that
have agreed to provide container syncing to one another.

The end user can then set the X-Container-Sync-To value on a
container to //realm/cluster/account/container instead of the
previously required URL.

The allowed hosts list is not used with this configuration and
instead every container sync request sent is signed using the realm
key and user key.

This offers better security as source hosts can be faked much more
easily than faking per request signatures. Replaying signed requests,
assuming it could easily be done, shouldn't be an issue as the
X-Timestamp is part of the signature and so would just short-circuit
as already current or as superceded.

This also makes configuration easier for the end user, especially
with difficult networking situations where a different host might
need to be used for the container sync daemon since it's connecting
from within a cluster. With this new configuration option, the end
user just specifies the realm and cluster names and that is resolved
to the proper endpoint configured by the operator. If the operator
changes their configuration (key or endpoint), the end user does not
need to change theirs.

DocImpact

Change-Id: Ie1704990b66d0434e4991e26ed1da8b08cb05a37
2014-01-10 23:48:00 +00:00
Jenkins
f310006fae Merge "Generic means for persisting system metadata." 2014-01-10 14:44:59 +00:00
Jenkins
c709cf90fc Merge "Adds a retry mechanism when deleting containers" 2014-01-08 21:09:33 +00:00
Peter Portante
a708295d82 Remove trailing slash for consistency
Change-Id: Idd4fd116b6be226e46e33f421883b6fb34947a84
Signed-off-by: Peter Portante <peter.portante@redhat.com>
2014-01-06 18:12:42 -05:00
anc
6164fa246d Generic means for persisting system metadata.
Middleware or core features may need to store metadata
against accounts or containers. This patch adds a
generic mechanism for system metadata to be persisted
in backend databases, without polluting the user
metadata namespace, by using the reserved header
namespace x-<server_type>-sysmeta-*.

Modifications are firstly that backend servers persist
system metadata headers alongside user metadata and
other system state.

For accounts and containers, system metadata in PUT
and POST requests is treated in a similar way to user
metadata. System metadata is not yet supported for
object requests.

Secondly, changes in the proxy controllers ensure that
headers in the system metadata namespace will pass through
in requests to backend servers.

Thirdly, system metadata returned from backend servers
in GET or HEAD responses is added to the cached info
dict, which middleware can access.

Finally, a gatekeeper middleware module is provided
which filters all system metadata headers from requests
and responses by removing headers with names starting
x-account-sysmeta-, x-container-sysmeta-. The gatekeeper
also removes headers starting x-object-sysmeta- in
anticipation of future support for system metadata being
set for objects. This prevents clients from writing or
reading system metadata.

The required_filters list in swift/proxy/server.py is
modified to include the gatekeeper middleware so that
if the gatekeeper has not been configured in the
pipeline then it will be automatically inserted close
to the start of the pipeline.

blueprint cluster-federation

Change-Id: I80b8b14243cc59505f8c584920f8f527646b5f45
2014-01-06 22:29:37 +00:00
Florian Hines
62254e42c4 Fix checkmount error parsing in swift-recon
- swift-recon now handles parsing instances where 'mounted' key (in unmounted
  and disk_usage) is an error message instead of a bool.
- Add's checkmount exception handling to the recon umounted endpoint.
- Updates existing unittest to have ismount throw an error.
- Updates unittests to cover the corner cases

Change-Id: Id51d14a8b98de69faaac84b2b34b7404b7df69e9
2013-12-28 20:58:27 -08:00
Jenkins
c9cb39282b Merge "Expose allowed tempurl methods in /info" 2013-12-23 19:01:31 +00:00
Samuel Merritt
aae254df55 Make POST for bulk delete actually work
Change-Id: I568e7e31df3dcbeac20dba6d543a13c0409de00e
Closes-Bug: 1232787
2013-12-21 11:32:34 -08:00
Samuel Merritt
979033a14e Expose allowed tempurl methods in /info
Clients can construct tempurls for any method, but they only work if
they're in this list, so it's helpful for clients to see the list.

Change-Id: Id852f457d65b62c4fe79db01b1d7029a5fa5aa09
2013-12-12 16:13:42 -08:00
Zhang Jinnan
bdc296abbc Remove start index 0 in range()
Remove the useless arg ("start index" = 0) in files, since its default
value is 0, to make code cleaner.

Fixes bug #1259750

Change-Id: I52afac28a3248895bb1c012a5934d39e7c2cc5a9
2013-12-10 16:16:44 -08:00
Cristian A Sanchez
96c9ff56fa Adds a retry mechanism when deleting containers
Bulk middleware now has a mechanism to retry a delete when the
HTTP response code is 409. This happens when the container still
has objects. It can be useful in a bulk delete where you delete
all the objects in a container and then try to delete the container.
It is very likely that at the end it will fail because the replica
objects have not been deleted by the time the middleware got a
successful response.

Change-Id: I1614fcb5cc511be26a9dda90753dd08ec9546a3c
Closes-Bug: #1253478
2013-12-09 19:16:35 -08:00
Jenkins
55dafa2f07 Merge "Prevent early auth deny in tempauth when using swift3 middleware." 2013-12-07 10:02:00 +00:00
Jenkins
c36eeb3502 Merge "Fix HEAD tempurls" 2013-12-06 03:33:08 +00:00
Samuel Merritt
d18b6d8d5d Fix HEAD tempurls
HEAD-only tempurls didn't work; tempurl only allowed a HEAD request if
the tempurl was generated for GET or PUT (that is, the method in the
HMAC-signed string was "GET" or "PUT").

The intent of the code was to allow a user with a GET or a PUT tempurl
to also perform a HEAD request; I think the breaking of HEAD tempurls
is just a bug.

Change-Id: I621ddaac03e0d058dd9e7c7c374cb5c4b6386d36
2013-12-05 16:52:25 -08:00
Peter Portante
60bf2dc675 Override spelling of tempauth test name
Change-Id: Ife571bbeb6dd4ba5c552a2f0725c36e133e64cf4
2013-12-05 15:31:09 -05:00
Jenkins
645d1c93c5 Merge "Add accurate timestamps in proxy log" 2013-12-03 03:51:42 +00:00
Samuel Merritt
0d45e99ff0 Expose bulk-operation limits in /info.
These will allow clients to perform the minimal number of requests
required to accomplish some bulk tasks. For example, a client with
many objects to delete can learn that the cluster's limit on
deletes-per-request is, say, 128, and then batch up their deletes in
groups of 128. Without this, the client has to either discover the
limit out-of-band somehow (and get notified if it changes), or do some
sort of binary search to figure out the limit.

Similar reasoning applies to the containers-per-request value.

The errors-per-request values are included so that clients may size
their requests such that everything is attempted regardless of
failure.

I split the 'bulk' entry into 'bulk_delete' and 'bulk_upload' because,
from a client's standpoint, they're separate operations. It so happens
that Swift implements both in one piece of middleware, but clients
don't care.

Bonus fix: documented a missing config setting for the bulk middleware.

Change-Id: Ic3549aef79682fd5b798145c3545c1609aa1592b
2013-11-30 11:09:34 -08:00
Peter Portante
d4e02a2e8b Add accurate timestamps in proxy log
Add accurate timestamps to the proxy-logging middleware log lines for
the start and end of a request. We use 9 digits of precision since on
some systems the clock resolution can be as fine as 1 ns.

The goal is to allow for log processing that can use the more accurate
timestamps to correlate requests, computing number of requests in
flight at a given historical point in time.

Change-Id: I61e8784b1c455d629f1299207fc4fc7e4a134814
Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-11-27 20:21:54 -05:00
Jenkins
7cc009735a Merge "Add a unit test to verify proxy logging fields" 2013-11-28 00:20:22 +00:00
Jenkins
34b4bf34d2 Merge "Added discoverable capabilities." 2013-11-28 00:11:35 +00:00
Peter Portante
e0147e60d8 Add a unit test to verify proxy logging fields
Also bring unit test coverage to 100% (well, at least every line is
reported as "covered").

Change-Id: I659d0c02008368897b1307a7a5c9aaba73b80588
Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-11-27 16:19:51 -05:00
Jenkins
7ea00f30bc Merge "Reorganize SLO unit tests" 2013-11-27 21:09:41 +00:00
Jenkins
2ce0e62c31 Merge "add bare except to catch errors" 2013-11-26 22:57:52 +00:00
John Dickinson
cdb6cd830a add bare except to catch errors
Change-Id: Ibe78912cf923591bddd6a8cf0e683cd028c9c4e8
2013-11-26 14:03:30 -05:00
Peter Portante
6e313e957d Fix for memcache middleware configuration
The documentation rightly said to use "memcache_max_connections", but
the code was looking for "max_connections", and only looking for it in
proxy-server.conf, not in memcache.conf as a fall back.

This commit brings the code coverage for the memcache middleware to
100%.

Closes-Bug: 1252893
Change-Id: I6ea64baa2f961a09d60b977b40d5baf842449ece
Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-11-26 18:03:33 +00:00
Richard (Rick) Hawkins
2c4bf81464 Added discoverable capabilities.
Swift can now optionally be configured to allow requests to '/info',
providing information about the swift cluster.  Additionally a HMAC
signed requests to
'/info?swiftinfo_sig=<sign>&swiftinfo_expires=<expires>' can be
configured allowing privileged access to more sensitive information
not meant to be public.

DocImpact
Change-Id: I2379360fbfe3d9e9e8b25f1dc34517d199574495
Implements: blueprint capabilities
Closes-Bug: #1245694
2013-11-22 15:54:13 -06:00
John Dickinson
b9efe1cd46 add an "inline" query parameter to tempurl
Giving the inline query parameter will cause the tempurl
response to be given a "Content-Disposition: inline" header,
regardless of other query parameters or metadata. This allows
easy in-line viewing, eg in browsers.

DocImpact

Change-Id: Icd5c544d6a749d4f58e8a921968f4e432a2185db
2013-11-22 09:46:40 -08:00
Jenkins
98e5371013 Merge "Do not format messages before they are logged" 2013-11-22 09:55:05 +00:00
Jenkins
62b693d133 Merge "Use POST in bulk-delete" 2013-11-22 09:11:05 +00:00
Peter Portante
a410730a2b Do not format messages before they are logged
Change-Id: Ia645c9eca47b7f404d9b987f68a96b4744031e9d
Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-11-21 22:34:36 -05:00
Jenkins
a73d5e4921 Merge "Handle COPY verb in account quota middleware" 2013-11-21 10:21:43 +00:00
Kun Huang
94090e8760 Use POST in bulk-delete
The DELETE verb applies to a single resource, and doesn't define any
semantics for the body.

http://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html#sec9.7

The swift Bulk Delete command affects multiple resources specified in a
DELETE body.

http://docs.openstack.org/developer/swift/misc.html#module-swift.common.middleware.bulk

While Bulk Delete is a welcome operation, its usage of DELETE is
unusual: affecting multiple resources and relying on reading content.

More typically, such an operation employs POST (or PUT), which folks
including api-craft usually agree is the best "catch-all" verb for
behaviors such as those affecting multiple resources. That's the TL;DR;
of the thread below.

https://groups.google.com/forum/#!searchin/api-craft/Regarding$20Bulk$20actions/api-craft/wY-W1NdZDRs/7YDwMhCR608J

Note that this topic isn't nasal or abstract. The current behavior is
unsupported using the built-in java http client. Even if third-party
libraries can work around this behavior, it is probably best to not be a
snowflake wrt http verb semantics where possible!

http://stackoverflow.com/questions/9100776/http-delete-with-request-body-issues

DocImpact

Closes-Bug: #1232787
Change-Id: I0fc74c85618fe4dd7ff5e7f9756c7f6f67aa0465
2013-11-20 11:28:14 +08:00
Samuel Merritt
9151bcc92c Reorganize SLO unit tests
This brings some sanity to the SLO test app (the thing the middleware
wraps in the unit tests) as well as splits things into multiple test
classes.

This is part of the effort to move all SLO functionality to
middleware, but it's a separate commit to prove that these test
changes don't harm anything when running against the old code.

Change-Id: I52a16f15a80dfaf9b3c595b0e634d52f418caf6c
2013-11-19 16:49:36 -08:00
Kun Huang
fd4843f8e7 catch OSError to prevent breaking request /recon/diskusage
swift.common.utils.ismount maybe raise some OSError in some special
cases; and the request against /recon/diskusage doesn't handle it
before. This patch let output of mounted keyword is the error's message.

Change-Id: I5d9018f580181e618a3fa072b7a760d41795d8eb
Closes-Bug: #1249181
2013-11-13 22:46:20 +08:00
Jenkins
fab1cd4d71 Merge "form post over XMLHttpRequest (cors) broken" 2013-10-30 15:22:49 +00:00
David Goetz
56c902c8de form post over XMLHttpRequest (cors) broken
Change-Id: Ia55e0d3974a96e11d49ab3cb26b6dcd7129b5cc8
2013-10-28 22:38:53 +00:00