SLO was setting it sometimes (the PUT path), but not others (the GET
path), and DLO just wasn't doing it at all. One could already
distinguish DLO/SLO internal requests by the user agent, but we should
set swift.source too.
Change-Id: I5d5b7dd49bd1522a9c830d0abd21fff92ae79a39
This is for the same reason that SLO got pulled into middleware, which
includes stuff like automatic retry of GETs on broken connection and
the multi-ring storage policy stuff.
The proxy will automatically insert the dlo middleware at an
appropriate place in the pipeline the same way it does with the
gatekeeper middleware. Clusters will still support DLOs after upgrade
even with an old config file that doesn't mention dlo at all.
Includes support for reading config values from the proxy server's
config section so that upgraded clusters continue to work as before.
Bonus fix: resolve 'after' vs. 'after_fn' in proxy's required filters
list. Having two was confusing, so I kept the more-general one.
DocImpact
blueprint multi-ring-large-objects
Change-Id: Ib3b3830c246816dd549fc74be98b4bc651e7bace
This is in case a cluster gets a problem user who has distributed the
writes to a bunch of containers but is just taking too much of the
cluster's resources.
Change-Id: Ibd2ffd0e911463a432117b478585b9f8bc4a2495
* Introduce a new privileged account header: X-Account-Access-Control
* Introduce JSON-based version 2 ACL syntax -- see below for discussion
* Implement account ACL authorization in TempAuth
X-Account-Access-Control Header
-------------------------------
Accounts now have a new privileged header to represent ACLs or any other
form of account-level access control. The value of the header is an opaque
string to be interpreted by the auth system, but it must be a JSON-encoded
dictionary. A reference implementation is given in TempAuth, with the
knowledge that historically other auth systems often use TempAuth as a
starting point.
The reference implementation describes three levels of account access:
"admin", "read-write", and "read-only". Adding new access control
features in a future patch (e.g. "write-only" account access) will
automatically be forward- and backward-compatible, due to the JSON
dictionary header format.
The privileged X-Account-Access-Control header may only be read or written
by a user with "swift_owner" status, traditionally the account owner but
now also any user on the "admin" ACL.
Access Levels:
Read-only access is intended to indicate to the auth system that this
list of identities can read everything (except privileged headers) in
the account. Specifically, a user with read-only account access can get
a list of containers in the account, list the contents of any container,
retrieve any object, and see the (non-privileged) headers of the
account, any container, or any object.
Read-write access is intended to indicate to the auth system that this
list of identities can read or write (or create) any container. A user
with read-write account access can create new containers, set any
unprivileged container headers, overwrite objects, delete containers,
etc. A read-write user can NOT set account headers (or perform any
PUT/POST/DELETE requests on the account).
Admin access is intended to indicate to the auth system that this list of
identities has "swift_owner" privileges. A user with admin account access
can do anything the account owner can, including setting account headers
and any privileged headers -- and thus changing the value of
X-Account-Access-Control and thereby granting read-only, read-write, or
admin access to other users.
The auth system is responsible for making decisions based on this header,
if it chooses to support its use. Therefore the above access level
descriptions are necessarily advisory only for other auth systems.
When setting the value of the header, callers are urged to use the new
format_acl() method, described below.
New ACL Format
--------------
The account ACLs introduce a new format for ACLs, rather than reusing the
existing format from X-Container-Read/X-Container-Write. There are several
reasons for this:
* Container ACL format does not support Unicode
* Container ACLs have a different structure than account ACLs
+ account ACLs have no concept of referrers or rlistings
+ accounts have additional "admin" access level
+ account access levels are structured as admin > rw > ro, which seems more
appropriate for how people access accounts, rather than reusing
container ACLs' orthogonal read and write access
In addition, the container ACL syntax is a bit arbitrary and highly custom,
so instead of parsing additional custom syntax, I'd rather propose a next
version and introduce a means for migration. The V2 ACL syntax has the
following benefits:
* JSON is a well-known standard syntax with parsers in all languages
* no artificial value restrictions (you can grant access to a user named
".rlistings" if you want)
* forward and backward compatibility: you may have extraneous keys, but
your attempt to parse the header won't raise an exception
I've introduced hooks in parse_acl and format_acl which currently default
to the old V1 syntax but tolerate the V2 syntax and can easily be flipped
to default to V2. I'm not changing the default or adding code to rewrite
V1 ACLs to V2, because this patch has suffered a lot of scope creep already,
but this seems like a sensible milestone in the migration.
TempAuth Account ACL Implementation
-----------------------------------
As stated above, core Swift is responsible for privileging the
X-Account-Access-Control header (making it only accessible to swift_owners),
for translating it to -sysmeta-* headers to trigger persistence by the
account server, and for including the header in the responses to requests
by privileged users. Core Swift puts no expectation on the *content* of
this header. Auth systems (including TempAuth) are responsible for
defining the content of the header and taking action based on it.
In addition to the changes described above, this patch defines a format
to be used by TempAuth for these headers in the common.middleware.acl
module, in the methods format_v2_acl() and parse_v2_acl(). This patch
also teaches TempAuth to take action based on the header contents. TempAuth
now sets swift_owner=True if the user is on the Admin ACL, authorizes
GET/HEAD/OPTIONS requests if the user is on any ACL, authorizes
PUT/POST/DELETE requests if the user is on the admin or read-write ACL, etc.
Note that the action of setting swift_owner=True triggers core Swift to
add or strip the privileged headers from the responses. Core Swift (not
the auth system) is responsible for that.
DocImpact: Documentation for the new ACL usage and format appears in
summary form in doc/source/overview_auth.rst, and in more detail in
swift/common/middleware/tempauth.py in the TempAuth class docstring.
I leave it to the Swift doc team to determine whether more is needed.
Change-Id: I836a99eaaa6bb0e92dc03e1ca46a474522e6e826
The SLO middleware supports deletion of an object and all its
segments; internally, it uses the same code as the bulk middleware to
do this (swift.common.middleware.bulk.Bulk).
If something goes wrong in the bulk delete, the Bulk object logs the
exception; however, SLO's Bulk object's logger always has the default
logging config, so if you've got a non-default logging setup
(e.g. syslog to a remote log host), this one exception message goes
somewhere else (e.g. local syslog).
This patch makes SLO and its bulk deleter share a logger so all the
messages go to the same place.
Change-Id: Idb01b5640257a02dbb9c698851a14ad8fba11a2d
These are the ones that affect what requests a client can make; the
others are just time and speed limits, so they weren't as interesting.
Change-Id: I21c19c950227f02725aafc309a3996fc6749a383
This way, with zero additional effort, SLO will support enhancements
to object storage and retrieval, such as:
* automatic resume of GETs on broken connection (today)
* storage policies (in the near future)
* erasure-coded object segments (in the far future)
This also lets SLOs work with other sorts of hypothetical third-party
middleware, for example object compression or encryption.
Getting COPY to work here is sort of a hack; the proxy's object
controller now checks for "swift.copy_response_hook" in the request's
environment and feeds the GET response (the source of the new object's
data) through it. This lets a COPY of a SLO manifest actually combine
the segments instead of merely copying the manifest document.
Updated ObjectController to expect a response's app_iter to be an
iterable, not just an iterator. (PEP 333 says "When called by the
server, the application object must return an iterable yielding zero
or more strings." ObjectController was just being too strict.) This
way, SLO can re-use the same response-generation logic for GET and
COPY requests.
Added a (sort of hokey) mechanism to allow middlewares to close
incompletely-consumed app iterators without triggering a warning. SLO
does this when it realizes it's performed a ranged GET on a manifest;
it closes the iterable, removes the range, and retries the
request. Without this change, the proxy logs would get 'Client
disconnected on read' in them.
DocImpact
blueprint multi-ring-large-objects
Change-Id: Ic11662eb5c7176fbf422a6fc87a569928d6f85a1
The copy source must be container/object.
This patch avoids the server to return
an internal server error when user provides
a path without a container.
Fixes: bug #1255049
Change-Id: I1a85c98d9b3a78bad40b8ceba9088cf323042412
Summary of the new configuration option:
The cluster operators add the container_sync middleware to their
proxy pipeline and create a container-sync-realms.conf for their
cluster and copy this out to all their proxy and container servers.
This file specifies the available container sync "realms".
A container sync realm is a group of clusters with a shared key that
have agreed to provide container syncing to one another.
The end user can then set the X-Container-Sync-To value on a
container to //realm/cluster/account/container instead of the
previously required URL.
The allowed hosts list is not used with this configuration and
instead every container sync request sent is signed using the realm
key and user key.
This offers better security as source hosts can be faked much more
easily than faking per request signatures. Replaying signed requests,
assuming it could easily be done, shouldn't be an issue as the
X-Timestamp is part of the signature and so would just short-circuit
as already current or as superceded.
This also makes configuration easier for the end user, especially
with difficult networking situations where a different host might
need to be used for the container sync daemon since it's connecting
from within a cluster. With this new configuration option, the end
user just specifies the realm and cluster names and that is resolved
to the proper endpoint configured by the operator. If the operator
changes their configuration (key or endpoint), the end user does not
need to change theirs.
DocImpact
Change-Id: Ie1704990b66d0434e4991e26ed1da8b08cb05a37
Middleware or core features may need to store metadata
against accounts or containers. This patch adds a
generic mechanism for system metadata to be persisted
in backend databases, without polluting the user
metadata namespace, by using the reserved header
namespace x-<server_type>-sysmeta-*.
Modifications are firstly that backend servers persist
system metadata headers alongside user metadata and
other system state.
For accounts and containers, system metadata in PUT
and POST requests is treated in a similar way to user
metadata. System metadata is not yet supported for
object requests.
Secondly, changes in the proxy controllers ensure that
headers in the system metadata namespace will pass through
in requests to backend servers.
Thirdly, system metadata returned from backend servers
in GET or HEAD responses is added to the cached info
dict, which middleware can access.
Finally, a gatekeeper middleware module is provided
which filters all system metadata headers from requests
and responses by removing headers with names starting
x-account-sysmeta-, x-container-sysmeta-. The gatekeeper
also removes headers starting x-object-sysmeta- in
anticipation of future support for system metadata being
set for objects. This prevents clients from writing or
reading system metadata.
The required_filters list in swift/proxy/server.py is
modified to include the gatekeeper middleware so that
if the gatekeeper has not been configured in the
pipeline then it will be automatically inserted close
to the start of the pipeline.
blueprint cluster-federation
Change-Id: I80b8b14243cc59505f8c584920f8f527646b5f45
- swift-recon now handles parsing instances where 'mounted' key (in unmounted
and disk_usage) is an error message instead of a bool.
- Add's checkmount exception handling to the recon umounted endpoint.
- Updates existing unittest to have ismount throw an error.
- Updates unittests to cover the corner cases
Change-Id: Id51d14a8b98de69faaac84b2b34b7404b7df69e9
Clients can construct tempurls for any method, but they only work if
they're in this list, so it's helpful for clients to see the list.
Change-Id: Id852f457d65b62c4fe79db01b1d7029a5fa5aa09
Remove the useless arg ("start index" = 0) in files, since its default
value is 0, to make code cleaner.
Fixes bug #1259750
Change-Id: I52afac28a3248895bb1c012a5934d39e7c2cc5a9
Bulk middleware now has a mechanism to retry a delete when the
HTTP response code is 409. This happens when the container still
has objects. It can be useful in a bulk delete where you delete
all the objects in a container and then try to delete the container.
It is very likely that at the end it will fail because the replica
objects have not been deleted by the time the middleware got a
successful response.
Change-Id: I1614fcb5cc511be26a9dda90753dd08ec9546a3c
Closes-Bug: #1253478
HEAD-only tempurls didn't work; tempurl only allowed a HEAD request if
the tempurl was generated for GET or PUT (that is, the method in the
HMAC-signed string was "GET" or "PUT").
The intent of the code was to allow a user with a GET or a PUT tempurl
to also perform a HEAD request; I think the breaking of HEAD tempurls
is just a bug.
Change-Id: I621ddaac03e0d058dd9e7c7c374cb5c4b6386d36
These will allow clients to perform the minimal number of requests
required to accomplish some bulk tasks. For example, a client with
many objects to delete can learn that the cluster's limit on
deletes-per-request is, say, 128, and then batch up their deletes in
groups of 128. Without this, the client has to either discover the
limit out-of-band somehow (and get notified if it changes), or do some
sort of binary search to figure out the limit.
Similar reasoning applies to the containers-per-request value.
The errors-per-request values are included so that clients may size
their requests such that everything is attempted regardless of
failure.
I split the 'bulk' entry into 'bulk_delete' and 'bulk_upload' because,
from a client's standpoint, they're separate operations. It so happens
that Swift implements both in one piece of middleware, but clients
don't care.
Bonus fix: documented a missing config setting for the bulk middleware.
Change-Id: Ic3549aef79682fd5b798145c3545c1609aa1592b
Add accurate timestamps to the proxy-logging middleware log lines for
the start and end of a request. We use 9 digits of precision since on
some systems the clock resolution can be as fine as 1 ns.
The goal is to allow for log processing that can use the more accurate
timestamps to correlate requests, computing number of requests in
flight at a given historical point in time.
Change-Id: I61e8784b1c455d629f1299207fc4fc7e4a134814
Signed-off-by: Peter Portante <peter.portante@redhat.com>
Also bring unit test coverage to 100% (well, at least every line is
reported as "covered").
Change-Id: I659d0c02008368897b1307a7a5c9aaba73b80588
Signed-off-by: Peter Portante <peter.portante@redhat.com>
The documentation rightly said to use "memcache_max_connections", but
the code was looking for "max_connections", and only looking for it in
proxy-server.conf, not in memcache.conf as a fall back.
This commit brings the code coverage for the memcache middleware to
100%.
Closes-Bug: 1252893
Change-Id: I6ea64baa2f961a09d60b977b40d5baf842449ece
Signed-off-by: Peter Portante <peter.portante@redhat.com>
Swift can now optionally be configured to allow requests to '/info',
providing information about the swift cluster. Additionally a HMAC
signed requests to
'/info?swiftinfo_sig=<sign>&swiftinfo_expires=<expires>' can be
configured allowing privileged access to more sensitive information
not meant to be public.
DocImpact
Change-Id: I2379360fbfe3d9e9e8b25f1dc34517d199574495
Implements: blueprint capabilities
Closes-Bug: #1245694
Giving the inline query parameter will cause the tempurl
response to be given a "Content-Disposition: inline" header,
regardless of other query parameters or metadata. This allows
easy in-line viewing, eg in browsers.
DocImpact
Change-Id: Icd5c544d6a749d4f58e8a921968f4e432a2185db
This brings some sanity to the SLO test app (the thing the middleware
wraps in the unit tests) as well as splits things into multiple test
classes.
This is part of the effort to move all SLO functionality to
middleware, but it's a separate commit to prove that these test
changes don't harm anything when running against the old code.
Change-Id: I52a16f15a80dfaf9b3c595b0e634d52f418caf6c
swift.common.utils.ismount maybe raise some OSError in some special
cases; and the request against /recon/diskusage doesn't handle it
before. This patch let output of mounted keyword is the error's message.
Change-Id: I5d9018f580181e618a3fa072b7a760d41795d8eb
Closes-Bug: #1249181