1454 Commits

Author SHA1 Message Date
Samuel Merritt
5fca0d07a8 Ensure swift.source is set for DLO/SLO requests
SLO was setting it sometimes (the PUT path), but not others (the GET
path), and DLO just wasn't doing it at all. One could already
distinguish DLO/SLO internal requests by the user agent, but we should
set swift.source too.

Change-Id: I5d5b7dd49bd1522a9c830d0abd21fff92ae79a39
2014-02-11 10:22:13 -08:00
Jenkins
8902aef591 Merge "Remove swiftclient dependency" 2014-02-11 02:18:06 +00:00
Jenkins
8fdb3217c9 Merge "Attempt to make test_memcache_pool_timeout stable" 2014-02-11 00:30:16 +00:00
Jenkins
9e9c0956bc Merge "Add tests for swift-ring-builder" 2014-02-07 02:59:11 +00:00
Christian Schwede
1f3ae6d8da Remove swiftclient dependency
Removes the requirement for swiftclient in swift-dispersion-report
and swift-dispersion-populate. To prevent a dependency on
keystoneclient and to avoid reinventing the wheel with an internal
keystoneclient, authentication with keystone is only supported if
swiftclient is available. If not, only auth v1 is supported.

The dependency in swift/container/sync.py has also been removed.

Implements: blueprint remove-swiftclient-dependency

Change-Id: I6ec3b3c85a67b9ab6eb04b90ffc16daf1600e8a7
2014-02-06 09:44:58 +00:00
Jenkins
6fec0dd735 Merge "Move all DLO functionality to middleware" 2014-02-05 22:59:15 +00:00
Christian Schwede
24657b2b39 Add tests for swift-ring-builder
Add some tests for essential methods in swift-ring-builder.
Tests for removing or changing device settings are executed
with different search values to cover many possible command
line arguments.

Currently tested methods:

- create ring
- add device
- remove device
- set weight
- set info
- set min_part_hours
- set replicas

Tests use swift.common.ring.RingBuilder to verify actions.

Catching and testing output from print statements is not
tested, because this requires redirecting sys.stdout during
tests and that might have some sideeffects for testing tools.

bin/swift-ring-builder has been moved to swift/cli/ringbuilder.py
and slightly modified to work as before (mainly due to no more
existing global variables since that part of the code has been
moved inside a main() function).

Change-Id: Ia63f59a8faca1fad990784f27532ca07a2125454
2014-02-05 16:20:09 +00:00
Jenkins
cef8e26ffc Merge "Fix up double mocking of swob.Request.__init__" 2014-02-04 20:25:44 +00:00
Jenkins
57603b7c58 Merge "Sanify handoff search depth with non-integer replica counts" 2014-02-04 20:25:33 +00:00
Jenkins
766ab4f0d0 Merge "Add some tests for bin/swift-recon" 2014-02-04 03:11:07 +00:00
Samuel Merritt
6acea29fa6 Move all DLO functionality to middleware
This is for the same reason that SLO got pulled into middleware, which
includes stuff like automatic retry of GETs on broken connection and
the multi-ring storage policy stuff.

The proxy will automatically insert the dlo middleware at an
appropriate place in the pipeline the same way it does with the
gatekeeper middleware. Clusters will still support DLOs after upgrade
even with an old config file that doesn't mention dlo at all.

Includes support for reading config values from the proxy server's
config section so that upgraded clusters continue to work as before.

Bonus fix: resolve 'after' vs. 'after_fn' in proxy's required filters
list. Having two was confusing, so I kept the more-general one.

DocImpact

blueprint multi-ring-large-objects

Change-Id: Ib3b3830c246816dd549fc74be98b4bc651e7bace
2014-02-03 18:29:48 -08:00
Jenkins
d2885febbf Merge "Fix container quota MW for handling a bad source path" 2014-02-04 01:53:36 +00:00
Jenkins
7e2f9bbe6a Merge "Reduce line count by two" 2014-02-03 23:40:45 +00:00
Jenkins
251b7b8734 Merge "Privileged acct ACL header, new ACL syntax, TempAuth impl." 2014-02-03 22:40:52 +00:00
Peter Portante
b881a1896f Fix up double mocking of swob.Request.__init__
From Pete Zaitcev's work on PBes (https://review.openstack.org/47713):

  Fixes up a problem with double mocking of Request.__init__, which
  should be seen in stock code as well.

Change-Id: Ibf8103425404b03aaab9a4e97a3a62d0fb8dbb14
2014-02-03 17:00:53 -05:00
Peter Portante
0d710140c5 Reduce line count by two
Change-Id: Ib0ee2bb5b202966d3d07c754656c92006149af9c
2014-02-03 16:01:14 -05:00
Peter Portante
204240222f Attempt to make test_memcache_pool_timeout stable
Too much time can pass after we create a set of co-routines and when
the parent co-routine continues is setup and staging. Instead, we
perform all the setup and staging, let the system run and then verify
the final state after all the dust settles.

Change-Id: I801148e380807119d3d6da5a24ba9cced39fb339
Closes-Bug: 1272503
2014-02-03 00:47:09 -05:00
Jenkins
e26e915330 Merge "Attempt to fix periodic memcache timeout test" 2014-02-01 02:55:20 +00:00
Jenkins
92e347813c Merge "Add a way to ratelimit all writes to an account" 2014-02-01 01:34:22 +00:00
Chuck Thier
15c04f6869 Attempt to fix periodic memcache timeout test
Not sure this will really fix the issue, but made a small change that
seemed to make more sense for the flow of the test.

Closes-Bug: #1272503
Change-Id: Ifb25ed7c4a42b11f74d865787f63fea4e0beb745
2014-01-31 19:54:02 +00:00
Christian Schwede
cd4b4da8b6 Add some tests for bin/swift-recon
Fix also minor bug in zone filtering when zone set to 0.

Moved bin/swift-recon to swift/cli/recon.py, which makes
it possible to import it without using some scary hacks.
bin/swift-recon is now created by setup.py install.

Closes-Bug: #1261692
Change-Id: Id0729991c8ece73604467480dbf93fec7d8eb196
2014-01-31 15:34:37 +00:00
Jenkins
4b9667d9d6 Merge "Container Sync: Simple HTTP Proxy load balancing" 2014-01-31 02:38:13 +00:00
Jenkins
ffd338e8d0 Merge "Skip delete_at_update for replication requests" 2014-01-30 11:11:10 +00:00
Jenkins
95ebac49a1 Merge "Config option to lower the timeout for recoverable object GETs." 2014-01-30 03:27:58 +00:00
David Goetz
b89ac55c05 Add a way to ratelimit all writes to an account
This is in case a cluster gets a problem user who has distributed the
writes to a bunch of containers but is just taking too much of the
cluster's resources.

Change-Id: Ibd2ffd0e911463a432117b478585b9f8bc4a2495
2014-01-29 14:08:19 -08:00
Jon Snitow
282fa0c398 Privileged acct ACL header, new ACL syntax, TempAuth impl.
* Introduce a new privileged account header: X-Account-Access-Control
 * Introduce JSON-based version 2 ACL syntax -- see below for discussion
 * Implement account ACL authorization in TempAuth

X-Account-Access-Control Header
-------------------------------

Accounts now have a new privileged header to represent ACLs or any other
form of account-level access control.  The value of the header is an opaque
string to be interpreted by the auth system, but it must be a JSON-encoded
dictionary.  A reference implementation is given in TempAuth, with the
knowledge that historically other auth systems often use TempAuth as a
starting point.

The reference implementation describes three levels of account access:
"admin", "read-write", and "read-only".  Adding new access control
features in a future patch (e.g. "write-only" account access) will
automatically be forward- and backward-compatible, due to the JSON
dictionary header format.

The privileged X-Account-Access-Control header may only be read or written
by a user with "swift_owner" status, traditionally the account owner but
now also any user on the "admin" ACL.

Access Levels:

Read-only access is intended to indicate to the auth system that this
list of identities can read everything (except privileged headers) in
the account.  Specifically, a user with read-only account access can get
a list of containers in the account, list the contents of any container,
retrieve any object, and see the (non-privileged) headers of the
account, any container, or any object.

Read-write access is intended to indicate to the auth system that this
list of identities can read or write (or create) any container.  A user
with read-write account access can create new containers, set any
unprivileged container headers, overwrite objects, delete containers,
etc.  A read-write user can NOT set account headers (or perform any
PUT/POST/DELETE requests on the account).

Admin access is intended to indicate to the auth system that this list of
identities has "swift_owner" privileges.  A user with admin account access
can do anything the account owner can, including setting account headers
and any privileged headers -- and thus changing the value of
X-Account-Access-Control and thereby granting read-only, read-write, or
admin access to other users.

The auth system is responsible for making decisions based on this header,
if it chooses to support its use.  Therefore the above access level
descriptions are necessarily advisory only for other auth systems.

When setting the value of the header, callers are urged to use the new
format_acl() method, described below.

New ACL Format
--------------

The account ACLs introduce a new format for ACLs, rather than reusing the
existing format from X-Container-Read/X-Container-Write.  There are several
reasons for this:
 * Container ACL format does not support Unicode
 * Container ACLs have a different structure than account ACLs
  + account ACLs have no concept of referrers or rlistings
  + accounts have additional "admin" access level
  + account access levels are structured as admin > rw > ro, which seems more
    appropriate for how people access accounts, rather than reusing
    container ACLs' orthogonal read and write access

In addition, the container ACL syntax is a bit arbitrary and highly custom,
so instead of parsing additional custom syntax, I'd rather propose a next
version and introduce a means for migration.  The V2 ACL syntax has the
following benefits:
 * JSON is a well-known standard syntax with parsers in all languages
 * no artificial value restrictions (you can grant access to a user named
    ".rlistings" if you want)
 * forward and backward compatibility: you may have extraneous keys, but
    your attempt to parse the header won't raise an exception

I've introduced hooks in parse_acl and format_acl which currently default
to the old V1 syntax but tolerate the V2 syntax and can easily be flipped
to default to V2.  I'm not changing the default or adding code to rewrite
V1 ACLs to V2, because this patch has suffered a lot of scope creep already,
but this seems like a sensible milestone in the migration.

TempAuth Account ACL Implementation
-----------------------------------

As stated above, core Swift is responsible for privileging the
X-Account-Access-Control header (making it only accessible to swift_owners),
for translating it to -sysmeta-* headers to trigger persistence by the
account server, and for including the header in the responses to requests
by privileged users.  Core Swift puts no expectation on the *content* of
this header.  Auth systems (including TempAuth) are responsible for
defining the content of the header and taking action based on it.

In addition to the changes described above, this patch defines a format
to be used by TempAuth for these headers in the common.middleware.acl
module, in the methods format_v2_acl() and parse_v2_acl().  This patch
also teaches TempAuth to take action based on the header contents.  TempAuth
now sets swift_owner=True if the user is on the Admin ACL, authorizes
GET/HEAD/OPTIONS requests if the user is on any ACL, authorizes
PUT/POST/DELETE requests if the user is on the admin or read-write ACL, etc.

Note that the action of setting swift_owner=True triggers core Swift to
add or strip the privileged headers from the responses.  Core Swift (not
the auth system) is responsible for that.

DocImpact: Documentation for the new ACL usage and format appears in
summary form in doc/source/overview_auth.rst, and in more detail in
swift/common/middleware/tempauth.py in the TempAuth class docstring.
I leave it to the Swift doc team to determine whether more is needed.

Change-Id: I836a99eaaa6bb0e92dc03e1ca46a474522e6e826
2014-01-29 13:02:54 -08:00
David Goetz
c0bf01afdb Config option to lower the timeout for recoverable object GETs.
Change-Id: I71f9824559126e4025e7629715ab9dac64231e09
2014-01-28 14:17:59 -08:00
Jenkins
9034558f0b Merge "Add secondary groups to user during privilege escalation" 2014-01-28 00:37:49 +00:00
Jenkins
61f9380225 Merge "Move test.unit.locale out of gettext's way" 2014-01-26 02:43:39 +00:00
David Goetz
89deaf5f3c Add flag to stop swob from always using absolute location.
This is needed for SOS (along with patch
https://github.com/dpgoetz/sos/pull/37)
to work with swift 1.12 . By spec you should always use the absolute
location but this causes a problem with staticweb over a cdn using a
cname. Basically you want to be able to forward the browser to a
relative location instead of whatever full url the proxy server
thinks you are using.

Change-Id: I3fa1d415bf9b566be069458b838f7e65db0c4f39
2014-01-24 13:37:45 -08:00
Jenkins
9b3c86fb70 Merge "ismount should not raise exceptions" 2014-01-24 21:07:09 +00:00
Jenkins
5f1d3ee82c Merge "Use a tempfile.mkdtemp() based temporary directory" 2014-01-24 15:23:48 +00:00
Clay Gerrard
bb8ffe17a5 Move test.unit.locale out of gettext's way
When I run unittests on my laptop I get an ImportError from gettext who tries
to "from locale import normalize" and says "ImportError: cannot import name
normalize".  Whit this patch unittests works, not sure why this problem
doesn't show up in my other dev environments.

Also this seems like good form since 'locale' is already a stdlib module.

Change-Id: If0fcd66ce391665b4e4127c610de3246db409a68
2014-01-24 01:24:08 -08:00
David Moreau Simard
c656e18949 Add secondary groups to user during privilege escalation
setgid provides the primary group, setgroups sets the secondary
groups. Prior to this patch, we would do a setgroups with an empty
list, effectively wiping secondary groups. We now verify which
secondary groups the user is member of and escalate the privileges
accordingly.

Change-Id: I33a10edd448b3ac5aa758a8d1d70e582cf421c7d
Closes-Bug: 1269473
2014-01-23 12:43:08 -05:00
Peter Portante
d6f0582c63 Add debug logger on per-test proxy-server
To help debug problems with the proxy-server app setup for each class,
we add a debug logger so that on failure we can see what requests were
actually sent.

Additionally, we move the set_http_connect() call to the same relative
position as the one before it for clarity of intent.

Related bug 1271962

Change-Id: Idc301c06e114b11c358ee4fbc0b2b70ec7743091
2014-01-23 10:06:10 -05:00
gholt
69d331d0d6 Container Sync: Simple HTTP Proxy load balancing
Change-Id: I021b043b927153bacff48cae648d4d8c5bbad765
2014-01-23 01:16:05 +00:00
gholt
b45e9a97ee Skip delete_at_update for replication requests
Requests through the object server that are from backend replication
should not send x-delete-at updates to the .expiring_objects account.
Replication is just moving data around or making new replicas, not
creating new data from nothing.

Change-Id: I324864face3ff559822c7a50c50e675e8b889b48
2014-01-23 01:05:16 +00:00
Jenkins
029790e8eb Merge "Adjust unit tests to clarify existing device usage" 2014-01-22 02:05:47 +00:00
Jenkins
7fead4161e Merge "Make test_close_error run much faster" 2014-01-21 22:00:54 +00:00
Jenkins
3d93dcb540 Merge "Make sure SLO's bulk-delete logger is configured" 2014-01-21 19:31:40 +00:00
Peter Portante
ca87827db9 Use a tempfile.mkdtemp() based temporary directory
Change-Id: Ie0e8137615348b130b67323d0d2913dc5ebfd5fb
2014-01-19 16:29:45 -05:00
Jenkins
8eabb13b8f Merge "Mute hash_cleanup_listdir common failures" 2014-01-19 20:56:02 +00:00
Samuel Merritt
a8ba751370 Make sure SLO's bulk-delete logger is configured
The SLO middleware supports deletion of an object and all its
segments; internally, it uses the same code as the bulk middleware to
do this (swift.common.middleware.bulk.Bulk).

If something goes wrong in the bulk delete, the Bulk object logs the
exception; however, SLO's Bulk object's logger always has the default
logging config, so if you've got a non-default logging setup
(e.g. syslog to a remote log host), this one exception message goes
somewhere else (e.g. local syslog).

This patch makes SLO and its bulk deleter share a logger so all the
messages go to the same place.

Change-Id: Idb01b5640257a02dbb9c698851a14ad8fba11a2d
2014-01-19 09:57:12 -05:00
Jenkins
0018f19524 Merge "Drop origin from get_info calls" 2014-01-19 07:44:34 +00:00
Jenkins
fc131c8b0e Merge "Mute hash_suffix rmdir failures" 2014-01-19 07:44:32 +00:00
Jenkins
f3382a2aa4 Merge "Remove an useless dependency to swiftclient" 2014-01-18 14:34:57 +00:00
David Goetz
04b1af215e Drop origin from get_info calls
The reason for this is that having origin in the get_info calls causes an
infinite loop. The way that code was written it relies on GETorHEAD_base to
populate the data- the only problem is that the HEAD call is wrapped by
cors_validation which calls get_info and round and round we go. imo get_info
should be refactored to not work this way (relying on this other call to do
stuff behind scenes and then magically your stuff is there) because it seems
pretty prone to breaking. But I'll let somebody else do that :).

Fixes bug 1270039

Change-Id: Idad3cedd965e0d5fb068b062fe8fef301c87b75a
2014-01-17 07:42:48 -08:00
Jenkins
ddb937cfc5 Merge "Consolidating and standardizing x-delete-at format" 2014-01-17 11:45:12 +00:00
Fabien Boucher
b6da40899c Remove an useless dependency to swiftclient
Unit tests in test/unit only have one dependency
to swiftclient in test_direct_client.py. This one
can be easily avoided and this patch removes it.

Change-Id: Ic1c78bc7f7fe426e8f7d8209a783342a0c4f071f
2014-01-16 14:07:48 +01:00
gholt
39b822873f ismount should not raise exceptions
The changes from using os.path.ismount to using
swift.common.utils.ismount has caused problems since the new one
raises exceptions in cases where the old one did not. Daemons have
been encountering this and exiting; servers have been 500ing instead
of 507ing in this case, changing handoff behaviors, etc.

Since the new one was specifically written and tested for this new
behavior, I left that original function as ismount_raw and made
ismount do what it did before.

If there really isn't some reason for this new behavior, I'll be glad
to get rid of ismount_raw and just keep ismount. I couldn't see any
reason for the new behavior myself.

Change-Id: I2b5b17f9ed9656cd8804a5ed568170697d0b183d
2014-01-16 08:01:10 +00:00