835 Commits

Author SHA1 Message Date
Samuel Merritt
d42a78a3aa Basic ring builder validation.
This prevents people from creating bogus ring builder files.

Example: "swift-ring-builder object.builder create 33 0.9 -4".

Fixes bug 924577.

Change-Id: I7bfc04f7fa5f55f70a4eaae96c414f6b2872e283
2013-03-18 09:50:45 -07:00
Jenkins
94defcbc53 Merge "Add tests and doc entry for request.environ[reseller_request]" 2013-03-16 19:29:47 +00:00
Jenkins
b115356af6 Merge "Remove check for valid Origin for the "actual request"." 2013-03-14 11:29:17 +00:00
David Goetz
f6e29b81d0 Remove check for valid Origin for the "actual request".
The only place in the spec that I could see the Origin being checked was
during the pre-flight OPTIONS request. If it gets to the actual request
let auth decide. Please correct me if this is wrong.

Change-Id: Ic31b71746ec056091c7778ebff3db7becc32bd9c
2013-03-13 13:43:19 -07:00
Samuel Merritt
ebcd60f7d9 Add a region tier to Swift's ring.
The region is one level above the zone; it is intended to represent a
chunk of machines that is distant from others with respect to
bandwidth and latency.

Old rings will default to having all their devices in region 1. Since
everything is in the same region by default, the ring builder will
simply distribute across zones as it did before, so your partition
assignment won't move because of this change. If you start adding
devices in other regions, of course, the assignment will change to
take that into account.

swift-ring-builder still accepts the same syntax as before, but will
default added devices to region 1 if no region is specified.

Examples:

$ swift-ring-builder foo.builder add r2z1-1.2.3.4:555/sda

$ swift-ring-builder foo.builder add r1z3-1.2.3.4:555/sda

$ swift-ring-builder foo.builder add z3-1.2.3.4:555/sda

Also, some updates to ring-overview doc.

Change-Id: Ifefbb839cdcf033e6c9201fadca95224c7303a29
2013-03-13 10:00:58 -07:00
gholt
f6d1fa1c15 Fixed bug with account_info
There was a bug where account_info wasn't converting the
container_count value to an int. Causes max container count cap to
get hit pretty quick since '0' > 0.

Change-Id: Ibfc6eebbff5a00aaebb47e9731dd053b60e3caa4
2013-03-12 19:15:35 +00:00
Christian Schwede
157c3c91ee Add tests and doc entry for request.environ[reseller_request]
The recent account_quotas (https://review.openstack.org/23434)
patch added a new setting request.environ[reseller_request].
This patch adds tests for tempauth and keystoneauth as well as
an updated overview_auth.rst.

Change-Id: Icdb7ec9948ae7424b0721fc51a143782b2fdc5a6
2013-03-10 19:21:13 +01:00
Christian Schwede
28c75db0e7 Account quotas
Add a new middleware implementing account quotas.

This middleware blocks write requests (PUT, POST) if a given quota (in bytes)
is exceeded while DELETE requests are still allowed.

Quotas are stored in the x-account-meta-quota-bytes metadata entry.
Write requests to this metadata setting are only allowed for resellers.

Change-Id: I57fd7c6209f34cc79d4bab72d500d43ba2a62083
2013-03-08 14:31:35 +01:00
Jenkins
48380c501a Merge "simplify the chexor function" 2013-03-07 22:20:40 +00:00
Michael Barton
0219b08b46 simplify the chexor function
Replace all that map(operator) nonsense.

It changes the error raised on invalid hashes, but we don't handle that
anywhere, and it shouldn't ever happen in real life.

Change-Id: Ib8cb549fac05e0b2725b4ea295326ac0c5e1f035
2013-03-07 01:34:46 -08:00
Sergey Lukjanov
7d5095c122 Support listing endpoints for an object.
Implements blueprint list-endpoints.

DocImpact: new middleware list_endpoints.

Change-Id: I0c4911ff726abd4cb8ce2b6245c99786ad46b410
2013-03-07 01:38:21 +04:00
Samuel Merritt
27dcaf2636 Spread handoffs out better around zones.
Before, you'd get your 3* primary nodes in 3 different zones, and then
get_more_nodes would give you everything it could from a non-primary
zone, and then finish up with stuff from the primary zones. It would
sort of look like this:

P: device in a primary node's zone
N: device not in a primary node's zone

PPPNNNNNNNNNNNNNNNNNNN...NNNNNNNNNPPP...PPPPPP

(The first three Ps are the primary nodes; they don't actually come
out of get_more_nodes(), but they're included for clarity.)

Now, the first few handoffs from get_more_nodes are in non-primary
zones, but only one per zone, and then the rest of the handoffs ignore
zones. It's still sampling the ring, so it's still taking weights into
consideration, but the zone distribution is more even early in the
handoff chain. It looks like this, assuming 10 zones:

P: device in a primary node's zone
N: device not in a primary node's zone
D: zone doesn't matter

PPPNNNNNNNDDDDDDDDDDD...DDD

* or whatever your replica count is

Change-Id: I31d2a2bc2cd6038386a2df85cd4fa37ccf2f650e
2013-03-05 13:28:12 -08:00
Jenkins
7117b744cc Merge "Static Large Object Support" 2013-03-05 04:09:47 +00:00
gholt
e064ba1915 Updated get_more_nodes algorithm
The handoff nodes will try to be in zones other than the primary
zones, will take into account the device weights, and will usually
keep the same sequences of handoffs even with ring changes.

On a real ring test the old get_more_nodes placed data mostly evenly
across zones, which is a problem for differently weighted zones. But
the real problem was that the extra partitions given to each device
was 0% to 0.77% with only 46.05% of the candidate devices getting
anything. Some of the devices increased in effective weight over 50%
in the test.

The new get_more_nodes placed closer to what the zone weights were
and the extra partitions given to each device was 0% to 0.24% with
90.58% of the candidate devices getting something. The worst off
device only increased in effective weight by 10.71%.

Change-Id: Iffb133a22db69074acaa2b90854cbfa92e4c2b9e
2013-03-04 08:52:24 +00:00
Jenkins
457ff9672d Merge "Make rings' replica counts adjustable." 2013-03-04 06:36:48 +00:00
Jenkins
9808a8744b Merge "TempURL filename options; bug fixes" 2013-03-02 05:20:57 +00:00
David Goetz
5d73da158b Static Large Object Support
DocImpact

Change-Id: I7edaa5e44208ab451f7f7566b64bb571b8eea1f9
2013-03-01 16:46:10 -08:00
Jenkins
cc63f8d791 Merge "Fixed formpost QUERY_STRING bugs." 2013-03-01 21:27:58 +00:00
gholt
c4c66d81bd TempURL filename options; bug fixes
- Prior to this commit, a Content-Disposition header was always set
  on responses to GET requests, with the filename based on the object
  name. Now, the header will only be set for 2xx responses and the
  filename can be overridden with a filename query parameter on the
  request.

- Fixed a bug where all query parameters on the request were being
  passed down the WSGI pipeline. Now, just the query parameters
  useful in log-based debugging are included. This becomes important
  with things like the Bulk middleware that act upon query
  parameters.

- Fixed bug where the Content-Disposition header wasn't following RFC
  spec.

DocImpact

Change-Id: I66ad809321dcdd03444324973c8b76869e3b0c8e
2013-03-01 15:47:20 +00:00
Jenkins
9e006183f8 Merge "make swift fsync" 2013-02-28 20:59:12 +00:00
Jenkins
b6b5d6670d Merge "Allow acl with a valid token." 2013-02-28 04:51:55 +00:00
Jenkins
61f4d077ae Merge "Add tests on keystone for invalid path requests." 2013-02-28 03:22:39 +00:00
Michael Barton
8bc065ec78 make swift fsync
Swift never fsyncs, it only fdatasyncs.  That is dumb, we have important
metadata we need to save.  Also, the code was weird and had no tests.

Change-Id: I6ec875c14560820b686266a28043a2b7631781e9
2013-02-27 17:43:19 -08:00
Jenkins
569bd1e4f6 Merge "Force log entries to be one line" 2013-02-27 22:45:48 +00:00
gholt
86220ba028 Force log entries to be one line
Different versions of syslog-ng and probably other syslog services
handle multi line log messages differently and sometimes quite
poorly. This patch collapses multi line log messages into single
lines before sending them on to syslog.

It's just a copy of what was already in Python's logging.Formatter
but altered to replace the newlines with #012. I used #012 since
that's a convention we've already used elsewhere in Swift.

Change-Id: I8d0509b7cf48e45c2cf6480b51c67eec5bc94fe2
2013-02-27 06:38:58 +00:00
Jenkins
1dc38b4672 Merge "timing-based affinity sorting for primary replicas" 2013-02-27 01:30:15 +00:00
Jenkins
b63620c442 Merge "Correct docstring for swift.common.ring.utils.build_tier_tree and add unit test for it." 2013-02-27 01:30:06 +00:00
Jenkins
ac4dd5608e Merge "Account Server: Refactor HEAD request handler" 2013-02-27 00:18:23 +00:00
Jenkins
5e1139b33c Merge "Account and container info fixes and improvement." 2013-02-27 00:17:35 +00:00
Dae S. Kim
89ab090434 Account Server: Refactor HEAD request handler
Deleted unused container checks. As method
swift::common::db::AccountBroker::get_container_timestamp becomes
unused, it is deleted too, along with the corresponding tests.

Change-Id: I61de4549b0abd7103226d6a13f1d9844abaa92d3
2013-02-26 21:42:06 +01:00
Kun Huang
d9130d79e5 Correct docstring for swift.common.ring.utils.build_tier_tree and add
unit test for it.

Some mistakes is in original docstring of that method. There's no unit
test for two methods in swift.common.ring.utils.

Fixes: bug #1070621

Change-Id: I6f4f211ea67d7fb8ccfe659f30bb0f5d394aca6b
2013-02-25 23:08:55 +08:00
Jenkins
d1381ba86b Merge "Adds first-byte latency timings for GET requests." 2013-02-23 01:10:58 +00:00
Samuel Merritt
7548cb9c47 Make rings' replica counts adjustable.
Example:

$ swift-ring-builder account.builder set_replicas 4
$ swift-ring-builder rebalance

This is a prerequisite for supporting globally-distributed clusters,
as operators of such clusters will probably want at least as many
replicas as they have regions. Therefore, adding a region requires
adding a replica. Similarly, removing a region lets an operator remove
a replica and save some money on disks.

In order to not hose clusters with lots of data, swift-ring-builder
now allows for setting of fractional replicas. Thus, one can gradually
increase the replica count at a rate that does not adversely affect
cluster performance.

Example:

$ swift-ring-builder object.builder set_replicas 3.01
$ swift-ring-builder object.builder rebalance
<distribute rings and wait>

$ swift-ring-builder object.builder set_replicas 3.02
$ swift-ring-builder object.builder rebalance
<distribute rings and wait>...

Obviously, fractional replicas are nonsensical for a single
partition. A fractional replica count is for the whole ring, not for
any individual partition, and indicates the average number of replicas
of each partition. For example, a replica count of 3.2 means that 20%
of partitions have 4 replicas and 80% have 3 replicas.

Changes do not take effect until after the ring is rebalanced. Thus,
if you mean to go from 3 replicas to 3.01 but you accidentally type
2.01, no data is lost.

Additionally, 'swift-ring-builder X.builder create' can now take a
decimal argument for the number of replicas.

DocImpact

Change-Id: I12b34dacf60350a297a46be493d5d171580243ff
2013-02-22 15:03:10 -08:00
Chmouel Boudjnah
a7e31eb5dd Add tests on keystone for invalid path requests.
- Add missing tests to make it coverage to 100%.

Change-Id: Ic182ad379e2c48cc4151b2532c509bd62d44bd00
2013-02-21 14:05:25 +01:00
Jenkins
44e3915564 Merge "Change in swift-drive-audit handling log rotation." 2013-02-21 02:41:37 +00:00
Chmouel Boudjnah
0f284e04e4 Allow acl with a valid token.
- When a user as a valid token it would go to authorize but the acl
  check was after the reseller_check and due fail. Check this before
  reseller_check and add a test for it.
- Fixes bug 1020722.

Change-Id: Iaff9f35f5ee690e9b729c36d05fb9adf3368dc79
2013-02-20 18:08:58 +01:00
gholt
ffada71137 Fixed formpost QUERY_STRING bugs.
Ensures that any QUERY_STRING to FormPost is not passed onward.
Handles a redirect with a query string properly.

Change-Id: If0a7d9b0a17314c6cd3852175362d4633f828d81
2013-02-19 19:55:50 +00:00
John Dickinson
69917347cf timing-based affinity sorting for primary replicas
This changes the way primary replicas can be sorted on GET requests.
Previously, replicas were shuffled. Now, if configured, the replicas
are sorted based on the most recent connection time data to that node.
This patch adds a config value that changes the sorting method.

get_more_nodes() (ie handoffs) is unaffected by this patch because
sorting by affinity would break the durability provided by the current
as-unique-as-possible handoff selection.

Timing data is collected for each node each time the proxy makes a
connection to that node (IP address). If timing data for a node doesn't
exist, then it is assumed at -1 (ie will sort earlier) so that timing
data can be collected for that node.

Change-Id: I837fa21c3a566b10cce33eb75788665e1d01cd8a
2013-02-19 10:25:25 -08:00
Chmouel Boudjnah
b62299376a Account and container info fixes and improvement.
- Fixes bug 1119282.
- Allow middleware accessing metadata of an account without having to
  store it separately in a new memcache namespace.
- Add tests for get_container_info that was previously missed.
- Add get_account_info method based on get_container_info, a function
  for other middleware to query accounts.
- Rename container_info['count'] as container_info['object_count'].

Change-Id: I43787916c7a812cb08d278edf45370521f12c912
2013-02-16 23:32:27 +01:00
Chuck Thier
e88ff34685 Cleanup of file permissions
Mostly removed execute on non-executable files

Change-Id: Ibfbe7e0cf0fbeabef602d70b20f75e1dd3bdf9c9
2013-02-15 11:06:11 -06:00
Jenkins
b66232a997 Merge "Swift MemcacheRing (set) interface is incompatible fixes" 2013-02-14 20:30:22 +00:00
Darrell Bishop
bce8443c9e Adds first-byte latency timings for GET requests.
This was an oustanding TODO for StatsD Swift metrics.  A new timing
metric is tracked for (only) GET requests for accounts, containers,
and objects:

  proxy-server.<req_type>.GET.<status_int>.first-byte.timing

Also updated StatsD documentation in the Admin Guide to clarify that
timing metrics are sent in units of milliseconds.

Change-Id: I5bb781c06cefcb5280f4fb1112a526c029fe0c20
2013-02-13 15:58:57 -08:00
David Goetz
3b341e375f fix large objects with unicode
Change-Id: I2a088e6cfe354d18cb2107a42bc626c2e7822e42
2013-02-13 12:05:25 -08:00
Tong Li
2267b030bf Swift MemcacheRing (set) interface is incompatible fixes
This patch fixes the Swift MemcacheRing set and set_multi
interface incompatible problem with python memcache. The fix
added two extra named parameters to both set and set_multi
method. When only time or timeout parameter is present, then one
of the value will be used. When both time and timeout are present,
the time parameter will be used.

Named parameter min_compress_len is added for pure compatibility
purposes. The current implementation ignores this parameter.

To make swift memcached methods all consistent cross the board,
method incr and decr have also been changed to include a new
named parameter time.

In future OpenStack releases, the named parameter timeout will be
removed, keep the named parameter timeout around for now is
to make sure that mismatched releases between client and server
will still work.

From now on, when a call is made to set, set_multi, decr, incr
by using timeout parametner, a warning message will be logged to
indicate the deprecation of the parameter.

Fixes: bug #1095730
Change-Id: I07af784a54d7d79395fc3265e74145f92f38a893
2013-02-13 13:56:04 -05:00
Jenkins
23f33b2069 Merge "Make statsd sample rate behave better." 2013-02-13 08:19:46 +00:00
Joe Gordon
45f0502b52 Fix spelling mistakes
git ls-files | misspellings -f -
Source: https://github.com/lyda/misspell-check

Change-Id: I4132e6a276e44e2a8985238358533d315ee8d9c4
2013-02-12 16:39:40 -08:00
jola-mirecka
902b66d3ae Change in swift-drive-audit handling log rotation.
Change supports kern.log rotation in order to avoid loss
of significant information.

There is a year change functionality added as kern.log
does not keep record of year.

There is also backwards function added which allows
reading logs from the back to the front, speeding up the
execution along with the unit test for it

Fixes Bug 1080682

Change-Id: I93436c405aff5625396514000cab774b66022dd0
2013-02-12 13:34:49 +00:00
Jenkins
3df9229dae Merge "Use a doubled proxy-logging instead of each middleware handling it differently (if at all)" 2013-02-12 03:47:13 +00:00
Jenkins
89d4a51359 Merge "Made internal client handle failed requests better" 2013-02-11 19:47:02 +00:00
David Goetz
a622349eda Use a doubled proxy-logging instead of each middleware handling it
differently (if at all)

Adding a swift.source to wsgi pre_auth funcs and all middleware that makes
subrequests to proxy server.

NOTE: This change will result in a change in the number of proxy logs made for
staticweb, formpost, tempurl, and any other middleware that performs sub
requests (including swauth and SOS).

Please see docs for details.

DocImpact

Change-Id: I80cf2806add1c3d34054147e2515944be340455b
2013-02-11 09:22:25 -08:00