2460 Commits

Author SHA1 Message Date
Marcelo Martins
1126e59c12 Adding a new optional variable called trans_id_suffix
The trans_id_suffix (default is empty) would be appended to the swift transaction
id allowing one to easily figure out from which cluster that X-Trans-Id
belongs to. This is very useful when one is managing more than one swift
cluster. Also updated sample and manpage to reflect the changes.

Change-Id: Icdf63643e9c1bde36a9ef5e3f41ee9fb20e55f5d
2013-04-10 06:37:32 -05:00
Jenkins
b115356af6 Merge "Remove check for valid Origin for the "actual request"." 2013-03-14 11:29:17 +00:00
Jenkins
8e58c9c2a9 Merge "Add missing CONTRIBUTING.md to source tarball" 2013-03-13 22:53:39 +00:00
Jenkins
fd3a8a6650 Merge "Fix swift-ring-builder's list_parts command." 2013-03-13 22:53:34 +00:00
David Goetz
f6e29b81d0 Remove check for valid Origin for the "actual request".
The only place in the spec that I could see the Origin being checked was
during the pre-flight OPTIONS request. If it gets to the actual request
let auth decide. Please correct me if this is wrong.

Change-Id: Ic31b71746ec056091c7778ebff3db7becc32bd9c
2013-03-13 13:43:19 -07:00
Samuel Merritt
35acd5912f Fix swift-ring-builder's list_parts command.
I broke it in 7548cb9 when a ring's replica count changed from an int
to a float.

Change-Id: I49fa7e2961a09daad0c96e19cf1e39fec248d998
2013-03-13 13:34:19 -07:00
Samuel Merritt
ebcd60f7d9 Add a region tier to Swift's ring.
The region is one level above the zone; it is intended to represent a
chunk of machines that is distant from others with respect to
bandwidth and latency.

Old rings will default to having all their devices in region 1. Since
everything is in the same region by default, the ring builder will
simply distribute across zones as it did before, so your partition
assignment won't move because of this change. If you start adding
devices in other regions, of course, the assignment will change to
take that into account.

swift-ring-builder still accepts the same syntax as before, but will
default added devices to region 1 if no region is specified.

Examples:

$ swift-ring-builder foo.builder add r2z1-1.2.3.4:555/sda

$ swift-ring-builder foo.builder add r1z3-1.2.3.4:555/sda

$ swift-ring-builder foo.builder add z3-1.2.3.4:555/sda

Also, some updates to ring-overview doc.

Change-Id: Ifefbb839cdcf033e6c9201fadca95224c7303a29
2013-03-13 10:00:58 -07:00
Thierry Carrez
b3732787ea Add missing CONTRIBUTING.md to source tarball
Fixes MANIFEST.in to include missing CONTRIBUTING.md into source
code tarballs.

Change-Id: I81876adad14e0f1b2211d39ec6471e35f5464120
2013-03-13 11:35:42 +01:00
gholt
f6d1fa1c15 Fixed bug with account_info
There was a bug where account_info wasn't converting the
container_count value to an int. Causes max container count cap to
get hit pretty quick since '0' > 0.

Change-Id: Ibfc6eebbff5a00aaebb47e9731dd053b60e3caa4
2013-03-12 19:15:35 +00:00
jola-mirecka
5e427e5e3b Add exception to swift-drive-audit if no date in line.
Sometimes there is no date at the beginning of a line in kern.log.
Although it does not happen often, there should be a check ensuring
the program doesn't crash in case it happens.

Added try-except block surrounding parsing string to date format.

Change-Id: I44a101266582eea2199189a006afa1037a9bd4ea
Fixes: bug #1152658
2013-03-08 16:38:11 +00:00
Jenkins
a8af3835c0 Merge "Account quotas" 2013-03-08 16:37:04 +00:00
Christian Schwede
28c75db0e7 Account quotas
Add a new middleware implementing account quotas.

This middleware blocks write requests (PUT, POST) if a given quota (in bytes)
is exceeded while DELETE requests are still allowed.

Quotas are stored in the x-account-meta-quota-bytes metadata entry.
Write requests to this metadata setting are only allowed for resellers.

Change-Id: I57fd7c6209f34cc79d4bab72d500d43ba2a62083
2013-03-08 14:31:35 +01:00
Samuel Merritt
d9fb84d209 Fix some unreadable code.
I thought it was readable back when I wrote it; turns out it's
not. Oops.

Change-Id: I1e1020935356522d6e07409aa867a5ffc8919787
Bonus: remove unused import.
2013-03-07 14:44:03 -08:00
Jenkins
48380c501a Merge "simplify the chexor function" 2013-03-07 22:20:40 +00:00
Jenkins
09f1387af5 Merge "Support tests for Apache" 2013-03-07 13:33:48 +00:00
Michael Barton
0219b08b46 simplify the chexor function
Replace all that map(operator) nonsense.

It changes the error raised on invalid hashes, but we don't handle that
anywhere, and it shouldn't ever happen in real life.

Change-Id: Ib8cb549fac05e0b2725b4ea295326ac0c5e1f035
2013-03-07 01:34:46 -08:00
David Hadas
40782ed20c Support tests for Apache
Add support for functional tests that work with Apache web front end

Change-Id: I72358a12016eeccc842d834461dbebaa188aa117
Implements: blueprint wsgi-application-interface
2013-03-07 01:32:16 +02:00
Sergey Lukjanov
7d5095c122 Support listing endpoints for an object.
Implements blueprint list-endpoints.

DocImpact: new middleware list_endpoints.

Change-Id: I0c4911ff726abd4cb8ce2b6245c99786ad46b410
2013-03-07 01:38:21 +04:00
Jenkins
34beb92edb Merge "Spread handoffs out better around zones." 2013-03-06 20:11:05 +00:00
Samuel Merritt
71a20d04cc Fix misspelled variable name
Change-Id: Ied6ed3cad16e9797df73a05f3df3ac9cc64299e0
2013-03-05 18:15:09 -08:00
Jenkins
28ac46ded3 Merge "Added per disk PUT timing monitoring support." 2013-03-06 01:59:23 +00:00
Samuel Merritt
27dcaf2636 Spread handoffs out better around zones.
Before, you'd get your 3* primary nodes in 3 different zones, and then
get_more_nodes would give you everything it could from a non-primary
zone, and then finish up with stuff from the primary zones. It would
sort of look like this:

P: device in a primary node's zone
N: device not in a primary node's zone

PPPNNNNNNNNNNNNNNNNNNN...NNNNNNNNNPPP...PPPPPP

(The first three Ps are the primary nodes; they don't actually come
out of get_more_nodes(), but they're included for clarity.)

Now, the first few handoffs from get_more_nodes are in non-primary
zones, but only one per zone, and then the rest of the handoffs ignore
zones. It's still sampling the ring, so it's still taking weights into
consideration, but the zone distribution is more even early in the
handoff chain. It looks like this, assuming 10 zones:

P: device in a primary node's zone
N: device not in a primary node's zone
D: zone doesn't matter

PPPNNNNNNNDDDDDDDDDDD...DDD

* or whatever your replica count is

Change-Id: I31d2a2bc2cd6038386a2df85cd4fa37ccf2f650e
2013-03-05 13:28:12 -08:00
tong li
7fd4323a69 David Hadas email address has been changed.
Change-Id: Ic3e9ad50f123eecb4abbe91340b0acff4a2d580c
2013-03-05 11:59:14 -05:00
Jenkins
7117b744cc Merge "Static Large Object Support" 2013-03-05 04:09:47 +00:00
gholt
e064ba1915 Updated get_more_nodes algorithm
The handoff nodes will try to be in zones other than the primary
zones, will take into account the device weights, and will usually
keep the same sequences of handoffs even with ring changes.

On a real ring test the old get_more_nodes placed data mostly evenly
across zones, which is a problem for differently weighted zones. But
the real problem was that the extra partitions given to each device
was 0% to 0.77% with only 46.05% of the candidate devices getting
anything. Some of the devices increased in effective weight over 50%
in the test.

The new get_more_nodes placed closer to what the zone weights were
and the extra partitions given to each device was 0% to 0.24% with
90.58% of the candidate devices getting something. The worst off
device only increased in effective weight by 10.71%.

Change-Id: Iffb133a22db69074acaa2b90854cbfa92e4c2b9e
2013-03-04 08:52:24 +00:00
Jenkins
457ff9672d Merge "Make rings' replica counts adjustable." 2013-03-04 06:36:48 +00:00
Jenkins
84bd434d8d Merge "Remove reduant check of list." 2013-03-02 18:46:57 +00:00
Jenkins
9808a8744b Merge "TempURL filename options; bug fixes" 2013-03-02 05:20:57 +00:00
David Goetz
5d73da158b Static Large Object Support
DocImpact

Change-Id: I7edaa5e44208ab451f7f7566b64bb571b8eea1f9
2013-03-01 16:46:10 -08:00
Jenkins
cc63f8d791 Merge "Fixed formpost QUERY_STRING bugs." 2013-03-01 21:27:58 +00:00
gholt
c4c66d81bd TempURL filename options; bug fixes
- Prior to this commit, a Content-Disposition header was always set
  on responses to GET requests, with the filename based on the object
  name. Now, the header will only be set for 2xx responses and the
  filename can be overridden with a filename query parameter on the
  request.

- Fixed a bug where all query parameters on the request were being
  passed down the WSGI pipeline. Now, just the query parameters
  useful in log-based debugging are included. This becomes important
  with things like the Bulk middleware that act upon query
  parameters.

- Fixed bug where the Content-Disposition header wasn't following RFC
  spec.

DocImpact

Change-Id: I66ad809321dcdd03444324973c8b76869e3b0c8e
2013-03-01 15:47:20 +00:00
ywang19
b715eb5d18 Remove reduant check of list.
Remove len() check and parenthesis.

Fixes Bug #1136893

Change-Id: Ib47ec4890c2f6a50e316a7fef204ef818c6c4d6e
2013-03-01 13:22:35 +08:00
Jenkins
9e006183f8 Merge "make swift fsync" 2013-02-28 20:59:12 +00:00
Jian Zhang
1d8a02f25c Added per disk PUT timing monitoring support.
Fixes bug 1104708

There could be severe performance drop for swift is one disk of one
storage node is problematic due to the tragic state of async disk I/O.

This patch provided PUT timing per kB transfered (ms/kB) monitoring
support for each non-zero-byte request of each disk and report to
statsD for alert.
-adding "object-server.PUT.<device>.timing" metrics for object-server.

DocImpact.

Change-Id: Ie94bddad28e8be52e71683bf6c9db988664abe47
2013-02-28 02:52:06 -08:00
Jenkins
b6b5d6670d Merge "Allow acl with a valid token." 2013-02-28 04:51:55 +00:00
Jenkins
61f4d077ae Merge "Add tests on keystone for invalid path requests." 2013-02-28 03:22:39 +00:00
Michael Barton
8bc065ec78 make swift fsync
Swift never fsyncs, it only fdatasyncs.  That is dumb, we have important
metadata we need to save.  Also, the code was weird and had no tests.

Change-Id: I6ec875c14560820b686266a28043a2b7631781e9
2013-02-27 17:43:19 -08:00
Jenkins
569bd1e4f6 Merge "Force log entries to be one line" 2013-02-27 22:45:48 +00:00
Chuck Thier
5d2b70b624 Updating classifiers status to stable and clean up
Change-Id: I72943a9575b873cef1bf193f197b390b544ed6bf
2013-02-27 15:08:46 -06:00
David Goetz
7e3adf5688 Adding a quick note to helper functions so its clear what they do and
they don't get misused.

Change-Id: Ie0292df9d9e565ccd608b6b61ead57dfff1d3797
2013-02-27 09:36:08 -08:00
Jenkins
2a119b650e Merge "Move the swift.1 manpage to python-swiftclient" 2013-02-27 16:50:17 +00:00
Jenkins
a06c71c624 Merge "Add cache=swift.cache for authtoken example." 2013-02-27 16:10:57 +00:00
gholt
86220ba028 Force log entries to be one line
Different versions of syslog-ng and probably other syslog services
handle multi line log messages differently and sometimes quite
poorly. This patch collapses multi line log messages into single
lines before sending them on to syslog.

It's just a copy of what was already in Python's logging.Formatter
but altered to replace the newlines with #012. I used #012 since
that's a convention we've already used elsewhere in Swift.

Change-Id: I8d0509b7cf48e45c2cf6480b51c67eec5bc94fe2
2013-02-27 06:38:58 +00:00
Jenkins
d3232d4fc5 Merge "Add debug level logs for TempAuth.authorize." 2013-02-27 02:05:36 +00:00
Jenkins
1dc38b4672 Merge "timing-based affinity sorting for primary replicas" 2013-02-27 01:30:15 +00:00
Jenkins
b63620c442 Merge "Correct docstring for swift.common.ring.utils.build_tier_tree and add unit test for it." 2013-02-27 01:30:06 +00:00
Jenkins
ca7468fd1b Merge "Improve bin/swift-account-audit" 2013-02-27 00:39:45 +00:00
Jenkins
ac4dd5608e Merge "Account Server: Refactor HEAD request handler" 2013-02-27 00:18:23 +00:00
Jenkins
5e1139b33c Merge "Account and container info fixes and improvement." 2013-02-27 00:17:35 +00:00
Dae S. Kim
89ab090434 Account Server: Refactor HEAD request handler
Deleted unused container checks. As method
swift::common::db::AccountBroker::get_container_timestamp becomes
unused, it is deleted too, along with the corresponding tests.

Change-Id: I61de4549b0abd7103226d6a13f1d9844abaa92d3
2013-02-26 21:42:06 +01:00