438 Commits

Author SHA1 Message Date
Jenkins
a186c53b24 Merge "fix slo docs" 2013-04-17 19:39:43 +00:00
Jenkins
dfcfb767d3 Merge "Add x-remove-versions-location for feature: disable versioning" 2013-04-17 17:57:53 +00:00
David Goetz
d2a4f75a95 fix slo docs
Change-Id: I49f8c272457547d8aae853986573549828d2503a
2013-04-17 07:03:46 -07:00
Jenkins
8ddfa5e533 Merge "Allow all headers requested for CORS." 2013-04-16 15:56:58 +00:00
Jenkins
c87576bb94 Merge "Refactored lists of nodes to contact for requests" 2013-04-15 23:05:47 +00:00
Jenkins
5140c0d5da Merge "Adding a new optional variable called trans_id_suffix" 2013-04-10 20:05:03 +00:00
Marcelo Martins
1126e59c12 Adding a new optional variable called trans_id_suffix
The trans_id_suffix (default is empty) would be appended to the swift transaction
id allowing one to easily figure out from which cluster that X-Trans-Id
belongs to. This is very useful when one is managing more than one swift
cluster. Also updated sample and manpage to reflect the changes.

Change-Id: Icdf63643e9c1bde36a9ef5e3f41ee9fb20e55f5d
2013-04-10 06:37:32 -05:00
gholt
d79a67ebf6 Refactored lists of nodes to contact for requests
Extensive refactor here to consolidate what nodes are contacted for
any request. This consolidation means reads will contact the same set
of nodes that writes would, giving a very good chance that
read-your-write behavior will succeed. This also means that writes
will not necessarily try all nodes in the cluster as it would
previously, which really wasn't desirable anyway. (If you really want
that, you can set request_node_count to a really big number, but
understand that also means reads will contact every node looking for
something that might not exist.)

* Added a request_node_count proxy-server conf value that allows
  control of how many nodes are contacted for a normal request.

In proxy.controllers.base.Controller:

* Got rid of error_increment since it was only used in one spot by
  another method and just served to confuse.

* Made error_occurred also log the device name.

* Made error_limit require an error message and also documented a bit
  better.

* Changed iter_nodes to just take a ring and a partition and yield
  all the nodes itself so it could control the number of nodes used
  in a given request. Also happens to consolidate where sort_nodes is
  called.

* Updated account_info and container_info to use all nodes from
  iter_nodes and to call error_occurred appropriately.

* Updated GETorHEAD_base to not track attempts on its own and just
  stop when iter_nodes tells it to stop. Also, it doesn't take the
  nodes to contact anymore; instead it takes the ring and gets the
  nodes from iter_nodes itself.

Elsewhere:

* Ring now has a get_part method.

* Made changes to reflect all of the above.

Change-Id: I37f76c99286b6456311abf25167cd0485bfcafac
2013-04-08 20:48:32 +00:00
Chmouel Boudjnah
c687f6956c Allow all headers requested for CORS.
- We allow all headers requested in preflight request. The CORS
  specification does leave the door open for this, as mentioned in
  http://www.w3.org/TR/cors/#resource-preflight-requests
  Note: Since the list of headers can be unbounded
  simply returning headers can be enough.
- This is a followup to review:
  https://review.openstack.org/#/c/24415/.
- Fixes bug 1155034.

Change-Id: If7b8f2f3a581c5209892d1ccc9f06ddb8fac92dd
2013-04-05 21:12:15 +01:00
tong li
f1aefcbc13 directory /var/cache/swift /srv/node should be created.
For multiple node swift cluster setup, these directories should be
created in the setup steps. Otherwise, when start up storage servers
there will be errors. The error messages won't tell if it is because
of missing the directory.

Change-Id: I67d74abb7743b76739b5e747d6dcbd3214b00774
Fixes: bug #1158519
2013-04-04 20:39:01 +00:00
Christian Schwede
e5cd78d902 Update documentation for account_quota middleware
Change-Id: I3dc7930ee2a1eb26b1f60e83fce2bc1bea0a8b0f
2013-04-03 15:10:42 +02:00
David Hadas
a979c8007b Add support for Hash Prefix
A new configuration parameter is added to /etc/swift/swift.conf
[swift-hash]
swift_hash_path_prefix = 'random unique string'

New installations are advised to set this parameter to a random secret,
which would not be disclosed ouside the organization.
The same secret needs to be used by all swift servers of the same cluster.

Existing installations should set this parameter to an empty string
(the default)

DocImpact

Fixes: Bug #1157454

Change-Id: I63b10d0b7d6dd3f74e0f10bb41b5f240fa03578a
2013-03-22 19:41:55 +02:00
tong li
c1dfe891e7 instruction command was not documented in right format.
Instructions for a Multiple Server Swift Installation (Ubuntu) doc
for creating /var/run/swift and changing owner command were not
using the right format. Missing two colons.

Change-Id: Ie23007a0da498373fbfb137c7edb3d80813c6ba5
Fixes: bug #1158310
2013-03-21 09:45:13 -04:00
Jenkins
85b7346808 Merge "Improved container-sync resiliency" 2013-03-19 07:20:57 +00:00
Jenkins
591a3a599d Merge "Small clarification to swift-recon section of admin guide." 2013-03-18 23:29:15 +00:00
tong li
68355d7ea5 Instruction for multi node setup misses /var/run/swift creation
Multiple Server Swift Installation (Ubuntu) instruction does not
indicate that the directory /var/run/swift needs to be created.
That directory actually needs to be created and the ownship needs
to be changed to the user/group which swift service runs under.

This patch will fix the document and gives the steps how to create
the directory and set the ownership right. It also gives instruction
on how the script can be added so that swift services can be resarted
after system reboots.

Change-Id: Id61aa67cc0d6f66d749701e6ea824b1ff3b2c478
Fixes: bug #1156631
2013-03-18 13:57:40 -04:00
Samuel Merritt
d88d12b120 Small clarification to swift-recon section of admin guide.
Apparently the use of port 6030 was causing some confusion.

Fixes bug 1095474.

Change-Id: I0cc71f4733ad91694e015a9b75c3eda080aca6fb
2013-03-17 15:58:06 -07:00
Kun Huang
5c3f75ba6a Add x-remove-versions-location for feature: disable versioning
We can set x-versions-location empty to remove this header in API, but
to keep consistent, we should allow x-remove-versions-location too. The
usage is post "x-remove-versions-locaion: x", just like ACL remove
headers.

Fixed: bug #1107592

Change-Id: I1271eec6401d4a0e8c1a7c2d63aeb8dfef00bf6d
2013-03-17 07:30:00 +08:00
Jenkins
94defcbc53 Merge "Add tests and doc entry for request.environ[reseller_request]" 2013-03-16 19:29:47 +00:00
Samuel Merritt
ebcd60f7d9 Add a region tier to Swift's ring.
The region is one level above the zone; it is intended to represent a
chunk of machines that is distant from others with respect to
bandwidth and latency.

Old rings will default to having all their devices in region 1. Since
everything is in the same region by default, the ring builder will
simply distribute across zones as it did before, so your partition
assignment won't move because of this change. If you start adding
devices in other regions, of course, the assignment will change to
take that into account.

swift-ring-builder still accepts the same syntax as before, but will
default added devices to region 1 if no region is specified.

Examples:

$ swift-ring-builder foo.builder add r2z1-1.2.3.4:555/sda

$ swift-ring-builder foo.builder add r1z3-1.2.3.4:555/sda

$ swift-ring-builder foo.builder add z3-1.2.3.4:555/sda

Also, some updates to ring-overview doc.

Change-Id: Ifefbb839cdcf033e6c9201fadca95224c7303a29
2013-03-13 10:00:58 -07:00
Christian Schwede
157c3c91ee Add tests and doc entry for request.environ[reseller_request]
The recent account_quotas (https://review.openstack.org/23434)
patch added a new setting request.environ[reseller_request].
This patch adds tests for tempauth and keystoneauth as well as
an updated overview_auth.rst.

Change-Id: Icdb7ec9948ae7424b0721fc51a143782b2fdc5a6
2013-03-10 19:21:13 +01:00
David Hadas
8b140033f0 Improved container-sync resiliency
container-sync now skips faulty objects in the first and second rounds.
All replicas try in the second round.
No server will give up until the faulty object suceeds

Fixes: bug #1068423

Change-Id: I0defc174b2ce3796a6acf410a2d2eae138e8193d
2013-03-08 22:28:06 +02:00
Christian Schwede
28c75db0e7 Account quotas
Add a new middleware implementing account quotas.

This middleware blocks write requests (PUT, POST) if a given quota (in bytes)
is exceeded while DELETE requests are still allowed.

Quotas are stored in the x-account-meta-quota-bytes metadata entry.
Write requests to this metadata setting are only allowed for resellers.

Change-Id: I57fd7c6209f34cc79d4bab72d500d43ba2a62083
2013-03-08 14:31:35 +01:00
Jenkins
09f1387af5 Merge "Support tests for Apache" 2013-03-07 13:33:48 +00:00
David Hadas
40782ed20c Support tests for Apache
Add support for functional tests that work with Apache web front end

Change-Id: I72358a12016eeccc842d834461dbebaa188aa117
Implements: blueprint wsgi-application-interface
2013-03-07 01:32:16 +02:00
Sergey Lukjanov
7d5095c122 Support listing endpoints for an object.
Implements blueprint list-endpoints.

DocImpact: new middleware list_endpoints.

Change-Id: I0c4911ff726abd4cb8ce2b6245c99786ad46b410
2013-03-07 01:38:21 +04:00
Jenkins
28ac46ded3 Merge "Added per disk PUT timing monitoring support." 2013-03-06 01:59:23 +00:00
David Goetz
5d73da158b Static Large Object Support
DocImpact

Change-Id: I7edaa5e44208ab451f7f7566b64bb571b8eea1f9
2013-03-01 16:46:10 -08:00
Jian Zhang
1d8a02f25c Added per disk PUT timing monitoring support.
Fixes bug 1104708

There could be severe performance drop for swift is one disk of one
storage node is problematic due to the tragic state of async disk I/O.

This patch provided PUT timing per kB transfered (ms/kB) monitoring
support for each non-zero-byte request of each disk and report to
statsD for alert.
-adding "object-server.PUT.<device>.timing" metrics for object-server.

DocImpact.

Change-Id: Ie94bddad28e8be52e71683bf6c9db988664abe47
2013-02-28 02:52:06 -08:00
Jenkins
2a119b650e Merge "Move the swift.1 manpage to python-swiftclient" 2013-02-27 16:50:17 +00:00
Jenkins
a06c71c624 Merge "Add cache=swift.cache for authtoken example." 2013-02-27 16:10:57 +00:00
Pete Zaitcev
9a3e0f41bd Move the swift.1 manpage to python-swiftclient
When someone split away python-swiftclient, he left behind the
client manpage.

Apparently all we need now is to delete it from swift and
add it to python-swiftclient. Except that of course the releases
have to be synchronized somehow or else files conflict.

Change-Id: I8b5f7b5557c28818048d8941df36473dacfb1d57
2013-02-26 12:41:27 -07:00
Jenkins
249a65461e Merge "Adding speed limit options for DB auditor" 2013-02-26 06:22:25 +00:00
Jenkins
d1381ba86b Merge "Adds first-byte latency timings for GET requests." 2013-02-23 01:10:58 +00:00
Chmouel Boudjnah
e69f3bef8f Add cache=swift.cache for authtoken example.
- Things swill go badly with swift if we leave the default to authtoken
  to use its own memcache cache connection based python-memcache c based
  binding.

Change-Id: I293b875acdcb06e5a7a0cfa9a9bb5d7678675da0
2013-02-21 22:58:27 +01:00
yuan-zhou
09370862ca Adding speed limit options for DB auditor
Fix bug 1129760

Without speed limit, DB auditor will likely consume high CPU% on
storage node. That will highly impact the cluster's performance.

This patch adds two options for account/container auditor:
 - containers_per_second: Maximum containers audited per second
 - accounts_per_second: Maximum accounts audited per second

DocImpact

Change-Id: I9faa506438185a83ca77db4906969328624d015f
2013-02-20 13:54:59 +08:00
Pete Zaitcev
93ea7c63b1 Documentation fixups
These are mostly cosmetic fixes for irritating imperfections:
- "separated with commas" was duplicated, leave just one
- extra whitespace here and there, man pages are not PEP8, drop
- weird extra commas, drop
- Fedora logs to /var/log/messages
- "drive is has failed", drop "is"

Change-Id: I5ceba2e61b16db4855d76c92cbc83663b9b2a0da
2013-02-18 10:54:27 -07:00
Chuck Thier
e88ff34685 Cleanup of file permissions
Mostly removed execute on non-executable files

Change-Id: Ibfbe7e0cf0fbeabef602d70b20f75e1dd3bdf9c9
2013-02-15 11:06:11 -06:00
Darrell Bishop
bce8443c9e Adds first-byte latency timings for GET requests.
This was an oustanding TODO for StatsD Swift metrics.  A new timing
metric is tracked for (only) GET requests for accounts, containers,
and objects:

  proxy-server.<req_type>.GET.<status_int>.first-byte.timing

Also updated StatsD documentation in the Admin Guide to clarify that
timing metrics are sent in units of milliseconds.

Change-Id: I5bb781c06cefcb5280f4fb1112a526c029fe0c20
2013-02-13 15:58:57 -08:00
Jenkins
23f33b2069 Merge "Make statsd sample rate behave better." 2013-02-13 08:19:46 +00:00
Joe Gordon
45f0502b52 Fix spelling mistakes
git ls-files | misspellings -f -
Source: https://github.com/lyda/misspell-check

Change-Id: I4132e6a276e44e2a8985238358533d315ee8d9c4
2013-02-12 16:39:40 -08:00
Jenkins
af3bd46679 Merge "Add generated keystone documentation to misc.rst." 2013-02-12 12:03:21 +00:00
Jenkins
e4e55588fb Merge "Replaces Copyright statements for LLC with Foundation, removes date." 2013-02-12 06:25:45 +00:00
annegentle
72428434f7 Replaces Copyright statements for LLC with Foundation, removes date.
Replaced GA code for cross-domain tracking.

Patchset addresses reviewer's comments
and follows new guidance from Foundation:
http://wiki.openstack.org/Documentation/Copyright

Adds current year to each Sphinx-built page.

Addresses only the docs copyright attribution, not code files.

Change-Id: Ib90fd1c92c8fafce2db821bc2b17cef1377cfc1e
2013-02-11 16:32:33 -06:00
Chmouel Boudjnah
1b86380e2e Add generated keystone documentation to misc.rst.
- Change a bit the formatting of the documention as well.
- Fix WARNING: Title underline too short. in misc.rst.

Change-Id: I2f4e36bcb5e01e984f0af0152bc5b3b9f7e942ce
2013-02-11 21:44:53 +01:00
David Hadas
83d89228a9 minor development doc changes
Reviewed development saio and guidelines while installing
a new dev and test node.

Change-Id: I59238a4d43fa08a89be14f43681ad4693f1261dd
2013-02-09 11:11:05 +02:00
Jenkins
c0e8ad609b Merge "Allow change the endpoint_type when use swift-dispersion tools" 2013-02-08 23:55:03 +00:00
Michael Barton
24ef12027c Basic container quotas
Add a new middleware implementing some basic container quotas.

Quotas are subject to several limitations: eventual consistency, the timeliness
of the cached container_info (60 second ttl by default), and it’s unable to
reject chunked transfer uploads that exceed the quota (though once the quota
is exceeded, new chunked transfers will be refused).

However, they get most of the way to container quotas fairly inexpensively.

Quotas are set by adding meta values to the container, and are validated when
set:

  X-Container-Meta-Quota-Bytes: Maximum size of the container, in bytes.
  X-Container-Meta-Quota-Count: Maximum object count of the container.

DocImpact

Change-Id: I77cfbf6dc231a2e522bd67328e4c082424a93eee
2013-02-05 06:03:38 -08:00
gholt
85529531d6 Remove tempauth allowed_sync_hosts conf option
Seems we missed these references when committing
357b12dc2ba7b19c66196a573ccb2489d2104b93

DocImpact

Change-Id: Ia226ce1d63e52769bc067d50ec4704cea4e11c5c
2013-01-31 18:30:10 +00:00
Mehdi Abaakouk
a1395ec672 Allow change the endpoint_type when use swift-dispersion tools
Fixes bug 1102319
DocImpact

Change-Id: I8fb0417ab9468e97ed01a6cb1e262630905e7f29
2013-01-31 16:10:37 +01:00