287 Commits

Author SHA1 Message Date
Florian Hines
5f72a8db4a Fix Dispersion report and swift-bench on saio
We're still using saio:11000 in a few spots so a few things
don't work out of the box on the saio. Fixes bug #1024561

Change-Id: I226de54c2785b0d0b681c8d0cc24260adbd3d663
2012-07-13 17:48:37 -05:00
Jenkins
217676c06c Merge "swift-bench should be able to use auth version 2.0" 2012-07-02 18:49:13 +00:00
Samuel Merritt
67af56b156 Make swift-get-nodes -a show all handoffs.
Commit 95786e5 made swift-get-nodes only output N handoff nodes, where
N is the ring's replica count. This change leaves that behavior in
place as the default, but adds the option -a/--all to print all the
handoff nodes.

This is handy if you're consuming the output with a script, e.g. if
you're doing some analysis of the distribution of primary+handoff
nodes for a particular set of things.

Change-Id: Ie0253575bb21ccf25559a1c5fd20570a58a2d9fa
2012-06-20 12:45:34 -07:00
Greg Lange
f4c319b94c Made dispersion report work with any replica count other than 3.
This changes the JSON output keys missing_one and missing_two to
missing_1 and missing_2 (if there are at least 2 replicas), drops
the missing_all key, and can add more missing_x keys depending on
the replica count. We will definitely need to post a prominent
notice of this change with the next release.

bug 862816

Change-Id: Ib747d944476638c33ee1d876b8f9db28953826e2
2012-06-18 20:15:51 +00:00
François Charlier
ef3e556b72 swift-bench should be able to use auth version 2.0
Fixes Bug #1011727

Change-Id: I6681bd85a5ddb82a1059913ae90696e5994aa906
2012-06-13 09:06:57 +02:00
gholt
95786e577c Update swift-get-nodes to return fewer more_nodes
Changed swift-get-nodes to just return up to replica_count of
more_nodes instead of all of them. When you have thousands of drives,
returning all possible handoff nodes is pretty crazy. :)

Change-Id: I864e7243726ca3ef2ac08c4ab660a05789f37878
2012-06-13 01:56:43 +00:00
Jenkins
22624aa50a Merge "added printing of 404s to dispersion report; fixed small, unrelated bug" 2012-06-11 23:42:43 +00:00
Greg Lange
27455cb15b added printing of 404s to dispersion report; fixed small, unrelated bug
also fixed bug where in error_log where identifier wasn't being set to anything meaningful, set it to the right thing

bug 612722

Change-Id: I53f237ea0db2a5b2b8979a7b41189faf1275e861
2012-06-07 21:15:18 +00:00
Chmouel Boudjnah
22572b506a Remove swift.common.client from here.
- It has been to its own gerrit project.
- direct_client should follow next.
- Implements blueprint clientbindings.

Change-Id: I3bb50c95eba81302bfec71cb7ce5288b85a41dc0
2012-06-07 16:36:49 +02:00
Florian Hines
ccb6334c17 Expand recon middleware support
Expand recon middleware to include support for account and container
servers in addition to the existing object servers. Also add support
for retrieving recent information from auditors, replicators, and
updaters. In the case of certain checks (such as container auditors)
the stats returned are only for the most recent path processed.

The middleware has also been refactored and should now also handle
errors better in cases where stats are unavailable.

While new check's have been added the output from pre-existing
check's has not changed. This should allow existing 3rd party
utilities such as the Swift ZenPack to continue to function.

Change-Id: Ib9893a77b9b8a2f03179f2a73639bc4a6e264df7
2012-05-24 14:50:00 -05:00
gholt
1c3b75c291 Reverted the pulling out of various middleware:
RateLimit
StaticWeb
TempURL/FormPOST

Change-Id: I988e93e6f4aacb817a2e354d43a04e47516fdf88
2012-05-16 21:25:10 +00:00
Jenkins
86f37c47d7 Merge "Let some swift-ring-builder commands take >1 arg." 2012-05-11 15:15:03 +00:00
Samuel Merritt
8e6f099daa Speed up swift-get-nodes by 2x.
It was loading the ring off disk once to print the primary nodes, and
then loading the whole thing off disk again to print the handoff
nodes. Changed it to only load the ring off disk once.

Change-Id: I6f4cd0af9762e1e69660c3eb20586590b5339e5f
2012-05-10 00:10:24 -07:00
Samuel Merritt
47f0dbb125 One PEP8 fix to make tox happy again.
Change-Id: I5ff2056f9f2eb99bfb98b020e3fc013332100e12
2012-05-09 12:28:00 -07:00
gholt
c0532a6ef2 Pulled out TempURL/FormPOST
TempURL/FormPOST is now at http://gholt.github.com/swift-tempurl/

For current users of TempURL/FormPOST, this will require installing the new
package and changing the "use" line of the tempurl and formpost conf section's
to:

[filter:tempurl]
use = egg:swifttempurl#tempurl

[filter:formpost]
use = egg:swifttempurl#formpost

And then 'swift-init proxy reload'.

Change-Id: I5bddf7f9e09ee07815530a41c46ff901fc21b447
2012-05-08 20:20:29 +00:00
Jenkins
742f3d1a8e Merge "Make scripts in bin/ PEP8-compliant." 2012-05-04 17:33:32 +00:00
Samuel Merritt
2ccf219ec1 Make scripts in bin/ PEP8-compliant.
Also made tox's PEP8 check look at the scripts in bin/ to keep them
PEP8-compliant.

Change-Id: I710365ea929d7fc15578d5f742a236bad47ef28e
2012-05-04 08:24:51 -07:00
Brent Roskos
0d640971cd Fix related to bug #994502
Added support for configuring lock file location.
added an additional error handler.

Change-Id: I3635ed9694b9fb8526e0ca0591dc731da03a7128
2012-05-04 06:59:45 -04:00
Samuel Merritt
0befccccdf Let some swift-ring-builder commands take >1 arg.
`swift-ring-builder X.builder [add|set_weight|set_info|remove]` can
now work on multiple devices at once.

This allows for faster bulk operations; for example, if an
administrator has to add 48 new devices to the ring, then this lets
them do it in one command invocation. Since each invocation of
swift-ring-builder follows a deserialize-operate-serialize pattern,
using the bulk form of the commands on N devices will save N-1
[de]serializations.

Change-Id: Ibf77528f28026866af3c94d0370181c8013d5bc5
2012-05-03 17:45:33 -07:00
Pete Zaitcev
6b91352007 Restore the syntax -U tenant:user
On a relatively recent build of swift, all my scripts blew up with
"No tenant specified". It was caused by the fix to add --os_tenant_name,
commit ID:  208b8e85a80e46ddb49dc2035cb292570a20c7db

This patch restores the old behavior. I tested it to work with old
swauth-based Swift, new Keystone-based Swift, in the latter case using
both -U and --os_tenant_name arguments.

Note that this patch permits to use a literal colon in the user name
with the new syntax, as long as tenant is specified. Empty tenant
names are not allowed with either syntax.

Change-Id: I7785e6981a9d6281d0421c43875ee19d61d5ff43
Bug: 982909
2012-04-26 10:58:07 -06:00
Jenkins
9115bbb17f Merge "Make swift client easier to wrap" 2012-04-19 02:49:49 +00:00
Jenkins
a77cbc2892 Merge "Use options.os_auth_url rather then environment variable" 2012-04-12 19:42:48 +00:00
Pete Zaitcev
3aabe86e34 Make swift client easier to wrap
In the process of wrapping for profiling, a trivial wrap with def main()
ended in a ton of these:
    NameError: global name 'parser' is not defined

A closer investigation revealed an apparent confusion in argument names.
The problem was introduced by commit 985968f76550251904b64d12de025fa5d65ed03f
of November 2010. Note that Greg fixed st_delete() correctly in that one,
but apparently forgot the others.

Unit tests pass for both current (bad) code, and the patched code.
By the magic of Python scoping, current code works, just impossible
to wrap for a quick hack.

Change-Id: I18f2494ce2a201d04d49f99ea259d141939dcb6a
2012-04-10 10:27:54 -06:00
Jenkins
3c2e85609b Merge "Add os_tenant_name to bin/swift and common.client" 2012-04-10 14:42:33 +00:00
Julien Danjou
50361f6a22 Use options.os_auth_url rather then environment variable
If you do specify --os_auth_url rather than using the environment variable
OS_AUTH_URL, you'll end up with different behaviour.

Change-Id: I7430c2a619487375427b1c377ce00245eff107ff
2012-04-06 15:53:43 +02:00
Julien Danjou
9a423d0b78 Allow to specify auth_version in swift-dispersion tools
Change-Id: I080d531471d8ea57c69a918d4f6930441f1e69f6
Signed-off-by: Julien Danjou <julien.danjou@enovance.com>
2012-04-02 16:42:05 +02:00
Julien Danjou
7a39fea989 Fix typo in swift-dispersion-report and populate
Change-Id: I5168942ad32b3461f4c46300e378b249dab6a2ee
Signed-off-by: Julien Danjou <julien.danjou@enovance.com>
2012-04-02 16:38:48 +02:00
Jenkins
549ca120fa Merge "Display total disk usage info in swift-recon" 2012-03-26 09:02:48 +00:00
Jenkins
6682138b0a Merge "Make ring class interface slightly more abstracted from implementation." 2012-03-22 20:25:06 +00:00
Florian Hines
83a12ece2a Display total disk usage info in swift-recon
Now display's cluster disk space used, free, and total.

Change-Id: I9c48dd6badaecd64cda4a2a8a51f41f1c29d328a
2012-03-21 17:30:44 -05:00
Chmouel Boudjnah
208b8e85a8 Add os_tenant_name to bin/swift and common.client
- Make sure we have a tenant_name when doing get_auth for 2.0
  auth_version.

Change-Id: I02dceb03f224c8fb91972e1f0e2925e5d1a06640
2012-03-20 12:11:21 +00:00
John Dickinson
1ecf5ebba1 updated copyright date for all files
Change-Id: Ifd909d3561c2647770a7e0caa3cd91acd1b4f298
2012-03-19 13:45:34 -05:00
Jenkins
a3638709aa Merge "Added optional max_containers_per_account restr..." 2012-03-19 18:04:16 +00:00
gholt
2c6de2ae52 Added optional max_containers_per_account restr...
Added optional max_containers_per_account restriction. If set to a
positive value and if a client tries to perform a container PUT when
at or above the max_containers_per_acount cap, a 403 Forbidden will
be returned with an explanatory message.

This only restricts the proxy server, not any of the background
processes that might need to create containers (replication, for
instance). Also, the container count is cached for the proxy's
recheck_account_existence number of seconds. For these reasons, a
given account could exceed this cap before the 403 Forbidden
responses kick in and therefore this feature should be considered a
"soft" limit.

You may also add accounts to the proxy's max_containers_whitelist
setting to have accounts that ignore this cap.

Change-Id: I74e8fb152de5e78d070ed30006ad4e53f82c8376
2012-03-15 19:41:31 +00:00
Chmouel Boudjnah
4f93c5d5e4 Update swift.common.client with bin/swift changes.
- Add auth version 2 to swift.common.client.
- Remove ununsed imports.
- Fix bug where auth_version should be a string.
- Add test for auth version 2.
- Allow to override the returns of http_connection for tests.
- Sync the passing of headers in bin/swift as well from client.
- Fixes bug 885011
- Previously it was review 3680 but abandoned.
- Address: Maru newby review.
- TODO: properly test auth_v1.

Change-Id: I579d8154828e892596fae9ab75f69d353f15e12c
2012-03-15 16:06:17 +00:00
Jenkins
7443f08f69 Merge "Change CLIAuth arg names" 2012-03-15 00:10:55 +00:00
Michael Barton
e008c2ebb8 Make ring class interface slightly more abstracted from implementation.
Change-Id: I0f55d61c7b8de30460f17a69e5d9946494dbda6e
2012-03-14 22:00:30 +00:00
Dean Troyer
ed02c900d8 Change CLIAuth arg names
Change the argument names used for common Keystone authentication per the
updated http://wiki.openstack.org/CLIAuth:

--auth_url -> --os_auth_url
--password -> --os_password
--username -> --os_username

Fixes bug 954528

Change-Id: I7e956e74de8192957eee87d155b2e5338c8942f0
2012-03-14 00:13:56 -05:00
Eamonn O'Toole
43ad1edd5d Body of http responses included in Swift error msg
The body of http responses is included in error messages from
swift.common.client and bin/swift if the body exists

Includes changes requested by Juan Martinez

Includes changes requested by David Goetz

Fixed pep8 warnings

Bug 949843

Change-Id: Ib998280762b084dd46f8c0f4524eed20513de82b
2012-03-12 11:25:41 +00:00
Florian Hines
892d63d8ec Fix "retries" key in dispersion json output.
Change-Id: I466302f05fd17c5b5460284c372c3667d7c46d7b
2012-03-08 21:08:15 +00:00
gholt
7141c52bd6 Remove old stats tools.
https://blueprints.launchpad.net/swift/+spec/remove-old-swift-stats-tools

Change-Id: Ibb57b3d71c7f18060d2d48a1d3754a7f1444624d
2012-03-06 17:57:16 +00:00
Chmouel Boudjnah
bcaecc574e Make sure auth is present before checking for /
- Fixes bug 945882

Change-Id: I22a688b01678ab93e888c9537a3f611e43e22c30
2012-03-03 21:56:02 +00:00
Jenkins
64cdc19490 Merge "Add json output option to swift-dispersion-report" 2012-02-29 22:09:10 +00:00
Jenkins
c51b8b2f20 Merge "swift-recon cli cleanup" 2012-02-29 21:57:03 +00:00
Jenkins
50ff676689 Merge "Adds support for metadata values with a ':'." 2012-02-29 21:49:10 +00:00
Florian Hines
5e4127ae2a Add json output option to swift-dispersion-report
Add's the configuration file option "dump_json" or command line
options [-j|--dump-json] to have swift-dispersion-report output
the report in json format. This allows the dispersion report to
be more easily consumed elsewhere.

There's also a few pep8 fixes and removal of unused imports.

Change-Id: I2374311ccbef43e6bbae24665c9584e60f3da173
2012-02-29 04:24:54 +00:00
Florian Hines
c18a4e4f43 swift-recon cli cleanup
Refactored swift-recon completely. It's broken out into Classes and no
longer uses globals. In addition, I pulled out the previous individual
scout_XXXX functions that where getting added for each check. All the
check's now use the same method from the Scout class to obtain stats
and telemetry.

Change-Id: I512ab50f29e1ef4d10bd3adbf1cac642308e7cf1
2012-02-28 16:33:57 +00:00
Doug Weimer
48303610a7 Adds support for metadata values with a ':'.
Changes the item.split(':') in st_post to item.split(':', 1). Fixes bug 930872.
Adds a helper function named split_metadata to have a common place for parsing
metadata options. Errors out if the metadata parameters do not contain a :.

Change-Id: I49bfb4dc8c34410e491cf55ef6e7f9a1466f201f
2012-02-23 18:22:31 +00:00
Dean Troyer
6800ec378b Blueprint cli-auth: common cli args
Add --auth_url, --username and --password and OS_* env vars as
aliases to --auth, --user and --key.

Default to --auth-version=2.0 if none of --auth, --user or --key
are set or if OS_AUTH_URL is set.

Ensure trailing '/' is present in --auth so URLs are created correctly.

Fixes lp925212

Change-Id: Ic0008c5d5c1ab4fddbaab5d982ab60fed2c50019
2012-02-08 15:06:40 -06:00
gholt
0406323f9b Added timeout option to swift-recon
Change-Id: I0d3c55a11c5e3252f0fd4b128ab840403c7f55f6
2012-02-08 00:06:55 +00:00