46 Commits

Author SHA1 Message Date
Christian Schwede
cbddec340e Add bin/swift-dispersion-report
Change-Id: I81736080fc478c2b69d5b71edd0cada39aad9400
2017-09-13 05:57:30 +00:00
Christian Schwede
b77de5393f Make swift-dispersion-report importable
This patch allows to import the dispersion report tool, and thus making
it more easily usable within other Python tools. This can be also used
in a follow up patch to add some tests for the report tool.

It also fixes a bug when using the "--dump-json" option - until now it
returned the policy name and made the JSON invalid.

Change-Id: Ie0d52a1a54fc152bb72cbb3f84dcc36a8dad972a
2017-09-13 05:56:34 +00:00
Tim Burke
9890184ea9 Turn on H233 and start using print function
As much as anything, I'm just tired of seeing a bunch or piecemeal
fixes.

Note that we *need* to include

   from __future__ import print_function

in order to support things like

   print()  # Would print "()" (the repr of an empty tuple) otherwise
   print(foo, end='')  # Would SyntaxError
   print(bar, file=sys.stderr)  # Would SyntaxError

Change-Id: I8fdf0740e292eb1ee785512d02e8c552781dcae1
2016-07-08 16:19:52 +00:00
Félix Cantournet
4f053c61d6 Fix swift-dispersion in multi-region setups
If you have 2 swift regions served by the same keystone,
then the client cannot get the correct URL for the swift endpoint
without specifying a region_name.

Closes-Bug: 1587088
Change-Id: Iaab883386e125c3ca6b9554389e63df17267a135
2016-06-01 15:35:47 +02:00
Hisashi Osanai
23c7a58f8f Fix ClientException handling in Container Sync
swift/container/sync.py uses swift.common.internal_client.delete_object
and put_object and expected these methods raise ClientException.
But delete_object and put_object never raise the exception so this patch
raises ClientException when urllib2 library raises HTTPError.

Co-Authored-By: Eran Rom <eranr@il.ibm.com>
Closes-Bug: #1419901
Change-Id: I58cbf77988979a07998a46d9d81be84d29b0d9bf
2016-01-07 10:17:20 +00:00
Zack M. Davis
1b8b08039a remove remaining simplejson uses, prefer standard library import
a1c32702, 736cf54a, and 38787d0f remove uses of `simplejson` from
various parts of Swift in favor of the standard libary `json`
module (introduced in Python 2.6). This commit performs the remaining
`simplejson` to `json` replacements, removes two comments highlighting
quirks of simplejson with respect to Unicode, and removes the references
to it in setup documentation and requirements.txt.

There were a lot of places where we were importing json from
swift.common.utils, which is less intuitive than a direct `import json`,
so that replacement is made as well.

(And in two more tiny drive-bys, we add some pretty-indenting to an XML
fragment and use `super` rather than naming a base class explicitly.)

Change-Id: I769e88dda7f76ce15cf7ce930dc1874d24f9498a
2015-11-16 12:34:24 -08:00
Jenkins
6a9b868ae6 Merge "Python3: Fix Remaining issues of python3 compatibility in bin directory" 2015-10-08 08:41:34 +00:00
Samuel Merritt
e02609c66a Preserve traceback in swift-dispersion-report
Commit c690bcb fixed a bug in the dispersion report, but changed this
from a bare "raise" to "raise err", which loses the traceback. Not a
big deal, but worth putting back IMO.

Change-Id: Id5b72153a4b8df8e3faaf1fa3fb2040e28ba85cc
2015-09-01 15:19:50 -07:00
Kazuhiro MIYAHARA
c690bcb683 Fix dispersion-reports error message
This patch fixes Swift to show message
"No objects to query. Has swift-dispersion-populate been run?"
for "swift-dispersion-report —object-only”
with no container for object dispersion.

Change-Id: I82da56709cfc296a27f5180681709bc56adbc13d
Closes-Bug: #1468120
2015-08-26 11:32:06 +09:00
Jenkins
e1683fdb2e Merge "Support keystone v3 domains in swift-dispersion" 2015-07-31 06:59:01 +00:00
Victor Stinner
d719064e78 Fix warning pep8 E128 warning of hacking 0.10
Fix the warning E128: "continuation line under-indented for visual
indent" of pep8.

Change-Id: Ie6c6ae341fe3d6281f2095c1d756d552fa5937f9
2015-07-30 09:33:41 +02:00
Falk Reimann
363a256e58 Support keystone v3 domains in swift-dispersion
This provides the capability to specify a project_name,
project_domain_name and user_domain_name in /etc/swift/dispersion.conf.
If this values are set in dispersion.conf they get populated to the
swift-client.  With this it is possible to have a specific dispersion
project specified, which is not the keystone default domain.  Changes
were applied to swift-dispersion-populate and swift-dispersion-report.
Relevant man pages, the example dispersion.conf and the admin guide were
updated accordingly.

DocImpact
Closes-Bug: #1468374

Change-Id: I0e716f8d281b4d0f510bc568bcee4a13fc480ff7
2015-07-24 13:40:24 -05:00
janonymous
78cb608ff7 Python3: Fix Remaining issues of python3 compatibility in bin directory
Changes Of py3 in bin :

* https://review.openstack.org/#/c/196835/
* ConfigParser from six.moves

Change-Id: Ic0374c8e09dfd595ec12c4d31b17dad30eaa803c
2015-07-24 19:30:04 +05:30
janonymous
cd20961abd Replace dict.iteritems() with dict.items() in bin directory of swift.
The iteritems() of Python 2 dictionaries has been renamed to items() on
Python 3.

Change-Id: I4bdc064c90bab56cd60f2dca2a5a78426ffbb31c
2015-07-05 11:08:35 +05:30
paul luse
e6165a7879 Add policy support to dispersion tools
Doesn't work for anything other than policy 0. updated to allow user
to specify policy name on cmd line (as with object-info) which
then makes populate/report work with 3x, 2x, or EC style policies

Change-Id: Ib7c298f0f6d666b1ecca25315b88539f45cf9f95
Closes-Bug: 1458688
2015-06-23 02:14:02 -07:00
Christian Schwede
1f3ae6d8da Remove swiftclient dependency
Removes the requirement for swiftclient in swift-dispersion-report
and swift-dispersion-populate. To prevent a dependency on
keystoneclient and to avoid reinventing the wheel with an internal
keystoneclient, authentication with keystone is only supported if
swiftclient is available. If not, only auth v1 is supported.

The dependency in swift/container/sync.py has also been removed.

Implements: blueprint remove-swiftclient-dependency

Change-Id: I6ec3b3c85a67b9ab6eb04b90ffc16daf1600e8a7
2014-02-06 09:44:58 +00:00
Samuel Merritt
a7e0a9d57d Fix swift-dispersion-report when partitions overlap
swift-dispersion-report tries to avoid checking a partition more than
once, so it keeps track of partitions already queried and skips
duplicates.

swift-dispersion-report also keeps track of the number of successful
responses; it counts the number of expected replicas to find, and also
counts the number of replicas actually found, and tells the operator
if the numbers differ.

However, in the case that a partition was duplicated, the
expected-responses counter was incremented, but the actual check was
skipped, so it looked as though some copies were missing. Now we only
increment the expected-responses counter if we're actually going to
perform the check.

Change-Id: I22ac2b8066b62ca7c8ebf099c9f602118bb1a298
2014-01-17 11:45:57 -08:00
Chmouel Boudjnah
150f338fc2 Remove swiftclient dep on direct_client
Partial Implements: blueprint remove-swiftclient-dependency
Change-Id: I9af7150e5d21d50e5f880e57796314b8f05822d2
2013-12-24 03:11:43 -08:00
ZhiQiang Fan
f72704fc82 Change OpenStack LLC to Foundation
Change-Id: I7c3df47c31759dbeb3105f8883e2688ada848d58
Closes-bug: #1214176
2013-09-20 01:02:31 +08:00
Dirk Mueller
3d36a76156 Use Python 3.x compatible except construct
except x,y: was deprected and is removed in Python 3.x.
Use "except x as y:" instead which works in any Python
version >= 2.6.

Change-Id: I7008c74b807340f3457d3a0c8bd0b83f23169d14
2013-09-07 10:50:54 +02:00
Vincent Untz
7f1aa9d1e8 Allow dispersion tools to use keystone server with insecure certificate
The swift-dispersion-populate and swift-dispersion-report tools now
accept a --insecure option.

Also, dispersion.conf now has a keystone_api_insecure option.

Default is obviously to use the secure path.

DocImpact

Change-Id: I4000352e547d9ce5b08ade54e0c886281caff891
2013-08-05 22:44:12 +02:00
Thomas Leaman
5449155fb0 Allow floating point value for dispersion_coverage
For systems with very large numbers of partitions, 1% dispersion
coverage may simply be too much/take too long. This fix allows <1
values to be used for dispersion_coverage.

DocImpact

Change-Id: I5ed35b69754d55a410e66e658b3854de57c7666b
2013-07-09 09:29:07 +00:00
Samuel Merritt
57d4e6e718 Fix swift-dispersion-report.
Fixes bug 1180680.

Change-Id: I1c0eaa6f934f73853e832a8e468e1b8369a21117
2013-06-24 10:58:22 -07:00
Monty Taylor
de2e5aa462 Update to flake8 instead of pep8.
Change-Id: I3d4a31111c3044da06611405ce80f208ef8a0ce3
2013-05-01 14:19:38 -04:00
Samuel Merritt
7548cb9c47 Make rings' replica counts adjustable.
Example:

$ swift-ring-builder account.builder set_replicas 4
$ swift-ring-builder rebalance

This is a prerequisite for supporting globally-distributed clusters,
as operators of such clusters will probably want at least as many
replicas as they have regions. Therefore, adding a region requires
adding a replica. Similarly, removing a region lets an operator remove
a replica and save some money on disks.

In order to not hose clusters with lots of data, swift-ring-builder
now allows for setting of fractional replicas. Thus, one can gradually
increase the replica count at a rate that does not adversely affect
cluster performance.

Example:

$ swift-ring-builder object.builder set_replicas 3.01
$ swift-ring-builder object.builder rebalance
<distribute rings and wait>

$ swift-ring-builder object.builder set_replicas 3.02
$ swift-ring-builder object.builder rebalance
<distribute rings and wait>...

Obviously, fractional replicas are nonsensical for a single
partition. A fractional replica count is for the whole ring, not for
any individual partition, and indicates the average number of replicas
of each partition. For example, a replica count of 3.2 means that 20%
of partitions have 4 replicas and 80% have 3 replicas.

Changes do not take effect until after the ring is rebalanced. Thus,
if you mean to go from 3 replicas to 3.01 but you accidentally type
2.01, no data is lost.

Additionally, 'swift-ring-builder X.builder create' can now take a
decimal argument for the number of replicas.

DocImpact

Change-Id: I12b34dacf60350a297a46be493d5d171580243ff
2013-02-22 15:03:10 -08:00
Mehdi Abaakouk
a1395ec672 Allow change the endpoint_type when use swift-dispersion tools
Fixes bug 1102319
DocImpact

Change-Id: I8fb0417ab9468e97ed01a6cb1e262630905e7f29
2013-01-31 16:10:37 +01:00
clayg
3dbc1a9b9c document correct config in dispersion-report help
The --help message of swift-dispersion-report used to say
/etc/swift/stats.conf instead of the correct and consistent
/etc/swift/dispersion.conf - this change updates that
commandline help message.

Change-Id: I69ad64d31bb86eb0d36fcf5b17aa8bf42f646ed1
2012-12-14 10:03:04 -08:00
Florian Hines
e474dfb720 Add dispersion report flags to limit reports
- Add two optional flags that let you limit swift-dispersion-report to only
reporting on containers OR objects.
- Also make dispersion.conf and swift-dispersion-report manpages
  current.

DocImpact

Change-Id: Iad56133cad261241db27d0e2103098e3c2f3c245
2012-12-09 18:20:08 -06:00
Jenkins
13941cb8eb Merge "dispersion report option to output missing parts" 2012-11-02 19:35:38 +00:00
gholt
dcc89e3ad6 dispersion report option to output missing parts
Adds a -p option to swift-dispersion-report that will output the
partitions missing copies to standard error. Another thing we've been
meaning to add for forever. It's useful when you want to do some
further research on whether a partition truly has fewer copies or if
they're just somewhere else in the cluster due to a ring change, for
instance.

DocImpact

Change-Id: I5e47aa5818483ecc34b39ef6f8cd83ad312ed9a0
2012-10-30 17:40:42 +00:00
John Dickinson
8ac292595f changed TRUE_VALUES references to utils.config_true_value() call
cleaned up pep8 (v1.3.3) in all files this patch touches

Change-Id: I30e8314dfdc23fb70ab83741a548db9905dfccff
2012-10-29 13:59:01 -07:00
Greg Lange
f4c319b94c Made dispersion report work with any replica count other than 3.
This changes the JSON output keys missing_one and missing_two to
missing_1 and missing_2 (if there are at least 2 replicas), drops
the missing_all key, and can add more missing_x keys depending on
the replica count. We will definitely need to post a prominent
notice of this change with the next release.

bug 862816

Change-Id: Ib747d944476638c33ee1d876b8f9db28953826e2
2012-06-18 20:15:51 +00:00
Jenkins
22624aa50a Merge "added printing of 404s to dispersion report; fixed small, unrelated bug" 2012-06-11 23:42:43 +00:00
Greg Lange
27455cb15b added printing of 404s to dispersion report; fixed small, unrelated bug
also fixed bug where in error_log where identifier wasn't being set to anything meaningful, set it to the right thing

bug 612722

Change-Id: I53f237ea0db2a5b2b8979a7b41189faf1275e861
2012-06-07 21:15:18 +00:00
Chmouel Boudjnah
22572b506a Remove swift.common.client from here.
- It has been to its own gerrit project.
- direct_client should follow next.
- Implements blueprint clientbindings.

Change-Id: I3bb50c95eba81302bfec71cb7ce5288b85a41dc0
2012-06-07 16:36:49 +02:00
Samuel Merritt
2ccf219ec1 Make scripts in bin/ PEP8-compliant.
Also made tox's PEP8 check look at the scripts in bin/ to keep them
PEP8-compliant.

Change-Id: I710365ea929d7fc15578d5f742a236bad47ef28e
2012-05-04 08:24:51 -07:00
Julien Danjou
9a423d0b78 Allow to specify auth_version in swift-dispersion tools
Change-Id: I080d531471d8ea57c69a918d4f6930441f1e69f6
Signed-off-by: Julien Danjou <julien.danjou@enovance.com>
2012-04-02 16:42:05 +02:00
Julien Danjou
7a39fea989 Fix typo in swift-dispersion-report and populate
Change-Id: I5168942ad32b3461f4c46300e378b249dab6a2ee
Signed-off-by: Julien Danjou <julien.danjou@enovance.com>
2012-04-02 16:38:48 +02:00
Jenkins
6682138b0a Merge "Make ring class interface slightly more abstracted from implementation." 2012-03-22 20:25:06 +00:00
John Dickinson
1ecf5ebba1 updated copyright date for all files
Change-Id: Ifd909d3561c2647770a7e0caa3cd91acd1b4f298
2012-03-19 13:45:34 -05:00
Michael Barton
e008c2ebb8 Make ring class interface slightly more abstracted from implementation.
Change-Id: I0f55d61c7b8de30460f17a69e5d9946494dbda6e
2012-03-14 22:00:30 +00:00
Florian Hines
892d63d8ec Fix "retries" key in dispersion json output.
Change-Id: I466302f05fd17c5b5460284c372c3667d7c46d7b
2012-03-08 21:08:15 +00:00
Florian Hines
5e4127ae2a Add json output option to swift-dispersion-report
Add's the configuration file option "dump_json" or command line
options [-j|--dump-json] to have swift-dispersion-report output
the report in json format. This allows the dispersion report to
be more easily consumed elsewhere.

There's also a few pep8 fixes and removal of unused imports.

Change-Id: I2374311ccbef43e6bbae24665c9584e60f3da173
2012-02-29 04:24:54 +00:00
gholt
1d866acdb1 PEP8 fixes 2011-05-12 01:10:16 +00:00
gholt
d0d98ba96e Updated to use standard shebang and explicitly flush output; added deprecation warning to stats.conf 2011-05-12 00:14:02 +00:00
gholt
6c13001244 Rename swift-stats-* to swift-dispersion-* to avoid confusion with log stats stuff 2011-03-31 22:32:41 +00:00