This patch allows to import the dispersion report tool, and thus making
it more easily usable within other Python tools. This can be also used
in a follow up patch to add some tests for the report tool.
It also fixes a bug when using the "--dump-json" option - until now it
returned the policy name and made the JSON invalid.
Change-Id: Ie0d52a1a54fc152bb72cbb3f84dcc36a8dad972a
As much as anything, I'm just tired of seeing a bunch or piecemeal
fixes.
Note that we *need* to include
from __future__ import print_function
in order to support things like
print() # Would print "()" (the repr of an empty tuple) otherwise
print(foo, end='') # Would SyntaxError
print(bar, file=sys.stderr) # Would SyntaxError
Change-Id: I8fdf0740e292eb1ee785512d02e8c552781dcae1
If you have 2 swift regions served by the same keystone,
then the client cannot get the correct URL for the swift endpoint
without specifying a region_name.
Closes-Bug: 1587088
Change-Id: Iaab883386e125c3ca6b9554389e63df17267a135
swift/container/sync.py uses swift.common.internal_client.delete_object
and put_object and expected these methods raise ClientException.
But delete_object and put_object never raise the exception so this patch
raises ClientException when urllib2 library raises HTTPError.
Co-Authored-By: Eran Rom <eranr@il.ibm.com>
Closes-Bug: #1419901
Change-Id: I58cbf77988979a07998a46d9d81be84d29b0d9bf
a1c32702, 736cf54a, and 38787d0f remove uses of `simplejson` from
various parts of Swift in favor of the standard libary `json`
module (introduced in Python 2.6). This commit performs the remaining
`simplejson` to `json` replacements, removes two comments highlighting
quirks of simplejson with respect to Unicode, and removes the references
to it in setup documentation and requirements.txt.
There were a lot of places where we were importing json from
swift.common.utils, which is less intuitive than a direct `import json`,
so that replacement is made as well.
(And in two more tiny drive-bys, we add some pretty-indenting to an XML
fragment and use `super` rather than naming a base class explicitly.)
Change-Id: I769e88dda7f76ce15cf7ce930dc1874d24f9498a
Commit c690bcb fixed a bug in the dispersion report, but changed this
from a bare "raise" to "raise err", which loses the traceback. Not a
big deal, but worth putting back IMO.
Change-Id: Id5b72153a4b8df8e3faaf1fa3fb2040e28ba85cc
This patch fixes Swift to show message
"No objects to query. Has swift-dispersion-populate been run?"
for "swift-dispersion-report —object-only”
with no container for object dispersion.
Change-Id: I82da56709cfc296a27f5180681709bc56adbc13d
Closes-Bug: #1468120
This provides the capability to specify a project_name,
project_domain_name and user_domain_name in /etc/swift/dispersion.conf.
If this values are set in dispersion.conf they get populated to the
swift-client. With this it is possible to have a specific dispersion
project specified, which is not the keystone default domain. Changes
were applied to swift-dispersion-populate and swift-dispersion-report.
Relevant man pages, the example dispersion.conf and the admin guide were
updated accordingly.
DocImpact
Closes-Bug: #1468374
Change-Id: I0e716f8d281b4d0f510bc568bcee4a13fc480ff7
Doesn't work for anything other than policy 0. updated to allow user
to specify policy name on cmd line (as with object-info) which
then makes populate/report work with 3x, 2x, or EC style policies
Change-Id: Ib7c298f0f6d666b1ecca25315b88539f45cf9f95
Closes-Bug: 1458688
Removes the requirement for swiftclient in swift-dispersion-report
and swift-dispersion-populate. To prevent a dependency on
keystoneclient and to avoid reinventing the wheel with an internal
keystoneclient, authentication with keystone is only supported if
swiftclient is available. If not, only auth v1 is supported.
The dependency in swift/container/sync.py has also been removed.
Implements: blueprint remove-swiftclient-dependency
Change-Id: I6ec3b3c85a67b9ab6eb04b90ffc16daf1600e8a7
swift-dispersion-report tries to avoid checking a partition more than
once, so it keeps track of partitions already queried and skips
duplicates.
swift-dispersion-report also keeps track of the number of successful
responses; it counts the number of expected replicas to find, and also
counts the number of replicas actually found, and tells the operator
if the numbers differ.
However, in the case that a partition was duplicated, the
expected-responses counter was incremented, but the actual check was
skipped, so it looked as though some copies were missing. Now we only
increment the expected-responses counter if we're actually going to
perform the check.
Change-Id: I22ac2b8066b62ca7c8ebf099c9f602118bb1a298
except x,y: was deprected and is removed in Python 3.x.
Use "except x as y:" instead which works in any Python
version >= 2.6.
Change-Id: I7008c74b807340f3457d3a0c8bd0b83f23169d14
The swift-dispersion-populate and swift-dispersion-report tools now
accept a --insecure option.
Also, dispersion.conf now has a keystone_api_insecure option.
Default is obviously to use the secure path.
DocImpact
Change-Id: I4000352e547d9ce5b08ade54e0c886281caff891
For systems with very large numbers of partitions, 1% dispersion
coverage may simply be too much/take too long. This fix allows <1
values to be used for dispersion_coverage.
DocImpact
Change-Id: I5ed35b69754d55a410e66e658b3854de57c7666b
Example:
$ swift-ring-builder account.builder set_replicas 4
$ swift-ring-builder rebalance
This is a prerequisite for supporting globally-distributed clusters,
as operators of such clusters will probably want at least as many
replicas as they have regions. Therefore, adding a region requires
adding a replica. Similarly, removing a region lets an operator remove
a replica and save some money on disks.
In order to not hose clusters with lots of data, swift-ring-builder
now allows for setting of fractional replicas. Thus, one can gradually
increase the replica count at a rate that does not adversely affect
cluster performance.
Example:
$ swift-ring-builder object.builder set_replicas 3.01
$ swift-ring-builder object.builder rebalance
<distribute rings and wait>
$ swift-ring-builder object.builder set_replicas 3.02
$ swift-ring-builder object.builder rebalance
<distribute rings and wait>...
Obviously, fractional replicas are nonsensical for a single
partition. A fractional replica count is for the whole ring, not for
any individual partition, and indicates the average number of replicas
of each partition. For example, a replica count of 3.2 means that 20%
of partitions have 4 replicas and 80% have 3 replicas.
Changes do not take effect until after the ring is rebalanced. Thus,
if you mean to go from 3 replicas to 3.01 but you accidentally type
2.01, no data is lost.
Additionally, 'swift-ring-builder X.builder create' can now take a
decimal argument for the number of replicas.
DocImpact
Change-Id: I12b34dacf60350a297a46be493d5d171580243ff
The --help message of swift-dispersion-report used to say
/etc/swift/stats.conf instead of the correct and consistent
/etc/swift/dispersion.conf - this change updates that
commandline help message.
Change-Id: I69ad64d31bb86eb0d36fcf5b17aa8bf42f646ed1
- Add two optional flags that let you limit swift-dispersion-report to only
reporting on containers OR objects.
- Also make dispersion.conf and swift-dispersion-report manpages
current.
DocImpact
Change-Id: Iad56133cad261241db27d0e2103098e3c2f3c245
Adds a -p option to swift-dispersion-report that will output the
partitions missing copies to standard error. Another thing we've been
meaning to add for forever. It's useful when you want to do some
further research on whether a partition truly has fewer copies or if
they're just somewhere else in the cluster due to a ring change, for
instance.
DocImpact
Change-Id: I5e47aa5818483ecc34b39ef6f8cd83ad312ed9a0
This changes the JSON output keys missing_one and missing_two to
missing_1 and missing_2 (if there are at least 2 replicas), drops
the missing_all key, and can add more missing_x keys depending on
the replica count. We will definitely need to post a prominent
notice of this change with the next release.
bug 862816
Change-Id: Ib747d944476638c33ee1d876b8f9db28953826e2
also fixed bug where in error_log where identifier wasn't being set to anything meaningful, set it to the right thing
bug 612722
Change-Id: I53f237ea0db2a5b2b8979a7b41189faf1275e861
- It has been to its own gerrit project.
- direct_client should follow next.
- Implements blueprint clientbindings.
Change-Id: I3bb50c95eba81302bfec71cb7ce5288b85a41dc0
Add's the configuration file option "dump_json" or command line
options [-j|--dump-json] to have swift-dispersion-report output
the report in json format. This allows the dispersion report to
be more easily consumed elsewhere.
There's also a few pep8 fixes and removal of unused imports.
Change-Id: I2374311ccbef43e6bbae24665c9584e60f3da173