41 Commits

Author SHA1 Message Date
Samuel Merritt
d69f929be9 Add "normal", optparse-style options to swift-ring-builder add.
The old format is still present and works just like it did before, so
your existing scripts won't break.

New format pros:
 * it's readable even for Swift newcomers
 * it's easy to extend
 * it's familiar to anyone who's used a Unix command line
 * we don't have to maintain the parser

New format cons:
 * you can't add multiple devices in one go

Old format pros:
 * you can add many devices with one command
 * it's compact

Old format cons:
 * it confuses newcomers
 * "wait, is that zone dash IP colon port slash device, or zone slash
   IP dash port colon meta underscore device?" Just try walking
   someone through adding a device over voice chat.
 * it's annoying to add new fields

Note that this only affects the command "swift-ring-builder
<builderfile> add". Other swift-ring-builder commands are unchanged.

DocImpact

Change-Id: I034b7f79eb6f4d81a5c4da193e1358741441c5b5
2013-06-10 14:09:27 -07:00
Ilya Kharin
3957dbc5d4 RingBuilder.add_dev returns device id
When added a new devices into builder the add_dev function returns it
unique id.

blueprint argparse-in-swift-ring-builder

Change-Id: I57080bb625e812f6cea71199df907a44b332b552
2013-05-24 17:34:24 +04:00
Ilya Kharin
43bf568f48 Move parse search logic outside from builder
Dramatic part of RingBuilder.search_devs which parse a complex format
of a search device string moved to the swift-ring-builder script.
Instead, the search_devs has a simple interface to search devices.

blueprint argparse-in-swift-ring-builder

Change-Id: If3dd77b297b474fb9a058e4693fef2dfb11fca3d
2013-05-24 17:12:34 +04:00
Ilya Kharin
cc040a9c29 Add ability to save builder data to a disk file
Instances of the RingBuilder class can store its data to a disk file by
the save method and load it by the load method.

blueprint argparse-in-swift-ring-builder

Change-Id: I69fdf0693ca9f520d235a795ecdd2da310dcd5d3
2013-05-16 19:49:00 +04:00
Sergey Kraynev
ea7858176b Implementation of replication servers
Support separate replication ip address:
- Added new function in utils. This function provides ability
  to select separate IP address for replication service.
- Db_replicator and object replicators were changed.
  Replication process uses new function now.

Replication network parameters:
- Replication network fields (replication_ip, replication_port)
  support was added to device dictionary in swift-ring-builder script.
- Changes were made to support new fields in search, show and set_info
  functions.

Implementation of replication servers:
- Separate replication servers use the same code as normal replication
  servers, but with replication_server parameter = True.  When using a
  separate replication network, the non-replication servers set
  replication_server = False.  When there is no separate replication
  network (the default case), replication_server is not included in the config.

DocImpact
Change-Id: Ie9af5bdcdf9241c355e36053ca4adfe49dc35bd0
Implements: blueprint dedicated-replication-network
2013-04-21 18:14:42 -04:00
Jenkins
4074ba93f5 Merge "Fix rebalance for zero weighted devices." 2013-05-04 17:03:31 +00:00
Monty Taylor
de2e5aa462 Update to flake8 instead of pep8.
Change-Id: I3d4a31111c3044da06611405ce80f208ef8a0ce3
2013-05-01 14:19:38 -04:00
Eohyung Lee
98acf42f92 Fix rebalance for zero weighted devices.
If we set device's weight to zero, currently balance will be set
special value(999.99) until zero weighted device return all
its partitions. So we cannot check balance has changed.
Thus we need to check balance or last_balance is special value.

Change-Id: I5b7db8b8e48db0c4771c51a764bda689869817d5
Fixes: bug #1171731
2013-04-26 16:36:08 +09:00
Jenkins
e3ccb3f52d Merge "Correcting output of swift-ring-builder" 2013-04-15 22:49:48 +00:00
Sergey Kraynev
5afd0c3bb6 Refactoring format_device function
* Algorithm format_device was changed for simplicity extension new
  ip addresses parameters.
* Some prints outputs was replacement by function format_device.

Change-Id: I8565d42fcdb62eeb398c4432bb6f499c27c05cf6
2013-04-12 12:40:27 -04:00
Sergey Kraynev
d7873b5ce3 Correcting output of swift-ring-builder
Indent swift-ring-builder output was changed according
with old style (before implementation of region tier).

Change-Id: I0d1cc7acdc5baf86f343745aea6fc2120838fd36
2013-04-11 09:32:56 -04:00
tong li
b530de2ad5 swift-ring-builder blows up when not supported command requested.
swift-ring-builder uses outdated unbounded class method to handle
not supported command. It worked for python 2.6 or older but not
for python 2.7 or newer. This patch fixes the problem.

Change-Id: I7dbc681ef6be44f6d79ff93189ccca13c51eab74
Fixes: bug #1154882
2013-03-18 09:04:50 -04:00
Ilya Kharin
e56832c138 Fix format device
Change-Id: Icef35175682c11a406ef85accc6c6e1acbf89f4d
2013-03-16 12:41:14 +04:00
Jenkins
fd3a8a6650 Merge "Fix swift-ring-builder's list_parts command." 2013-03-13 22:53:34 +00:00
Samuel Merritt
35acd5912f Fix swift-ring-builder's list_parts command.
I broke it in 7548cb9 when a ring's replica count changed from an int
to a float.

Change-Id: I49fa7e2961a09daad0c96e19cf1e39fec248d998
2013-03-13 13:34:19 -07:00
Samuel Merritt
ebcd60f7d9 Add a region tier to Swift's ring.
The region is one level above the zone; it is intended to represent a
chunk of machines that is distant from others with respect to
bandwidth and latency.

Old rings will default to having all their devices in region 1. Since
everything is in the same region by default, the ring builder will
simply distribute across zones as it did before, so your partition
assignment won't move because of this change. If you start adding
devices in other regions, of course, the assignment will change to
take that into account.

swift-ring-builder still accepts the same syntax as before, but will
default added devices to region 1 if no region is specified.

Examples:

$ swift-ring-builder foo.builder add r2z1-1.2.3.4:555/sda

$ swift-ring-builder foo.builder add r1z3-1.2.3.4:555/sda

$ swift-ring-builder foo.builder add z3-1.2.3.4:555/sda

Also, some updates to ring-overview doc.

Change-Id: Ifefbb839cdcf033e6c9201fadca95224c7303a29
2013-03-13 10:00:58 -07:00
Samuel Merritt
7548cb9c47 Make rings' replica counts adjustable.
Example:

$ swift-ring-builder account.builder set_replicas 4
$ swift-ring-builder rebalance

This is a prerequisite for supporting globally-distributed clusters,
as operators of such clusters will probably want at least as many
replicas as they have regions. Therefore, adding a region requires
adding a replica. Similarly, removing a region lets an operator remove
a replica and save some money on disks.

In order to not hose clusters with lots of data, swift-ring-builder
now allows for setting of fractional replicas. Thus, one can gradually
increase the replica count at a rate that does not adversely affect
cluster performance.

Example:

$ swift-ring-builder object.builder set_replicas 3.01
$ swift-ring-builder object.builder rebalance
<distribute rings and wait>

$ swift-ring-builder object.builder set_replicas 3.02
$ swift-ring-builder object.builder rebalance
<distribute rings and wait>...

Obviously, fractional replicas are nonsensical for a single
partition. A fractional replica count is for the whole ring, not for
any individual partition, and indicates the average number of replicas
of each partition. For example, a replica count of 3.2 means that 20%
of partitions have 4 replicas and 80% have 3 replicas.

Changes do not take effect until after the ring is rebalanced. Thus,
if you mean to go from 3 replicas to 3.01 but you accidentally type
2.01, no data is lost.

Additionally, 'swift-ring-builder X.builder create' can now take a
decimal argument for the number of replicas.

DocImpact

Change-Id: I12b34dacf60350a297a46be493d5d171580243ff
2013-02-22 15:03:10 -08:00
Christopher MacGown
e189723fec Allow rebalance to take a seed.
Passing a seed into rebalance makes the rebalance deterministic
which allows us to generate identical rings across disparate
nodes without having to copy the ring files around.

Change-Id: Ie5ae46ac030e61284bc501fdef9d77eeb5243afd
2013-01-29 17:08:20 -08:00
Florian Hines
00dbad0825 Add optional locking to swift-ring-builder
If invoked as 'swift-ring-builder-safe' the directory containing the builder
file provided will be locked (via lock_parent_directory()). This provides a
small safe guard against multiple instances of the swift-ring-builder (or
other utilities that observe this lock) from attempting to write to or read
the builder/ring files while operations are in progress.

This is particularly useful in environments where ring management has been
automated (via Chef or custom solutions) but the operator still occasionally
needs to manually interact with the ring.

DocImpact

Change-Id: Ia362744a8151a91bfb586d01da582906726852e6
2013-01-25 08:00:33 -08:00
Florian Hines
c4f5761101 builder.add_devs gets next id if not provided
Have builder.add_devs get the next id to use when adding a new device
if its not specified in the dict.

Change-Id: I5a0defab43f5cfc5d997080bfd8563bfe72368ad
2012-09-14 16:11:50 -05:00
Florian Hines
c0537ac6e0 Breakout search_devs & add get_builder() for reuse
This moves search_devs into RingBuilder to make it accessible to other utils
that need to search the builder. Along the same lines this also adds a
load() call to get a RingBuilder instance when working with the builder files.

- This adds python-mock >= 0.7 as a dependency for unittests. On Ubuntu
  10.04 you'll have to pip install it, on 12.04 you can apt-get install
  it. Fedora 17+ should be able to yum install it.
- new pep8 compliance
- Fixed a small issue (undefined var) in swift-ring-builder when remove was
called but failed to find a match.

Change-Id: I2e02684235aa2f4e901a00858ae037091594c545
2012-09-06 20:16:46 -05:00
Darrell Bishop
f8ce43a218 Use custom encoding for RingData, not pickle.
Serialize RingData in a versioned, custom format which is a combination
of a JSON-encoded header and .tostring() dumps of the
replica2part2dev_id arrays.  This format deserializes hundreds of times
faster than rings serialized with Python 2.7's pickle (a significant
performance regression for ring loading between Python 2.6 and Python
2.7).  Fixes bug 1031954.

swift.common.ring.ring.RingData is now responsible for serialization and
deserialization of its data via a new load() class method and save()
object method.  The new implementation is backward-compatible; if a ring
does not begin with a new-style magic string, it is assumed to be an
old-style pickle-dumped ring and is handled as before.  So new Swift
code can read old rings, but old Swift code will not be able to read
newly-serialized rings. THIS SHOULD BE MENTIONED PROMINENTLY IN THE
RELEASE NOTES.

I didn't want to bite of more than necessary, so I didn't mess with
builder file serialization.

Change-Id: I799b9a4c894d54fb16592443904ac055b2638e2d
2012-08-05 00:51:49 -07:00
Samuel Merritt
0befccccdf Let some swift-ring-builder commands take >1 arg.
`swift-ring-builder X.builder [add|set_weight|set_info|remove]` can
now work on multiple devices at once.

This allows for faster bulk operations; for example, if an
administrator has to add 48 new devices to the ring, then this lets
them do it in one command invocation. Since each invocation of
swift-ring-builder follows a deserialize-operate-serialize pattern,
using the bulk form of the commands on N devices will save N-1
[de]serializations.

Change-Id: Ibf77528f28026866af3c94d0370181c8013d5bc5
2012-05-03 17:45:33 -07:00
John Dickinson
1ecf5ebba1 updated copyright date for all files
Change-Id: Ifd909d3561c2647770a7e0caa3cd91acd1b4f298
2012-03-19 13:45:34 -05:00
Mark Gius
2f50a0798e Add more specific error messages to swift-ring-builder
Replace existing Exceptions in ring builder with more specific exceptions.
Abstracted out some behavior in ring-builder that is likely to cause an
exception. Add try/except blocks to swift-ring-builder to catch specific
exceptions and provide the user with some information about how to deal
with the error.

This change begins to address blueprint friendly-error-messages

Change-Id: I8fc9cfa4899421fe04bba23ac52523778e902321
2011-09-27 10:20:51 -07:00
Mark Gius
1ad31af887 Changes swift-ring-builder exit codes
swift-ring-builder will now return zero for successful operations and
non-zero for operations that have errored or need attention.

Fixes bug 836922

Change-Id: I8d18a49fcf239f7cc7bfe4cdb468ac849436122e
2011-09-20 14:53:53 -07:00
gholt
2d3f409190 swift-ring-builder: Updated help for list_parts 2011-05-18 15:11:08 +00:00
gholt
a062abd593 PEP8 fix 2011-05-12 01:07:57 +00:00
gholt
97ba935ec3 Added list_parts command to swift-ring-builder which lists common partitions for a list of devices. 2011-05-12 01:01:38 +00:00
gholt
fc6391ea5c ring: pickles now use only stdlib objects; old and really old pickles can still be read 2011-02-15 18:43:55 -08:00
Michael Barton
c62842bfd1 update all ring-builder grammars 2011-02-03 19:50:16 +00:00
Michael Barton
67de0c88f4 ipv6 support 2011-01-16 09:52:08 +00:00
Russ Nelson
954dcf7c99 At least in theory, the code is not functionally different. It's just physically restructured internally so that the help strings are now docstrings within each command/function. There are other improvements possible, such as including test code in the docstring. I love Python! 2011-01-07 17:07:34 +00:00
Anne Gentle
8823427161 Changed copyright notices on py files and the single rst file with a copyright notice 2011-01-04 17:34:43 -06:00
gholt
68cc8fba9d Fix a couple bugs; rework the rework a bit more; PEP8 2011-01-04 12:04:18 -08:00
Russ Nelson
c97e7b6054 Refactor the command and help structure so it uses Python's docstrings. Also put the commands into a
class so they can be listed and their docstrings automatically printed.
2010-12-31 12:34:22 -05:00
gholt
57970bdeb5 Cleaned up the bins; patched the broken test (when run standalone) 2010-12-29 12:00:08 -08:00
Michael Barton
f432269013 add gettext to all binaries 2010-12-22 17:18:30 +00:00
Caleb Tennis
db90da2763 Remove the exception from the unit test, since we don't bomb out anymore. Also, add a warning to swift-ring-builder if you're building an empty ring, or do a write_ring and you aren't rebalanced 2010-08-21 18:21:59 +00:00
Cory Wright
f2b694b7ab Fixed a few typos in the deployment guide (s/ring_builder/ring-builder/g) 2010-08-13 11:19:43 -04:00
Monty Taylor
76e6c6ea6c Renamed bin files in the tree so that setup.py install does the same thing
as a deb install.
2010-07-14 09:58:38 -05:00