14 Commits

Author SHA1 Message Date
Samuel Merritt
ccf0758ef1 Add ring-builder analyzer.
This is a tool to help developers quantify changes to the ring
builder. It takes a scenario (JSON file) describing the builder's
basic parameters (part_power, replicas, etc.) and a number of
"rounds", where each round is a set of operations to perform on the
builder. For each round, the operations are applied, and then the
builder is rebalanced until it reaches a steady state.

The idea is that a developer observes the ring builder behaving
suboptimally, writes a scenario to reproduce the behavior, modifies
the ring builder to fix it, and references the scenario with the
commit so that others can see that things have improved.

I decided to write this after writing my fourth or fifth hacky one-off
script to reproduce some bad behavior in the ring builder.

Change-Id: I114242748368f142304aab90a6d99c1337bced4c
2015-07-02 08:16:03 -07:00
Christian Schwede
d3213fb1fe Check if device name is valid when adding to the ring
Currently device names can be empty or start and/or end with spaces.
This can create unexpected results, for example these three commands
are all valid:

swift-ring-builder account.builder add "r1z1-127.0.0.1:6000/" 1
swift-ring-builder account.builder add "r1z1-127.0.0.1:6000/sda " 1
swift-ring-builder account.builder add "r1z1-127.0.0.1:6000/ meta" 1

This patch validates device names and prevents empty names or names
starting and/or ending with spaces.

Also fixed the test "test_warn_at_risk" - the test passed if the
exception was not raised.

Closes-Bug: 1438579

Change-Id: I811b0eae7db503279e6429d985275bbab8b29c9f
2015-04-14 13:15:15 -07:00
Samuel Merritt
8d3b3b2ee0 Add some debug output to the ring builder
Sometimes, I get handed a builder file in a support ticket and a
question of the form "why is the balance [not] doing $thing?". When
that happens, I add a bunch of print statements to my local
swift/common/ring/builder.py, figure things out, and then delete the
print statements. This time, instead of deleting the print statements,
I turned them into debug() calls and added a "--debug" flag to the
rebalance command in hopes that someone else will find it useful.

Change-Id: I697af90984fa5b314ddf570280b4585ba0ba363c
2015-03-30 17:47:28 -07:00
Jenkins
bc7c496f71 Merge "Allow hostnames for nodes in Rings" 2015-02-10 04:32:38 +00:00
Hisashi Osanai
efb39a5665 Allow hostnames for nodes in Rings
This change modifies the swift-ring-builder and introduces new format
of sub-commands (search, list_parts, set_weight, set_info and remove)
in addition to add sub-command so that hostnames can be used in place
of an ip-address for the sub-commands.
The account reaper, container synchronizer, and replicators were also
updated so that they still have a way to identify a particular device
as being "local".

Previously this was Change-Id:
Ie471902413002872fc6755bacd36af3b9c613b74

Change-Id: Ieff583ffb932133e3820744a3f8f9f491686b08d
Co-Authored-By: Alex Pecoraro <alex.pecoraro@emc.com>
Implements: blueprint allow-hostnames-for-nodes-in-rings
2015-02-02 05:06:03 +09:00
sarvesh-ranjan
d8fdbc2b2d Typos fixed
Change-Id: I2c216a870ce299039dec9948dcdef3de0721b4da
2015-01-29 18:20:31 -08:00
Jenkins
df529a225f Merge "Allow set_overload to take value as percent" 2015-01-29 06:19:22 +00:00
Pete Zaitcev
562f7e8906 Allow set_overload to take value as percent
...and output overload as a percent like dispersion and balance.

Also raise a warning if someone tries to set overload higher than 100%
(unless the specifically requested a percent value great than 100).

Change-Id: Id030123153ea746671a8f1ca306d4b86e903fa22
2015-01-28 13:49:55 -08:00
Clay Gerrard
376dc5adc3 don't print cached dispersion if it's a lie
Change-Id: I551fcaf274876861feb12848749590f220842d68
2015-01-27 10:19:41 -08:00
Clay Gerrard
a8bd2f737c Add dispersion command to swift-ring-builder
Output a dispersion report that shows how many parts have each replica count
at each tier along with some additional context.  Also the max_dispersion is a
good canary for what a reasonable overload might be.

Also display a warning on rebalance if the ring's dispersion is sub-optimal.

The primitive form of the dispersion graph is cached on the builder, but the
dispersion command will build it on the fly if you have a ring that was last
rebalanced before the change.

Also add --force option to rebalance to make it write a ring even if less than
1% of parts moved.

Try to clarify some dispersion and balance a little bit in the ring section of
the architectural overview.

Co-Authored-By: Christian Schwede <christian.schwede@enovance.com>
Co-Authored-By: Darrell Bishop <darrell@swiftstack.com>

Change-Id: I7696df25d092fac56588080722e0a4167ed2c824
2015-01-08 18:40:27 -08:00
Samuel Merritt
bcf26f5209 Add notion of overload to swift-ring-builder
The ring builder's placement algorithm has two goals: first, to ensure
that each partition has its replicas as far apart as possible, and
second, to ensure that partitions are fairly distributed according to
device weight. In many cases, it succeeds in both, but sometimes those
goals conflict. When that happens, operators may want to relax the
rules a little bit in order to reach a compromise solution.

Imagine a cluster of 3 nodes (A, B, C), each with 20 identical disks,
and using 3 replicas. The ring builder will place 1 replica of each
partition on each node, as you'd expect.

Now imagine that one disk fails in node C and is removed from the
ring. The operator would probably be okay with remaining at 1 replica
per node (unless their disks are really close to full), but to
accomplish that, they have to multiply the weights of the other disks
in node C by 20/19 to make C's total weight stay the same. Otherwise,
the ring builder will move partitions around such that some partitions
have replicas only on nodes A and B.

If 14 more disks failed in node C, the operator would probably be okay
with some data not living on C, as a 4x increase in storage
requirements is likely to fill disks.

This commit introduces the notion of "overload": how much extra
partition space can be placed on each disk *over* what the weight
dictates.

For example, an overload of 0.1 means that a device can take up to 10%
more partitions than its weight would imply in order to make the
replica dispersion better.

Overload only has an effect when replica-dispersion and device weights
come into conflict.

The overload is a single floating-point value for the builder
file. Existing builders get an overload of 0.0, so there will be no
behavior change on existing rings.

In the example above, imagine the operator sets an overload of 0.112
on his rings. If node C loses a drive, each other drive can take on up
to 11.2% more data. Splitting the dead drive's partitions among the
remaining 19 results in a 5.26% increase, so everything that was on
node C stays on node C. If another disk dies, then we're up to an
11.1% increase, and so everything still stays on node C. If a third
disk dies, then we've reached the limits of the overload, so some
partitions will begin to reside solely on nodes A and B.

DocImpact

Change-Id: I3593a1defcd63b6ed8eae9c1c66b9d3428b33864
2015-01-07 14:16:08 -08:00
Hisashi Osanai
d742b610df Fix the behavior of swift-ring-builder list_parts before rebalance
The swift-ring-builder list_parts before rebalance failed abnormally so
this patch fix the behavior. After this patch applies the behavior is
completion normally with the following messages.

Specified builder file "<builder_file>" is not rebalanced yet.
Please rebalance first.

Closes-Bug: #1399529
Change-Id: I9e5db6da85de4188915c51bc401604733f0e1b77
2014-12-06 02:44:59 +09:00
Keshava Bharadwaj
38ba5790fb Provides proper error handling on builder unpickle
This patch provides the necessary error handling while unpickling
a builder file. Earlier if a builder file is empty/invalid/corrupted,
the stacktrace was shown to user with an exit code of 1. This fixes it
to show a user-friendly message and also returns the exit code of 2,
indicating there was a failure.

Change-Id: I51eb24702c422299629f8053d4591dd10f5863f8
Closes-Bug: #1370680
2014-09-26 09:44:35 +05:30
Christian Schwede
24657b2b39 Add tests for swift-ring-builder
Add some tests for essential methods in swift-ring-builder.
Tests for removing or changing device settings are executed
with different search values to cover many possible command
line arguments.

Currently tested methods:

- create ring
- add device
- remove device
- set weight
- set info
- set min_part_hours
- set replicas

Tests use swift.common.ring.RingBuilder to verify actions.

Catching and testing output from print statements is not
tested, because this requires redirecting sys.stdout during
tests and that might have some sideeffects for testing tools.

bin/swift-ring-builder has been moved to swift/cli/ringbuilder.py
and slightly modified to work as before (mainly due to no more
existing global variables since that part of the code has been
moved inside a main() function).

Change-Id: Ia63f59a8faca1fad990784f27532ca07a2125454
2014-02-05 16:20:09 +00:00