Last time we did this was nearly 4 years ago; drag ourselves into
something approaching the present. Address a few new pyflakes issues
that seem reasonable to enforce:
E275 missing whitespace after keyword
E231 missing whitespace after ','
E721 do not compare types, for exact checks use `is` / `is not`,
for instance checks use `isinstance()`
Main motivator is that the old hacking kept us on an old version
of flake8 et al., which no longer work with newer Pythons.
Change-Id: I54b46349fabb9776dcadc6def1cfb961c123aaa0
Previously, when fetching a shard range from a container DB using
ContainerBroker.get_own_shard_range(), the stats of the returned shard
range were updated as a side effect. However, the stats were not
persisted in the own shard range row in the DB. Often the extra DB
queries to get the stats are unnecessary because we don't need
up-to-date stats in the returned shard range. The behavior also leads
to potential confusion because the object stats of the returned shard
range object are not necessarily consistent with the object stats of
the same shard range in the DB.
This patch therefore removes the stats updating behavior from
get_own_shard_range() and makes the stats updating happen as an
explicit separate operation, when needed. This is also more consistent
with how the tombstone count is updated.
Up-to-date own shard range stats are persisted when a container is
first enabled for sharding, and then each time a shard container
reports its stats to the root container.
Change-Id: Ib10ef918c8983ca006a3740db8cfd07df2dfecf7
Some subcommands of 'swift-manage-shard-ranges' prompt a user before
taking action. For example, 'find_and_replace --enable' will warn a
user and ask for input if existing shard ranges are found. This patch
improves user input handling to exit gracefully (with err code 3) if
either an EOFError or KeyboardInterrupt is raised while waiting for
user input.
Co-Authored-By: Tim Burke <tim.burke@gmail.com>
Change-Id: I75c8acd372d449b8a4531ff816d839dfe40d8a56
We've seen shards become stuck while sharding because they had
incomplete or stale deleted shard ranges. The root container had more
complete and useful shard ranges into which objects could have been
cleaved, but the shard never merged the root's shard ranges.
While the sharder is auditing shard container DBs it would previously
only merge shard ranges fetched from root into the shard DB if the
shard was shrinking or the shard ranges were known to be children of
the shard. With this patch the sharder will now merge other shard
ranges from root during sharding as well as shrinking.
Shard ranges from root are only merged if they would not result in
overlaps or gaps in the set of shard ranges in the shard DB. Shard
ranges that are known to be ancestors of the shard are never merged,
except the root shard range which may be merged into a shrinking
shard. These checks were not previously applied when merging
shard ranges into a shrinking shard.
The two substantive changes with this patch are therefore:
- shard ranges from root are now merged during sharding,
subject to checks.
- shard ranges from root are still merged during shrinking,
but are now subjected to checks.
Change-Id: I066cfbd9062c43cd9638710882ae9bd85a5b4c37
Stuck shard ranges have been seen in the production, root cause has been
traced back to that s-m-s-r failed to detect parent-child relationship
in overlaps and it either shrinked child shard ranges into parents or
the other way around. A patch has been added to check minimum age before
s-m-s-r performs repair, which will most likely prevent this from
happening again, but we also need to check for parent-child relationship
in overlaps explicitly during repairs. This patch will do that and
remove parent or child shard ranges from doners, and prevent s-m-s-r
from shrinking them into acceptor shard ranges.
Drive-by 1: fixup gap repair probe test.
The probe test is no longer appropriate because we're no longer
allowed to repair parent-child overlaps, so replace the test with a
manually created gap.
Drive-by 2: address probe test TODOs.
The commented assertion would fail because the node filtering
comparison failed to account for the same node having different indexes
when generated for the root versus the shard. Adding a new iterable
function filter_nodes makes the node filtering behave as expected.
Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Change-Id: Iaa89e94a2746ba939fb62449e24bdab9666d7bab
Adds a subcommand to swift-manage-shard-ranges that allows arbitrary
shard ranges to be read from a file and merged into a container
DB. The file should be a JSON serialized list of dicts.
The merge subcommand is only recommended for emergency shard range
manipulation by expert users.
Change-Id: Ic9ffcc042399f3834027a7935b64292d1fffe8d5
Add an option for the swift-manage-shard-ranges repair tool to ignore
overlaps where the overlapping shards appear to be recently created,
since this may indicate that they are shards of the parent with
which they overlap.
The new option is --min-shard-age with a default of 14400 seconds.
Change-Id: Ib82219a506732303a1157c2c9e1c452b4a56061b
Add checks that more of the sharder config values make sense with
respect to each other.
For the daemon, all config options are checked. For
swift-manage-shard-ranges only those options relevant to the current
subcommand are checked.
Change-Id: I2d4dab1d56c31f904161e5338f1bb89a03fd88f7
A container is typically sharded when it has grown to have an object
count of shard_container_threshold + N, where N <<
shard_container_threshold. If sharded using the default
rows_per_shard of shard_container_threshold / 2 then this would
previously result in 3 shards: the tail shard would typically be
small, having only N rows. This behaviour caused more shards to be
generated than desirable.
This patch adds a minimum-shard-size option to
swift-manage-shard-ranges, and a corresponding option in the sharder
config, which can be used to avoid small tail shards. If set to
greater than one then the final shard range may be extended to more
than rows_per_shard in order to avoid a further shard range with less
than minimum-shard-size rows. In the example given, if
minimum-shard-size is set to M > N then the container would shard into
two shards having rows_per_shard rows and rows_per_shard + N
respectively.
The default value for minimum-shard-size is rows_per_shard // 5. If
all options have their default values this results in
minimum-shard-size being 100000.
Closes-Bug: #1928370
Co-Authored-By: Matthew Oliver <matt@oliver.net.au>
Change-Id: I3baa278c6eaf488e3f390a936eebbec13f2c3e55
Make rows_per_shard an option that can be configured
in the [container-sharder] section of a config file.
For auto-sharding, this option was previously hard-coded to
shard_container_threshold // 2.
The swift-manage-shard-ranges command line tool already supported
rows_per_shard on the command line and will now also load it from a
config file if specified. Any value given on the command line takes
precedence over any value found in a config file.
Change-Id: I820e133a4e24400ed1e6a87ebf357f7dac463e38
Add two new sharder config options for configuring shrinking
behaviour:
- shrink_threshold: the size below which a shard may shrink
- expansion_limit: the maximum size to which an acceptor shard
may grow
The new options match the 'swift-manage-shard-ranges' command line
options and take absolute values.
The new options provide alternatives to the current equivalent options
'shard_shrink_point' and 'shard_shrink_merge_point', which are
expressed as percentages of 'shard_container_threshold'.
'shard_shrink_point' and 'shard_shrink_merge_point' are deprecated and
will be overridden by the new options if the new options are
explicitly set in a config file.
The default values of the new options are the same as the values that
would result from the default 'shard_container_threshold',
'shard_shrink_point' and 'shard_shrink_merge_point' i.e.:
- shrink_threshold: 100000
- expansion_limit: 750000
Change-Id: I087eac961c1eab53540fe56be4881e01ded1f60e
Change the swift-manage-shard-ranges default expansion-limit to equal
the sharder daemon default merge_size i.e 750000. The previous default
of 500000 had erroneously differed from the sharder default value.
Introduce a ContainerSharderConf class to encapsulate loading of
sharder conf and the definition of defaults. ContainerSharder inherits
this and swift-manage-shard-ranges instantiates it.
Rename ContainerSharder member vars to match the equivalent vars and
cli options in manage_shard_ranges:
shrink_size -> shrink_threshold
merge_size -> expansion_limit
split_size -> rows_per_shard
(This direction of renaming is chosen so that the manage_shard_ranges
cli options are not changed.)
Rename ContainerSharder member vars to match the conf file option name:
scanner_batch_size -> shard_scanner_batch_size
Remove some ContainerSharder member vars that were not used outside of
the __init__ method:
shrink_merge_point
shard_shrink_point
Change-Id: I8a58a82c08ac3abaddb43c11d26fda9fb45fe6c1
After individual compactible sequences have been listed, add a summary
line such as:
Total of 123 shard sequences identified for compaction.
Change-Id: I087c2c1885f86ff8163ce358e25dc3c092897223
Previously a shard might be shrunk if its object_count was fell below
the shrink_threshold. However, it is possible that a shard with few
objects has a large number of tombstones, which would result in a
larger than anticipated replication of rows to the acceptor shard.
With this patch, a shard's row count (i.e. the sum of tombstones and
objects) must be below the shrink_threshold before the shard will be
considered for shrinking.
A number of changes are made to enable tombstone count to be used in
shrinking decisions:
- DatabaseBroker reclaim is enhanced to count remaining tombstones
after rows have been reclaimed. A new TombstoneReclaimer class is
added to encapsulate the reclaim process and tombstone count.
- ShardRange has new 'tombstones' and 'row_count' attributes.
- A 'tombstones' column is added to the Containerbroker shard_range
table.
- The sharder performs a reclaim prior to reporting shard container
stats to the root container so that the tombstone count can be
included.
- The sharder uses 'row_count' rather than 'object_count' when
evaluating if a shard range is a shrink candidate.
Change-Id: I41b86c19c243220b7f1c01c6ecee52835de972b6
Previously swift-manage-shard-ranges would variously return either 1
or 2 in cases of invalid arguments, errors or unexpected outcomes, or
if the user chose to quit before a change was applied. This patch
applies a more consistent pattern to the exit codes:
0 = success
1 = an unexpected outcome, including errors
2 = invalid command line arguments or conf file options
3 = user quit
Some errors that previously resulted in an exit code 2 will now exit
with code 1.
Change-Id: Icf170fef26ed36aab3bf845c5560f1e579a69c2b
swift-container-info does not currently include 'epoch' among the
shard range info that it outputs, which prevents shard data scraped
from that tool being read by the swift-manage-shard-ranges 'analyze'
command. The 'epoch' key is not significant to the 'analyze' results
so it is safe to add a default value when it is missing.
Change-Id: I7be30c3247a53969a17fed63d0eba5ab4f4027e4
Adds a repair command that reads shard ranges from the container DB,
identifies overlapping shard ranges and recommends how overlaps can be
repaired by shrinking into a single chosen path. Prompted user input
or a '-y' option will then cause the appropriately modified shard
ranges to be merged into the container DB.
The repair command does not fix gaps in shard range paths and can
therefore only succeed if there is at least one unbroken path of
shards through the entire namespace.
Also adds an analyze command that loads shard data from a file and
reports overlapping shard ranges, but takes no action. The analyze
command is similar to the repair command but reads shard data from
file rather than a container db and makes no changes to the db.
e.g.:
swift-manage-shard-ranges <db-file-name> repair
swift-manage-shard-ranges <shard-data.json> analyze
to see more detail:
swift-manage-shard-ranges -v <shard-data.json> analyze
For consistency with the new repair command, and to be more cautious,
this patch changes the user input required to apply the compact
command changes to the DB from 'y' to 'yes'.
Change-Id: I9ec411462e4aaf9f21aba6c5fd7698ff75a07de3
Currently when you get to the final collapse into root with the compact
command, it can look a little screwy:
$ swift-manage-shard-ranges --conf /etc/swift/container-server/1.conf /mnt/sdb1/1/node/sdb1/containers/925/c2a/e77a449c51d01ef47471efb208c20c2a/e77a449c51d01ef47471efb208c20c2a.db compact
Loaded db broker for AUTH_admin/shardme.
Donor shard range(s) with total of 0 objects:
objects: 0 lower: ''
state: active upper: ''
can be compacted into acceptor shard range:
objects: 0 lower: ''
state: sharded upper: ''
Once applied to the broker these changes will result in shard range compaction the next time the
sharder runs.
But it's fixed if we add the name to the compact output. It also makes
gives it just a little bit more useful information:
$ swift-manage-shard-ranges --conf /etc/swift/container-server/1.conf /mnt/sdb1/1/node/sdb1/containers/925/c2a/e77a449c51d01ef47471efb208c20c2a/e77a449c51d01ef47471efb208c20c2a.db compact
Loaded db broker for AUTH_admin/shardme.
Donor shard range(s) with total of 0 objects:
'.shards_AUTH_admin/shardme-7820d6e9550a1661e01e0538aea8cc1b-1614319760.63229-5'
objects: 0 lower: ''
state: active upper: ''
can be compacted into acceptor shard range:
'AUTH_admin/shardme'
objects: 0 lower: ''
state: sharded upper: ''
Once applied to the broker these changes will result in shard range compaction the next time the sharder runs.
Do you want to apply these changes? [y/N]
This is what a non root collapse would look like with more ranges:
$ swift-manage-shard-ranges --conf /etc/swift/container-server/1.conf /mnt/sdb1/1/node/sdb1/containers/925/c2a/e77a449c51d01ef47471efb208c20c2a/e77a449c51d01ef47471efb208c20c2a.db compact
Loaded db broker for AUTH_admin/shardme.
Donor shard range(s) with total of 0 objects:
'.shards_AUTH_admin/shardme-7820d6e9550a1661e01e0538aea8cc1b-1614565810.95419-0'
objects: 0 lower: ''
state: active upper: 'e9'
can be compacted into acceptor shard range:
'.shards_AUTH_admin/shardme-7820d6e9550a1661e01e0538aea8cc1b-1614565810.95419-1'
objects: 0 lower: 'e9'
state: active upper: 'j9'
Donor shard range(s) with total of 0 objects:
'.shards_AUTH_admin/shardme-7820d6e9550a1661e01e0538aea8cc1b-1614565810.95419-2'
objects: 0 lower: 'j9'
state: active upper: 'o9'
can be compacted into acceptor shard range:
'.shards_AUTH_admin/shardme-7820d6e9550a1661e01e0538aea8cc1b-1614565810.95419-3'
objects: 0 lower: 'o9'
state: active upper: 't9'
Donor shard range(s) with total of 0 objects:
'.shards_AUTH_admin/shardme-7820d6e9550a1661e01e0538aea8cc1b-1614565810.95419-4'
objects: 0 lower: 't9'
state: active upper: 'y9'
can be compacted into acceptor shard range:
'.shards_AUTH_admin/shardme-7820d6e9550a1661e01e0538aea8cc1b-1614565810.95419-5'
objects: 0 lower: 'y9'
state: active upper: ''
Once applied to the broker these changes will result in shard range compaction the next time the sharder runs.
Do you want to apply these changes? [y/N]
Change-Id: Ie8e2b03f93ace0283c7a759d2404022208c0c553
While working on the shrinking recon drops, we want to display numbers
that directly relate to how tool should behave. But currently all
options of the s-m-s-r tool is driven by cli options.
This creates a disconnect, defining what should be used in the sharder
and in the tool via options are bound for failure. It would be much
better to be able to define the required default options for your
environment in one place that both the sharder and tool could use.
This patch does some refactoring and adding max_shrinking and
max_expanding options to the sharding config. As well as adds a
--config option to the tool.
The --config option expects a config with at '[container-sharder]'
section. It only supports the shard options:
- max_shrinking
- max_expanding
- shard_container_threshold
- shard_shrink_point
- shard_merge_point
The latter 2 are used to generate the s-m-s-r's:
- shrink_threshold
- expansion_limit
- rows_per_shard
Use of cli arguments take precedence over that of the config.
Change-Id: I4d0147ce284a1a318b3cd88975e060956d186aec
The sharder find_compactible_shard_sequences function was vulnerable
to looping with some combinations of shrink_threshold and merge_size
parameters. The inner loop might not consume a shard range, resulting
in the same shard range being submitted to the inner loop again.
This patch simplifies the function in an attempt to make it more
obvious that the loops are always making progress towards termination
by consuming shard ranges from the list.
Change-Id: Ia87ab6feaf5172d91f1c60c2e0f72e03182e3c9b
The output from swift-manage-shard-ranges compact was hard to read,
particularly with long lower and upper bounds typically seen in
production. This patch changes the output, most significantly by
aligning lower and upper bounds at the end of lines and printing
donors before acceptors, so that the continuity of shard range bounds
is easier to see.
Example of new ouput:
Donor shard range(s) with total of 2 objects:
objects: 2 lower: ''
state: active upper: 'obj-0001'
can be compacted into acceptor shard range:
objects: 2 lower: 'obj-0001'
state: active upper: 'obj-0003'
Donor shard range(s) with total of 2 objects:
objects: 2 lower: 'obj-0003'
state: active upper: 'obj-0005'
can be compacted into acceptor shard range:
objects: 2 lower: 'obj-0005'
state: active upper: ''
Change-Id: I90f3ab7a29b87d0959c94a05c9ee9ad97b60014d
If a sequence of shard ranges is already shrinking then in some
circumstances we do not want to report it as a candidate for
shrinking. For backwards compatibility allow already shrinking
sequences to be optionally included in return value of
find_compactible_shard_sequences.
Also refactor to add an is_shrinking_candidate() function.
Change-Id: Ifa20b7c08aba7254185918dfcee69e8206f51cea
Fix the find_compactible_shard_sequences function to prevent skipping
a shard range after finding a sequence of compactible shard ranges
that approaches the merge size.
Previously a compactible sequence would correctly terminate on the nth
shard range if the n+1th shard range would take the object count over
the merge_size, but the n+1th shard range would then be skipped and
not considered for the start of the next sequence.
Change-Id: I670441e7426b28ab2247563c7fa854d1cd502316
This patch adds a 'compact' command to swift-manage-shard-ranges that
enables sequences of contiguous shards with low object counts to be
compacted into another existing shard, or into the root container.
Change-Id: Ia8f3297d610b5a5cf5598d076fdaf30211832366
The idea is, if none of
- timestamp,
- object_count,
- bytes_used,
- state, or
- epoch
has changed, we shouldn't need to send an update back to the root
container.
This is more-or-less comparable to what the container-updater does to
avoid unnecessary writes to the account.
Closes-Bug: #1834097
Change-Id: I1ee7ba5eae3c508064714c4deb4f7c6bbbfa32af
When an operator does a `find_and_replace` on a DB that already has
shard ranges, they get a prompt like:
This will delete existing 58 shard ranges.
Do you want to show the existing ranges [s], delete the existing
ranges [yes] or quit without deleting [q]?
Previously, if they selected `q`, we would skip the delete but still do
the merge (!) and immediately warn about how there are now invalid shard
ranges. Now, quit without merging.
Change-Id: I7d869b137a6fbade59bb8ba16e4f3e9663e18822
This is a follow up patch from the cleaning up cleave context's patch
(patch 681970). Instead of tracking a last_modified timestamp, and storing
it in the context metadata, use the timestamp we use when storing any
metadata.
Reducing duplication is nice, but there's a more significant reason to
do this: affected container DBs can start getting cleaned up as soon as
they're running the new code rather than needing to wait for an
additional reclaim_age.
Change-Id: I2cdbe11f06ffb5574e573c4a60ba4e5d41a00c50
There is a sharding edge case where more CleaveContext are generated and
stored in the sharding container DB. If this number get's high enough,
like in the linked bug. If enough CleaveContects build up in the DB then
this can lead to the 503's when attempting to list the container due to
all the `X-Container-Sysmeta-Shard-Context-*` headers.
This patch resolves this by tracking the a CleaveContext's last
modified. And during the sharding audit, any context's that hasn't been
touched after reclaim_age are deleted.
This plus the skip empty ranges patches should improve these handoff
shards.
Change-Id: I1e502c328be16fca5f1cca2186b27a0545fecc16
Closes-Bug: #1843313
This started with ShardRanges and its CLI. The sharder is at the
bottom of the dependency chain. Even container backend needs it.
Once we started tinkering with the sharder, it all snowballed to
include the rest of the container services.
Beware, this does affect some of Python 2 code. Mostly it's trivial
and obviously correct, but needs checking by reviewers.
About killing the stray "from __future__ import unicode_literals":
we do not do it in general. The specific problem it caused was
a failure of functional tests because unicode leaked into a field
that was supposed to be encoded. It is just too hard to track the
types when rules change from file to file, so off with its head.
Change-Id: Iba4e65d0e46d8c1f5a91feb96c2c07f99ca7c666
The sharder daemon visits container dbs and when necessary executes
the sharding workflow on the db.
The workflow is, in overview:
- perform an audit of the container for sharding purposes.
- move any misplaced objects that do not belong in the container
to their correct shard.
- move shard ranges from FOUND state to CREATED state by creating
shard containers.
- move shard ranges from CREATED to CLEAVED state by cleaving objects
to shard dbs and replicating those dbs. By default this is done in
batches of 2 shard ranges per visit.
Additionally, when the auto_shard option is True (NOT yet recommeneded
in production), the sharder will identify shard ranges for containers
that have exceeded the threshold for sharding, and will also manage
the sharding and shrinking of shard containers.
The manage_shard_ranges tool provides a means to manually identify
shard ranges and merge them to a container in order to trigger
sharding. This is currently the recommended way to shard a container.
Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Co-Authored-By: Tim Burke <tim.burke@gmail.com>
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Change-Id: I7f192209d4d5580f5a0aa6838f9f04e436cf6b1f
