Currently when you get to the final collapse into root with the compact
command, it can look a little screwy:
$ swift-manage-shard-ranges --conf /etc/swift/container-server/1.conf /mnt/sdb1/1/node/sdb1/containers/925/c2a/e77a449c51d01ef47471efb208c20c2a/e77a449c51d01ef47471efb208c20c2a.db compact
Loaded db broker for AUTH_admin/shardme.
Donor shard range(s) with total of 0 objects:
objects: 0 lower: ''
state: active upper: ''
can be compacted into acceptor shard range:
objects: 0 lower: ''
state: sharded upper: ''
Once applied to the broker these changes will result in shard range compaction the next time the
sharder runs.
But it's fixed if we add the name to the compact output. It also makes
gives it just a little bit more useful information:
$ swift-manage-shard-ranges --conf /etc/swift/container-server/1.conf /mnt/sdb1/1/node/sdb1/containers/925/c2a/e77a449c51d01ef47471efb208c20c2a/e77a449c51d01ef47471efb208c20c2a.db compact
Loaded db broker for AUTH_admin/shardme.
Donor shard range(s) with total of 0 objects:
'.shards_AUTH_admin/shardme-7820d6e9550a1661e01e0538aea8cc1b-1614319760.63229-5'
objects: 0 lower: ''
state: active upper: ''
can be compacted into acceptor shard range:
'AUTH_admin/shardme'
objects: 0 lower: ''
state: sharded upper: ''
Once applied to the broker these changes will result in shard range compaction the next time the sharder runs.
Do you want to apply these changes? [y/N]
This is what a non root collapse would look like with more ranges:
$ swift-manage-shard-ranges --conf /etc/swift/container-server/1.conf /mnt/sdb1/1/node/sdb1/containers/925/c2a/e77a449c51d01ef47471efb208c20c2a/e77a449c51d01ef47471efb208c20c2a.db compact
Loaded db broker for AUTH_admin/shardme.
Donor shard range(s) with total of 0 objects:
'.shards_AUTH_admin/shardme-7820d6e9550a1661e01e0538aea8cc1b-1614565810.95419-0'
objects: 0 lower: ''
state: active upper: 'e9'
can be compacted into acceptor shard range:
'.shards_AUTH_admin/shardme-7820d6e9550a1661e01e0538aea8cc1b-1614565810.95419-1'
objects: 0 lower: 'e9'
state: active upper: 'j9'
Donor shard range(s) with total of 0 objects:
'.shards_AUTH_admin/shardme-7820d6e9550a1661e01e0538aea8cc1b-1614565810.95419-2'
objects: 0 lower: 'j9'
state: active upper: 'o9'
can be compacted into acceptor shard range:
'.shards_AUTH_admin/shardme-7820d6e9550a1661e01e0538aea8cc1b-1614565810.95419-3'
objects: 0 lower: 'o9'
state: active upper: 't9'
Donor shard range(s) with total of 0 objects:
'.shards_AUTH_admin/shardme-7820d6e9550a1661e01e0538aea8cc1b-1614565810.95419-4'
objects: 0 lower: 't9'
state: active upper: 'y9'
can be compacted into acceptor shard range:
'.shards_AUTH_admin/shardme-7820d6e9550a1661e01e0538aea8cc1b-1614565810.95419-5'
objects: 0 lower: 'y9'
state: active upper: ''
Once applied to the broker these changes will result in shard range compaction the next time the sharder runs.
Do you want to apply these changes? [y/N]
Change-Id: Ie8e2b03f93ace0283c7a759d2404022208c0c553
While working on the shrinking recon drops, we want to display numbers
that directly relate to how tool should behave. But currently all
options of the s-m-s-r tool is driven by cli options.
This creates a disconnect, defining what should be used in the sharder
and in the tool via options are bound for failure. It would be much
better to be able to define the required default options for your
environment in one place that both the sharder and tool could use.
This patch does some refactoring and adding max_shrinking and
max_expanding options to the sharding config. As well as adds a
--config option to the tool.
The --config option expects a config with at '[container-sharder]'
section. It only supports the shard options:
- max_shrinking
- max_expanding
- shard_container_threshold
- shard_shrink_point
- shard_merge_point
The latter 2 are used to generate the s-m-s-r's:
- shrink_threshold
- expansion_limit
- rows_per_shard
Use of cli arguments take precedence over that of the config.
Change-Id: I4d0147ce284a1a318b3cd88975e060956d186aec
The sharder find_compactible_shard_sequences function was vulnerable
to looping with some combinations of shrink_threshold and merge_size
parameters. The inner loop might not consume a shard range, resulting
in the same shard range being submitted to the inner loop again.
This patch simplifies the function in an attempt to make it more
obvious that the loops are always making progress towards termination
by consuming shard ranges from the list.
Change-Id: Ia87ab6feaf5172d91f1c60c2e0f72e03182e3c9b
The output from swift-manage-shard-ranges compact was hard to read,
particularly with long lower and upper bounds typically seen in
production. This patch changes the output, most significantly by
aligning lower and upper bounds at the end of lines and printing
donors before acceptors, so that the continuity of shard range bounds
is easier to see.
Example of new ouput:
Donor shard range(s) with total of 2 objects:
objects: 2 lower: ''
state: active upper: 'obj-0001'
can be compacted into acceptor shard range:
objects: 2 lower: 'obj-0001'
state: active upper: 'obj-0003'
Donor shard range(s) with total of 2 objects:
objects: 2 lower: 'obj-0003'
state: active upper: 'obj-0005'
can be compacted into acceptor shard range:
objects: 2 lower: 'obj-0005'
state: active upper: ''
Change-Id: I90f3ab7a29b87d0959c94a05c9ee9ad97b60014d
If a sequence of shard ranges is already shrinking then in some
circumstances we do not want to report it as a candidate for
shrinking. For backwards compatibility allow already shrinking
sequences to be optionally included in return value of
find_compactible_shard_sequences.
Also refactor to add an is_shrinking_candidate() function.
Change-Id: Ifa20b7c08aba7254185918dfcee69e8206f51cea
Fix the find_compactible_shard_sequences function to prevent skipping
a shard range after finding a sequence of compactible shard ranges
that approaches the merge size.
Previously a compactible sequence would correctly terminate on the nth
shard range if the n+1th shard range would take the object count over
the merge_size, but the n+1th shard range would then be skipped and
not considered for the start of the next sequence.
Change-Id: I670441e7426b28ab2247563c7fa854d1cd502316
This patch adds a 'compact' command to swift-manage-shard-ranges that
enables sequences of contiguous shards with low object counts to be
compacted into another existing shard, or into the root container.
Change-Id: Ia8f3297d610b5a5cf5598d076fdaf30211832366
The idea is, if none of
- timestamp,
- object_count,
- bytes_used,
- state, or
- epoch
has changed, we shouldn't need to send an update back to the root
container.
This is more-or-less comparable to what the container-updater does to
avoid unnecessary writes to the account.
Closes-Bug: #1834097
Change-Id: I1ee7ba5eae3c508064714c4deb4f7c6bbbfa32af
When an operator does a `find_and_replace` on a DB that already has
shard ranges, they get a prompt like:
This will delete existing 58 shard ranges.
Do you want to show the existing ranges [s], delete the existing
ranges [yes] or quit without deleting [q]?
Previously, if they selected `q`, we would skip the delete but still do
the merge (!) and immediately warn about how there are now invalid shard
ranges. Now, quit without merging.
Change-Id: I7d869b137a6fbade59bb8ba16e4f3e9663e18822
This is a follow up patch from the cleaning up cleave context's patch
(patch 681970). Instead of tracking a last_modified timestamp, and storing
it in the context metadata, use the timestamp we use when storing any
metadata.
Reducing duplication is nice, but there's a more significant reason to
do this: affected container DBs can start getting cleaned up as soon as
they're running the new code rather than needing to wait for an
additional reclaim_age.
Change-Id: I2cdbe11f06ffb5574e573c4a60ba4e5d41a00c50
There is a sharding edge case where more CleaveContext are generated and
stored in the sharding container DB. If this number get's high enough,
like in the linked bug. If enough CleaveContects build up in the DB then
this can lead to the 503's when attempting to list the container due to
all the `X-Container-Sysmeta-Shard-Context-*` headers.
This patch resolves this by tracking the a CleaveContext's last
modified. And during the sharding audit, any context's that hasn't been
touched after reclaim_age are deleted.
This plus the skip empty ranges patches should improve these handoff
shards.
Change-Id: I1e502c328be16fca5f1cca2186b27a0545fecc16
Closes-Bug: #1843313
This started with ShardRanges and its CLI. The sharder is at the
bottom of the dependency chain. Even container backend needs it.
Once we started tinkering with the sharder, it all snowballed to
include the rest of the container services.
Beware, this does affect some of Python 2 code. Mostly it's trivial
and obviously correct, but needs checking by reviewers.
About killing the stray "from __future__ import unicode_literals":
we do not do it in general. The specific problem it caused was
a failure of functional tests because unicode leaked into a field
that was supposed to be encoded. It is just too hard to track the
types when rules change from file to file, so off with its head.
Change-Id: Iba4e65d0e46d8c1f5a91feb96c2c07f99ca7c666
The sharder daemon visits container dbs and when necessary executes
the sharding workflow on the db.
The workflow is, in overview:
- perform an audit of the container for sharding purposes.
- move any misplaced objects that do not belong in the container
to their correct shard.
- move shard ranges from FOUND state to CREATED state by creating
shard containers.
- move shard ranges from CREATED to CLEAVED state by cleaving objects
to shard dbs and replicating those dbs. By default this is done in
batches of 2 shard ranges per visit.
Additionally, when the auto_shard option is True (NOT yet recommeneded
in production), the sharder will identify shard ranges for containers
that have exceeded the threshold for sharding, and will also manage
the sharding and shrinking of shard containers.
The manage_shard_ranges tool provides a means to manually identify
shard ranges and merge them to a container in order to trigger
sharding. This is currently the recommended way to shard a container.
Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Co-Authored-By: Tim Burke <tim.burke@gmail.com>
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Change-Id: I7f192209d4d5580f5a0aa6838f9f04e436cf6b1f
