The output from swift-manage-shard-ranges compact was hard to read,
particularly with long lower and upper bounds typically seen in
production. This patch changes the output, most significantly by
aligning lower and upper bounds at the end of lines and printing
donors before acceptors, so that the continuity of shard range bounds
is easier to see.
Example of new ouput:
Donor shard range(s) with total of 2 objects:
objects: 2 lower: ''
state: active upper: 'obj-0001'
can be compacted into acceptor shard range:
objects: 2 lower: 'obj-0001'
state: active upper: 'obj-0003'
Donor shard range(s) with total of 2 objects:
objects: 2 lower: 'obj-0003'
state: active upper: 'obj-0005'
can be compacted into acceptor shard range:
objects: 2 lower: 'obj-0005'
state: active upper: ''
Change-Id: I90f3ab7a29b87d0959c94a05c9ee9ad97b60014d
If a sequence of shard ranges is already shrinking then in some
circumstances we do not want to report it as a candidate for
shrinking. For backwards compatibility allow already shrinking
sequences to be optionally included in return value of
find_compactible_shard_sequences.
Also refactor to add an is_shrinking_candidate() function.
Change-Id: Ifa20b7c08aba7254185918dfcee69e8206f51cea
Fix the find_compactible_shard_sequences function to prevent skipping
a shard range after finding a sequence of compactible shard ranges
that approaches the merge size.
Previously a compactible sequence would correctly terminate on the nth
shard range if the n+1th shard range would take the object count over
the merge_size, but the n+1th shard range would then be skipped and
not considered for the start of the next sequence.
Change-Id: I670441e7426b28ab2247563c7fa854d1cd502316
This patch adds a 'compact' command to swift-manage-shard-ranges that
enables sequences of contiguous shards with low object counts to be
compacted into another existing shard, or into the root container.
Change-Id: Ia8f3297d610b5a5cf5598d076fdaf30211832366
The idea is, if none of
- timestamp,
- object_count,
- bytes_used,
- state, or
- epoch
has changed, we shouldn't need to send an update back to the root
container.
This is more-or-less comparable to what the container-updater does to
avoid unnecessary writes to the account.
Closes-Bug: #1834097
Change-Id: I1ee7ba5eae3c508064714c4deb4f7c6bbbfa32af
When an operator does a `find_and_replace` on a DB that already has
shard ranges, they get a prompt like:
This will delete existing 58 shard ranges.
Do you want to show the existing ranges [s], delete the existing
ranges [yes] or quit without deleting [q]?
Previously, if they selected `q`, we would skip the delete but still do
the merge (!) and immediately warn about how there are now invalid shard
ranges. Now, quit without merging.
Change-Id: I7d869b137a6fbade59bb8ba16e4f3e9663e18822
This is a follow up patch from the cleaning up cleave context's patch
(patch 681970). Instead of tracking a last_modified timestamp, and storing
it in the context metadata, use the timestamp we use when storing any
metadata.
Reducing duplication is nice, but there's a more significant reason to
do this: affected container DBs can start getting cleaned up as soon as
they're running the new code rather than needing to wait for an
additional reclaim_age.
Change-Id: I2cdbe11f06ffb5574e573c4a60ba4e5d41a00c50
There is a sharding edge case where more CleaveContext are generated and
stored in the sharding container DB. If this number get's high enough,
like in the linked bug. If enough CleaveContects build up in the DB then
this can lead to the 503's when attempting to list the container due to
all the `X-Container-Sysmeta-Shard-Context-*` headers.
This patch resolves this by tracking the a CleaveContext's last
modified. And during the sharding audit, any context's that hasn't been
touched after reclaim_age are deleted.
This plus the skip empty ranges patches should improve these handoff
shards.
Change-Id: I1e502c328be16fca5f1cca2186b27a0545fecc16
Closes-Bug: #1843313
This started with ShardRanges and its CLI. The sharder is at the
bottom of the dependency chain. Even container backend needs it.
Once we started tinkering with the sharder, it all snowballed to
include the rest of the container services.
Beware, this does affect some of Python 2 code. Mostly it's trivial
and obviously correct, but needs checking by reviewers.
About killing the stray "from __future__ import unicode_literals":
we do not do it in general. The specific problem it caused was
a failure of functional tests because unicode leaked into a field
that was supposed to be encoded. It is just too hard to track the
types when rules change from file to file, so off with its head.
Change-Id: Iba4e65d0e46d8c1f5a91feb96c2c07f99ca7c666
The sharder daemon visits container dbs and when necessary executes
the sharding workflow on the db.
The workflow is, in overview:
- perform an audit of the container for sharding purposes.
- move any misplaced objects that do not belong in the container
to their correct shard.
- move shard ranges from FOUND state to CREATED state by creating
shard containers.
- move shard ranges from CREATED to CLEAVED state by cleaving objects
to shard dbs and replicating those dbs. By default this is done in
batches of 2 shard ranges per visit.
Additionally, when the auto_shard option is True (NOT yet recommeneded
in production), the sharder will identify shard ranges for containers
that have exceeded the threshold for sharding, and will also manage
the sharding and shrinking of shard containers.
The manage_shard_ranges tool provides a means to manually identify
shard ranges and merge them to a container in order to trigger
sharding. This is currently the recommended way to shard a container.
Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Co-Authored-By: Tim Burke <tim.burke@gmail.com>
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Change-Id: I7f192209d4d5580f5a0aa6838f9f04e436cf6b1f
