Unit tests using O_TMPFILE only rely on the kernel version to check
for the feature. This is wrong, as some filesystem, like tmpfs, doesn't
support O_TMPFILE.
So, instead of checking kernel version, this patch actually attempts to
open a file using O_TMPFILE and see if that's supported. If not, then
the test is skipped.
Change-Id: I5d652f1634b1ef940838573cfdd799ea17b8b572
This patch ports the form_signature cli `swift-form-signature`
and it's corrisponding tests to py3.
In essence in Py3 the HMAC function expects binary strings.
Change-Id: I5dded4ceb80f0cc595403775e8f9c17873e1e37b
We've already got it in the response, may as well apply it now rather
than wait for the other end to get around to running its replicators.
Change-Id: Ie36a6dd075beda04b9726dfa2bba9ffed025c9ef
test.unit.obj.test_replicator.TestObjectReplicator.test_replicate_lockup_detector
is failing a lot in the gate. Let's disable it for now so that other
patches can continue to land.
Change-Id: I1790ebcbc0c8d075c2786aebca4e8ccf7547b178
Similar to the object replicator and reconstructor, these arguments
are comma-separated lists of device names and partitions,
respectively, on which the account or container replicator will
operate. Other devices and partitions are ignored.
Change-Id: Ic108f5c38f700ac4c7bcf8315bf4c55306951361
Also, add some guards against a NameError in particularly-bad races.
Change-Id: If90662b6996e25bde74e0a202301b52a1d266e92
Related-Change: Ifd14ce82de1f7ebb636d6131849e0fadb113a701
In the related changes, we switched to using
Timestamp.normal representations for the X-If-Delete-At header.
However, the object-server required that the header be an int,
and the trailing '.00000' would cause trip the
"Bad X-If-Delete-At header value" error handling.
Now, we'll convert both the expirer header and the stored X-Delete-At to
Timestamps, even though we expect them to have no fractional value.
Note that we *could* have changed the expirer to continue sending
headers that are valid ints, but Timestamps are already the normal
Swift-way of passing and comparing times -- we should use that.
Related-Change: Ibf61eb1f767a48cb457dd494e1f7c12acfd205de
Related-Change: Ie82622625d13177e08a363686ec632f63d24f4e9
Change-Id: Ida22c1c8c5bf21bdc72c33e225e75fb750f8444b
Reviewer, beware: we determined that the test was using the
facilities improperly. This patch adjusts the test but does
not fix the code under test.
The time.time() output looks like this:
[zaitcev@lembas swift-tsrep]$ python2
Python 2.7.14 (default, Dec 11 2017, 14:52:53)
[GCC 7.2.1 20170915 (Red Hat 7.2.1-2)] on linux2
>>> import time
>>> time.time()
1519861559.96239
>>> time.time()
1519861561.046204
>>> time.time()
1519861561.732341
>>>
(it's never beyond 6 digits on py2)
[zaitcev@lembas swift-tsrep]$ python3
Python 3.6.3 (default, Oct 9 2017, 12:07:10)
[GCC 7.2.1 20170915 (Red Hat 7.2.1-2)] on linux
>>> import time
>>> time.time()
1519861541.7662468
>>> time.time()
1519861542.893482
>>> time.time()
1519861546.56222
>>> time.time()
1519861547.3297756
>>>
(can go beyond 6 digits on py3)
When fraction is too long on py3, you get:
>>> now = 1519830570.6949349
>>> now
1519830570.6949348
>>> timestamp = Timestamp(now, offset=1)
>>> timestamp
1519830570.69493_0000000000000001
>>> value = '%f' % now
>>> value
'1519830570.694935'
>>> timestamp > value
False
>>>
Note that the test fails in exactly the same way on py2, if time.time()
returns enough digits. Therefore, rounding changes are not the culprit.
The real problem is the assumption that you can take a float T, print
it with '%f' into S, then do arithmetic on T to get O, convert S, T,
and O into Timestamp, then make comparisons. This does not work,
because rounding happens twice: once when you interpolate %f, and
then when you construct a Timestamp. The only valid operation is
to accept a timestamp (e.g. from X-Delete-At) as a floating point
number as a decimal string, and convert it once. Only then you can
do arithmetics to find the expiration.
Change-Id: Ie3b002abbd4734c675ee48a7535b8b846032f9d1
Bring under test
- test/unit/cli/test_dispersion_report.py
- test/unit/cli/test_info.py and
- test/unit/cli/test_relinker.py
I've verified that swift-*-info (at least) behave reasonably under
py3, even swift-object-info when there's non-utf8 metadata on the
data/meta file.
Change-Id: Ifed4b8059337c395e56f5e9f8d939c34fe4ff8dd
To prepare for implement general task queue mode to expirer,
this patch splits expirer's method into smaller ones and parametrize task
account. This change will make expirer's general task queue patch [1] more
simple.
This patch has following approaches:
1: Split methods into smaller ones
2: Parameterize task account name to adapt many task accounts
in general task queue
3: Include task account names in log messages
4: Skip task account when the account has no task containers
[1]: https://review.openstack.org/#/c/517389/
Change-Id: I907612f7c258495e9ccc53c1d57de4791b3e7ab7
Because the replicator in the master doesn't propergate the kill
signal to the subprocess in the coroutine. With the behavior, the lockup
detector causes a lot of rsync processes even it tries to reset the process.
This patch fixes the replicator kill rsync procs when the lockup detector
calls kill of eventlet threads.
Change-Id: Ifd14ce82de1f7ebb636d6131849e0fadb113a701
Object-expirer's task name should be in format of
"<timestamp>-<account>/<container>/<obj>". In object-expirer
implementation, ValueError is catched and handled when expirer's task
objects have invalid name. But in actual swift cluster, invalid task
object name is not created because task object is created by
object-server.
However, without the ValueError catching, some unit tests fail,
because the unit tests create invalid task object names.
This patch fixes invalid task object names in unit tests. The
ValueError catch is remained for unexpected errors, but in the case
the task will be skipped.
This patch will help to refactor expirer's task object parsing.
Change-Id: I8fab8fd180481ce9e97c945904c5c89eec037110
In particular, test that each work item is only done *once*.
Change-Id: I9cc610bffb2aa9a2f2b05f4c49e574ab56d05201
Related-Change: Ic0075a3718face8c509ed0524b63d9171f5b7d7a
In test_expirer.TestObjectExpirer.test_process_based_concurrency,
an assertion checks that expirer execute tasks in round-robin order
for target containers. But the assertion depends on task object path,
because task assignation for each process depends on md5 of task
object path. The dependency makes the assetion confusing.
Now, we have test_expirer.TestObjectExpirer.test_round_robin_order which
is added in [1]. So this patch remove the confusing assertion.
This patch will help to refactor expirer's task object parsing.
I will push patch for the refactoring after this patch.
[1]: https://review.openstack.org/#/c/538171
Change-Id: Ic0075a3718face8c509ed0524b63d9171f5b7d7a
Object-expirer changes order of expiration tasks to avoid deleting
objects in a certain container continuously.
To make review for expirer's task queue update patch [1] easy,
this patch refactors the implementation of the order change. In this
patch, the order change is divided as a function.
In [1], there will be two implementations for legacy task queue
and for general task queue. The two implementations have similar
codes. This patch helps to avoid copying codes in the two implementations.
Other than dividing function, this patch tries to resolve:
- Separate container iteration and object iteration to avoid the generator
termination with (container, None) tuple.
- Using Timestamp class for delete_timestamp to be consist with other modules
- Change yielded delete task object info from tuple to dict because that
includes several complex info (e.g. task_container, task_object,
and target_path)
- Fix minor docs and tests depends on the changes above
[1]: https://review.openstack.org/#/c/517389
Co-Authored-By: Kota Tsuyuzaki <tsuyuzaki.kota@lab.ntt.co.jp>
Change-Id: Ibf61eb1f767a48cb457dd494e1f7c12acfd205de
The object reconstructor has a handoffs-only mode that is very useful
when a cluster requires rapid rebalancing, like when disks are nearing
fullness. This mode's goal is to remove handoff partitions from disks
without spending effort on primary partitions. The object replicator
has a similar mode, though it varies in some details.
This commit adds a handoffs-only mode to the account and container
replicators.
Change-Id: I588b151ee65ae49d204bd6bf58555504c15edf9f
Closes-Bug: 1668399
If a cluster operator has some tooling that makes directories in
/srv/node/<disk>/accounts, then the account replicator will treat
those directories as partition dirs and may remove empty
subdirectories contained therein. This wastes time and confuses the
operator.
This commit makes DB replicators skip partition directories whose
names don't look like positive integers. This doesn't completely avoid
the problem since an operator can still use an all-digit name, but it
will skip directories like "tmp21945".
Change-Id: I8d6682915a555f537fc0ce8c39c3d52c99ff3056
I can't imagine us *not* having a py3 proxy server at some point, and
that proxy server is going to need a ring.
While we're at it (and since they were so close anyway), port
* cli/ringbuilder.py and
* common/linkat.py
* common/daemon.py
Change-Id: Iec8d97e0ce925614a86b516c4c6ed82809d0ba9b