3468 Commits

Author SHA1 Message Date
Zuul
b20893f540 Merge "Support -d <devs> and -p <partitions> in DB replicators." 2018-03-20 00:04:01 +00:00
Zuul
5fa1c9e8bb Merge "py3: port cli form_signature and unit tests" 2018-03-15 02:19:37 +00:00
Zuul
787c114ec4 Merge "Fix tests using O_TMPFILE" 2018-03-15 00:40:55 +00:00
Zuul
44070c0eec Merge "Add handoffs-only mode to DB replicators." 2018-03-14 20:36:31 +00:00
Thomas Goirand
22b9a4a943 Fix tests using O_TMPFILE
Unit tests using O_TMPFILE only rely on the kernel version to check
for the feature. This is wrong, as some filesystem, like tmpfs, doesn't
support O_TMPFILE.

So, instead of checking kernel version, this patch actually attempts to
open a file using O_TMPFILE and see if that's supported. If not, then
the test is skipped.

Change-Id: I5d652f1634b1ef940838573cfdd799ea17b8b572
2018-03-13 12:06:07 +00:00
Matthew Oliver
e84ed57a72 py3: port cli form_signature and unit tests
This patch ports the form_signature cli `swift-form-signature`
and it's corrisponding tests to py3.

In essence in Py3 the HMAC function expects binary strings.

Change-Id: I5dded4ceb80f0cc595403775e8f9c17873e1e37b
2018-03-13 06:19:44 +00:00
Zuul
2c7e12289f Merge "Optionally drop common prefixes in swift-*-info output" 2018-03-13 03:39:02 +00:00
Zuul
4fd7b0354c Merge "Apply remote metadata in _handle_sync_response" 2018-03-12 23:42:57 +00:00
Zuul
800e585176 Merge "py3: have Timestamp accept bytestrings, better validate input" 2018-03-08 05:13:23 +00:00
Tim Burke
e4640495d8 py3: have Timestamp accept bytestrings, better validate input
Change-Id: I627894b9496cc85b3b1852e9df4812257df4ac6a
2018-03-07 12:53:36 -08:00
Zuul
cb51a0a00b Merge "py3: Port more CLI tools" 2018-03-06 23:40:00 +00:00
Tim Burke
b640631daf Apply remote metadata in _handle_sync_response
We've already got it in the response, may as well apply it now rather
than wait for the other end to get around to running its replicators.

Change-Id: Ie36a6dd075beda04b9726dfa2bba9ffed025c9ef
2018-03-06 19:52:59 +00:00
Zuul
1a5cafb887 Merge "Make DB replicators ignore non-partition directories" 2018-03-06 05:11:30 +00:00
Zuul
6f230839a7 Merge "Temporarily disable flaky test." 2018-03-06 04:29:08 +00:00
Zuul
199937a158 Merge "py3: port common/wsgi.py" 2018-03-06 02:04:04 +00:00
Samuel Merritt
63e6c75efb Temporarily disable flaky test.
test.unit.obj.test_replicator.TestObjectReplicator.test_replicate_lockup_detector
is failing a lot in the gate. Let's disable it for now so that other
patches can continue to land.

Change-Id: I1790ebcbc0c8d075c2786aebca4e8ccf7547b178
2018-03-05 17:51:03 -08:00
Samuel Merritt
b08c70d38e Support -d <devs> and -p <partitions> in DB replicators.
Similar to the object replicator and reconstructor, these arguments
are comma-separated lists of device names and partitions,
respectively, on which the account or container replicator will
operate. Other devices and partitions are ignored.

Change-Id: Ic108f5c38f700ac4c7bcf8315bf4c55306951361
2018-03-05 16:26:19 -08:00
Zuul
35cca77526 Merge "Try to avoid leaving (killed) long-running rsyncs in the process table" 2018-03-05 23:55:35 +00:00
Zuul
488fc6627a Merge "Kill rsync coros when lockup detector tries to kill the process" 2018-03-04 00:43:59 +00:00
Zuul
14a8797228 Merge "Make test_greater_with_offset not fail on py36" 2018-03-03 23:55:53 +00:00
Tim Burke
afb6cb5835 Try to avoid leaving (killed) long-running rsyncs in the process table
Also, add some guards against a NameError in particularly-bad races.

Change-Id: If90662b6996e25bde74e0a202301b52a1d266e92
Related-Change: Ifd14ce82de1f7ebb636d6131849e0fadb113a701
2018-03-03 21:56:56 +00:00
Zuul
cfb893eb87 Merge "Cleanup for iterators in SegmentedIterable" 2018-03-03 03:21:10 +00:00
Zuul
ceb3c01bf6 Merge "Make statsd errors correspond to 5xx only" 2018-03-03 03:08:41 +00:00
Zuul
95225d3b12 Merge "Solve the zombie process problem of Auditor" 2018-03-03 02:43:43 +00:00
Tim Burke
57b632fbb5 Fix object-server to not 400 all expirer DELETEs
In the related changes, we switched to using
Timestamp.normal representations for the X-If-Delete-At header.
However, the object-server required that the header be an int,
and the trailing '.00000' would cause trip the
"Bad X-If-Delete-At header value" error handling.

Now, we'll convert both the expirer header and the stored X-Delete-At to
Timestamps, even though we expect them to have no fractional value.

Note that we *could* have changed the expirer to continue sending
headers that are valid ints, but Timestamps are already the normal
Swift-way of passing and comparing times -- we should use that.

Related-Change: Ibf61eb1f767a48cb457dd494e1f7c12acfd205de
Related-Change: Ie82622625d13177e08a363686ec632f63d24f4e9
Change-Id: Ida22c1c8c5bf21bdc72c33e225e75fb750f8444b
2018-03-02 15:25:38 +00:00
Pete Zaitcev
fdaf052d73 Make test_greater_with_offset not fail on py36
Reviewer, beware: we determined that the test was using the
facilities improperly. This patch adjusts the test but does
not fix the code under test.

The time.time() output looks like this:

[zaitcev@lembas swift-tsrep]$ python2
Python 2.7.14 (default, Dec 11 2017, 14:52:53)
[GCC 7.2.1 20170915 (Red Hat 7.2.1-2)] on linux2
>>> import time
>>> time.time()
1519861559.96239
>>> time.time()
1519861561.046204
>>> time.time()
1519861561.732341
>>>

(it's never beyond 6 digits on py2)

[zaitcev@lembas swift-tsrep]$ python3
Python 3.6.3 (default, Oct  9 2017, 12:07:10)
[GCC 7.2.1 20170915 (Red Hat 7.2.1-2)] on linux
>>> import time
>>> time.time()
1519861541.7662468
>>> time.time()
1519861542.893482
>>> time.time()
1519861546.56222
>>> time.time()
1519861547.3297756
>>>

(can go beyond 6 digits on py3)

When fraction is too long on py3, you get:

>>> now = 1519830570.6949349
>>> now
1519830570.6949348
>>> timestamp = Timestamp(now, offset=1)
>>> timestamp
1519830570.69493_0000000000000001
>>> value = '%f' % now
>>> value
'1519830570.694935'
>>> timestamp > value
False
>>>

Note that the test fails in exactly the same way on py2, if time.time()
returns enough digits. Therefore, rounding changes are not the culprit.

The real problem is the assumption that you can take a float T, print
it with '%f' into S, then do arithmetic on T to get O, convert S, T,
and O into Timestamp, then make comparisons. This does not work,
because rounding happens twice: once when you interpolate %f, and
then when you construct a Timestamp. The only valid operation is
to accept a timestamp (e.g. from X-Delete-At) as a floating point
number as a decimal string, and convert it once. Only then you can
do arithmetics to find the expiration.

Change-Id: Ie3b002abbd4734c675ee48a7535b8b846032f9d1
2018-03-01 21:04:42 -06:00
Tim Burke
36c42974d6 py3: Port more CLI tools
Bring under test

 - test/unit/cli/test_dispersion_report.py
 - test/unit/cli/test_info.py and
 - test/unit/cli/test_relinker.py

I've verified that swift-*-info (at least) behave reasonably under
py3, even swift-object-info when there's non-utf8 metadata on the
data/meta file.

Change-Id: Ifed4b8059337c395e56f5e9f8d939c34fe4ff8dd
2018-02-28 21:10:01 +00:00
Zuul
78439d95f4 Merge "py3: port common/memcached.py" 2018-02-28 19:35:15 +00:00
Tim Burke
624b5310b4 py3: port common/wsgi.py
Note that we're punting on configuring socket buffer sizes (for now)

Change-Id: I285a9b521fd0af381a227e0e824bc391817547f4
2018-02-28 12:49:13 -05:00
Kazuhiro MIYAHARA
1fadffeae0 Split expirer methods and parametrize task account
To prepare for implement general task queue mode to expirer,
this patch splits expirer's method into smaller ones and parametrize task
account. This change will make expirer's general task queue patch [1] more
simple.

This patch has following approaches:
    1: Split methods into smaller ones
    2: Parameterize task account name to adapt many task accounts
       in general task queue
    3: Include task account names in log messages
    4: Skip task account when the account has no task containers

[1]: https://review.openstack.org/#/c/517389/

Change-Id: I907612f7c258495e9ccc53c1d57de4791b3e7ab7
2018-02-27 22:49:31 +00:00
Kota Tsuyuzaki
9e5f434574 Kill rsync coros when lockup detector tries to kill the process
Because the replicator in the master doesn't propergate the kill
signal to the subprocess in the coroutine. With the behavior, the lockup
detector causes a lot of rsync processes even it tries to reset the process.

This patch fixes the replicator kill rsync procs when the lockup detector
calls kill of eventlet threads.

Change-Id: Ifd14ce82de1f7ebb636d6131849e0fadb113a701
2018-02-27 19:34:51 +09:00
Zuul
f1f8591c6a Merge "Fix expirer's invalid task object names in unit tests" 2018-02-26 23:27:01 +00:00
Zuul
d0f4fd6db5 Merge "py3: port common/storage_policy.py" 2018-02-26 18:12:32 +00:00
Kazuhiro MIYAHARA
b3f1558acd Fix expirer's invalid task object names in unit tests
Object-expirer's task name should be in format of
"<timestamp>-<account>/<container>/<obj>". In object-expirer
implementation, ValueError is catched and handled when expirer's task
objects have invalid name. But in actual swift cluster, invalid task
object name is not created because task object is created by
object-server.
However, without the ValueError catching, some unit tests fail,
because the unit tests create invalid task object names.

This patch fixes invalid task object names in unit tests. The
ValueError catch is remained for unexpected errors, but in the case
the task will be skipped.

This patch will help to refactor expirer's task object parsing.

Change-Id: I8fab8fd180481ce9e97c945904c5c89eec037110
2018-02-26 16:10:40 +00:00
Tim Burke
5cb0869743 py3: port common/memcached.py
Change-Id: I7f04b3977971f0581b04180e5372686d8186346f
2018-02-26 12:39:16 +00:00
Zuul
54509a6791 Merge "Tighten up assertions around expirer's concurrency" 2018-02-26 11:13:12 +00:00
Zuul
353a7ad07b Merge "Remove confusing assertion from expirer's unit test" 2018-02-26 11:13:08 +00:00
Tim Burke
4b19ac7723 py3: port common/storage_policy.py
Change-Id: I7030280a8495628df9ed8edcc8abc31f901da72e
2018-02-26 10:57:41 +00:00
Tim Burke
25540a415e Tighten up assertions around expirer's concurrency
In particular, test that each work item is only done *once*.

Change-Id: I9cc610bffb2aa9a2f2b05f4c49e574ab56d05201
Related-Change: Ic0075a3718face8c509ed0524b63d9171f5b7d7a
2018-02-26 10:37:23 +00:00
Kazuhiro MIYAHARA
58f5d89066 Remove confusing assertion from expirer's unit test
In test_expirer.TestObjectExpirer.test_process_based_concurrency,
an assertion checks that expirer execute tasks in round-robin order
for target containers. But the assertion depends on task object path,
because task assignation for each process depends on md5 of task
object path. The dependency makes the assetion confusing.

Now, we have test_expirer.TestObjectExpirer.test_round_robin_order which
is added in [1]. So this patch remove the confusing assertion.

This patch will help to refactor expirer's task object parsing.
I will push patch for the refactoring after this patch.

[1]: https://review.openstack.org/#/c/538171

Change-Id: Ic0075a3718face8c509ed0524b63d9171f5b7d7a
2018-02-22 11:05:46 -08:00
Tim Burke
6060af8db9 Add more tests around ObjectExpirer.round_robin_order
Change-Id: I43b5e8d9513fd0566a61ff585dfdc1dde5b28343
Related-Change: Ibf61eb1f767a48cb457dd494e1f7c12acfd205de
2018-02-22 10:26:00 -08:00
Zuul
d296ec8be4 Merge "Refactor expirer's task round robin implementation" 2018-02-22 12:35:07 +00:00
Kazuhiro MIYAHARA
303635348b Refactor expirer's task round robin implementation
Object-expirer changes order of expiration tasks to avoid deleting
objects in a certain container continuously.

To make review for expirer's task queue update patch [1] easy,
this patch refactors the implementation of the order change. In this
patch, the order change is divided as a function.

In [1], there will be two implementations for legacy task queue
and for general task queue. The two implementations have similar
codes. This patch helps to avoid copying codes in the two implementations.

Other than dividing function, this patch tries to resolve:
- Separate container iteration and object iteration to avoid the generator
  termination with (container, None) tuple.
- Using Timestamp class for delete_timestamp to be consist with other modules
- Change yielded delete task object info from tuple to dict because that
  includes several complex info (e.g. task_container, task_object,
  and target_path)
- Fix minor docs and tests depends on the changes above

[1]: https://review.openstack.org/#/c/517389

Co-Authored-By: Kota Tsuyuzaki <tsuyuzaki.kota@lab.ntt.co.jp>

Change-Id: Ibf61eb1f767a48cb457dd494e1f7c12acfd205de
2018-02-22 18:43:11 +09:00
Zuul
ee282c1166 Merge "Fix suffix-byte-range responses for zero-byte replicated objects." 2018-02-20 07:38:10 +00:00
Zuul
db03703443 Merge "Using assertIsNone() instead of assertEqual(None)" 2018-02-19 20:45:18 +00:00
Samuel Merritt
47fed6f2f9 Add handoffs-only mode to DB replicators.
The object reconstructor has a handoffs-only mode that is very useful
when a cluster requires rapid rebalancing, like when disks are nearing
fullness. This mode's goal is to remove handoff partitions from disks
without spending effort on primary partitions. The object replicator
has a similar mode, though it varies in some details.

This commit adds a handoffs-only mode to the account and container
replicators.

Change-Id: I588b151ee65ae49d204bd6bf58555504c15edf9f
Closes-Bug: 1668399
2018-02-16 16:56:13 -08:00
Samuel Merritt
2bfd9c6a9b Make DB replicators ignore non-partition directories
If a cluster operator has some tooling that makes directories in
/srv/node/<disk>/accounts, then the account replicator will treat
those directories as partition dirs and may remove empty
subdirectories contained therein. This wastes time and confuses the
operator.

This commit makes DB replicators skip partition directories whose
names don't look like positive integers. This doesn't completely avoid
the problem since an operator can still use an all-digit name, but it
will skip directories like "tmp21945".

Change-Id: I8d6682915a555f537fc0ce8c39c3d52c99ff3056
2018-02-16 16:56:13 -08:00
Tim Burke
642f79965a py3: port common/ring/ and common/utils.py
I can't imagine us *not* having a py3 proxy server at some point, and
that proxy server is going to need a ring.

While we're at it (and since they were so close anyway), port

* cli/ringbuilder.py and
* common/linkat.py
* common/daemon.py

Change-Id: Iec8d97e0ce925614a86b516c4c6ed82809d0ba9b
2018-02-12 06:42:24 +00:00
Zuul
6544ae1848 Merge "Move eventlet patch before call to loadapp" 2018-02-09 22:29:27 +00:00
Zuul
1038ddacae Merge "Fix typos in swift" 2018-02-09 07:35:22 +00:00