Commit Graph

17 Commits

Author SHA1 Message Date
e199192cae Replace replication_one_per_device by custom count
This commit replaces boolean replication_one_per_device by an integer
replication_concurrency_per_device. The new configuration parameter is
passed to utils.lock_path() which now accept as an argument a limit for
the number of locks that can be acquired for a specific path.

Instead of trying to lock path/.lock, utils.lock_path() now tries to lock
files path/.lock-X, where X is in the range (0, N), N being the limit for
the number of locks allowed for the path. The default value of limit is
set to 1.

Change-Id: I3c3193344c7a57a8a4fc7932d1b10e702efd3572
2017-10-24 16:17:41 +01:00
cf3c970a77 Trivial fix typos
Change-Id: I7e1e3b2f92183b2a249299659f0778fe838212e2
2017-05-30 14:26:23 +07:00
36b1a2f69f Fix some reST field lists in docstrings
Probably the most common format for documenting arguments is reST field
lists [1]. This change updates some docstrings to comply with the field
lists syntax.

 [1] http://sphinx-doc.org/domains.html#info-field-lists

Change-Id: I87e77a9bbd5bcb834b35460ce0adff5bc59d9168
2017-03-29 09:11:07 +08:00
04502a9f64 Fix test comment and remove extra parameter
Fixed the comment in the test to match exactly what's
being removed and what the expected result is.

Also, removed that extra '/' parameter which was causing
the assert to test at the wrong directory level.

Change-Id: I2f27f0d12c08375c61047a3f861c94a3dd3915c6
Signed-off-by: Thiago da Silva <thiago@redhat.com>
2017-02-12 00:48:53 +00:00
5bb3ea162c Ignore cache files when comparing synced parts
Prior to the Related-Change no suffixes were written to hashes.invalid
until after initial suffix hashing created hashes.pkl - and in our probe
test the only updates to the partition occurred before replication.

Before the related change with sync_method = rsync it was possible when
starting from a clean slate to write data, and replicate from a handoff
partition without generating a hashes.invalid file in any primary.

After the Related-Change it was no longer possible to write data without
generating a hashes.invalid file; however with sync_method = rsync the
replicator could still replicate data into a partition that never
received an update directly and therefore no hashes.invalid.

When using sync_method = ssync replication updates the hashes.invalid
like any normal update to the partition and therefore all partitions
always have a hashes.invalid.

This change opts to ignores these implementation details in the probe
tests when comparing the files between synced partitions by
black-listing these known cache files and only validates that the disk
file's on disk files are in sync.

Related-Change-Id: I2b48238d9d684e831d9777a7b18f91a3cef57cd1

Change-Id: Ia9c50d7bc1a74a17c608a3c3cfb8f93196fb709d
Closes-Bug: #1663021
2017-02-10 11:28:43 -06:00
300d388825 Use more specific asserts in test/probe tests
I changed asserts with more specific assert methods.

e.g.: from assertTrue(sth == None) to assertIsNone(*) or
assertTrue(isinstance(inst, type)) to assertIsInstace(inst, type) or
assertTrue(not sth) to assertFalse(sth).

The code gets more readable, and a better description will be shown on fail.

Change-Id: I3768faa568e3964e726ecc48ac8cb133cb088284
2016-11-02 18:13:22 +00:00
647b66a2ce Erasure Code Reconstructor
This patch adds the erasure code reconstructor. It follows the
design of the replicator but:
  - There is no notion of update() or update_deleted().
  - There is a single job processor
  - Jobs are processed partition by partition.
  - At the end of processing a rebalanced or handoff partition, the
    reconstructor will remove successfully reverted objects if any.

And various ssync changes such as the addition of reconstruct_fa()
function called from ssync_sender which performs the actual
reconstruction while sending the object to the receiver

Co-Authored-By: Alistair Coles <alistair.coles@hp.com>
Co-Authored-By: Thiago da Silva <thiago@redhat.com>
Co-Authored-By: John Dickinson <me@not.mn>
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Co-Authored-By: Tushar Gohad <tushar.gohad@intel.com>
Co-Authored-By: Samuel Merritt <sam@swiftstack.com>
Co-Authored-By: Christian Schwede <christian.schwede@enovance.com>
Co-Authored-By: Yuan Zhou <yuan.zhou@intel.com>
blueprint ec-reconstructor
Change-Id: I7d15620dc66ee646b223bb9fff700796cd6bef51
2015-04-14 00:52:17 -07:00
ca0fce8542 more probe test refactoring
* move get_to_final_state into ProbeTest
* get rid of kill_servers
* add replicators manager and updaters manager to ProbeTest

(this is all going someplace, i promise)

Change-Id: I8393a2ebc0d04051cae48cc3c49580f70818dbf2
2015-02-13 16:55:45 -08:00
2c1b5af062 refactor probe tests
* refactor probe tests to use probe.common.ProbeTest
* move reset_environment functionality to ProbeTest.setUp()
* choose rings and policies that meet the criteria - raise SkipTest if
nothing matches
* replace all AssertionErrors in setup with SkipTest

Change-Id: Id56c497d58083f5fd55f5283cdd346840df039d3
2015-02-12 11:30:21 -08:00
ad2a9cefe5 Fixes probe tests with non-zero default storage policy
Add headers param to direct_client.direct_get_object, which is used in
probetests to passthrough the X-Storage-Policy-Index header.

DocImpact
Implements: blueprint storage-policies
Change-Id: I19adbbcefbc086c8467bd904a275d55cde596412
2014-06-18 21:09:53 -07:00
032f0bfc7c Fix several typos in the codebase.
These were found using https://github.com/intgr/topy

Change-Id: I0dc7b76c44b8b17b1dcd79184dad1516fb11173c
2014-04-25 20:14:09 -07:00
0efac0cac2 make probetests work with conf.d configs
Change-Id: I451ff4629730a334ac1bd8fc6cd75de95314e153
2014-03-12 12:04:45 -07:00
f72704fc82 Change OpenStack LLC to Foundation
Change-Id: I7c3df47c31759dbeb3105f8883e2688ada848d58
Closes-bug: #1214176
2013-09-20 01:02:31 +08:00
0b7432bc4d Pep8 the probe tests (1 of 12)
Change-Id: I20571816d9838e125a60a2775219b2f3cbb36ddd
Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-09-01 15:12:12 -04:00
ff5a6d0111 Corrected many style violations in the tests.
I focussed primarily on F-category violations, they are all but all fixed with
this patch.

Change-Id: I343f6945c97984ed1093bc347b6def6994297041
2013-07-24 10:18:47 -07:00
7a02c218bb Fixed probetests
* Fixed issue with running probetests with the latest update
  of python-swiftclient that removed eventlet
* Fixed issue with replication server tests to not require hard
  coded paths

Change-Id: Ibbf727ae99c0f3893ae58e270e2f879a1f618e49
2013-05-23 22:41:04 +00:00
ea7858176b Implementation of replication servers
Support separate replication ip address:
- Added new function in utils. This function provides ability
  to select separate IP address for replication service.
- Db_replicator and object replicators were changed.
  Replication process uses new function now.

Replication network parameters:
- Replication network fields (replication_ip, replication_port)
  support was added to device dictionary in swift-ring-builder script.
- Changes were made to support new fields in search, show and set_info
  functions.

Implementation of replication servers:
- Separate replication servers use the same code as normal replication
  servers, but with replication_server parameter = True.  When using a
  separate replication network, the non-replication servers set
  replication_server = False.  When there is no separate replication
  network (the default case), replication_server is not included in the config.

DocImpact
Change-Id: Ie9af5bdcdf9241c355e36053ca4adfe49dc35bd0
Implements: blueprint dedicated-replication-network
2013-04-21 18:14:42 -04:00