4451 Commits

Author SHA1 Message Date
Alistair Coles
fa89064933 Per-policy DiskFile classes
Adds specific disk file classes for EC policy types.

The new ECDiskFile and ECDiskFileWriter classes are used by the
ECDiskFileManager.

ECDiskFileManager is registered with the DiskFileRouter for use with
EC_POLICY type policies.

Refactors diskfile tests into BaseDiskFileMixin and BaseDiskFileManagerMixin
classes which are then extended in subclasses for the legacy
replication-type DiskFile* and ECDiskFile* classes.

Refactor to prefer use of a policy instance reference over a policy_index
int to refer to a policy.

Add additional verification to DiskFileManager.get_dev_path to validate the
device root with common.constraints.check_dir, even when mount_check is
disabled for use in on a virtual swift-all-in-one.

Co-Authored-By: Thiago da Silva <thiago@redhat.com>
Co-Authored-By: John Dickinson <me@not.mn>
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Co-Authored-By: Tushar Gohad <tushar.gohad@intel.com>
Co-Authored-By: Paul Luse <paul.e.luse@intel.com>
Co-Authored-By: Samuel Merritt <sam@swiftstack.com>
Co-Authored-By: Christian Schwede <christian.schwede@enovance.com>
Co-Authored-By: Yuan Zhou <yuan.zhou@intel.com>
Change-Id: I22f915160dc67a9e18f4738c1ddf068344e8ad5d
2015-04-14 00:52:16 -07:00
Jenkins
8a58bbf75f Merge "Update test infrastructure" into feature/ec_review 2015-04-14 07:42:37 +00:00
Jenkins
4e6c095135 Merge "Add support for policy types, 'erasure_coding' policy" into feature/ec_review 2015-04-14 07:41:34 +00:00
Clay Gerrard
a707829334 Update test infrastructure
* Get FakeConn ready for expect 100 continue
 * Use debug_logger more and with better interfaces
 * Fix patch_policies to be less annoying

Co-Authored-By: Alistair Coles <alistair.coles@hp.com>
Co-Authored-By: Thiago da Silva <thiago@redhat.com>
Co-Authored-By: Tushar Gohad <tushar.gohad@intel.com>
Co-Authored-By: Paul Luse <paul.e.luse@intel.com>
Co-Authored-By: Samuel Merritt <sam@swiftstack.com>
Co-Authored-By: Christian Schwede <christian.schwede@enovance.com>
Co-Authored-By: Yuan Zhou <yuan.zhou@intel.com>
Change-Id: I28c0a3539d994cbb8e6b94d63a23ed4ea6cb956d
2015-04-13 22:57:42 -07:00
Tushar Gohad
ed54066288 Add support for policy types, 'erasure_coding' policy
This patch extends the StoragePolicy class for non-replication storage
policies, the first one being "erasure coding".

Changes:

 - Add 'policy_type' support to BaseStoragePolicy class
 - Disallow direct instantiation of BaseStoragePolicy class

 - Subclass BaseStoragePolicy

   - "StoragePolicy":
     . Replication policy, default
     . policy_type = 'replication'

   - "ECStoragePolicy":
     . Erasure Coding policy
     . policy_type = 'erasure_coding'
     . Private member variables
       ec_type (EC backend),
       ec_num_data_fragments (number of fragments original
         data split into after erasure coding operation),
       ec_num_parity_fragments (number of parity fragments
       generated during erasure coding)
     . Private methods
       EC specific attributes and ring validator methods.

 - Swift will use PyECLib, a Python Erasure Coding library, for
   erasure coding operations. PyECLib is already an approved
   OpenStack core requirement.
   (https://bitbucket.org/kmgreen2/pyeclib/)

 - Add test cases for
   - 'policy_type' StoragePolicy member
   - policy_type == 'erasure_coding'

DocImpact

Co-Authored-By: Alistair Coles <alistair.coles@hp.com>
Co-Authored-By: Thiago da Silva <thiago@redhat.com>
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Co-Authored-By: Paul Luse <paul.e.luse@intel.com>
Co-Authored-By: Samuel Merritt <sam@swiftstack.com>
Co-Authored-By: Christian Schwede <christian.schwede@enovance.com>
Co-Authored-By: Yuan Zhou <yuan.zhou@intel.com>
Change-Id: Ie0e09796e3ec45d3e656fb7540d0e5a5709b8386
Implements: blueprint ec-proxy-work
2015-04-13 22:57:42 -07:00
Jenkins
ce596684f6 Merge "Set connection timeout in container sync" 2015-04-13 03:16:54 +00:00
Christian Schwede
16ee994c1e Set connection timeout in container sync
Container sync might get stuck without a connection timeout if the remote proxy
is not responding.

This patch sets a default timeout of 5.0 seconds for the connection attempt. The
value is much higher than other connection timeouts inside Swift (0.5); however
there might be a much higher latency to the remote peer, thus playing it safe.
There is also a retry if the attempt timed out.

Note that this setting only applies to the connection request itself. Setting
this timeout does not apply when the remote proxy goes away during a request.

Also added a short test to ensure urlopen is called with the timeout value.

Co-Authored-By: Alistair Coles <alistair.coles@hp.com>

Change-Id: Ic08a55157fa91fe1316653781adf4d66eead61bc
Partial-Bug: 1419916
2015-04-10 10:06:40 +00:00
Jenkins
39c1362a4f Merge "Add some debug output to the ring builder" 2015-04-05 17:28:50 +00:00
Jenkins
c785d04b3e Merge "Tighten up recon middleware unit tests" 2015-04-05 16:35:43 +00:00
John Dickinson
376ecc74fc make git review easier
Change-Id: I6032a7915586db6da25aafccf3dced581b37df9c
2015-04-01 12:41:44 -07:00
Samuel Merritt
8d3b3b2ee0 Add some debug output to the ring builder
Sometimes, I get handed a builder file in a support ticket and a
question of the form "why is the balance [not] doing $thing?". When
that happens, I add a bunch of print statements to my local
swift/common/ring/builder.py, figure things out, and then delete the
print statements. This time, instead of deleting the print statements,
I turned them into debug() calls and added a "--debug" flag to the
rebalance command in hopes that someone else will find it useful.

Change-Id: I697af90984fa5b314ddf570280b4585ba0ba363c
2015-03-30 17:47:28 -07:00
Clay Gerrard
cf21db4bc6 Add Swift Design Principles to CONTRIBUTING.md
This is based of work started a few design summits ago [1].

With EC getting close I felt like I could the inspiration.

1. https://etherpad.openstack.org/p/juno_swift_core_principles

Change-Id: I90fa56d67003080fe3d7bc4e0fad053e3b0c8504
Signed-off-by: Thiago da Silva <thiago@redhat.com>
2015-03-27 13:13:31 -04:00
Jenkins
eb6a4cbec5 Merge "Bump eventlet version to 0.16.1" 2015-03-25 23:15:58 +00:00
Tushar Gohad
a812941003 Bump eventlet version to 0.16.1
This version adds support for multiple 100-continue responses
via the API send_hundred_continue_response(), which we need for
the two-phase scheme for PUTs with Erasure Coding

Change-Id: I7154ef1dd96ab761894b09d5e6abf98a453f7b64
2015-03-25 12:02:48 -07:00
Jenkins
4dca4459c2 Merge "Handle ENOSPC in mkstemp()" 2015-03-25 11:35:24 +00:00
Jenkins
77671d5de0 Merge "Fix common misspellings" 2015-03-25 00:53:44 +00:00
Jenkins
963379924c Merge "Imported Translations from Transifex" 2015-03-24 17:38:42 +00:00
Jenkins
3973eb38e0 Merge "Add swift-recon feature to track swift-drive-audit error count" 2015-03-24 10:59:24 +00:00
Martin Kletzander
76b106fc01 Fix common misspellings
Wikipedia's list of common misspellings [1] has a machine-readable
version.  This patch fixes those misspellings mentioned in the list
which don't have multiple right variants (as e.g. "accension", which can
be both "accession" and "ascension"), such misspellings are left
untouched.  The list of changes was manually re-checked for false
positives.

[1] https://en.wikipedia.org/wiki/Wikipedia:Lists_of_common_misspellings/For_machines

Change-Id: Ic9a5438629664f7cea216413a28acc0e8992da05
Signed-off-by: Martin Kletzander <mkletzan@redhat.com>
2015-03-24 11:07:56 +01:00
OpenStack Proposal Bot
a288d05ca5 Imported Translations from Transifex
For more information about this automatic import see:
https://wiki.openstack.org/wiki/Translations/Infrastructure

Change-Id: I6ae0dd0c34ac1d7173acacdd0dd4896619592e0c
2015-03-24 06:06:33 +00:00
Jenkins
3e8925ae04 Merge "Refactoring the PUT method" 2015-03-24 04:19:20 +00:00
Jenkins
d28c799d17 Merge "Test swift-object-info opens meta and ts files" 2015-03-23 12:50:16 +00:00
Lorcan
0a46793662 Add swift-recon feature to track swift-drive-audit error count
This is a follow-on from a previous commit which added recon info
for swift-drive-audit (https://review.openstack.org/#/c/122468/).

Here, the "--drievaudit" option is added to swift-recon tool. This
feature gives the statistics for the system-wide drive errors flagged
by swift-drive-audit. An example of the output is as follows:
(verbose mode)

swift-recon --driveaudit -v
===============================================================================
--> Starting reconnaissance on 5 hosts
===============================================================================
[2015-03-11 17:13:39] Checking drive-audit errors
-> http://1.2.3.4:6000/recon/driveaudit: {'drive_audit_errors': 14}
-> http://1.2.3.5:6000/recon/driveaudit: {'drive_audit_errors': 0}
-> http://1.2.3.6:6000/recon/driveaudit: {'drive_audit_errors': 37}
-> http://1.2.3.7:6000/recon/driveaudit: {'drive_audit_errors': 101}
-> http://1.2.3.8:6000/recon/driveaudit: {'drive_audit_errors': 0}
[drive_audit_errors] low: 0, high: 101, avg: 30.4, total: 152, Failed: 0.0%, no_result: 0, reported: 5
===============================================================================

Change-Id: Ia16c52a9d613eeb3de1a5a428d88dd1233631912
2015-03-23 11:38:32 +00:00
Jenkins
fcabad2c3f Merge "Fix typo in swift/test/unit/account/test_backend.py" 2015-03-23 06:53:03 +00:00
Thiago da Silva
23d0842dec Refactoring the PUT method
Extracting large chunks of the PUT method into smaller
methods to improve maintainability and reuse of code.

Based on the work that Clay Gerrard started:
https://review.openstack.org/#/c/77812/

Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>

Change-Id: Id479fc5b159a2782361ac4a6e4a6d8bbaee4fe85
Signed-off-by: Thiago da Silva <thiago@redhat.com>
2015-03-21 12:30:58 -04:00
Jenkins
1bafdd9f53 Merge "Enable in-process functional test policy to be configured" 2015-03-21 01:44:36 +00:00
Mitsuhiro SHIGEMATSU
c3d2602c60 Fix typo in swift/test/unit/account/test_backend.py
Change-Id: I6cab110599019471b4005f1584fc26fa2a85d02e
2015-03-21 08:43:19 +09:00
Alistair Coles
7e64c811fe Enable in-process functional test policy to be configured
Currently the in-process tests build a 2 replica, 4 partition,
2 device object ring. This patch allows an alternative policy
and ring to be specified for testing via environment variables
that may optionally be set.

SWIFT_TEST_IN_PROCESS_CONF_DIR - This points the test setup to a
  directory which may have a swift.conf file and ring file. The
  test setup will then prefer these conf files over the samples
  in '/etc'.

SWIFT_TEST_POLICY - This causes the in-process test to
  use the specified policy from the swift.conf file and its
  associated ring for testing (first copying the conf and ring file
  and modifying device parameters to suit in-process testing). If
  not set, the tests will use the default policy.

The in-process tests now start sufficient object servers for the
ring file being tested against.

This should allow in-process functional testing of various policies
and rings (e.g. EC policies) without needing to reconfigure an SAIO
for each test scenario.

The refactoring of the in_process test setup code should also
allow easier addition of other 'hard-coded' test policies/rings
in the future.

Change-Id: I24f5a13de3d296b400da1691dcb53423a9f8a463
2015-03-20 16:13:57 +00:00
Kota Tsuyuzaki
9ae0a0702c Fix a lack of method arguments at tempauth
This fixes a kind of trivial issue with TypeError caused by a lack
of method argument at calling.

When using 'Authorization' header with tempauth, tempauth calls
_get_user_groups with 3 arguments (includes self) on current code
but the method requires 4 arguments so it will go to fail with 500.

This patch fixes to take correct arguments and adds an unit test
for the case.

Closes-Bug:1434465

Change-Id: Ibf00c41fa8e5cfdaf49bc4d571e575d7f229ebfe
2015-03-20 09:21:09 +00:00
Alistair Coles
a624a593df Tighten up recon middleware unit tests
Adds a test for get_expirer_info() and tightens up
the test for get_async_info().

While reviewing [1] I noticed that the existing tests on
master will not catch a bug creeping into the arg strings
that these methods pass to from_recon_cache().

[1] https://review.openstack.org/#/c/163889

Change-Id: Iee6f12aeff684d70435dbeea769ccc181021be79
2015-03-19 11:08:06 +00:00
Jenkins
b7fabcc84a Merge "Imported Translations from Transifex" 2015-03-18 13:24:52 +00:00
Alistair Coles
15b83f67d2 Test swift-object-info opens meta and ts files
Adds a unit test to verify the change made in [1], i.e. that
swift-object-info will read from .meta and .ts files as well
as .data files.

[1] change I43966d371218ad39414e9282cde579e48370a2a7

Change-Id: I82dde36e3a96db1a21cfe9a4cca0d941e543dfd0
Related-Bug: 1425679
2015-03-18 12:43:03 +00:00
Jenkins
bbc00d9c60 Merge "Allow swift-object-info to inspect .meta and .ts files" 2015-03-18 12:15:54 +00:00
OpenStack Proposal Bot
3f1feaee42 Imported Translations from Transifex
For more information about this automatic import see:
https://wiki.openstack.org/wiki/Translations/Infrastructure

Change-Id: I2e0a6053fec208b3eea9a68416fdc616da0631a9
2015-03-18 06:11:23 +00:00
Jenkins
e939b38c2c Merge "Add the missing "file" keyword for print" 2015-03-18 05:38:07 +00:00
Christian Schwede
3d3db0ab78 Bump PyECLib version to >= 1.0.3
ECPyECLibException has been removed in 1.0.1, among other
API/error handling enhancements in 1.0.3. Let's make sure
the right version is used.

Change-Id: Ie6e0c1b491017dd5096b14c23b610abc22b03d6a
2015-03-16 14:16:49 -07:00
Jenkins
2fa4906cb8 Merge "Fix the prefix of messages caputured from stderr" 2015-03-16 20:28:27 +00:00
Jenkins
f86bb65545 Merge "Fix ContainerBroker to use policy-0 in default" 2015-03-16 20:02:47 +00:00
Jiangmiao Gao
fdb2ec0cc7 Add the missing "file" keyword for print
All non-keyword arguments are converted to strings and written to
the stream, the argument "sys.stderr" should be given keyword
"file" as keyword argument.

Change-Id: I1f9c4e41ed7a1d3246eae9179fd9221c0a834292
2015-03-16 23:30:37 +08:00
OpenStack Proposal Bot
51a8f9c64f Imported Translations from Transifex
For more information about this automatic import see:
https://wiki.openstack.org/wiki/Translations/Infrastructure

Change-Id: I0058932520e008aa4bde7ce34d6c9f25424684b4
2015-03-16 06:10:17 +00:00
Kota Tsuyuzaki
99a3d1a1c5 Fix the prefix of messages caputured from stderr
Swift caputures both sys.stdout and sys.stderr messages and
transfers them to syslog. However, whole of the messages
uses the "STDOUT:" prefix, currently.

It will be quite confusable for debugging purpose because we
cannot make sure whether "Swift didn't caputure stderr" or
"Swift captured stderr but didn't show it" when we want to log
some stderr messages.

This patch enables to allow a new prefix "STDERR:" and use it
for sys.stderr.

Change-Id: Idf4649e9860b77c3600a5cd0a3e0bd674b31fb4f
2015-03-13 04:39:16 -07:00
Jenkins
09474f28b7 Merge "fsync() on directories" 2015-03-13 01:55:15 +00:00
Ricardo Ferreira
9baafe7f30 Allow swift-object-info to inspect .meta and .ts files
Removing the check for ".data" extension makes it check
*any* file for metadata, so it works with .meta and .ts
filetypes.

Change-Id: I43966d371218ad39414e9282cde579e48370a2a7
Closes-Bug:1425679
2015-03-10 10:51:40 +00:00
Jenkins
61f14f0e90 Merge "Fix param description on _make_request" 2015-03-10 00:50:28 +00:00
Jenkins
85bad0b6a0 Merge "Small optimization to ring builder." 2015-03-09 13:55:28 +00:00
Kota Tsuyuzaki
b73e49e790 Fix param description on _make_request
make_requests requires a list of headers but _make_request needs
just headers dictionary (not a list).

Change-Id: Iaf7ba32de3702bf2189f41e42041b561c9d1b57b
2015-03-09 03:18:25 -07:00
Samuel Merritt
98911be6d7 Small optimization to ring builder.
We were already checking this condition a few lines up; no need to do
it again.

Change-Id: I066c635c8dfa3c3a1e9a944decae2f41e2c689c9
2015-03-06 15:06:50 -08:00
Thiago da Silva
7f482e9a56 fixing small typos in associated projects doc
Change-Id: Ia4555acc1763a58cdf605530219ea4e4586ad517
Signed-off-by: Thiago da Silva <thiago@redhat.com>
2015-03-06 14:35:40 -05:00
Jenkins
a3e5c85ba6 Merge "Minor change to the service token sample test config" 2015-03-05 21:17:28 +00:00
Prashanth Pai
0eb27b96f2 Handle ENOSPC in mkstemp()
mkstemp() can fail with ENOSPC when filesystem runs out of inodes.
And fallocate() used to raise DiskFileNoSpace for all OSErrors.

Change-Id: I8c95cb710107d8e481d068b00eda53dd805c00a5
Signed-off-by: Prashanth Pai <ppai@redhat.com>
2015-03-05 19:01:38 +05:30