28 Commits

Author SHA1 Message Date
Ondřej Nový
33c18c579e Remove executable flag from some test modules
Change-Id: I36560c2b54c43d1674b007b8105200869b5f7987
2016-10-31 21:22:10 +00:00
Janie Richling
96a0e07753 Enable object body and metadata encryption
Adds encryption middlewares.

All object servers and proxy servers should be upgraded before
introducing encryption middleware.

Encryption middleware should be first introduced with the
encryption middleware disable_encryption option set to True.
Once all proxies have encryption middleware installed this
option may be set to False (the default).

Increases constraints.py:MAX_HEADER_COUNT by 4 to allow for
headers generated by encryption-related middleware.

Co-Authored-By: Tim Burke <tim.burke@gmail.com>
Co-Authored-By: Christian Cachin <cca@zurich.ibm.com>
Co-Authored-By: Mahati Chamarthy <mahati.chamarthy@gmail.com>
Co-Authored-By: Peter Chng <pchng@ca.ibm.com>
Co-Authored-By: Alistair Coles <alistair.coles@hpe.com>
Co-Authored-By: Jonathan Hinson <jlhinson@us.ibm.com>
Co-Authored-By: Hamdi Roumani <roumani@ca.ibm.com>

UpgradeImpact

Change-Id: Ie6db22697ceb1021baaa6bddcf8e41ae3acb5376
2016-06-30 23:31:15 -07:00
Samuel Merritt
99305b9300 Fix probe tests from commit cf48e75
Commit cf48e75 changed the default account/container/object ports in a
lot of places, including the probetests. However, it didn't change
them in doc/saio/bin/remakerings, and since the probe tests must match
the rings, they started failing.

This commit just backs out the changes to the test/probe directory so
that remakerings and the probe tests match again.

Change-Id: I316a09e6ee1a911f37ce9df3d641644739f88eeb
2016-05-02 17:29:32 -07:00
Shashirekha Gundur
cf48e75c25 change default ports for servers
Changing the recommended ports for Swift services
from ports 6000-6002 to unused ports 6200-6202;
so they do not conflict with X-Windows or other services.

Updated SAIO docs.

DocImpact
Closes-Bug: #1521339
Change-Id: Ie1c778b159792c8e259e2a54cb86051686ac9d18
2016-04-29 14:47:38 -04:00
Bill Huber
239e94e625 pep8 fix: assertEquals -> assertEqual
assertEquals is deprecated in py3 in the following dir:
test/probe/*

Change-Id: Ie08dd7a8a6c48e3452dfe4f2b41676330ce455d5
2015-08-06 09:28:51 -05:00
Victor Stinner
e5c962a28c Replace xrange() with six.moves.range()
Patch generated by the xrange operation of the sixer tool:
https://pypi.python.org/pypi/sixer

Manual changes:

* Fix indentation for pep8 checks
* Fix TestGreenthreadSafeIterator.test_access_is_serialized of
  test.unit.common.test_utils:
  replace range(1, 11) with list(range(1, 11))
* Fix UnsafeXrange docstring, revert change

Change-Id: Icb7e26135c5e57b5302b8bfe066b33cafe69fe4d
2015-06-23 07:29:15 +00:00
paul luse
647b66a2ce Erasure Code Reconstructor
This patch adds the erasure code reconstructor. It follows the
design of the replicator but:
  - There is no notion of update() or update_deleted().
  - There is a single job processor
  - Jobs are processed partition by partition.
  - At the end of processing a rebalanced or handoff partition, the
    reconstructor will remove successfully reverted objects if any.

And various ssync changes such as the addition of reconstruct_fa()
function called from ssync_sender which performs the actual
reconstruction while sending the object to the receiver

Co-Authored-By: Alistair Coles <alistair.coles@hp.com>
Co-Authored-By: Thiago da Silva <thiago@redhat.com>
Co-Authored-By: John Dickinson <me@not.mn>
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Co-Authored-By: Tushar Gohad <tushar.gohad@intel.com>
Co-Authored-By: Samuel Merritt <sam@swiftstack.com>
Co-Authored-By: Christian Schwede <christian.schwede@enovance.com>
Co-Authored-By: Yuan Zhou <yuan.zhou@intel.com>
blueprint ec-reconstructor
Change-Id: I7d15620dc66ee646b223bb9fff700796cd6bef51
2015-04-14 00:52:17 -07:00
Leah Klearman
2c1b5af062 refactor probe tests
* refactor probe tests to use probe.common.ProbeTest
* move reset_environment functionality to ProbeTest.setUp()
* choose rings and policies that meet the criteria - raise SkipTest if
nothing matches
* replace all AssertionErrors in setup with SkipTest

Change-Id: Id56c497d58083f5fd55f5283cdd346840df039d3
2015-02-12 11:30:21 -08:00
Paul Luse
873c52e608 Replace POLICY and POLICY_INDEX with string literals
Replaced throughout code base &  tox'd. Functional as well
as probe tests pass with and without policies defined.

POLICY --> 'X-Storage-Policy'
POLICY_INDEX --> 'X-Backend-Storage-Policy-Index'

Change-Id: Iea3d06de80210e9e504e296d4572583d7ffabeac
2014-06-23 12:52:50 -07:00
Yuan Zhou
ad2a9cefe5 Fixes probe tests with non-zero default storage policy
Add headers param to direct_client.direct_get_object, which is used in
probetests to passthrough the X-Storage-Policy-Index header.

DocImpact
Implements: blueprint storage-policies
Change-Id: I19adbbcefbc086c8467bd904a275d55cde596412
2014-06-18 21:09:53 -07:00
Clay Gerrard
0efac0cac2 make probetests work with conf.d configs
Change-Id: I451ff4629730a334ac1bd8fc6cd75de95314e153
2014-03-12 12:04:45 -07:00
Chmouel Boudjnah
150f338fc2 Remove swiftclient dep on direct_client
Partial Implements: blueprint remove-swiftclient-dependency
Change-Id: I9af7150e5d21d50e5f880e57796314b8f05822d2
2013-12-24 03:11:43 -08:00
gholt
985c7bf38b Fix probe test
Fix for a probe test that failed every once in a
while due to the early-majority change previously
committed. Sometimes a write would return success
before the third node had succeeded and the probe
test would look for on-disk evidence and fail,
when it would've been fine had it waited just a
bit longer for the third node to complete.

Since there's no real way for the probe test to
know when all three nodes are done, I just made
it retry once a second for several seconds before
reporting an error.

There may be more tests like this we'll have to
fix as we run across them.

Change-Id: I749e43d4580a7c726a9a8648f71bafefa70a05f5
2013-11-12 18:09:36 +00:00
Samuel Merritt
0a26bb20b1 Simplify callers of diskfile.[read|write]_metadata()
As it happens, diskfile.read_metadata() and diskfile.write_metadata()
can take either an open file or a filename as their first arguments
(since xattr.[get|set]xattr() can), so we can clean up a couple places
where we were opening a file just to call read_metadata() or
write_metadata() on it. This results in 2 fewer system calls.

Example strace output:

/* read_metadata(filename) */
getxattr("/mnt/sdb1/1/node/sdb1/afile", "user.some.key", 0x0, 0) = 10
getxattr("/mnt/sdb1/1/node/sdb1/afile", "user.some.key", "some-value", 10) = 10

/* fp = open(filename); read_metadata(fp) */
open("/mnt/sdb1/1/node/sdb1/afile", O_RDONLY) = 4
fstat(4, {st_mode=S_IFREG|0664, st_size=0, ...}) = 0
fgetxattr(4, "user.some.key", 0x0, 0)   = 10
fgetxattr(4, "user.some.key", "some-value", 10) = 10

Change-Id: I321d8663b9e9e47b8f3ee6c21a1b65b408bb80e6
2013-10-24 11:23:37 -07:00
Peter Portante
9411a24ba7 Revert "Refactor common/utils methods to common/ondisk"
This reverts commit 7760f41c3ce436cb23b4b8425db3749a3da33d32

Change-Id: I95e57a2563784a8cd5e995cc826afeac0eadbe62
Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-10-07 17:18:09 -04:00
Jenkins
0b594bc3af Merge "Change OpenStack LLC to Foundation" 2013-10-07 16:09:37 +00:00
gholt
58efcb3b3e Fix probe tests
Change-Id: I03573bf24baf031b1874c3ff2e2d89d34473c266
2013-09-19 17:14:35 +00:00
ZhiQiang Fan
f72704fc82 Change OpenStack LLC to Foundation
Change-Id: I7c3df47c31759dbeb3105f8883e2688ada848d58
Closes-bug: #1214176
2013-09-20 01:02:31 +08:00
Dirk Mueller
3d36a76156 Use Python 3.x compatible except construct
except x,y: was deprected and is removed in Python 3.x.
Use "except x as y:" instead which works in any Python
version >= 2.6.

Change-Id: I7008c74b807340f3457d3a0c8bd0b83f23169d14
2013-09-07 10:50:54 +02:00
Clay Gerrard
c9de9f2b8d Forklift the DiskFile interface into it's own module
* new module swift.obj.diskfile

I parameterized two constants from obj.server into the DiskFile's __init__

 * DATADIR -> obj_dir
 * DISALLOWED_HEADERS -> disallowed_metadata_keys

I'm not sure if this is the right long term abstraction but for now it avoids
circular imports.

Change-Id: I3962202c07c4b2fbfc26f9776c8a5c96292ae199
2013-07-18 08:00:14 -07:00
Sergey Kraynev
ea7858176b Implementation of replication servers
Support separate replication ip address:
- Added new function in utils. This function provides ability
  to select separate IP address for replication service.
- Db_replicator and object replicators were changed.
  Replication process uses new function now.

Replication network parameters:
- Replication network fields (replication_ip, replication_port)
  support was added to device dictionary in swift-ring-builder script.
- Changes were made to support new fields in search, show and set_info
  functions.

Implementation of replication servers:
- Separate replication servers use the same code as normal replication
  servers, but with replication_server parameter = True.  When using a
  separate replication network, the non-replication servers set
  replication_server = False.  When there is no separate replication
  network (the default case), replication_server is not included in the config.

DocImpact
Change-Id: Ie9af5bdcdf9241c355e36053ca4adfe49dc35bd0
Implements: blueprint dedicated-replication-network
2013-04-21 18:14:42 -04:00
gholt
4a9d19197c Updated probe tests
The probe tests were woefully out of date with all the changes that
have ocurred since they were written. I've updated most of them and
removed some that are hopeless outdated.

I also greatly improved the timing issues (hopefully completely
solved them? I ran them 25 times with no problems) and made them pep8
1.3.1 safe.

Change-Id: I8e9dbb6e7d6e04e293843b1dce1ded99d84e0348
2012-07-06 23:30:12 +00:00
gholt
be79b0884e Fixes for probe tests
Updated the imports and added a head_account to the "is the cluster
started yet?" checks. Hopefully this fixes the notorious timing
issues of these tests where auth answers requests just a bit before
the rest of the cluster is ready.

Fixes bug 1014931

Change-Id: Iea1d62db2317560371da49af5e94a0279b646294
2012-06-27 05:09:53 +00:00
John Dickinson
1ecf5ebba1 updated copyright date for all files
Change-Id: Ifd909d3561c2647770a7e0caa3cd91acd1b4f298
2012-03-19 13:45:34 -05:00
David Goetz
be01135c75 make executable 2011-03-28 21:03:19 -07:00
David Goetz
863393c823 pep8 2011-03-16 09:04:00 -07:00
David Goetz
5482035bc0 change ite logic and add more tests 2011-03-16 07:55:07 -07:00
David Goetz
86c84c3bc3 unittests workng and added probe test 2011-03-15 22:12:03 -07:00