Adds encryption middlewares.
All object servers and proxy servers should be upgraded before
introducing encryption middleware.
Encryption middleware should be first introduced with the
encryption middleware disable_encryption option set to True.
Once all proxies have encryption middleware installed this
option may be set to False (the default).
Increases constraints.py:MAX_HEADER_COUNT by 4 to allow for
headers generated by encryption-related middleware.
Co-Authored-By: Tim Burke <tim.burke@gmail.com>
Co-Authored-By: Christian Cachin <cca@zurich.ibm.com>
Co-Authored-By: Mahati Chamarthy <mahati.chamarthy@gmail.com>
Co-Authored-By: Peter Chng <pchng@ca.ibm.com>
Co-Authored-By: Alistair Coles <alistair.coles@hpe.com>
Co-Authored-By: Jonathan Hinson <jlhinson@us.ibm.com>
Co-Authored-By: Hamdi Roumani <roumani@ca.ibm.com>
UpgradeImpact
Change-Id: Ie6db22697ceb1021baaa6bddcf8e41ae3acb5376
Commit cf48e75 changed the default account/container/object ports in a
lot of places, including the probetests. However, it didn't change
them in doc/saio/bin/remakerings, and since the probe tests must match
the rings, they started failing.
This commit just backs out the changes to the test/probe directory so
that remakerings and the probe tests match again.
Change-Id: I316a09e6ee1a911f37ce9df3d641644739f88eeb
Changing the recommended ports for Swift services
from ports 6000-6002 to unused ports 6200-6202;
so they do not conflict with X-Windows or other services.
Updated SAIO docs.
DocImpact
Closes-Bug: #1521339
Change-Id: Ie1c778b159792c8e259e2a54cb86051686ac9d18
This patch adds the erasure code reconstructor. It follows the
design of the replicator but:
- There is no notion of update() or update_deleted().
- There is a single job processor
- Jobs are processed partition by partition.
- At the end of processing a rebalanced or handoff partition, the
reconstructor will remove successfully reverted objects if any.
And various ssync changes such as the addition of reconstruct_fa()
function called from ssync_sender which performs the actual
reconstruction while sending the object to the receiver
Co-Authored-By: Alistair Coles <alistair.coles@hp.com>
Co-Authored-By: Thiago da Silva <thiago@redhat.com>
Co-Authored-By: John Dickinson <me@not.mn>
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Co-Authored-By: Tushar Gohad <tushar.gohad@intel.com>
Co-Authored-By: Samuel Merritt <sam@swiftstack.com>
Co-Authored-By: Christian Schwede <christian.schwede@enovance.com>
Co-Authored-By: Yuan Zhou <yuan.zhou@intel.com>
blueprint ec-reconstructor
Change-Id: I7d15620dc66ee646b223bb9fff700796cd6bef51
* refactor probe tests to use probe.common.ProbeTest
* move reset_environment functionality to ProbeTest.setUp()
* choose rings and policies that meet the criteria - raise SkipTest if
nothing matches
* replace all AssertionErrors in setup with SkipTest
Change-Id: Id56c497d58083f5fd55f5283cdd346840df039d3
Replaced throughout code base & tox'd. Functional as well
as probe tests pass with and without policies defined.
POLICY --> 'X-Storage-Policy'
POLICY_INDEX --> 'X-Backend-Storage-Policy-Index'
Change-Id: Iea3d06de80210e9e504e296d4572583d7ffabeac
Add headers param to direct_client.direct_get_object, which is used in
probetests to passthrough the X-Storage-Policy-Index header.
DocImpact
Implements: blueprint storage-policies
Change-Id: I19adbbcefbc086c8467bd904a275d55cde596412
Fix for a probe test that failed every once in a
while due to the early-majority change previously
committed. Sometimes a write would return success
before the third node had succeeded and the probe
test would look for on-disk evidence and fail,
when it would've been fine had it waited just a
bit longer for the third node to complete.
Since there's no real way for the probe test to
know when all three nodes are done, I just made
it retry once a second for several seconds before
reporting an error.
There may be more tests like this we'll have to
fix as we run across them.
Change-Id: I749e43d4580a7c726a9a8648f71bafefa70a05f5
As it happens, diskfile.read_metadata() and diskfile.write_metadata()
can take either an open file or a filename as their first arguments
(since xattr.[get|set]xattr() can), so we can clean up a couple places
where we were opening a file just to call read_metadata() or
write_metadata() on it. This results in 2 fewer system calls.
Example strace output:
/* read_metadata(filename) */
getxattr("/mnt/sdb1/1/node/sdb1/afile", "user.some.key", 0x0, 0) = 10
getxattr("/mnt/sdb1/1/node/sdb1/afile", "user.some.key", "some-value", 10) = 10
/* fp = open(filename); read_metadata(fp) */
open("/mnt/sdb1/1/node/sdb1/afile", O_RDONLY) = 4
fstat(4, {st_mode=S_IFREG|0664, st_size=0, ...}) = 0
fgetxattr(4, "user.some.key", 0x0, 0) = 10
fgetxattr(4, "user.some.key", "some-value", 10) = 10
Change-Id: I321d8663b9e9e47b8f3ee6c21a1b65b408bb80e6
This reverts commit 7760f41c3ce436cb23b4b8425db3749a3da33d32
Change-Id: I95e57a2563784a8cd5e995cc826afeac0eadbe62
Signed-off-by: Peter Portante <peter.portante@redhat.com>
except x,y: was deprected and is removed in Python 3.x.
Use "except x as y:" instead which works in any Python
version >= 2.6.
Change-Id: I7008c74b807340f3457d3a0c8bd0b83f23169d14
* new module swift.obj.diskfile
I parameterized two constants from obj.server into the DiskFile's __init__
* DATADIR -> obj_dir
* DISALLOWED_HEADERS -> disallowed_metadata_keys
I'm not sure if this is the right long term abstraction but for now it avoids
circular imports.
Change-Id: I3962202c07c4b2fbfc26f9776c8a5c96292ae199
Support separate replication ip address:
- Added new function in utils. This function provides ability
to select separate IP address for replication service.
- Db_replicator and object replicators were changed.
Replication process uses new function now.
Replication network parameters:
- Replication network fields (replication_ip, replication_port)
support was added to device dictionary in swift-ring-builder script.
- Changes were made to support new fields in search, show and set_info
functions.
Implementation of replication servers:
- Separate replication servers use the same code as normal replication
servers, but with replication_server parameter = True. When using a
separate replication network, the non-replication servers set
replication_server = False. When there is no separate replication
network (the default case), replication_server is not included in the config.
DocImpact
Change-Id: Ie9af5bdcdf9241c355e36053ca4adfe49dc35bd0
Implements: blueprint dedicated-replication-network
The probe tests were woefully out of date with all the changes that
have ocurred since they were written. I've updated most of them and
removed some that are hopeless outdated.
I also greatly improved the timing issues (hopefully completely
solved them? I ran them 25 times with no problems) and made them pep8
1.3.1 safe.
Change-Id: I8e9dbb6e7d6e04e293843b1dce1ded99d84e0348
Updated the imports and added a head_account to the "is the cluster
started yet?" checks. Hopefully this fixes the notorious timing
issues of these tests where auth answers requests just a bit before
the rest of the cluster is ready.
Fixes bug 1014931
Change-Id: Iea1d62db2317560371da49af5e94a0279b646294