This is an alternative approach to that proposed in [1]
Adds support for optional per-policy config sections
to be added in proxy-server.conf. This is highly desirable
to allow per-policy affinity options to be set for use with
duplicated EC policies [2] and composite rings [3].
Certain options found in per-policy conf sections will
override their equivalents that may be set in the
[app:proxy-server] section. Currently the options
handled that way are:
sorting_method
read_affinity
write_affinity
write_affinity_node_count
For example:
[proxy-server:policy:0]
sorting_method = affinity
read_affinity = r1=100
write_affinity = r1
write_affinity_node_count = 1 * replicas
The corresponding attributes of the proxy-server Application
are now available from instances of an OverrideConf object
that is obtained from Application.get_policy_options(policy).
[1] Related-Change: I9104fc789ba85ab3ab5ccd34096125b482821389
[2] Related-Change: Idd155401982a2c48110c30b480966a863f6bd305
[3] Related-Change: I0d8928b55020592f8e75321d1f7678688301d797
Co-Authored-By: Kota Tsuyuzaki <tsuyuzaki.kota@lab.ntt.co.jp>
Change-Id: I3f718f425f525baa80045ba067950c752bcaaefc
We've got these lovely __enter__ and __exit__ methods; let's use them!
Note that this also changes how we patch classes' setUp methods so we
don't set self._orig_POLICIES when the class is already patched. I
hope this may fix some sporadic failures that include tracebacks
that look like
proxy ERROR: ERROR 500 Traceback (most recent call last):
File ".../swift/obj/server.py", line 1105, in __call__
res = getattr(self, req.method)(req)
File ".../swift/common/utils.py", line 1626, in _timing_stats
resp = func(ctrl, *args, **kwargs)
File ".../swift/obj/server.py", line 880, in GET
policy=policy, frag_prefs=frag_prefs)
File ".../swift/obj/server.py", line 211, in get_diskfile
return self._diskfile_router[policy].get_diskfile(
File ".../swift/obj/diskfile.py", line 555, in __getitem__
return self.policy_to_manager[policy]
KeyError: ECStoragePolicy(...)
... and try to unpatch more gracefully with TestCase.addCleanup
Change-Id: Iaa3d42ec21758b0707155878a645e665aa36696c
The ismount_raw method does not work inside containers if disks are
mounted on the hostsystem and only mountpoints are exposed inside the
containers. In this case the inode and device checks fail, making this
option unusable.
Mounting devices into the containers would solve this. However, this
would require that all processes that require access to a device are
running inside the same container, which counteracts the container
concept.
This patch adds the possiblity to place stubfiles named ".ismount" into
the root directory of any device, and Swift assumes a given device to be
mounted if that file exists. This should be transparent to existing
clusters.
Change-Id: I9d9fc0a4447a8c5dd39ca60b274c119af6b4c28f
Add entries for these options in the deployment guide and
make the text in proxy-server.conf-sample and man page
consistent.
Change-Id: I5854ddb3e5864ddbeaf9ac2c930bfafdb47517c3
This is a follow up for https://review.openstack.org/#/c/441921
all of this patch is for minor fixes, I found in my self-review.
Change-Id: Ib3a1dc983c3da69dea592114e25a5047ec91a2b9
Previously, we were testing that a 254 (!?) character name would be valid
when the maximum configured is 500. Now we'll test that 500 character
names are valid.
While we're at it, stop patching self.test_check. It was unnecessary,
and we were doing it badly.
Change-Id: Ia604fa7b809a97fbce176c82606af73cdb92828c
* Adds a composite_builder module which provides the functionality to
build a composite ring from a number of component ring builders.
* Add id to RingBuilder to differentiate rings in composite.
A RingBuilder now gets a UUID when it is saved to file if
it does not already have one. A RingBuilder loaded from
file does NOT get a UUID assigned unless it was previously persisted in
the file. This forces users to explicitly assign an id to
existing ring builders by saving the state back to file.
The UUID is included in first line of the output from:
swift-ring-builder <builder-file>
Background:
This is another implementation for Composite Ring [1]
to enable better dispersion for global erasure coded cluster.
The most significant difference from the related-change [1] is that this
solution attempts to solve the problem as an offline tool rather than
dynamic compositing on the running servers. Due to the change, we gain
advantages such as:
- Less code and being simple
- No complex state validation on the running server
- Easy deployments with an offline tool
This patch does not provide a command line utility for managing
composite rings. The interface for such a tool is still under
discussion; this patch provides the enabling functionality first.
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
[1] Related-Change: I80ef36d3ac4d4b7c97a1d034b7fc8e0dc2214d16
Change-Id: I0d8928b55020592f8e75321d1f7678688301d797
The patch that introduced this (with my own help, no less),
also pasted these in tests, but someone else has already
cleaned that up.
Change-Id: I87c0a110831b5b5f75cb6fde1badec338f2150a9
In previous 'rsync_then_merge' remote objects are merged with
rsync'ed local objects, but remote metadata is not merged with local
one. Account/Container replicator sometimes uses rsync for db sync if
there is a big difference of record history in db files between 'local'
and 'remote' servers. If replicator needs to rsync local db to remote
but metadata in local db is older, older info of metadata can be
distributed then some metadata values can be missing or go back to older.
This patch fixes this problem by merging 'remote' metadata with
rsync'ed local db file.
Closes-Bug: #1570118
Change-Id: Icdf0a936fc456c5462471938cbc365bd012b05d4
cname_lookup and domain_remap are currently not catching redirections
(eg: staticweb). This behavior makes the domain to change when a call
through cname_lookup and domain_remap end up being redirected. Example:
commit fixes it.
- original request: http://mysite.com/subdir
- redirected: http://cont.acct.storage.domain.com/v1/AUTH_acct/cont/subdir/
- expected: http://mysite.com/subdir/
This patch is fixing this.
Closes-Bug: #1190625
Co-Authored-By: Tim Burke <tim.burke@gmail.com>
Change-Id: I67f642b8b070bc21e7760477d0a1e3b902ba7896
domain_remap strip all starting/ending slashes. This behavior does not
allow to access objects starting or ending with slash. It is also
impacting staticweb middleware as staticweb tries to redirect
pseudo-directory to an URL with an ending slash, but as domain_remap
strip it, it goes to an infinite loop.
With this commit, the path of the request is passed as-is when
reconstructing the new request path. Example
http://www.example.com//obj/ was previously rewritten to
http://storage.example.com/v1/AUTH_abc/container/obj. It is now
rewritten to http://storage.example.com/v1/AUTH_abc/container//obj/
Closes-Bug: #1682293
Co-Authored-By: Christian Schwede <cschwede@redhat.com>
Change-Id: I1ef6b8752183d27103a3b0e720edcb4ce06fb837
Recently out gate started blowing up intermittently with a strange
case of ports mixed up. Sometimes a functional tests tries to
authorize on a port that's clearly an object server port, and
the like. As it turns out, eventlet developers added an unavoidable
SO_REUSEPORT into listen(), which makes listen(("localhost",0)
to reuse ports.
There's an issue about it:
https://github.com/eventlet/eventlet/issues/411
This patch is working around the problem while eventlet people
consider the issue.
Change-Id: I67522909f96495a6a30e1acdb79835dce2189549
We've seen some failures in the gate like
==============================
Failed 1 tests - output below:
==============================
setUpModule (test.functional.test_account)
------------------------------------------
Captured traceback:
~~~~~~~~~~~~~~~~~~~
Traceback (most recent call last):
File "test/functional/test_account.py", line 33, in setUpModule
tf.setup_package()
File "test/functional/__init__.py", line 716, in setup_package
mem_object_server if in_mem_obj else object_server))
File "test/functional/__init__.py", line 621, in in_process_setup
create_account(AUTH_test)
File "test/functional/__init__.py", line 619, in create_account
assert(resp.status == 201)
AssertionError
...which aren't terribly useful in figuring out what went wrong.
Change-Id: I3cd31bb480dc8508828fe21416bfae33bc0985b7
Module setup() and teardown() functions are found by nosetests [1] but
unittests expects setUpModule() and tearDownModule() [2]. The latter
function names are also found by nosetests, so using those function
names enables the test module to be run with either nosetests or
unittest.
Although the tox test envs and .unittests script use nosetests, this
change allows the convenience of using unittest, for example when it
is the default test runner in a development environment such as
PyCharm.
This change also makes it unnecessary to explicitly call the setup()
and teardown() functions when executing the module directly.
[1] http://nose.readthedocs.io/en/latest/writing_tests.html#test-modules
[2] https://docs.python.org/2/library/unittest.html#setupmodule-and-teardownmodule
Change-Id: Ib2e5470a339af1f937b25d643b64356e8848ed36
We don't know the order in which connections will be made, so grab the
ip/port/dev from the mocked conn's request log.
Change-Id: I3b4486c99ad85173d5027b46e1c0613202d0f75a
Currently, EC GET responses from proxy to clients, unlike any other
response, include a "Connection: close" header. If the client has sent
a "Connection: keep-alive" header then eventlet.wsgi appends this to
the client response, so clients can receive a response with both
headers:
Connection: close
Connection: keep-alive
This patch fixes the proxy EC GET path to remove any Connection header
from it's response, but does not change the behaviour of eventlet.wsgi
with respect to returning any client provided 'Connection: keep-alive'
header.
Change-Id: I43cd27c978edb4a1a587f031dbbee26e9acdc920
Co-Authored-By: Matthew Oliver <matt@oliver.net.au>
Closes-Bug: #1680731
In unit/container/test_backend.py test_policy_stat_tracking in
classes TestContainerBroker[BeforeMetadata|BeforeSPI|BeforeXSync]
fails if the default policy in /etc/swift/swift.conf has an index >10.
Those classes monkey patch the container broker to pre-storage-policy
index behaviour, so that it's policy index will always be 0. The test
fails with a KeyError when asserting that the broker should have stats
for the POLICIES.default index even when no object have been PUT with
that policy index. When the default policy in swift.conf has index
>10, that is neither the broker's policy index (0) nor the policy
index of any other object that has been PUT during the test.
The test need a patch_policies decorator to remove the coupling with
swift.conf policies. However, the assertion that the broker has stats
for it's policy index even when no objects were PUT to that index is
then extremely unlikely to ever be tested, because the broker's
default policy index of 0 is very likely to have used for a PUT
object. So this patch also repeats that assertion before any object
have been PUT
Closes-Bug: #1687029
Change-Id: I8b3678dac83f7329d835059c9973b994bc975a33
From the docs for LogRecord.message [1],
> This is set when Formatter.format() is invoked.
Apparently we may find ourselves in a situation [2] where that never
happens? Really weird that it failed *midway* through the test though;
maybe some concurrent test removed all formatters?
ERROR: test_known_bad_ec_config
(test.unit.common.test_storage_policy.TestStoragePolicies)
----------------------------------------------------------------------
Traceback (most recent call last):
File ".../mock/mock.py", line 1305, in patched
return func(*args, **keywargs)
File ".../test/unit/common/test_storage_policy.py", line 688, in test_known_bad_ec_config
self.assertIn(msg, records[0].message)
AttributeError: 'LogRecord' object has no attribute 'message'
[1] https://docs.python.org/2/library/logging.html#logrecord-attributes
[2] http://logs.openstack.org/59/460359/1/check/gate-swift-tox-xfs-tmp-py27-ubuntu-xenial/5ecc2cb/console.html#_2017-04-27_01_06_43_346096
Change-Id: I8f5ac0ec1195a233f14edc0126de1d1cea7a6e2f
The name of the default domain in keystone is 'Default', not 'default',
which is its ID. If a user copy and pastes the user_domain_name and
project_domain_name settings from example [filter:authtoken] section
in the install guide, the first time they run 'swift stat' they will
meet a 401 error because keystone is trying to look up domains by name
rather than ID. This patch corrects the docs so that they are
copy-and-paste-proof, and also updates the keystone user creation
example to demonstrate the correct domain ID.
Change-Id: Ic595c2923b71d1c2ff2b4c9a773ea7742fdd029b
Often, we want the current timestamp. May as well improve the ergonomics
a bit and provide a class method for it.
Change-Id: I3581c635c094a8c4339e9b770331a03eab704074
