6755 Commits

Author SHA1 Message Date
Tim Burke
0b60b0bd7e Close all versioned_writes subrequests' app_iters
While we're at it, drop some unused expect_exception code from
test_versioned_writes.py

Change-Id: Id232880c7259839159c057471b61083d5862d189
Related-Change: I430c48c4a81e8392fa271160bcbc1817ef0a88f7
2017-06-15 14:32:44 -07:00
Romain LE DISEZ
a7c8becd4e Fix a socket leak in copy middleware
When the "copy" middleware tries to copy a segmented object which is
bigger than max_file_size, it immediatly returns "413 Request Entity Too
Large". But at that point, connections have already been established by
the proxy server to the object servers. These connections must be closed
before returning.

Closes-Bug: #1698207
Change-Id: I430c48c4a81e8392fa271160bcbc1817ef0a88f7
2017-06-15 22:49:17 +02:00
Jenkins
6181351a65 Merge "Make mount_check option usable in containerized environments" 2017-06-14 19:49:23 +00:00
Jenkins
aed3edc522 Merge "Follow up for affinity config per policy" 2017-06-14 02:54:11 +00:00
Jenkins
73da215bdf Merge "Ring doc cleanups" 2017-06-14 02:54:03 +00:00
Jenkins
4315093a28 Merge "More Global EC doc updates" 2017-06-13 21:13:07 +00:00
Tim Burke
b5ee8c88d0 Ring doc cleanups
Change-Id: Ie51ea5c729341da793887e1e25c1e45301a96751
2017-06-13 09:23:23 -07:00
Kota Tsuyuzaki
066f44323d Follow up for affinity config per policy
This changes:
- Add assertions for write_affinity values in load_app tests
- Add a test case that policy override read_affinity default by timing
  strategy
- Avoid 'scope' but it's 'label'

Related-Change: I3f718f425f525baa80045ba067950c752bcaaefc
Change-Id: Ia8262490895d60da345f3679fc53653b2c2a2b3e
2017-06-13 11:53:11 +01:00
Clay Gerrard
4c7839d256 More Global EC doc updates
Soften the language about inefficiency on read and strengthen the
language encouraging the use of read affinity and composite rings.

Change-Id: Idc81a8c71e74ae28d384759700c5268d77ae3c85
2017-06-13 10:08:20 +01:00
Jenkins
41c8f1330f Merge "Update Global EC docs with reference to composite rings" 2017-06-13 06:26:54 +00:00
Alistair Coles
9665252352 Update Global EC docs with reference to composite rings
* In light of the composite rings feature being added [1],
  downgrade the warnings about EC Duplication [2] being
  experimental.

* Add links from Global EC docs to composite rings and
  per-policy proxy config features.

* Add discussion of using EC duplication with composite
  rings.

* Update Known Issues.

[1] Related-Change: I0d8928b55020592f8e75321d1f7678688301d797
[2] Related-Change: Idd155401982a2c48110c30b480966a863f6bd305

Change-Id: Id97a4899255945a6eaeacfef12fd29a2580588df
2017-06-12 16:58:02 -07:00
Jenkins
75c1bea7a5 Merge "Cleanup db replicator probetest" 2017-06-12 21:13:15 +00:00
Jenkins
7bbe02b290 Merge "Allow to configure the nameservers in cname_lookup" 2017-06-12 19:48:45 +00:00
Jenkins
53b356b1ee Merge "Fix unit test failing when swift.conf has default policy index >10" 2017-06-12 19:22:26 +00:00
Jenkins
8aa560dc9c Merge "remove remote qualifier from release notes branch scanning" 2017-06-12 19:18:47 +00:00
Jenkins
d46b0f29f9 Merge "Limit number of revert tombstone SSYNC requests" 2017-06-08 18:10:08 +00:00
Mahati Chamarthy
188c07e12a Limit number of revert tombstone SSYNC requests
Revert tombstone only parts try to talk to all primary nodes - this
fixes it to randomize selection within part_nodes. Corresponding probe
test is modified to reflect this change.

The primary improvement of this patch is the reconstuctor at a handoff
node is being able to delete local tombstones when it succeeds to sync
to less than all primary nodes. (Before this patch, it requires all
nodes are responsible for the REVERT requests)

The number of primary nodes to communicate with the reconstructor can be
in dicsussion more but, right now with this patch, it's (replicas - k + 1)
that is able to prevent stale read.

*BONUS*

- Fix mis-testsetting (was setting less replicas than ec_k + ec_m)
  for reconstructor ring in the unit test

Co-Authored-By: Kota Tsuyuzaki <tsuyuzaki.kota@lab.ntt.co.jp>
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>

Change-Id: I05ce8fe75f1c4a7971cc8995b003df818b69b3c1
Closes-Bug: #1668857
2017-06-08 07:07:42 +00:00
Jenkins
bb80773153 Merge "Using assertIsNone() instead of assertEqual(None)" 2017-06-08 01:13:49 +00:00
Jenkins
c36438aa26 Merge "Use config_number method instead of node_id + 1" 2017-06-08 01:13:41 +00:00
Jenkins
e640563b18 Merge "Sync metadata in 'rsync_then_merge' in db_replicator" 2017-06-08 00:43:43 +00:00
Clay Gerrard
6687d2fcd0 Cleanup db replicator probetest
Use a manager and config_number helper for clarity - then make a one
last assertion on the final consistent state.

Change-Id: I5030c314076003d17c41b8b136bcbda252474bad
Related-Change-Id: Icdf0a936fc456c5462471938cbc365bd012b05d4
2017-06-07 16:21:08 -07:00
Kota Tsuyuzaki
c23b6db264 Use config_number method instead of node_id + 1
Change-Id: I596a2ac947f7e5f9c0cb3286779ece5e40feefd0
2017-06-07 01:44:04 -07:00
lingyongxu
ee9458a250 Using assertIsNone() instead of assertEqual(None)
Following OpenStack Style Guidelines:
[1] http://docs.openstack.org/developer/hacking/#unit-tests-and-assertraises
[H203] Unit test assertions tend to give better messages for more specific
assertions. As a result, assertIsNone(...) is preferred over
assertEqual(None, ...) and assertIs(..., None)

Change-Id: If4db8872c4f5705c1fff017c4891626e9ce4d1e4
2017-06-07 14:05:53 +08:00
lingyongxu
8d6eb36779 Using assertIsNone() instead of assertEqual(None)
Following OpenStack Style Guidelines:
[1] http://docs.openstack.org/developer/hacking/#unit-tests-and-assertraises
[H203] Unit test assertions tend to give better messages for more specific
assertions. As a result, assertIsNone(...) is preferred over
assertEqual(None, ...) and assertIs(..., None)

Change-Id: I5b5900c509956cf13ec3d1f2105fc7c89531f22a
2017-06-07 09:08:46 +08:00
Jenkins
e11eb88ad9 Merge "Remove deprecated vm_test_mode option" 2017-06-06 05:42:14 +00:00
Romain LE DISEZ
5d7b0d1edb Delay port binding to reduce wait at process start
With the new default behavior of eventlet 0.20 to set SO_REUSEPORT on listening
sockets, it becomes easy to have a zero-downtime reload.

By binding the listening socket as late as we can, we can reduce the
time an incoming connection will stay in queue, waiting for the process
to be ready to handle the connection.

This commit moves the socket binding after the load of the application.

On some benchmarks, it reduced the wait-time for incoming connections from
about 1.1s to about 350ms.

Change-Id: I045f4c9aa2c07cdc5ed95afd028540f1b3874637
2017-06-05 11:42:52 +02:00
Jenkins
0d5b2a867d Merge "Follow-up for per-policy proxy configs" 2017-06-05 08:50:45 +00:00
Tim Burke
d51ecb4ecc Remove threads_per_disk from object-server.conf manpages
That option was removed entirely in 2.8.0.

Change-Id: Ib40f816936429a78e622d3737bb0b064225d2d44
Related-Change: Ie76be5c8a74d60a1330627caace19e06d1b9383c
2017-06-01 14:07:54 -07:00
Romain LE DISEZ
420e73fabd Allow to configure the nameservers in cname_lookup
For various reasons, an operator might want to use specifics nameservers
instead of the systems ones to resolve CNAME in cname_lookup. This patch
creates a new configuration variable nameservers which accepts a list of
nameservers separated by commas. If not specified or empty, systems
namservers are used as previously.

Co-Authored-By: Tim Burke <tim.burke@gmail.com>
Change-Id: I34219e6ab7e45678c1a80ff76a1ac0730c64ddde
2017-06-01 14:02:08 -07:00
Jenkins
6fc45361c5 Merge "README: dedent the numbered list" 2017-06-01 20:54:20 +00:00
Jenkins
a6fb2076bb Merge "Rewrite redirection in cname_lookup & domain_remap" 2017-06-01 20:49:14 +00:00
Tim Burke
5ecf828b17 Follow-up for per-policy proxy configs
* Only use one StringIO in ConfigString
* Rename the write_affinity_node_count function to be
  write_affinity_node_count_fn
* Use comprehensions instead of six.moves.filter
* Rename OverrideConf to ProxyOverrideOptions
* Make ProxyOverrideOptions's __repr__ eval()able
* Various conf -> options renames
* Stop trying to handle a KeyError that should never come up
* Be explicit about how deep we need to copy in proxy/test_server.py
* Drop an unused return value
* Add a test for a non-"proxy-server" app name
* Combine bad-section-name tests
* Try to clean up (at least a little) a self-described "hokey test"

Related-Change: I3f718f425f525baa80045ba067950c752bcaaefc
Change-Id: I4e81175d5445049bc1f48b3ac02c5bc0f77e6f59
2017-06-01 20:36:19 +00:00
Tim Burke
61bcd75971 README: dedent the numbered list
It looks funny otherwise.

Change-Id: Ib00872e050a2885e43085f907780bc9ff0d2b381
2017-06-01 18:15:24 +00:00
Jenkins
1b24b0eb7a Merge "Make probe tests work when policy-0 is EC" 2017-06-01 17:26:14 +00:00
Doug Hellmann
a3aba99a29 remove remote qualifier from release notes branch scanning
Reno will automatically add the origin remote prefix to the branch name
if it does not find a local branch with the same name, so the input
instructions don't need to include "origin/". Removing them allows a
local developer to experiment with changes on local branches, and should
have no impact on the CI system build results.

Depends-On: Id30888b77f46b1f163c0a8175033b96aa3f9bff8
Change-Id: I819fa75d55f96320a24fc4182e5f1cb63c678d16
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
2017-06-01 15:16:56 +00:00
Jenkins
f5ded0519c Merge "Trivial fix typos" 2017-06-01 03:09:01 +00:00
Tim Burke
f64fa46f3a Make probe tests work when policy-0 is EC
test_object_metadata_replication requires a replicated policy. We even
have it subclass ReplProbeTest, but then we hardcoded the policy index!
Stop doing that.

Change-Id: I8871cc0beceb0909abaf59babe40c3cafbcd0cc9
2017-05-31 17:07:39 -07:00
Jenkins
45b17d89c7 Merge "Fix SSYNC failing to replicate unexpired object" 2017-05-31 22:49:55 +00:00
Jenkins
db45e1dd69 Merge "Add structure to storage policy configuration guide" 2017-05-31 21:36:13 +00:00
Tim Burke
c34ac98c90 Stop having Sphinx treat warnings as errors
Warnings-as-errors breaks our docs gate currently, as the gate is using
an old liberasurecode. Stop treating warnings as errors until the gate
can be updated with a more-recent version.

Related-Change: I3985d027f0ac2119ceaeb4daba5964f937de6cea
Change-Id: I4869ec370f08e338be78d2c0f930106081f21a0a
2017-05-31 09:47:40 -07:00
Alistair Coles
37ba21face Add structure to storage policy configuration guide
The description of storage policy config options was
unstructured and repetitive. This patch attempts to
improve the doc by gathering the notes for each option
into a structured list.

Change-Id: I57090b35a70f365e82fb0e29ab42e533d6359a7b
2017-05-31 11:11:32 +01:00
Jenkins
b9322a2f08 Merge "Add link from policies overview to per-policy proxy-server conf" 2017-05-30 19:13:56 +00:00
Vu Cong Tuan
cf3c970a77 Trivial fix typos
Change-Id: I7e1e3b2f92183b2a249299659f0778fe838212e2
2017-05-30 14:26:23 +07:00
Alistair Coles
227cef9933 Add link from policies overview to per-policy proxy-server conf
- add proxy server per policy config as an optional
  step in the configuration of a policy, with link to
  the deployment guide

- add reverse link from deployment guide per-policy
  config doc section to storage policies docs

Drive-by fix an incorrect test comment

Change-Id: Ib95310193270a63c9d1e321c6e7de240e00b387f
Related-Change: I3f718f425f525baa80045ba067950c752bcaaefc
2017-05-26 10:41:35 +01:00
Tim Burke
675145ef4a Remove deprecated vm_test_mode option
This was deprecated in the 2.5.0 release (i.e. Liberty cycle), and we've
been warning about it ever since. A year and a half seems like a long
enough time.

Change-Id: I5688e8f7dedb534071e67d799252bf0b2ccdd9b6
Related-Change: Iad91df50dadbe96c921181797799b4444323ce2e
2017-05-25 13:02:42 -07:00
Tim Burke
d487bf7fb1 Remove tempauth docs from deployment guide
Instead, link to the middleware list and auth overview, as well as
referring readers to proxy-server.conf-sample

TempAuth-related content that was previously in the deployment guide has
been moved to TempAuth's own docs, which have been cleaned up a bit.

Change-Id: I00070bb09294362c069f7ee9426ac570bc1b3ddb
2017-05-25 12:35:46 -07:00
Jenkins
96de9ad126 Merge "Clean up how PatchPolicies works" 2017-05-25 09:20:52 +00:00
Jenkins
263dc8a3f3 Merge "Enable per policy proxy config options" 2017-05-25 06:34:48 +00:00
Jenkins
24e8789689 Merge "Small minor fixes for composite ring functionality" 2017-05-24 22:57:30 +00:00
Jenkins
6f7b1f9ee2 Merge "Use setUpModule instead of setup for module level unit test setup" 2017-05-24 21:29:09 +00:00