Commit Graph

983 Commits

Author SHA1 Message Date
2596b3ca9d Merge "Let clients request heartbeats during SLO PUTs" 2017-11-03 16:05:18 +00:00
77a8a4455d Let clients request heartbeats during SLO PUTs
An SLO PUT requires that we HEAD every referenced object; as a result, it
can be a very time-intensive operation. This makes it difficult as a
client to differentiate between a proxy-server that's still doing work and
one that's crashed but left the socket open.

Now, clients can opt-in to receiving heartbeats during long-running PUTs
by including the query parameter

    heartbeat=on

With heartbeating turned on, the proxy will start its response immediately
with 202 Accepted then send a single whitespace character periodically
until the request completes. At that point, a final summary chunk will be
sent which includes a "Response Status" key indicating success or failure
and (if successful) an "Etag" key indicating the Etag of the resulting SLO.

This mechanism is very similar to the way bulk extractions and deletions
work, and even the way SLO behaves for ?multipart-manifest=delete requests.

Note that this is opt-in: this prevents us from sending the 202 response
to existing clients that may mis-interpret it as an immediate indication
of success.

Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Related-Bug: 1718811
Change-Id: I65cee5f629c87364e188aa05a06d563c3849c8f3
2017-11-03 09:42:48 +00:00
92705bb36b Fix indent in overview_policies.rst
Change-Id: I7f070956d8b996db798837392adfca4483067aea
2017-11-02 12:39:19 +01:00
449d83fb0c Doc uses alias instead of aliases
The overview_policies doc makes reference to an `alias` option when in
fact the option is `aliases`.
The sample storage policy snippet is correct, it's just incorrect when
listing the possible options.

This change changes the listed option to `aliases`.

Change-Id: Iddf0f19f4d50819ff6abd46e6a1156dc8e4a451d
2017-10-31 15:55:13 +11:00
e199192cae Replace replication_one_per_device by custom count
This commit replaces boolean replication_one_per_device by an integer
replication_concurrency_per_device. The new configuration parameter is
passed to utils.lock_path() which now accept as an argument a limit for
the number of locks that can be acquired for a specific path.

Instead of trying to lock path/.lock, utils.lock_path() now tries to lock
files path/.lock-X, where X is in the range (0, N), N being the limit for
the number of locks allowed for the path. The default value of limit is
set to 1.

Change-Id: I3c3193344c7a57a8a4fc7932d1b10e702efd3572
2017-10-24 16:17:41 +01:00
1e68eaa4f5 Merge "Document keystone role element in container ACL" 2017-10-19 20:08:27 +00:00
9a09641a7c Merge "Add cautionary note re delay_reaping in account-server.conf-sample" 2017-09-28 01:19:33 +00:00
93fc9d2de8 Add cautionary note re delay_reaping in account-server.conf-sample
Change-Id: I2c3eea783321338316eecf467d30ba0b3217256c
Related-Bug: #1514528
2017-09-27 22:52:47 +01:00
5c76b9e691 Add concurrent_gets to proxy.conf man page
Change-Id: Iab1beff4899d096936c0e5915f3ec32364b3e517
Closes-Bug: #1559347
2017-09-27 14:11:14 +01:00
69a90dcd75 Remove reference to EC being in beta
Closes-Bug: #1719095

Change-Id: I8051895987bf72c8095e72b5a521042a13993174
Signed-off-by: Thiago da Silva <thiago@redhat.com>
2017-09-25 13:27:50 -04:00
3fda32470b Merge "Remove all post_as_copy related code and configes" 2017-09-19 19:19:00 +00:00
1e79f828ad Remove all post_as_copy related code and configes
It was deprecated and we discussed on this topic in Denver PTG
for Queen cycle. Main motivation for this work is that deprecated
post_as_copy option and its gate blocks future symlink work.

Change-Id: I411893db1565864ed5beb6ae75c38b982a574476
2017-09-16 05:50:41 +00:00
d2e32b39e8 Merge "doc migration: update the doc link address[2/3]" 2017-09-15 11:09:28 +00:00
df00122e74 doc migration: update the doc link address[2/3]
Update the doc link brought by the doc migration.
Although we had some effort to fix these, it still left lots of bad
doc link, I separate these changes into 3 patches aim to fix all of
these, this is the 2st patch for doc/manpages.

Change-Id: Id426c5dd45a812ef801042834c93701bb6e63a05
2017-09-15 06:31:00 +00:00
5d5f665cd6 Merge "Correct the unused doc link address" 2017-09-14 22:32:20 +00:00
6305993317 Correct the unused doc link address
Update the doc link brought by the doc migration.
Although we had some effort to fix these, it still left lots of bad
doc link, I separate these changes into 3 patches aim to fix all of
these, this is the 3rd patch for doc/source/install.

Change-Id: I1b0c12cd5f893f1a84d12782ddc39f6d06beb2aa
2017-09-13 08:54:42 +00:00
4ddd4c8a63 Update ubuntu version to be correct target
- saio describes both 14.04 and 16.04 procedure
- currently we're testing on 16.04 (xenial) envrionment on the gate

Remaining task (probably another work):
- review the installation guide which adjusts to the ubuntu 14.04 LTS

Change-Id: Id690a1deabeb24bfc1af3ba3a3019794fe4b8eb9
2017-09-12 02:21:41 +09:00
834b733d29 Merge "Remove vestigate HUDSON_PUBLISH_DOCS reference" 2017-09-06 20:11:26 +00:00
7e5a39a5bd Merge "Add explanation about Content-Length header limit with DLO to docs" 2017-09-06 18:32:15 +00:00
a2f557e921 Add explanation about Content-Length header limit with DLO to docs
If a number of DLO segments is larger than container listing limit,
Content-Length header will not be included in GET or HEAD response.
However, this fact is not explained in document of large objects.

This patch add explanation about this fact to the document.

Change-Id: Ia45fad05797f38fa8b6b0ed917b4f9d7fb337149
Closes-Bug: 1680219
2017-09-06 07:18:45 +00:00
ddec5f5373 Update Container-Sync Docs for internal_client.conf
At patch [1], container sync daemon is updated to use internal client.
However, the "Configuring Container Sync" section [2] of swift docs doesn't
updated for this change.

This patch adds an explanation about the change.

[1]: https://review.openstack.org/#/c/452773/
[2]: http://docs.openstack.org/developer/swift/overview_container_sync.html#configuring-container-sync

Change-Id: Ia1aabeab250cbe27414d7877138ad38708f5ce3f
Closes-Bug: 1446721
2017-09-05 07:26:11 +00:00
03c54d505f Remove vestigate HUDSON_PUBLISH_DOCS reference
This isn't actually used (and in swift is commented out already)
and is a leftover from a thing we did about seven years ago.

Change-Id: I9889bcfd29054f14679ae7430b077ad3afb25b98
2017-09-02 13:09:17 -05:00
4b6e8bc7ec Merge "Add OpenSuse SAIO build instructions" 2017-08-25 22:23:22 +00:00
b0142d0cd2 Merge "Retrieve encryption root secret from Barbican" 2017-08-21 21:19:09 +00:00
164074325f Merge "Inconsistencies in endpoint definitions" 2017-08-18 07:54:24 +00:00
f6b78f6de8 Merge "Clarify usage of replicator and reconstructor override options" 2017-08-18 03:08:06 +00:00
ca56090ba1 Inconsistencies in endpoint definitions
Change-Id: Ie3feea8ed31bfeb9a1047995efc44424071f8224
Closes-Bug: #1709930
2017-08-16 15:00:11 +05:30
0fead33ba1 Add OpenSuse SAIO build instructions
This patch adds OpenSuse to the build a SAIO development page.

OpenSuse's libssl.so naming is different then other Linux distros
and as such it can't simply use pip's cryptography wheel/binary,
which by default is linked to libssl.so.10.
To fix this, --no-binary cryptography was added to pip install:

  pip install --no-binary cryptography -r requirements.txt

Which forces the cryptography module's binding to be compiled
against the correct libssl.so library.

Change-Id: I6a070f33d670edbb887433530c44e2cb509f0c58
2017-08-16 11:35:41 +10:00
a959d24bf5 Document keystone role element in container ACL
The use of a keystone role name in container ACLs is supported
and tested. This patch adds documentation.

[1] fb3d01a974/swift/common/middleware/keystoneauth.py (L491-L497)
[2] test.unit.common.middleware.test_keystoneauth.TestAuthorize.test_authorize_succeeds_for_user_role_in_roles

Change-Id: I77df27393a10f1d8c5a43161fdd4eb08be632566
Closes-Bug: #1705300
2017-08-15 12:37:33 +01:00
f7c55c169a Merge "Turn on warning-is-error in doc build" 2017-08-08 03:33:14 +00:00
c93c0c0c6e [Trivialfix]Fix typos in swift
Fix typos that found in swift.

Change-Id: I52fad1a4882cec4456f22174b46d54e42ec66d97
2017-08-04 07:50:10 +00:00
77bd74da09 Retrieve encryption root secret from Barbican
This patch adds support for retrieving the encryption root secret from
an external key management system. In practice, this is currently
limited to Barbican.

Change-Id: I1700e997f4ae6fa1a7e68be6b97539a24046e80b
2017-08-02 15:53:09 +03:00
019515e153 Merge "Update and optimize documentation links" 2017-08-01 21:27:18 +00:00
0d07e3fdb1 Update and optimize documentation links
* Update URLs according to document migration
 * Update the dead and outdated links

Change-Id: Id92552f4a2d0fb79ddefc55a08636f2e7aeb07cb
2017-08-01 15:12:00 +01:00
d18e847c94 Turn on warning-is-error in doc build
* Fixes warnings in RST file
* Suppress warning log from pyeclib during the doc build.
  pyeclib emits a warning message on an older liberasurecode [1]
  and sphinx treats this as error (when warning-is-error is set).
  There is no need to check warnings during the doc build,
  so we can safely suppress the warning.

This is a part of the doc migration community-wide effort.
http://specs.openstack.org/openstack/docs-specs/specs/pike/os-manuals-migration.html

[1] https://github.com/openstack/pyeclib/commit/d163972b

Change-Id: I9adaee29185a2990cc3985bbe0dd366e22f4f1a2
2017-07-30 14:56:12 +00:00
0e3926f471 Merge "added .htaccess redirect file for docs" 2017-07-28 00:53:40 +00:00
9155e6f0d7 added .htaccess redirect file for docs
also added a test redirect so we know if things are working

Change-Id: I8fd848c23053e52e45fa55637f280124eba74927
2017-07-27 14:06:51 -07:00
701a172afa Add multiple worker processes strategy to reconstructor
This change adds a new Strategy concept to the daemon module similar to
how we manage WSGI workers.  We need to leverage multiple python
processes to get the concurrency properties we need.  More workers will
rebalance much faster on dense chassis with many devices.

Currently the default is still only one process, and no workers.  Set
reconstructor_workers in the [object-reconstructor] section to some
whole number <= the number of devices on a node to get that many
reconstructor workers.

Each worker will operate on a different subset of disks.

Once mode works as before, but tends to want to update recon drops a
little bit more.

If you change the rings, the strategy will shutdown workers and spawn
new ones.

You can kill the worker pids and the daemon strategy will respawn them.

New per-disk reconstructor stats are dumped to recon under the
object_reconstruction_per_disk key.  To maintain legacy compatibility
and replication monitoring based on cycle times they are aggregated
every stats_interval (default 5 mins).

Change-Id: I28925a37f3985c9082b5a06e76af4dc3ec813abe
2017-07-26 16:55:10 -07:00
b599b48f69 Clarify usage of replicator and reconstructor override options
Clarify in usage statement and man pages that CLI override options for
swift-object-reconstructor and swift-object-replicator only have
effect when --once is used.

Also add a link to object reconstructor source code docs to the doc
index page for consistency with the other object services.

Change-Id: If348b340d59a672d3a19d4df231ebdb74f4aed51
2017-07-26 12:53:46 +01:00
4e4028e59f Merge "moved install guide and removed tox env definition" 2017-07-21 23:21:52 +00:00
5b10cf530b Add more structure to the deployment guide
Previously it was hard to navigate to a particular config section in
the deployment guide, and not possible to provide a link directly to
one section.

This patch makes each config section a heading so that it appears in
navigation tables and can be easily linked to. A list of config
sections is also added at the start of each server section.

Change-Id: Iecb0637fde521600a9163fa66b3dbdc176a71dff
Related-Bug: #1626290
2017-07-20 17:01:36 +01:00
9c5628b4f1 Add reconstructor section to deployment guide
Change-Id: I062998e813718828b7adf4e7c3f877b6a31633c0
Closes-Bug: #1626290
2017-07-20 11:40:17 +01:00
13a07aa77a Misc doc cleanup
* Change some absolute URLs to internal links
* Fix some bulletted list indentation
* Choose a better lexer for some syntax highlighting
* Use ``inline code`` instead of `italics` for some example command
  lines
* Change some quoted paragraphs that only included inlined code to be
  proper code blocks

Change-Id: Iaaa7eefb690122f5af9dcb1c871358c22335c743
2017-07-12 12:14:45 -07:00
63c4215994 Fix rst for link
Change-Id: I2f63ce16ae67277504a47e68c7dca3c3412fe92c
2017-07-11 18:38:43 -07:00
239dfb559c moved install guide and removed tox env definition
Change-Id: I966f849804cd5eeeae566bc7e732c7bf56bc9eaf
2017-07-10 13:49:28 -07:00
d112b7d29d First pass at admin-guide cleanup
Change-Id: I005232d95a3e1d181271eef488a828ad330e6006
2017-07-07 17:33:54 +00:00
4cb76a41ce docs migration from openstack-manuals
Context for this is at
https://specs.openstack.org/openstack/docs-specs/specs/pike/os-manuals-migration.html

Change-Id: I9a4da27ce1d56b6406e2db979698038488f3cf6f
2017-07-06 16:44:25 -07:00
37a1935198 Merge "Switch from oslosphinx to openstackdocstheme" 2017-07-06 18:16:43 +00:00
c3f6e82ae1 Merge "Write-affinity aware object deletion" 2017-07-06 14:00:05 +00:00
e94b383655 Merge "Add support to increase object ring partition power" 2017-07-05 14:40:42 +00:00