1019 Commits

Author SHA1 Message Date
David Rabel
5257ed033c Fix indentation in overview_backin_store.rst
Fix indentation of nested list.
Old indentation was wrongly interpreted as quotation

Change-Id: Ia9625bc3b7984501f4931b3428fa930a156bfe66
2018-05-02 16:24:27 +02:00
Zuul
3313392462 Merge "Import swift3 into swift repo as s3api middleware" 2018-04-30 16:00:56 +00:00
Kota Tsuyuzaki
636b922f3b Import swift3 into swift repo as s3api middleware
This attempts to import openstack/swift3 package into swift upstream
repository, namespace. This is almost simple porting except following items.

1. Rename swift3 namespace to swift.common.middleware.s3api
1.1 Rename also some conflicted class names (e.g. Request/Response)

2. Port unittests to test/unit/s3api dir to be able to run on the gate.

3. Port functests to test/functional/s3api and setup in-process testing

4. Port docs to doc dir, then address the namespace change.

5. Use get_logger() instead of global logger instance

6. Avoid global conf instance

Ex. fix various minor issue on those steps (e.g. packages, dependencies,
  deprecated things)

The details and patch references in the work on feature/s3api are listed
at https://trello.com/b/ZloaZ23t/s3api (completed board)

Note that, because this is just a porting, no new feature is developed since
the last swift3 release, and in the future work, Swift upstream may continue
to work on remaining items for further improvements and the best compatibility
of Amazon S3. Please read the new docs for your deployment and keep track to
know what would be changed in the future releases.

Change-Id: Ib803ea89cfee9a53c429606149159dd136c036fd
Co-Authored-By: Thiago da Silva <thiago@redhat.com>
Co-Authored-By: Tim Burke <tim.burke@gmail.com>
2018-04-27 15:53:57 +09:00
Zuul
47efb5b969 Merge "Multiprocess object replicator" 2018-04-25 00:41:21 +00:00
Samuel Merritt
c28004deb0 Multiprocess object replicator
Add a multiprocess mode to the object replicator. Setting the
"replicator_workers" setting to a positive value N will result in the
replicator using up to N worker processes to perform replication
tasks.

At most one worker per disk will be spawned, so one can set
replicator_workers=99999999 to always get one worker per disk
regardless of the number of disks in each node. This is the same
behavior that the object reconstructor has.

Worker process logs will have a bit of information prepended so
operators can tell which messages came from which worker. It looks
like this:

  [worker 1/2 pid=16529] 154/154 (100.00%) partitions replicated in 1.02s (150.87/sec, 0s remaining)

The prefix is "[worker M/N pid=P] ", where M is the worker's index, N
is the total number of workers, and P is the process ID. Every message
from the replicator's logger will have the prefix; this includes
messages from down in diskfile, but does not include things printed to
stdout or stderr.

Drive-by fix: don't dump recon stats when replicating only certain
policies. When running the object replicator with replicator_workers >
0 and "--policies=X,Y,Z", the replicator would update recon stats
after running. Since it only ran on a subset of objects, it should not
update recon, much like it doesn't update recon when run with
--devices or --partitions.

Change-Id: I6802a9ad9f1f9b9dafb99d8b095af0fdbf174dc5
2018-04-24 04:05:08 +00:00
Tovin Seven
9b90498b07 Trivial: Update pypi url to new url
Pypi url changed from [1] to [2]

[1] https://pypi.python.org/pypi/<package>
[2] https://pypi.org/project/<package>

Change-Id: Ic639298e83189410395f31cc9ef60076af479efd
2018-04-20 17:30:47 +07:00
John Dickinson
87dad294cc added another golang client to associated projects
Change-Id: I04f5c14586ad4ecac03d18992b8cf351d6eab1e3
2018-04-19 17:40:33 -07:00
Zuul
260bd2601b Merge "Deprecate auth_uri option" 2018-04-19 01:56:03 +00:00
wangqi
708b24aef1 Deprecate auth_uri option
Option auth_uri from group keystone_authtoken is deprecated[1].
Use option www_authenticate_uri from group keystone_authtoken.

[1]https://review.openstack.org/#/c/508522/

Change-Id: I43bbc8b8c986e54a9a0829a0631d78d4077306f8
2018-04-18 02:07:11 +00:00
John Dickinson
6a428c4266 updated associated projects page
* Cleaned up the SDK/library links
* Added a few projects
* Fixed some existing links
* Removed some very old, unmaintained projects

Change-Id: I3effd920e978eb7af39ab27b4877a7bfc8c64b8b
2018-04-17 15:19:35 -07:00
Zuul
f8175ea345 Merge "Add full working example of sharing a container with another user" 2018-04-11 00:15:30 +00:00
Tim Burke
9eeacbf150 Fix new doc warnings resulting from Sphinx 1.7.1
Change-Id: Id6678dd7e59a85bc2a7b3ba3d41d28e08956ae3e
2018-03-02 14:56:57 +00:00
melissaml
5017864133 Fix the incorrect reference links
TrivialFix
[1] is the installation guide for OpenStack components, obviously,
we need [1] in the docs.

[1] https://docs.openstack.org/latest/install/

Change-Id: I3c6fe7327f5552cc2b8f0f0e42b41f8e989a0a7e
2018-02-26 15:49:10 +08:00
Tim Burke
0a993437d1 Update install-guide URLs to point to stable/queens
Change-Id: I7b2dd19f18cbfac3bc3ea763cd3333e3fb68f1cd
2018-02-08 10:57:49 -08:00
Zuul
d800c177cc Merge "Replace Chinese punctuation with English punctuation" 2018-01-31 01:48:12 +00:00
Zuul
7fee65c360 Merge "Add container-reconciler to source doc tree" 2018-01-26 21:48:29 +00:00
gaofei
10542d00ea Replace Chinese punctuation with English punctuation
Curly quotes(Chinese punctuation) usually input from Chinese input method.
When read from english context, it makes some confusion.

Change-Id: Ibd50299ee287c56ec4759ea8ff53d47d006144f8
2018-01-25 05:33:04 +00:00
gaofei
05b2a14fe0 Update broken link
Change-Id: I8d13696970593d47d0f07cf97aa1284eed5f7a15
2018-01-23 14:57:18 +08:00
Zuul
22f700d1cc Merge "Modify redirection URL and broken URL" 2018-01-18 19:21:23 +00:00
Zuul
17eb570a6c Merge "Improve object-updater's stats logging" 2018-01-18 19:21:20 +00:00
chengebj5238
222df91857 Modify redirection URL and broken URL
Change-Id: I9a04cb2fbe61e1fbd8185ab2fac9abbcea4d55cc
2018-01-18 17:05:10 +08:00
Samuel Merritt
f64c00b00a Improve object-updater's stats logging
The object updater has five different stats, but its logging only told
you two of them (successes and failures), and it only told you after
finishing all the async_pendings for a device. If you have a cluster
that's been sick and has millions upon millions of async_pendings
laying around, then your object-updaters are frustratingly
silent. I've seen one cluster with around 8 million async_pendings per
disk where the object-updaters only emitted stats every 12 hours.

Yes, if you have StatsD logging set up properly, you can go look at
your graphs and get real-time feedback on what it's doing. If you
don't have that, all you get is a frustrating silence.

Now, the object updater tells you all of its stats (successes,
failures, quarantines due to bad pickles, unlinks, and errors), and it
tells you incremental progress every five minutes. The logging at the
end of a pass remains and has been expanded to also include all stats.

Also included is a small change to what counts as an error: unmounted
drives no longer do. The goal is that only abnormal things count as
errors, like permission problems, malformed filenames, and so
on. These are things that should never happen, but if they do, may
require operator intervention. Drives fail, so logging an error upon
encountering an unmounted drive is not useful.

Change-Id: Idbddd507f0b633d14dffb7a9834fce93a10359ab
2018-01-17 13:59:23 -08:00
Tim Burke
5347c3069d Add container-reconciler to source doc tree
Change-Id: I98e7e55839e5697d975af0a4756fd09933af1c5f
2018-01-17 20:39:22 +00:00
Samuel Merritt
56b84c9295 Minor cleanup in monitoring doc.
Change-Id: Ia21f8743bfd745f2579db8658624f888461c2cc2
2018-01-16 12:02:13 -08:00
xhancar
b22d3c1115 fix of type error
There was incorrect path starting /home/swift, but /home/<your-user-name> is correct for common users.

Change-Id: Ia81b2119c87dd88417428e55c82dac1ab7c028b3
Closes-Bug: 1741378
2018-01-06 20:48:10 +00:00
Alistair Coles
582460ecf9 Document that x-delete-after takes precedence over x-delete-at
Change-Id: Ib2483444d3999e13ba83ca2edd3a8ef8e5c48548
2018-01-05 19:50:50 +00:00
Zuul
507a4fab10 Merge "Represent dispersion worse than one replicanth" 2018-01-03 07:10:41 +00:00
Clay Gerrard
7013e70ca6 Represent dispersion worse than one replicanth
With a sufficiently undispersed ring it's possible to move an entire
replicas worth of parts and yet the value of dispersion may not get any
better (even though in reality dispersion has dramatically improved).
The problem is dispersion will currently only represent up to one whole
replica worth of parts being undispersed.

However with EC rings it's possible for more than one whole replicas
worth of partitions to be undispersed, in these cases the builder will
require multiple rebalance operations to fully disperse replicas - but
the dispersion value should improve with every rebalance.

N.B. with this change it's possible for rings with a bad dispersion
value to measure as having a significantly smaller dispersion value
after a rebalance (even though they may not have had their dispersion
change) because the total amount of bad dispersion we can measure has
been increased but we're normalizing within a similar range.

Closes-Bug: #1697543

Change-Id: Ifefff0260deac0c3e8b369a1e158686c89936686
2017-12-28 11:16:17 -08:00
Alistair Coles
8df263184b Symlink doc clean up
Cleanup for docs and docstrings.

Related-Change: I838ed71bacb3e33916db8dd42c7880d5bb9f8e18
Change-Id: Ie8de0565dfaca5bd8a5693a75e6ee14ded5b7161
2017-12-14 20:12:55 +00:00
Robert Francis
99b89aea10 Symlink implementation.
Add a symbolic link ("symlink") object support to Swift. This
object will reference another object. GET and HEAD
requests for a symlink object will operate on the referenced object.
DELETE and PUT requests for a symlink object will operate on the
symlink object, not the referenced object, and will delete or
overwrite it, respectively.
POST requests are *not* forwarded to the referenced object and should
be sent directly. POST requests sent to a symlink object will
result in a 307 Error.

Historical information on symlink design can be found here:
https://github.com/openstack/swift-specs/blob/master/specs/in_progress/symlinks.rst.
https://etherpad.openstack.org/p/swift_symlinks

Co-Authored-By: Thiago da Silva <thiago@redhat.com>
Co-Authored-By: Janie Richling <jrichli@us.ibm.com>
Co-Authored-By: Kazuhiro MIYAHARA <miyahara.kazuhiro@lab.ntt.co.jp>
Co-Authored-By: Kota Tsuyuzaki <tsuyuzaki.kota@lab.ntt.co.jp>

Change-Id: I838ed71bacb3e33916db8dd42c7880d5bb9f8e18
Signed-off-by: Thiago da Silva <thiago@redhat.com>
2017-12-13 21:26:12 +00:00
Alistair Coles
fc12d63c76 Remove repeated text from deployment guide
Fix what appears to be a cut and paste error.

Change-Id: Iccf97ebbf75c8f97095a4493ea6a8beb074df099
2017-12-06 10:29:52 -08:00
Robin Naundorf
4f3abec614 Fix small formatting issue
TrivialFix

Change-Id: Iacd341a1f3dbf43c6fc2ebcf546aed54f866face
2017-11-20 10:59:05 +01:00
Robin Naundorf
de81861108 Fix formatting of SAIO docs
* Fix formatting error in SAIO docs so that list is displaying
  correct

Change-Id: If88869290252a2c86f60fdf89d78ef4a418ae23d
2017-11-17 13:30:08 +01:00
Zuul
13e932066c Merge "Add checksum to object extended attributes" 2017-11-06 19:06:09 +00:00
Samuel Merritt
728b4ba140 Add checksum to object extended attributes
Currently, our integrity checking for objects is pretty weak when it
comes to object metadata. If the extended attributes on a .data or
.meta file get corrupted in such a way that we can still unpickle it,
we don't have anything that detects that.

This could be especially bad with encrypted etags; if the encrypted
etag (X-Object-Sysmeta-Crypto-Etag or whatever it is) gets some bits
flipped, then we'll cheerfully decrypt the cipherjunk into plainjunk,
then send it to the client. Net effect is that the client sees a GET
response with an ETag that doesn't match the MD5 of the object *and*
Swift has no way of detecting and quarantining this object.

Note that, with an unencrypted object, if the ETag metadatum gets
mangled, then the object will be quarantined by the object server or
auditor, whichever notices first.

As part of this commit, I also ripped out some mocking of
getxattr/setxattr in tests. It appears to be there to allow unit tests
to run on systems where /tmp doesn't support xattrs. However, since
the mock is keyed off of inode number and inode numbers get re-used,
there's lots of leakage between different test runs. On a real FS,
unlinking a file and then creating a new one of the same name will
also reset the xattrs; this isn't the case with the mock.

The mock was pretty old; Ubuntu 12.04 and up all support xattrs in
/tmp, and recent Red Hat / CentOS releases do too. The xattr mock was
added in 2011; maybe it was to support Ubuntu Lucid Lynx?

Bonus: now you can pause a test with the debugger, inspect its files
in /tmp, and actually see the xattrs along with the data.

Since this patch now uses a real filesystem for testing filesystem
operations, tests are skipped if the underlying filesystem does not
support setting xattrs (eg tmpfs or more than 4k of xattrs on ext4).

References to "/tmp" have been replaced with calls to
tempfile.gettempdir(). This will allow setting the TMPDIR envvar in
test setup and getting an XFS filesystem instead of ext4 or tmpfs.

THIS PATCH SIGNIFICANTLY CHANGES TESTING ENVIRONMENTS

With this patch, every test environment will require TMPDIR to be
using a filesystem that supports at least 4k of extended attributes.
Neither ext4 nor tempfs support this. XFS is recommended.

So why all the SkipTests? Why not simply raise an error? We still need
the tests to run on the base image for OpenStack's CI system. Since
we were previously mocking out xattr, there wasn't a problem, but we
also weren't actually testing anything. This patch adds functionality
to validate xattr data, so we need to drop the mock.

`test.unit.skip_if_no_xattrs()` is also imported into `test.functional`
so that functional tests can import it from the functional test
namespace.

The related OpenStack CI infrastructure changes are made in
https://review.openstack.org/#/c/394600/.

Co-Authored-By: John Dickinson <me@not.mn>

Change-Id: I98a37c0d451f4960b7a12f648e4405c6c6716808
2017-11-03 13:30:05 -04:00
Zuul
2596b3ca9d Merge "Let clients request heartbeats during SLO PUTs" 2017-11-03 16:05:18 +00:00
Tim Burke
77a8a4455d Let clients request heartbeats during SLO PUTs
An SLO PUT requires that we HEAD every referenced object; as a result, it
can be a very time-intensive operation. This makes it difficult as a
client to differentiate between a proxy-server that's still doing work and
one that's crashed but left the socket open.

Now, clients can opt-in to receiving heartbeats during long-running PUTs
by including the query parameter

    heartbeat=on

With heartbeating turned on, the proxy will start its response immediately
with 202 Accepted then send a single whitespace character periodically
until the request completes. At that point, a final summary chunk will be
sent which includes a "Response Status" key indicating success or failure
and (if successful) an "Etag" key indicating the Etag of the resulting SLO.

This mechanism is very similar to the way bulk extractions and deletions
work, and even the way SLO behaves for ?multipart-manifest=delete requests.

Note that this is opt-in: this prevents us from sending the 202 response
to existing clients that may mis-interpret it as an immediate indication
of success.

Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Related-Bug: 1718811
Change-Id: I65cee5f629c87364e188aa05a06d563c3849c8f3
2017-11-03 09:42:48 +00:00
David Rabel
92705bb36b Fix indent in overview_policies.rst
Change-Id: I7f070956d8b996db798837392adfca4483067aea
2017-11-02 12:39:19 +01:00
Matthew Oliver
449d83fb0c Doc uses alias instead of aliases
The overview_policies doc makes reference to an `alias` option when in
fact the option is `aliases`.
The sample storage policy snippet is correct, it's just incorrect when
listing the possible options.

This change changes the listed option to `aliases`.

Change-Id: Iddf0f19f4d50819ff6abd46e6a1156dc8e4a451d
2017-10-31 15:55:13 +11:00
Romain LE DISEZ
e199192cae Replace replication_one_per_device by custom count
This commit replaces boolean replication_one_per_device by an integer
replication_concurrency_per_device. The new configuration parameter is
passed to utils.lock_path() which now accept as an argument a limit for
the number of locks that can be acquired for a specific path.

Instead of trying to lock path/.lock, utils.lock_path() now tries to lock
files path/.lock-X, where X is in the range (0, N), N being the limit for
the number of locks allowed for the path. The default value of limit is
set to 1.

Change-Id: I3c3193344c7a57a8a4fc7932d1b10e702efd3572
2017-10-24 16:17:41 +01:00
Zuul
1e68eaa4f5 Merge "Document keystone role element in container ACL" 2017-10-19 20:08:27 +00:00
Jenkins
9a09641a7c Merge "Add cautionary note re delay_reaping in account-server.conf-sample" 2017-09-28 01:19:33 +00:00
Alistair Coles
93fc9d2de8 Add cautionary note re delay_reaping in account-server.conf-sample
Change-Id: I2c3eea783321338316eecf467d30ba0b3217256c
Related-Bug: #1514528
2017-09-27 22:52:47 +01:00
Alistair Coles
5c76b9e691 Add concurrent_gets to proxy.conf man page
Change-Id: Iab1beff4899d096936c0e5915f3ec32364b3e517
Closes-Bug: #1559347
2017-09-27 14:11:14 +01:00
Thiago da Silva
69a90dcd75 Remove reference to EC being in beta
Closes-Bug: #1719095

Change-Id: I8051895987bf72c8095e72b5a521042a13993174
Signed-off-by: Thiago da Silva <thiago@redhat.com>
2017-09-25 13:27:50 -04:00
Jenkins
3fda32470b Merge "Remove all post_as_copy related code and configes" 2017-09-19 19:19:00 +00:00
Kota Tsuyuzaki
1e79f828ad Remove all post_as_copy related code and configes
It was deprecated and we discussed on this topic in Denver PTG
for Queen cycle. Main motivation for this work is that deprecated
post_as_copy option and its gate blocks future symlink work.

Change-Id: I411893db1565864ed5beb6ae75c38b982a574476
2017-09-16 05:50:41 +00:00
Jenkins
d2e32b39e8 Merge "doc migration: update the doc link address[2/3]" 2017-09-15 11:09:28 +00:00
junboli
df00122e74 doc migration: update the doc link address[2/3]
Update the doc link brought by the doc migration.
Although we had some effort to fix these, it still left lots of bad
doc link, I separate these changes into 3 patches aim to fix all of
these, this is the 2st patch for doc/manpages.

Change-Id: Id426c5dd45a812ef801042834c93701bb6e63a05
2017-09-15 06:31:00 +00:00
Jenkins
5d5f665cd6 Merge "Correct the unused doc link address" 2017-09-14 22:32:20 +00:00