686 Commits

Author SHA1 Message Date
Jenkins
90b139db19 Merge "fix api overview document" 2015-07-08 19:16:59 +00:00
Jenkins
74455ad74c Merge "Correcting minor grammatical errors" 2015-07-06 18:48:04 +00:00
asettle
0a6f2ab870 Correcting minor grammatical errors
1. Removing unnecessary semi colon
2. Removing unnecessary content

Change-Id: Ie95403a4d96db2b8465e75495061fc059098c922
2015-07-02 10:31:48 +10:00
ZhiQiang Fan
a88bd6e6b9 fix api overview document
The content about Object Storage HTTP requests constraints seems to
be a table but broken, this patch fixes it into correct rst table format.

Change-Id: I1b4c62da3e6d91add3ee0218707c1628c8f04b33
2015-06-28 22:08:35 -07:00
Jenkins
20a582a6e5 Merge "Fixed Errors in Swift-Form Post middleware section rst." 2015-06-26 18:40:16 +00:00
Jenkins
f5d6f8c23a Merge "Add policy support to dispersion tools" 2015-06-25 09:42:00 +00:00
Jenkins
383a3e4823 Merge "Add note about updatedb to the docs" 2015-06-24 22:19:13 +00:00
Jenkins
2d33edde6f Merge "Fixed Formatting Error in Swift -Form Post middleware section." 2015-06-24 22:10:35 +00:00
janonymous
c2c1366b07 Fixed Errors in Swift-Form Post middleware section rst.
Inline markup cannot be nested in RST.
Fixing markups by using a single option for the whole line and making
some minor error corrections.

Change-Id: I42bfd7dfe5c93a69436ecc5d154f2e61ca83fa82
2015-06-24 23:58:50 +05:30
janonymous
514daea631 Fixed Formatting Error in Swift -Form Post middleware section.
Fixed Formatting error in ``action=”SWIFT_URL``” => action=”``SWIFT_URL``”

Change-Id: Id461129ef7f5b6412f94d36920c942a4181c0eb7
Closes-Bug: #1464740
2015-06-24 19:07:44 +05:30
paul luse
e6165a7879 Add policy support to dispersion tools
Doesn't work for anything other than policy 0. updated to allow user
to specify policy name on cmd line (as with object-info) which
then makes populate/report work with 3x, 2x, or EC style policies

Change-Id: Ib7c298f0f6d666b1ecca25315b88539f45cf9f95
Closes-Bug: 1458688
2015-06-23 02:14:02 -07:00
Christian Schwede
4a5b851207 Add note about updatedb to the docs
Change-Id: Ia1aa0bb1f93ee487e2f7ddf76a7a08efa8f3ba41
2015-06-22 14:49:59 +02:00
Jenkins
0009a43eb4 Merge "Allow 1+ object-servers-per-disk deployment" 2015-06-22 03:45:11 +00:00
Jenkins
b32abab56c Merge "Allow default reseller prefix in domain_remap middleware" 2015-06-19 00:43:15 +00:00
Darrell Bishop
df134df901 Allow 1+ object-servers-per-disk deployment
Enabled by a new > 0 integer config value, "servers_per_port" in the
[DEFAULT] config section for object-server and/or replication server
configs.  The setting's integer value determines how many different
object-server workers handle requests for any single unique local port
in the ring.  In this mode, the parent swift-object-server process
continues to run as the original user (i.e. root if low-port binding
is required), binds to all ports as defined in the ring, and forks off
the specified number of workers per listen socket.  The child, per-port
servers drop privileges and behave pretty much how object-server workers
always have, except that because the ring has unique ports per disk, the
object-servers will only be handling requests for a single disk.  The
parent process detects dead servers and restarts them (with the correct
listen socket), starts missing servers when an updated ring file is
found with a device on the server with a new port, and kills extraneous
servers when their port is found to no longer be in the ring.  The ring
files are stat'ed at most every "ring_check_interval" seconds, as
configured in the object-server config (same default of 15s).

Immediately stopping all swift-object-worker processes still works by
sending the parent a SIGTERM.  Likewise, a SIGHUP to the parent process
still causes the parent process to close all listen sockets and exit,
allowing existing children to finish serving their existing requests.
The drop_privileges helper function now has an optional param to
suppress the setsid() call, which otherwise screws up the child workers'
process management.

The class method RingData.load() can be told to only load the ring
metadata (i.e. everything except replica2part2dev_id) with the optional
kwarg, header_only=True.  This is used to keep the parent and all
forked off workers from unnecessarily having full copies of all storage
policy rings in memory.

A new helper class, swift.common.storage_policy.BindPortsCache,
provides a method to return a set of all device ports in all rings for
the server on which it is instantiated (identified by its set of IP
addresses).  The BindPortsCache instance will track mtimes of ring
files, so they are not opened more frequently than necessary.

This patch includes enhancements to the probe tests and
object-replicator/object-reconstructor config plumbing to allow the
probe tests to work correctly both in the "normal" config (same IP but
unique ports for each SAIO "server") and a server-per-port setup where
each SAIO "server" must have a unique IP address and unique port per
disk within each "server".  The main probe tests only work with 4
servers and 4 disks, but you can see the difference in the rings for the
EC probe tests where there are 2 disks per server for a total of 8
disks.  Specifically, swift.common.ring.utils.is_local_device() will
ignore the ports when the "my_port" argument is None.  Then,
object-replicator and object-reconstructor both set self.bind_port to
None if server_per_port is enabled.  Bonus improvement for IPv6
addresses in is_local_device().

This PR for vagrant-swift-all-in-one will aid in testing this patch:
https://github.com/swiftstack/vagrant-swift-all-in-one/pull/16/

Also allow SAIO to answer is_local_device() better; common SAIO setups
have multiple "servers" all on the same host with different ports for
the different "servers" (which happen to match the IPs specified in the
rings for the devices on each of those "servers").

However, you can configure the SAIO to have different localhost IP
addresses (e.g. 127.0.0.1, 127.0.0.2, etc.) in the ring and in the
servers' config files' bind_ip setting.

This new whataremyips() implementation combined with a little plumbing
allows is_local_device() to accurately answer, even on an SAIO.

In the default case (an unspecified bind_ip defaults to '0.0.0.0') as
well as an explict "bind to everything" like '0.0.0.0' or '::',
whataremyips() behaves as it always has, returning all IP addresses for
the server.

Also updated probe tests to handle each "server" in the SAIO having a
unique IP address.

For some (noisy) benchmarks that show servers_per_port=X is at least as
good as the same number of "normal" workers:
https://gist.github.com/dbishop/c214f89ca708a6b1624a#file-summary-md

Benchmarks showing the benefits of I/O isolation with a small number of
slow disks:
https://gist.github.com/dbishop/fd0ab067babdecfb07ca#file-results-md

If you were wondering what the overhead of threads_per_disk looks like:
https://gist.github.com/dbishop/1d14755fedc86a161718#file-tabular_results-md

DocImpact

Change-Id: I2239a4000b41a7e7cc53465ce794af49d44796c6
2015-06-18 12:43:50 -07:00
Jenkins
47b611e1d0 Merge "Change usage help and Attention messages to warnings" 2015-06-10 04:15:31 +00:00
janonymous
037a0c5dbe Object are sorted and concatenated by swift in cardinal order.
Modified values in example in decimal to be more precise,
added a small description on Object are sorted and concatenated
by swift in cardinal order as a small phrase would be sufficient.

Change-Id: I914ded4e5726e50bb93b05759c3bfb76edda53ab
backport: none
Closes-Bug: #1383893
2015-06-07 12:53:25 +05:30
Koert van der Veer
11e5c4adf0 Allow default reseller prefix in domain_remap middleware
Previously, the reseller prefix needed to be provided in the host name
even when the domain was unique to that reseller. With the
default_reseller_prefix, any domain which matches in this middleware,
will will be passed on with a reseller prefix, whether or not it was
provided.

Change-Id: I5aa5ce78ad1ee2e3660cce4c3e07306f8999f02a
Implements: blueprint domainremap-reseller-domains
2015-06-06 12:54:41 -07:00
Mark Seger
af734b3fb6 Change usage help and Attention messages to warnings
Change-Id: I1396aaffe36e739606f15f7fef37b11bd83f1fc1
2015-06-03 15:32:25 -04:00
Christian Schwede
f1f4bb30cd Fix testing issues
When functional tests are run in tox and an exception is raised when connecting
to Swift (for example: Swift not running, missing python-keystoneclient package
used by python-swiftclient) 0 tests are executed, but tox returns a success.

An exception is raised during tests, caused by a missing python-keystoneclient
in python-swiftclient. Instead of adding python-keystoneclient as a dependency
in python-swiftclient the package is added to the test-requirements.txt in Swift
itself. Note that adding python-keystoneclient to the test-requirements in
python-swiftclient is not sufficient (it's already in there).

The exception in setup_package() is catched by the openstack.nose_plugin, thus
disabling this plugin for now as well.

Also fixing two test errors seen on the gate regarding the tempurl middleware.

There was also an update to tox, environment variables were no longer passed
with versions >= 2.0 (http://tox.readthedocs.org/en/latest/changelog.html).
Swift test environment variables have been added to the passenv to re-enable the
former behavior, as well as environment variables required to pass proxy
settings.

This also led to skipped tempauth tests, and together with the missing
python-keystoneclient no tests were executed.

Related-Bug: 1461440
Related-Bug: 1455102

Co-Authored-By: Alistair Coles <alistair.coles@hp.com>

Change-Id: Ideea071017d04912c60ed0bc76532adbb446c31d
2015-06-03 14:13:14 +01:00
Jenkins
d25c398e0b Merge "Replaced setting run_pause with standard interval" 2015-06-01 07:37:50 +00:00
John Dickinson
5374ba3a80 drop Python 2.6 testing support
Change-Id: I78f21e5794e8ba7a095f03d279247516a241f555
2015-05-27 12:28:04 -07:00
Jenkins
15fd063ac3 Merge "Add swift-durability-calculator line to docs" 2015-05-26 20:09:26 +00:00
Joanna H. Huang
af8d842076 Replaced setting run_pause with standard interval
The deprecated directive `run_pause` should be replaced with the more
standard one `interval`. The `run_pause` should be still supported for
backward compatibility. This patch updates object replicator to use
`interval` and support `run_pause`. It also updates its sample config
and documentation.

Co-Authored-By: Joanna H. Huang <joanna.huitzu.huang@gmail.com>
Co-Authored-By: Kamil Rykowski <kamil.rykowski@intel.com>

Change-Id: Ie2a3414a96a94efb9273ff53a80b9d90c74fff09
Closes-Bug: #1364735
2015-05-25 11:47:47 +02:00
Kota Tsuyuzaki
f11d92d566 Add swift-durability-calculator line to docs
This commits add a line (link and small doc) for
swift-durability-calculator which provides a browser based
durability calculation tool to docs as an associated project.

Change-Id: I4ea8015f512616dc25072080bef79b8734971ccf
2015-05-22 16:58:04 -07:00
Clay Gerrard
f864092455 Add Swift Inspector to assoicated projects
Change-Id: I5b5448674ea455119a51509ab5e7cd11a764b5a7
2015-05-20 17:50:07 -07:00
Pete Zaitcev
0b20a18e52 Spell "rebalance" right in swift-ring-builder.1
See Red Hat bug #1218269.

Change-Id: I814eb4b3c0821f5a8df5feea2bda3a964aace536
2015-05-07 13:17:15 -06:00
Christian Schwede
55dd705a86 Add missing statsd metrics section for object-reconstructor
Change-Id: Id3f98e5f637ff537a387262b40f21c05876fca91
2015-05-06 19:53:09 +02:00
Jenkins
085c262194 Merge "SAIO instructions: ensure ~/bin exists before copying into it" 2015-04-21 04:19:59 +00:00
Samuel Merritt
0c391d6daf SAIO instructions: ensure ~/bin exists before copying into it
Change-Id: I16cd211b00b529ccc4b46f6b10497c32b6741896
2015-04-20 16:47:10 -07:00
Minwoo B
c5c281ba6d Included step in development_saio.rst for installing dependencies in requirements.txt.
Change-Id: I6ed1704148e5ae1e3164d10080c350d81856f7a9
2015-04-20 17:03:25 -05:00
Lorcan
193fe9a5f9 Update swift-recon doc with more options
The swit-recon tool has had several functional additions
added recently but not all of these have been added to the docs.

This change adds the following options to the manpages:

  --human-readable
  --validate-servers
  --sockstat
  --driveaudit
  --region
  --timeout

Also fixes a typo on line 78 (cop -> copy)

Change-Id: Id083b32a60473ad5a2b9ac1d092528d230521c86
2015-04-15 13:44:42 +01:00
Lorcan
66fb207fcb Add lowest option to swift-recon disk usage output
Currently there is a "--top" option when running swift-recon for
disk usage stats. This option lists the x disks with the highest
disk usage in descending order.

This feature adds a "--lowest" option which does the opposite and
lists the y disks with lowest disk usage in ascending order.

Have also updated the docs section with --top and --lowest options

Change-Id: Ic15d407fe010a31995c2bdd9fb88548a1057f569
2015-04-14 16:00:37 -07:00
Paul Luse
8f5d4d2455 Erasure Code Documentation
This patch adds all the relevant EC documentation to
the source tree. Notable additions are:
  - Updated SAIO documentation
  - Updates to existing swift documentation; and
  - Erasure Coding overview

Co-Authored-By: Alistair Coles <alistair.coles@hp.com>
Co-Authored-By: Thiago da Silva <thiago@redhat.com>
Co-Authored-By: John Dickinson <me@not.mn>
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Co-Authored-By: Tushar Gohad <tushar.gohad@intel.com>
Co-Authored-By: Samuel Merritt <sam@swiftstack.com>
Co-Authored-By: Christian Schwede <christian.schwede@enovance.com>
Co-Authored-By: Yuan Zhou <yuan.zhou@intel.com>
Change-Id: I0403016a4bb7dad9535891632753b0e5e9d402eb
Implements: blueprint swift-ec
Signed-off-by: Thiago da Silva <thiago@redhat.com>
2015-04-14 00:52:17 -07:00
Yuan Zhou
61a9d35fd5 Update container sync to use internal client
This patch changes container sync to use Internal Client instead
of Direct Client.

In the current design, container sync uses direct_get_object to
get the newest source object(which talks to storage node directly).
This works fine for replication storage policies however in
erasure coding policies, direct_get_object would only return part
of the object(it's encoded as several pieces). Using Internal
Client can get the original object in EC case.

Note that for the container sync put/delete part, it's working in
EC since it's using Simple Client.

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

DocImpact
Change-Id: I91952bc9337f354ce6024bf8392046a1ecf6ecc9
2015-04-14 00:52:17 -07:00
Samuel Merritt
8d3b3b2ee0 Add some debug output to the ring builder
Sometimes, I get handed a builder file in a support ticket and a
question of the form "why is the balance [not] doing $thing?". When
that happens, I add a bunch of print statements to my local
swift/common/ring/builder.py, figure things out, and then delete the
print statements. This time, instead of deleting the print statements,
I turned them into debug() calls and added a "--debug" flag to the
rebalance command in hopes that someone else will find it useful.

Change-Id: I697af90984fa5b314ddf570280b4585ba0ba363c
2015-03-30 17:47:28 -07:00
Martin Kletzander
76b106fc01 Fix common misspellings
Wikipedia's list of common misspellings [1] has a machine-readable
version.  This patch fixes those misspellings mentioned in the list
which don't have multiple right variants (as e.g. "accension", which can
be both "accession" and "ascension"), such misspellings are left
untouched.  The list of changes was manually re-checked for false
positives.

[1] https://en.wikipedia.org/wiki/Wikipedia:Lists_of_common_misspellings/For_machines

Change-Id: Ic9a5438629664f7cea216413a28acc0e8992da05
Signed-off-by: Martin Kletzander <mkletzan@redhat.com>
2015-03-24 11:07:56 +01:00
Alistair Coles
7e64c811fe Enable in-process functional test policy to be configured
Currently the in-process tests build a 2 replica, 4 partition,
2 device object ring. This patch allows an alternative policy
and ring to be specified for testing via environment variables
that may optionally be set.

SWIFT_TEST_IN_PROCESS_CONF_DIR - This points the test setup to a
  directory which may have a swift.conf file and ring file. The
  test setup will then prefer these conf files over the samples
  in '/etc'.

SWIFT_TEST_POLICY - This causes the in-process test to
  use the specified policy from the swift.conf file and its
  associated ring for testing (first copying the conf and ring file
  and modifying device parameters to suit in-process testing). If
  not set, the tests will use the default policy.

The in-process tests now start sufficient object servers for the
ring file being tested against.

This should allow in-process functional testing of various policies
and rings (e.g. EC policies) without needing to reconfigure an SAIO
for each test scenario.

The refactoring of the in_process test setup code should also
allow easier addition of other 'hard-coded' test policies/rings
in the future.

Change-Id: I24f5a13de3d296b400da1691dcb53423a9f8a463
2015-03-20 16:13:57 +00:00
Thiago da Silva
7f482e9a56 fixing small typos in associated projects doc
Change-Id: Ia4555acc1763a58cdf605530219ea4e4586ad517
Signed-off-by: Thiago da Silva <thiago@redhat.com>
2015-03-06 14:35:40 -05:00
Matthew Oliver
4f9a3a3342 Fixed links in multi-server Swift documentation
The link to the ubuntu installation guide was incorrect and 404'ing.

This change updates page to point (anchor) to the install guide
section of docs.openstack.org, so it will always point to the latest
versions of the install documentation.

It also updates the deep links to the current install guides for the
past few releases.

As links continue to get outdated or we forget to add new ones, the
anchor link at the top of the page will probably still work.

Change-Id: I38299c03370936eb41992a953aa9d10b01a8396b
Closes-Bug: #1147331
2015-03-03 16:18:58 +11:00
Jenkins
d6467d3385 Merge "Add multiple reseller prefixes and composite tokens" 2015-02-24 16:12:01 +00:00
Donagh McCabe
89397c5b67 Add multiple reseller prefixes and composite tokens
This change is in support of Composite Tokens and Service Accounts
(see http://specs.openstack.org/openstack/swift-specs/specs/in_progress/
service_token.html)

During coding, minor changes were made compared to the original
specification. See https://review.openstack.org/138771 for these changes.

DocImpact

Change-Id: I6072b4efb3a479a8e0cc2d9c11ffda5764b55e30
2015-02-23 15:57:20 +00:00
Alistair Coles
4ca08cc395 Update guest VM OS recommendation in SAIO doc
The target development platform has changed to Ubuntu 14.04 [1].
This patch makes the suggested SAIO platform the same.

Also, remove pointer to wiki page for other platform install
instructions that either redirects back to this SAIO doc or
to another wike page and then a dead link.

[1] I0a96bcf692bb240f3ab5aab7fefd294a07735a83

DocImpact

Change-Id: I9f96104b5437c1f1f28f924c048ef83cf03338f4
2015-02-19 17:46:10 +00:00
John Dickinson
949804eda4 update the getting started doc
Change-Id: I0a96bcf692bb240f3ab5aab7fefd294a07735a83
2015-02-17 13:24:33 -08:00
John Dickinson
7acc291129 added swift_source to ratelimit info calls
Change-Id: I2b4ccb809c0f4505098e261f546f53c27440dd43
2015-02-15 17:14:31 -08:00
Jenkins
f2e2326b5f Merge "Update TempURL docs to include containers" 2015-02-12 22:32:07 +00:00
Richard Hawkins
f5b533fb2f Update TempURL docs to include containers
Change-Id: Ifb7c26b23ec81e3bde96b6e3bad0e950c0ca9408
2015-02-12 11:27:31 -06:00
Clay Gerrard
ca90be414e Add developer tools section to associated projects
vagrant-swift-all-in-one is being used and maintained by a number of swift
developers, it has an open source license.

The ansible playbook project serves a similar goal but it's based on a Fedora
distribution and includes Swift-on-File support.

Drive-by fix for the Swift-on-File link which has migrated to stackforge.

Change-Id: Id7478d58adcead57cf56ac4e1d05c6556c8c9b7b
2015-02-11 16:43:44 -08:00
Donagh McCabe
843ce7e301 API document update for container-level tempurl/formpost
Tempurl and Formpost now support secret keys at the container
level [1]. Adds these headers:

    X-Container-Meta-Temp-Url-Key
    X-Container-Meta-Temp-Url-Key-2

DocImpact -- because headers need to be added to [2]

[1] https://review.openstack.org/#/c/154293
[2] http://developer.openstack.org/api-ref-objectstorage-v1.html

Change-Id: I157cd4a540dcc4a9b45bc3d806df152ab56e2354
2015-02-11 12:02:38 +00:00
Jenkins
0186194e96 Merge "Output logs of policy index" 2015-02-10 22:21:13 +00:00