148 Commits

Author SHA1 Message Date
Matthew Oliver
e11a38c63a Bind SAIO services on different loopback addresses
Currently all devices in the ring and all services in a SAIO
all bind to the same loopback address 127.0.0.1. But this
breaks servers_per_port if you want to do any testing on that.

This change binds each service to a different loopback address
and updates the rings (remakerings) accordingly.

To make sure rysncd binds correctly the bind address needed
to be changed to listen on all addresses (0.0.0.0).

Change-Id: I7e77434f275df1e2699de495d8b622b90157a9d7
2017-06-27 16:08:08 -04:00
Alistair Coles
adcb4c270e Document SAIO rsync service setup for ubuntu 16
SAIO docs do suggest using Ubuntu 14.04, but if using
16.04 then systemctl needs to be used to have rsync service
restart on reboot.

Change-Id: I4fb0d3d063df61fbdfca981f06911148f3c4dc04
2017-03-08 15:07:13 +00:00
Nam Nguyen Hoai
44fc037f97 Fix_typo: "subsitute" -> "substitute"
There is a wrong word, it should be updated.

Change-Id: I17a1fed844bcd8ab3ed668a34a8471e5f6032963
2017-02-07 15:09:41 +07:00
Alistair Coles
bfc8c59a08 Add encryption package requirements to SAIO instructions
libssl-dev/openssl-devel are already listed in other-requirements.txt;
add them to installation instructions in the SAIO docs.

Change-Id: I3dc07213ff8dac1299d3eb68d3448a77e15c79af
2016-07-07 18:40:10 +01:00
Matthew Oliver
876df35f84 disable_fallocate also disables fallocate_reserve
Currently when disable_fallocate is true it disables calling the
fallocate syscall, but it doesn't disable fallocate_reserve. This
patch fixes this.

This problem has caused functional tests to fail in our SAIOs, since
SAIOs have disable_fallocate set but the fallocate_reserve space free
checking was still being run creating 507 responses. This is thanks
to the change in fallocate_reserve default changing from 0 to 1%.

Because fallocate_reserve and disable_fallocate causes SAIO functional
tests to fail a section called 'Known Issues' has been added to the
SAIO developer documentation which includes a warning about
using fallocate_reserve on SAIOs.

Change-Id: I727bfb0861ea26fe2f16ad55f4d36ae088864d8f
2016-05-19 11:56:29 +10:00
Andrew Welleck
c790db8936 Additional step for Debugging Issues in SAIO
Adding an additional debugging step for the SAIO "Debugging Issues"
section. Adding step to check if memcache is running and to start
it if its not.

Change-Id: I0bfa563f595dbefcf907494451fad45e6310db88
2016-01-25 10:09:08 -06:00
Samuel Merritt
d5ff5447be Install liberasurecode packages in SAIO.
Change-Id: If673afa2b61a3e388612debf4860d561960963a3
2016-01-12 16:42:06 -08:00
Zack M. Davis
1b8b08039a remove remaining simplejson uses, prefer standard library import
a1c32702, 736cf54a, and 38787d0f remove uses of `simplejson` from
various parts of Swift in favor of the standard libary `json`
module (introduced in Python 2.6). This commit performs the remaining
`simplejson` to `json` replacements, removes two comments highlighting
quirks of simplejson with respect to Unicode, and removes the references
to it in setup documentation and requirements.txt.

There were a lot of places where we were importing json from
swift.common.utils, which is less intuitive than a direct `import json`,
so that replacement is made as well.

(And in two more tiny drive-bys, we add some pretty-indenting to an XML
fragment and use `super` rather than naming a base class explicitly.)

Change-Id: I769e88dda7f76ce15cf7ce930dc1874d24f9498a
2015-11-16 12:34:24 -08:00
Ondřej Nový
9cafa472a3 Autodetect systemctl in SAIO and use it on systemd distros
Change-Id: I84a9b27baac89327749d8774032860f8ad5166f2
2015-10-12 22:39:52 +02:00
Eran Rom
69e7424d3c Add container sync probe test to SAIO default set
SAIO Configuration and documentation changes enabling to run the
container sync probe test by default

Change-Id: Iccf59533d0d4fe72549d318339ab125d04dde006
Related-Bug: #1476623
2015-08-22 22:50:50 +03:00
Ben Martin
89f5906286 +Document method to avoid rsync filling root drive
When rsync pushes to a remote node with an unmounted drive and if
certain steps are not taken, rsync may attempt to write files to
the local drive at the location where the drive was mounted.

There are two suggested solutions for this issue:
  1) Set the permissions for all mount points in /srv/node/
       to root:root 755
  2) Mount the drives elsewhere and symlink the drives to /srv/.../

The first method ensures that only root and not the swift user
can write in the /srv/.../ directories.

The second method will prompt a broken link issue if rsync
attempts to write to an unmounted drive.

Change-Id: I60ce4ed9ef8401768d5f78b6806cbb2e2a65303e
Closes-Bug: #1470576
2015-08-05 09:29:07 -05:00
Azhagu Selvan SP
317fa1af49 Minor change in development saio setup doc
Make the test-requirements install command uniform with the rest of the
setup commands in the page by using the $HOME, instead of hardcoding
the folder path.

Change-Id: I8da7a191a301601e27f894f8268238d77b150ede
2015-07-18 15:19:43 +05:30
asettle
0a6f2ab870 Correcting minor grammatical errors
1. Removing unnecessary semi colon
2. Removing unnecessary content

Change-Id: Ie95403a4d96db2b8465e75495061fc059098c922
2015-07-02 10:31:48 +10:00
Jenkins
085c262194 Merge "SAIO instructions: ensure ~/bin exists before copying into it" 2015-04-21 04:19:59 +00:00
Samuel Merritt
0c391d6daf SAIO instructions: ensure ~/bin exists before copying into it
Change-Id: I16cd211b00b529ccc4b46f6b10497c32b6741896
2015-04-20 16:47:10 -07:00
Minwoo B
c5c281ba6d Included step in development_saio.rst for installing dependencies in requirements.txt.
Change-Id: I6ed1704148e5ae1e3164d10080c350d81856f7a9
2015-04-20 17:03:25 -05:00
Paul Luse
8f5d4d2455 Erasure Code Documentation
This patch adds all the relevant EC documentation to
the source tree. Notable additions are:
  - Updated SAIO documentation
  - Updates to existing swift documentation; and
  - Erasure Coding overview

Co-Authored-By: Alistair Coles <alistair.coles@hp.com>
Co-Authored-By: Thiago da Silva <thiago@redhat.com>
Co-Authored-By: John Dickinson <me@not.mn>
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Co-Authored-By: Tushar Gohad <tushar.gohad@intel.com>
Co-Authored-By: Samuel Merritt <sam@swiftstack.com>
Co-Authored-By: Christian Schwede <christian.schwede@enovance.com>
Co-Authored-By: Yuan Zhou <yuan.zhou@intel.com>
Change-Id: I0403016a4bb7dad9535891632753b0e5e9d402eb
Implements: blueprint swift-ec
Signed-off-by: Thiago da Silva <thiago@redhat.com>
2015-04-14 00:52:17 -07:00
Alistair Coles
4ca08cc395 Update guest VM OS recommendation in SAIO doc
The target development platform has changed to Ubuntu 14.04 [1].
This patch makes the suggested SAIO platform the same.

Also, remove pointer to wiki page for other platform install
instructions that either redirects back to this SAIO doc or
to another wike page and then a dead link.

[1] I0a96bcf692bb240f3ab5aab7fefd294a07735a83

DocImpact

Change-Id: I9f96104b5437c1f1f28f924c048ef83cf03338f4
2015-02-19 17:46:10 +00:00
Samuel Merritt
466403723c Make resetswift customizable via environment
Instead of recommending to edit resetswift to replace "/dev/sdb1" with
"/srv/swift-disk", use an environment variable instead. This way I can
set SAIO_BLOCK_DEVICE=/srv/swift-disk in my .bashrc, and then when I'm
testing out changes to resetswift, I don't need to remember to edit
the modified script, nor do I end up submitting changes with the wrong
default in there.

The variable defaults to /dev/sdb1, so if you use the script unmodified
and don't set SAIO_BLOCK_DEVICE, nothing changes for you.

Change-Id: I741a8c91c2c54a4f32bc391cd794ef4206402753
2014-11-10 15:07:44 -08:00
Peter Portante
3e04606f87 Add a env var to use in-memory obj server in func
Add an environment variable to enable the use of the in-memory object
server during in-process functional test runs.

It might be worth-while to just run under both object servers in-tree,
but this at least enables it, without having to figure out how to make
two test runs in two different environments.

DocImpact

Change-Id: Id76b008e1f273c639ae61550affddc32c5d7c419
Signed-off-by: Thiago da Silva <thiago@redhat.com>
2014-08-04 16:54:50 +01:00
Matthew Oliver
7904358a43 Add SAIO fix for Ubuntu 12.04
The version of setup tools in Ubuntu 12.04 is too old and is causing
the SAIO instructions to fail when installing python-swiftclient.

The work around is to install python-swiftclient's dependencies
before running running:

  python setup.py develop

This change adds a note to users of Ubuntu 12.04 to replace step 2
of "Getting the code" with:

  cd $HOME/python-swiftclient; sudo pip install -r requirements.txt; \
                               python setup.py develop; cd -

Change-Id: I63f57bbf1f1158f8740f6137ad55ff49f12a316c
Closes-Bug: #1217288
2014-06-30 12:55:37 +10:00
Paul Luse
e52e8bc917 Add Storage Policy Documentation
Add overview and example information for using Storage Policies.

DocImpact
Implements: blueprint storage-policies
Change-Id: I6f11f7a1bdaa6f3defb3baa56a820050e5f727f1
2014-06-19 10:18:34 -07:00
Peter Portante
deb0dfd090 Add includes of referenced SAIO bin scripts
Change-Id: I6810e69a757336a3aed0a38146c27f270fe2dde1
2014-04-10 12:27:25 -07:00
Shane Wang
a94be9443d Fix misspellings in swift
Fix misspellings detected by:
* pip install misspellings
* git ls-files | grep -v locale | misspellings -f -

Change-Id: I6594fc4ca5ae10bd30eac8a2f2493a376adcadee
Closes-Bug: #1257295
2014-02-20 16:15:48 +08:00
Peter Portante
ba5fe5f39e Use files in the source tree instead of cut/paste
Many of the large files are included in the tree and the script now
leverages a checked out swift tree to provide those files so that
users don't have to cut/paste text from the document. The contents of
those files are still included in the document for reference.

Updated to add sudo in appropriate places so that the entire script
can be run as the user instead of as root.

We also simplify the steps needed to get resetswift script working
(don't need to edit the user name).

Change-Id: Ie5b5a815870edcc205d273e35e0bbd2426d3b002
Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-12-09 20:55:08 -05:00
Samuel Merritt
26483a2fd1 Add more stuff to SAIO doc's proxy pipeline.
If you're setting one of these up, you're probably going to use it for
development, in which case you want everything but the kitchen sink
turned on so you can just start hacking away.

Change-Id: I98d178ff545cbf8d853c102e9fce76fb9f6773ac
2013-11-05 16:15:07 -08:00
Peter Portante
3d8f0f1805 Fedora 19: need to use /etc/rc.d/rc.local
Change-Id: I80e9a4c40ff99ec09a8eeef935447c6393ea78ec
Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-10-03 11:36:31 -04:00
Clay Gerrard
02e247c1b3 Add "note" box callouts to SAIO for user changes.
The SAIO is purpously cut into two parts, so that you don't have to switch
back and forth between root and your unprivledged user.  Add some "note" box
callouts to highlight this changeover.

Change-Id: I8b1a8f0539eac60d4121bdd4dab01df75ecca207
2013-10-02 11:39:35 -07:00
Peter Portante
e8a07c4ca7 Fedora 19 updates
Change-Id: I95138852e45aa7632218a7107e0e7ba1f6ef373c
Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-09-30 10:23:04 -04:00
Samuel Merritt
d9d7b2135a Install libffi-dev in SAIO docs.
If you don't, then newer versions of xattr won't install, and since
our xattr requirement is simply ">= 0.4" in requirements.txt, this
affects anyone setting up a new SAIO.

This happened with xattr 0.7, which was released on 2013-07-19.

Change-Id: Iaf335fa25a2908953d1fd218158ebedf5d01cc27
2013-09-24 16:54:58 -07:00
Samuel Merritt
ce5e810fed Update SAIO doc to have double proxy-logging in pipeline.
Change-Id: I0a034ca1420761cbf4e35dcea1d9cd18a92f90bd
2013-09-24 16:54:58 -07:00
Clay Gerrard
de3acec4bf Set default wsgi workers to cpu_count
Change the default value of wsgi workers from 1 to auto.  The new default
value for workers in the proxy, container, account & object wsgi servers will
spawn as many workers per process as you have cpu cores.

This will not be ideal for some configurations, but it's much more likely to
produce a successful out of the box deployment.

Inspect the number of cpu_cores using python's multiprocessing when available.
Multiprocessing was added in python 2.6, but I know I've compiled python
without it before on accident.  The cpu_count method seems to be pretty system
agnostic, but it says it can raise NotImplementedError or sometimes return 0.

Add a new utility method 'config_auto_int_value' to pull an integer out of the
config which has a dynamic default.

 * drive by s/container/proxy/ in proxy-server.conf.5
 * fix misplaced max_clients in *-server.conf-sample
 * update doc/development_saio to force workers = 1

DocImpact

Change-Id: Ifa563d22952c902ab8cbe1d339ba385413c54e95
2013-07-18 22:57:18 -07:00
Chuck Thier
581f7f5517 Update docs to use default XFS inode size
In past couple of years, the XFS team has greatly improved inode use in
xfs.  With more recent kernels, there is no performance penalty for
using the default inode size, and a smaller inode size gives us
improvements in other areas where disk access is involved.

DocImpact

Change-Id: Ie9da53a6e8bf43d1d02881befbb52595462c9f2e
2013-06-28 19:52:17 +00:00
Chuck Thier
b012fd998c Change ring partition size for SAIO
Making this smaller (10 instead of 18) can make some of the tests run
faster and makes rebuilding of the rings faster.

Change-Id: Ibe46011d8e6a6482d39b3a20ac9c091d9fbc6ef7
2013-06-26 15:52:18 +00:00
Pete Zaitcev
11aaaf1f3f Remove Lucid/ppa instructions from SAIO guide
Lucid won't EOL until May of 2014; but I stopped trusting that ppa a long time
ago. Besides with the requires for dnspython and mock where they're at you
almost can't install swift from source on any stock distro and expect tests to
pass with system packages - so we're looking at pypi for depends regardless.

While I'm in there:

* more explanation of <your-user-name> and a helpful find/sed for configs
* group the "setup ~/.bashrc" stuff with the "setup ~/bin" stuff
* some updates/fixes from my experience installing on CentOS
* remove region warnings from remakerings

Change-Id: Ie2e6b06959ab699d853e07e5b7e8cda7036a44fe
2013-06-13 18:29:06 -06:00
Kun Huang
90c422deae Improve SAIO deploy document.
improving points:
1. Remove yum install swift in Fedora; Use installing from source for
both Ubuntu and Fedora.
2. Explain you could use all users including root, your own guest. An
d the points developer have to care.

Change-Id: Id6d683441bd790a21734624e29eb7c98bb40de85
Fixes: bug #1126389
2013-06-13 11:55:03 +08:00
David Hadas
39fc4c37bf Doc changes for new pip-requirements
dnspython was added recently to the list of requirments.

This patch completes the change:
1. The new requirement is documented
2. The need for tox --recreate is documented
(assuming there is no way to automate 2 above)

Change-Id: I49f0cb4b01b79dd7cf4329984a2f8416378ebe19
Fixes: Bug #1176627
2013-05-06 19:21:53 +03:00
David Hadas
a979c8007b Add support for Hash Prefix
A new configuration parameter is added to /etc/swift/swift.conf
[swift-hash]
swift_hash_path_prefix = 'random unique string'

New installations are advised to set this parameter to a random secret,
which would not be disclosed ouside the organization.
The same secret needs to be used by all swift servers of the same cluster.

Existing installations should set this parameter to an empty string
(the default)

DocImpact

Fixes: Bug #1157454

Change-Id: I63b10d0b7d6dd3f74e0f10bb41b5f240fa03578a
2013-03-22 19:41:55 +02:00
Pete Zaitcev
93ea7c63b1 Documentation fixups
These are mostly cosmetic fixes for irritating imperfections:
- "separated with commas" was duplicated, leave just one
- extra whitespace here and there, man pages are not PEP8, drop
- weird extra commas, drop
- Fedora logs to /var/log/messages
- "drive is has failed", drop "is"

Change-Id: I5ceba2e61b16db4855d76c92cbc83663b9b2a0da
2013-02-18 10:54:27 -07:00
Chuck Thier
e88ff34685 Cleanup of file permissions
Mostly removed execute on non-executable files

Change-Id: Ibfbe7e0cf0fbeabef602d70b20f75e1dd3bdf9c9
2013-02-15 11:06:11 -06:00
David Hadas
83d89228a9 minor development doc changes
Reviewed development saio and guidelines while installing
a new dev and test node.

Change-Id: I59238a4d43fa08a89be14f43681ad4693f1261dd
2013-02-09 11:11:05 +02:00
Michael Barton
c37954d935 saio truncate instead of dd (for reals)
Changing that complex dd to a functionally equivalent truncate that's easier
to read and modify.

DocImpact

Change-Id: I64404318364608e62a1d80f6a0550271eb4cd03a
2012-12-20 20:12:50 -08:00
Michael Barton
5d5b262be8 Revert "suggest fallocate instead of dd in saio"
Change-Id: I3742c4580d920e09de434e86b75b134c62f1732e
2012-12-20 13:35:45 -08:00
Michael Barton
e81ca8daf3 suggest fallocate instead of dd in saio
Change-Id: I4d5ba8e58216749c2858dc1c0b8225b3619a7287
2012-12-19 14:10:07 -08:00
Chuck Thier
4c6a354483 Add config option to turn eventlet debug on/off
By default, this will be turned off.  This will cause eventlet to not
print stack traces to stderr which can be very annoying on production
systems.  It is still recommended to turn it on for development or
debuging purposes.

DocImpact
Change-Id: I5e5b902d3d9ed85f784549e53f2ee2fc87cbe2e5
2012-12-10 10:22:09 -06:00
Michael Barton
c7e2e9a1f3 remove configobj from deps
Change-Id: Ie566e712ba0d3dca543e99efbdb576b19623e73c
2012-10-08 14:20:00 -07:00
Michael Barton
5e3e9a882d local WSGI Request and Response classes
This change replaces WebOb with a mostly compatible local library,
swift.common.swob.  Subtle changes to WebOb's API over the years have been a
huge headache.  Swift doesn't even run on the current version.

There are a few incompatibilities to simplify the implementation/interface:
 * It only implements the header properties we use.  More can be easily added.
 * Casts header values to str on assignment.
 * Response classes ("HTTPNotFound") are no longer subclasses, but partials
   on Response, so things like isinstance no longer work on them.
 * Unlike newer webob versions, will never return unicode objects.

Change-Id: I76617a0903ee2286b25a821b3c935c86ff95233f
2012-09-28 14:48:48 -07:00
Florian Hines
c0537ac6e0 Breakout search_devs & add get_builder() for reuse
This moves search_devs into RingBuilder to make it accessible to other utils
that need to search the builder. Along the same lines this also adds a
load() call to get a RingBuilder instance when working with the builder files.

- This adds python-mock >= 0.7 as a dependency for unittests. On Ubuntu
  10.04 you'll have to pip install it, on 12.04 you can apt-get install
  it. Fedora 17+ should be able to yum install it.
- new pep8 compliance
- Fixed a small issue (undefined var) in swift-ring-builder when remove was
called but failed to find a match.

Change-Id: I2e02684235aa2f4e901a00858ae037091594c545
2012-09-06 20:16:46 -05:00
Darrell Bishop
4a2ae2b460 Upating proxy-server StatsD logging.
Removed many StatsD logging calls in proxy-server and added
swift-informant-style catch-all logging in the proxy-logger middleware.
Many errors previously rolled into the "proxy-server.<type>.errors"
counter will now appear broken down by response code and with timing
data at: "proxy-server.<type>.<verb>.<status>.timing".  Also, bytes
transferred (sum of in + out) will be at:
"proxy-server.<type>.<verb>.<status>.xfer".  The proxy-logging
middleware can get its StatsD config from standard vars in [DEFAULT] or
from access_log_statsd_* config vars in its config section.

Similarly to Swift Informant, request methods ("verbs") are filtered
using the new proxy-logging config var, "log_statsd_valid_http_methods"
which defaults to GET, HEAD, POST, PUT, DELETE, and COPY.  Requests with
methods not in this list use "BAD_METHOD" for <verb> in the metric name.
To avoid user error, access_log_statsd_valid_http_methods is also
accepted.

Previously, proxy-server metrics used "Account", "Container", and
"Object" for the <type>, but these are now all lowercase.

Updated the admin guide's StatsD docs to reflect the above changes and
also include the "proxy-server.<type>.handoff_count" and
"proxy-server.<type>.handoff_all_count" metrics.

The proxy server now saves off the original req.method and proxy_logging
will use this if it can (both for request logging and as the "<verb>" in
the statsd timing metric).  This fixes bug 1025433.

Removed some stale access_log_* related code in proxy/server.py.  Also
removed the BaseApplication/Application distinction as it's no longer
necessary.

Fixed up the sample config files a bit (logging lines, mostly).

Fixed typo in SAIO development guide.

Got proxy_logging.py test coverage to 100%.

Fixed proxy_logging.py for PEP8 v1.3.2.

Enhanced test.unit.FakeLogger to track more calls to enable testing
StatsD metric calls.

Change-Id: I45d94cb76450be96d66fcfab56359bdfdc3a2576
2012-08-29 16:08:30 -07:00
gholt
c509ac2371 Added ability to disable fallocate
Change-Id: Id8872c581ed23378a8e14cbf3bf049b5c0d21577
2012-08-29 19:57:26 +00:00