Currently all devices in the ring and all services in a SAIO
all bind to the same loopback address 127.0.0.1. But this
breaks servers_per_port if you want to do any testing on that.
This change binds each service to a different loopback address
and updates the rings (remakerings) accordingly.
To make sure rysncd binds correctly the bind address needed
to be changed to listen on all addresses (0.0.0.0).
Change-Id: I7e77434f275df1e2699de495d8b622b90157a9d7
SAIO docs do suggest using Ubuntu 14.04, but if using
16.04 then systemctl needs to be used to have rsync service
restart on reboot.
Change-Id: I4fb0d3d063df61fbdfca981f06911148f3c4dc04
libssl-dev/openssl-devel are already listed in other-requirements.txt;
add them to installation instructions in the SAIO docs.
Change-Id: I3dc07213ff8dac1299d3eb68d3448a77e15c79af
Currently when disable_fallocate is true it disables calling the
fallocate syscall, but it doesn't disable fallocate_reserve. This
patch fixes this.
This problem has caused functional tests to fail in our SAIOs, since
SAIOs have disable_fallocate set but the fallocate_reserve space free
checking was still being run creating 507 responses. This is thanks
to the change in fallocate_reserve default changing from 0 to 1%.
Because fallocate_reserve and disable_fallocate causes SAIO functional
tests to fail a section called 'Known Issues' has been added to the
SAIO developer documentation which includes a warning about
using fallocate_reserve on SAIOs.
Change-Id: I727bfb0861ea26fe2f16ad55f4d36ae088864d8f
Adding an additional debugging step for the SAIO "Debugging Issues"
section. Adding step to check if memcache is running and to start
it if its not.
Change-Id: I0bfa563f595dbefcf907494451fad45e6310db88
a1c32702, 736cf54a, and 38787d0f remove uses of `simplejson` from
various parts of Swift in favor of the standard libary `json`
module (introduced in Python 2.6). This commit performs the remaining
`simplejson` to `json` replacements, removes two comments highlighting
quirks of simplejson with respect to Unicode, and removes the references
to it in setup documentation and requirements.txt.
There were a lot of places where we were importing json from
swift.common.utils, which is less intuitive than a direct `import json`,
so that replacement is made as well.
(And in two more tiny drive-bys, we add some pretty-indenting to an XML
fragment and use `super` rather than naming a base class explicitly.)
Change-Id: I769e88dda7f76ce15cf7ce930dc1874d24f9498a
SAIO Configuration and documentation changes enabling to run the
container sync probe test by default
Change-Id: Iccf59533d0d4fe72549d318339ab125d04dde006
Related-Bug: #1476623
When rsync pushes to a remote node with an unmounted drive and if
certain steps are not taken, rsync may attempt to write files to
the local drive at the location where the drive was mounted.
There are two suggested solutions for this issue:
1) Set the permissions for all mount points in /srv/node/
to root:root 755
2) Mount the drives elsewhere and symlink the drives to /srv/.../
The first method ensures that only root and not the swift user
can write in the /srv/.../ directories.
The second method will prompt a broken link issue if rsync
attempts to write to an unmounted drive.
Change-Id: I60ce4ed9ef8401768d5f78b6806cbb2e2a65303e
Closes-Bug: #1470576
Make the test-requirements install command uniform with the rest of the
setup commands in the page by using the $HOME, instead of hardcoding
the folder path.
Change-Id: I8da7a191a301601e27f894f8268238d77b150ede
The target development platform has changed to Ubuntu 14.04 [1].
This patch makes the suggested SAIO platform the same.
Also, remove pointer to wiki page for other platform install
instructions that either redirects back to this SAIO doc or
to another wike page and then a dead link.
[1] I0a96bcf692bb240f3ab5aab7fefd294a07735a83
DocImpact
Change-Id: I9f96104b5437c1f1f28f924c048ef83cf03338f4
Instead of recommending to edit resetswift to replace "/dev/sdb1" with
"/srv/swift-disk", use an environment variable instead. This way I can
set SAIO_BLOCK_DEVICE=/srv/swift-disk in my .bashrc, and then when I'm
testing out changes to resetswift, I don't need to remember to edit
the modified script, nor do I end up submitting changes with the wrong
default in there.
The variable defaults to /dev/sdb1, so if you use the script unmodified
and don't set SAIO_BLOCK_DEVICE, nothing changes for you.
Change-Id: I741a8c91c2c54a4f32bc391cd794ef4206402753
Add an environment variable to enable the use of the in-memory object
server during in-process functional test runs.
It might be worth-while to just run under both object servers in-tree,
but this at least enables it, without having to figure out how to make
two test runs in two different environments.
DocImpact
Change-Id: Id76b008e1f273c639ae61550affddc32c5d7c419
Signed-off-by: Thiago da Silva <thiago@redhat.com>
The version of setup tools in Ubuntu 12.04 is too old and is causing
the SAIO instructions to fail when installing python-swiftclient.
The work around is to install python-swiftclient's dependencies
before running running:
python setup.py develop
This change adds a note to users of Ubuntu 12.04 to replace step 2
of "Getting the code" with:
cd $HOME/python-swiftclient; sudo pip install -r requirements.txt; \
python setup.py develop; cd -
Change-Id: I63f57bbf1f1158f8740f6137ad55ff49f12a316c
Closes-Bug: #1217288
Add overview and example information for using Storage Policies.
DocImpact
Implements: blueprint storage-policies
Change-Id: I6f11f7a1bdaa6f3defb3baa56a820050e5f727f1
Many of the large files are included in the tree and the script now
leverages a checked out swift tree to provide those files so that
users don't have to cut/paste text from the document. The contents of
those files are still included in the document for reference.
Updated to add sudo in appropriate places so that the entire script
can be run as the user instead of as root.
We also simplify the steps needed to get resetswift script working
(don't need to edit the user name).
Change-Id: Ie5b5a815870edcc205d273e35e0bbd2426d3b002
Signed-off-by: Peter Portante <peter.portante@redhat.com>
If you're setting one of these up, you're probably going to use it for
development, in which case you want everything but the kitchen sink
turned on so you can just start hacking away.
Change-Id: I98d178ff545cbf8d853c102e9fce76fb9f6773ac
The SAIO is purpously cut into two parts, so that you don't have to switch
back and forth between root and your unprivledged user. Add some "note" box
callouts to highlight this changeover.
Change-Id: I8b1a8f0539eac60d4121bdd4dab01df75ecca207
If you don't, then newer versions of xattr won't install, and since
our xattr requirement is simply ">= 0.4" in requirements.txt, this
affects anyone setting up a new SAIO.
This happened with xattr 0.7, which was released on 2013-07-19.
Change-Id: Iaf335fa25a2908953d1fd218158ebedf5d01cc27
Change the default value of wsgi workers from 1 to auto. The new default
value for workers in the proxy, container, account & object wsgi servers will
spawn as many workers per process as you have cpu cores.
This will not be ideal for some configurations, but it's much more likely to
produce a successful out of the box deployment.
Inspect the number of cpu_cores using python's multiprocessing when available.
Multiprocessing was added in python 2.6, but I know I've compiled python
without it before on accident. The cpu_count method seems to be pretty system
agnostic, but it says it can raise NotImplementedError or sometimes return 0.
Add a new utility method 'config_auto_int_value' to pull an integer out of the
config which has a dynamic default.
* drive by s/container/proxy/ in proxy-server.conf.5
* fix misplaced max_clients in *-server.conf-sample
* update doc/development_saio to force workers = 1
DocImpact
Change-Id: Ifa563d22952c902ab8cbe1d339ba385413c54e95
In past couple of years, the XFS team has greatly improved inode use in
xfs. With more recent kernels, there is no performance penalty for
using the default inode size, and a smaller inode size gives us
improvements in other areas where disk access is involved.
DocImpact
Change-Id: Ie9da53a6e8bf43d1d02881befbb52595462c9f2e
Making this smaller (10 instead of 18) can make some of the tests run
faster and makes rebuilding of the rings faster.
Change-Id: Ibe46011d8e6a6482d39b3a20ac9c091d9fbc6ef7
Lucid won't EOL until May of 2014; but I stopped trusting that ppa a long time
ago. Besides with the requires for dnspython and mock where they're at you
almost can't install swift from source on any stock distro and expect tests to
pass with system packages - so we're looking at pypi for depends regardless.
While I'm in there:
* more explanation of <your-user-name> and a helpful find/sed for configs
* group the "setup ~/.bashrc" stuff with the "setup ~/bin" stuff
* some updates/fixes from my experience installing on CentOS
* remove region warnings from remakerings
Change-Id: Ie2e6b06959ab699d853e07e5b7e8cda7036a44fe
improving points:
1. Remove yum install swift in Fedora; Use installing from source for
both Ubuntu and Fedora.
2. Explain you could use all users including root, your own guest. An
d the points developer have to care.
Change-Id: Id6d683441bd790a21734624e29eb7c98bb40de85
Fixes: bug #1126389
dnspython was added recently to the list of requirments.
This patch completes the change:
1. The new requirement is documented
2. The need for tox --recreate is documented
(assuming there is no way to automate 2 above)
Change-Id: I49f0cb4b01b79dd7cf4329984a2f8416378ebe19
Fixes: Bug #1176627
A new configuration parameter is added to /etc/swift/swift.conf
[swift-hash]
swift_hash_path_prefix = 'random unique string'
New installations are advised to set this parameter to a random secret,
which would not be disclosed ouside the organization.
The same secret needs to be used by all swift servers of the same cluster.
Existing installations should set this parameter to an empty string
(the default)
DocImpact
Fixes: Bug #1157454
Change-Id: I63b10d0b7d6dd3f74e0f10bb41b5f240fa03578a
These are mostly cosmetic fixes for irritating imperfections:
- "separated with commas" was duplicated, leave just one
- extra whitespace here and there, man pages are not PEP8, drop
- weird extra commas, drop
- Fedora logs to /var/log/messages
- "drive is has failed", drop "is"
Change-Id: I5ceba2e61b16db4855d76c92cbc83663b9b2a0da
Changing that complex dd to a functionally equivalent truncate that's easier
to read and modify.
DocImpact
Change-Id: I64404318364608e62a1d80f6a0550271eb4cd03a
By default, this will be turned off. This will cause eventlet to not
print stack traces to stderr which can be very annoying on production
systems. It is still recommended to turn it on for development or
debuging purposes.
DocImpact
Change-Id: I5e5b902d3d9ed85f784549e53f2ee2fc87cbe2e5
This change replaces WebOb with a mostly compatible local library,
swift.common.swob. Subtle changes to WebOb's API over the years have been a
huge headache. Swift doesn't even run on the current version.
There are a few incompatibilities to simplify the implementation/interface:
* It only implements the header properties we use. More can be easily added.
* Casts header values to str on assignment.
* Response classes ("HTTPNotFound") are no longer subclasses, but partials
on Response, so things like isinstance no longer work on them.
* Unlike newer webob versions, will never return unicode objects.
Change-Id: I76617a0903ee2286b25a821b3c935c86ff95233f
This moves search_devs into RingBuilder to make it accessible to other utils
that need to search the builder. Along the same lines this also adds a
load() call to get a RingBuilder instance when working with the builder files.
- This adds python-mock >= 0.7 as a dependency for unittests. On Ubuntu
10.04 you'll have to pip install it, on 12.04 you can apt-get install
it. Fedora 17+ should be able to yum install it.
- new pep8 compliance
- Fixed a small issue (undefined var) in swift-ring-builder when remove was
called but failed to find a match.
Change-Id: I2e02684235aa2f4e901a00858ae037091594c545
Removed many StatsD logging calls in proxy-server and added
swift-informant-style catch-all logging in the proxy-logger middleware.
Many errors previously rolled into the "proxy-server.<type>.errors"
counter will now appear broken down by response code and with timing
data at: "proxy-server.<type>.<verb>.<status>.timing". Also, bytes
transferred (sum of in + out) will be at:
"proxy-server.<type>.<verb>.<status>.xfer". The proxy-logging
middleware can get its StatsD config from standard vars in [DEFAULT] or
from access_log_statsd_* config vars in its config section.
Similarly to Swift Informant, request methods ("verbs") are filtered
using the new proxy-logging config var, "log_statsd_valid_http_methods"
which defaults to GET, HEAD, POST, PUT, DELETE, and COPY. Requests with
methods not in this list use "BAD_METHOD" for <verb> in the metric name.
To avoid user error, access_log_statsd_valid_http_methods is also
accepted.
Previously, proxy-server metrics used "Account", "Container", and
"Object" for the <type>, but these are now all lowercase.
Updated the admin guide's StatsD docs to reflect the above changes and
also include the "proxy-server.<type>.handoff_count" and
"proxy-server.<type>.handoff_all_count" metrics.
The proxy server now saves off the original req.method and proxy_logging
will use this if it can (both for request logging and as the "<verb>" in
the statsd timing metric). This fixes bug 1025433.
Removed some stale access_log_* related code in proxy/server.py. Also
removed the BaseApplication/Application distinction as it's no longer
necessary.
Fixed up the sample config files a bit (logging lines, mostly).
Fixed typo in SAIO development guide.
Got proxy_logging.py test coverage to 100%.
Fixed proxy_logging.py for PEP8 v1.3.2.
Enhanced test.unit.FakeLogger to track more calls to enable testing
StatsD metric calls.
Change-Id: I45d94cb76450be96d66fcfab56359bdfdc3a2576