131 Commits

Author SHA1 Message Date
Alistair Coles
4ca08cc395 Update guest VM OS recommendation in SAIO doc
The target development platform has changed to Ubuntu 14.04 [1].
This patch makes the suggested SAIO platform the same.

Also, remove pointer to wiki page for other platform install
instructions that either redirects back to this SAIO doc or
to another wike page and then a dead link.

[1] I0a96bcf692bb240f3ab5aab7fefd294a07735a83

DocImpact

Change-Id: I9f96104b5437c1f1f28f924c048ef83cf03338f4
2015-02-19 17:46:10 +00:00
Samuel Merritt
466403723c Make resetswift customizable via environment
Instead of recommending to edit resetswift to replace "/dev/sdb1" with
"/srv/swift-disk", use an environment variable instead. This way I can
set SAIO_BLOCK_DEVICE=/srv/swift-disk in my .bashrc, and then when I'm
testing out changes to resetswift, I don't need to remember to edit
the modified script, nor do I end up submitting changes with the wrong
default in there.

The variable defaults to /dev/sdb1, so if you use the script unmodified
and don't set SAIO_BLOCK_DEVICE, nothing changes for you.

Change-Id: I741a8c91c2c54a4f32bc391cd794ef4206402753
2014-11-10 15:07:44 -08:00
Peter Portante
3e04606f87 Add a env var to use in-memory obj server in func
Add an environment variable to enable the use of the in-memory object
server during in-process functional test runs.

It might be worth-while to just run under both object servers in-tree,
but this at least enables it, without having to figure out how to make
two test runs in two different environments.

DocImpact

Change-Id: Id76b008e1f273c639ae61550affddc32c5d7c419
Signed-off-by: Thiago da Silva <thiago@redhat.com>
2014-08-04 16:54:50 +01:00
Matthew Oliver
7904358a43 Add SAIO fix for Ubuntu 12.04
The version of setup tools in Ubuntu 12.04 is too old and is causing
the SAIO instructions to fail when installing python-swiftclient.

The work around is to install python-swiftclient's dependencies
before running running:

  python setup.py develop

This change adds a note to users of Ubuntu 12.04 to replace step 2
of "Getting the code" with:

  cd $HOME/python-swiftclient; sudo pip install -r requirements.txt; \
                               python setup.py develop; cd -

Change-Id: I63f57bbf1f1158f8740f6137ad55ff49f12a316c
Closes-Bug: #1217288
2014-06-30 12:55:37 +10:00
Paul Luse
e52e8bc917 Add Storage Policy Documentation
Add overview and example information for using Storage Policies.

DocImpact
Implements: blueprint storage-policies
Change-Id: I6f11f7a1bdaa6f3defb3baa56a820050e5f727f1
2014-06-19 10:18:34 -07:00
Peter Portante
deb0dfd090 Add includes of referenced SAIO bin scripts
Change-Id: I6810e69a757336a3aed0a38146c27f270fe2dde1
2014-04-10 12:27:25 -07:00
Shane Wang
a94be9443d Fix misspellings in swift
Fix misspellings detected by:
* pip install misspellings
* git ls-files | grep -v locale | misspellings -f -

Change-Id: I6594fc4ca5ae10bd30eac8a2f2493a376adcadee
Closes-Bug: #1257295
2014-02-20 16:15:48 +08:00
Peter Portante
ba5fe5f39e Use files in the source tree instead of cut/paste
Many of the large files are included in the tree and the script now
leverages a checked out swift tree to provide those files so that
users don't have to cut/paste text from the document. The contents of
those files are still included in the document for reference.

Updated to add sudo in appropriate places so that the entire script
can be run as the user instead of as root.

We also simplify the steps needed to get resetswift script working
(don't need to edit the user name).

Change-Id: Ie5b5a815870edcc205d273e35e0bbd2426d3b002
Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-12-09 20:55:08 -05:00
Samuel Merritt
26483a2fd1 Add more stuff to SAIO doc's proxy pipeline.
If you're setting one of these up, you're probably going to use it for
development, in which case you want everything but the kitchen sink
turned on so you can just start hacking away.

Change-Id: I98d178ff545cbf8d853c102e9fce76fb9f6773ac
2013-11-05 16:15:07 -08:00
Peter Portante
3d8f0f1805 Fedora 19: need to use /etc/rc.d/rc.local
Change-Id: I80e9a4c40ff99ec09a8eeef935447c6393ea78ec
Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-10-03 11:36:31 -04:00
Clay Gerrard
02e247c1b3 Add "note" box callouts to SAIO for user changes.
The SAIO is purpously cut into two parts, so that you don't have to switch
back and forth between root and your unprivledged user.  Add some "note" box
callouts to highlight this changeover.

Change-Id: I8b1a8f0539eac60d4121bdd4dab01df75ecca207
2013-10-02 11:39:35 -07:00
Peter Portante
e8a07c4ca7 Fedora 19 updates
Change-Id: I95138852e45aa7632218a7107e0e7ba1f6ef373c
Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-09-30 10:23:04 -04:00
Samuel Merritt
d9d7b2135a Install libffi-dev in SAIO docs.
If you don't, then newer versions of xattr won't install, and since
our xattr requirement is simply ">= 0.4" in requirements.txt, this
affects anyone setting up a new SAIO.

This happened with xattr 0.7, which was released on 2013-07-19.

Change-Id: Iaf335fa25a2908953d1fd218158ebedf5d01cc27
2013-09-24 16:54:58 -07:00
Samuel Merritt
ce5e810fed Update SAIO doc to have double proxy-logging in pipeline.
Change-Id: I0a034ca1420761cbf4e35dcea1d9cd18a92f90bd
2013-09-24 16:54:58 -07:00
Clay Gerrard
de3acec4bf Set default wsgi workers to cpu_count
Change the default value of wsgi workers from 1 to auto.  The new default
value for workers in the proxy, container, account & object wsgi servers will
spawn as many workers per process as you have cpu cores.

This will not be ideal for some configurations, but it's much more likely to
produce a successful out of the box deployment.

Inspect the number of cpu_cores using python's multiprocessing when available.
Multiprocessing was added in python 2.6, but I know I've compiled python
without it before on accident.  The cpu_count method seems to be pretty system
agnostic, but it says it can raise NotImplementedError or sometimes return 0.

Add a new utility method 'config_auto_int_value' to pull an integer out of the
config which has a dynamic default.

 * drive by s/container/proxy/ in proxy-server.conf.5
 * fix misplaced max_clients in *-server.conf-sample
 * update doc/development_saio to force workers = 1

DocImpact

Change-Id: Ifa563d22952c902ab8cbe1d339ba385413c54e95
2013-07-18 22:57:18 -07:00
Chuck Thier
581f7f5517 Update docs to use default XFS inode size
In past couple of years, the XFS team has greatly improved inode use in
xfs.  With more recent kernels, there is no performance penalty for
using the default inode size, and a smaller inode size gives us
improvements in other areas where disk access is involved.

DocImpact

Change-Id: Ie9da53a6e8bf43d1d02881befbb52595462c9f2e
2013-06-28 19:52:17 +00:00
Chuck Thier
b012fd998c Change ring partition size for SAIO
Making this smaller (10 instead of 18) can make some of the tests run
faster and makes rebuilding of the rings faster.

Change-Id: Ibe46011d8e6a6482d39b3a20ac9c091d9fbc6ef7
2013-06-26 15:52:18 +00:00
Pete Zaitcev
11aaaf1f3f Remove Lucid/ppa instructions from SAIO guide
Lucid won't EOL until May of 2014; but I stopped trusting that ppa a long time
ago. Besides with the requires for dnspython and mock where they're at you
almost can't install swift from source on any stock distro and expect tests to
pass with system packages - so we're looking at pypi for depends regardless.

While I'm in there:

* more explanation of <your-user-name> and a helpful find/sed for configs
* group the "setup ~/.bashrc" stuff with the "setup ~/bin" stuff
* some updates/fixes from my experience installing on CentOS
* remove region warnings from remakerings

Change-Id: Ie2e6b06959ab699d853e07e5b7e8cda7036a44fe
2013-06-13 18:29:06 -06:00
Kun Huang
90c422deae Improve SAIO deploy document.
improving points:
1. Remove yum install swift in Fedora; Use installing from source for
both Ubuntu and Fedora.
2. Explain you could use all users including root, your own guest. An
d the points developer have to care.

Change-Id: Id6d683441bd790a21734624e29eb7c98bb40de85
Fixes: bug #1126389
2013-06-13 11:55:03 +08:00
David Hadas
39fc4c37bf Doc changes for new pip-requirements
dnspython was added recently to the list of requirments.

This patch completes the change:
1. The new requirement is documented
2. The need for tox --recreate is documented
(assuming there is no way to automate 2 above)

Change-Id: I49f0cb4b01b79dd7cf4329984a2f8416378ebe19
Fixes: Bug #1176627
2013-05-06 19:21:53 +03:00
David Hadas
a979c8007b Add support for Hash Prefix
A new configuration parameter is added to /etc/swift/swift.conf
[swift-hash]
swift_hash_path_prefix = 'random unique string'

New installations are advised to set this parameter to a random secret,
which would not be disclosed ouside the organization.
The same secret needs to be used by all swift servers of the same cluster.

Existing installations should set this parameter to an empty string
(the default)

DocImpact

Fixes: Bug #1157454

Change-Id: I63b10d0b7d6dd3f74e0f10bb41b5f240fa03578a
2013-03-22 19:41:55 +02:00
Pete Zaitcev
93ea7c63b1 Documentation fixups
These are mostly cosmetic fixes for irritating imperfections:
- "separated with commas" was duplicated, leave just one
- extra whitespace here and there, man pages are not PEP8, drop
- weird extra commas, drop
- Fedora logs to /var/log/messages
- "drive is has failed", drop "is"

Change-Id: I5ceba2e61b16db4855d76c92cbc83663b9b2a0da
2013-02-18 10:54:27 -07:00
Chuck Thier
e88ff34685 Cleanup of file permissions
Mostly removed execute on non-executable files

Change-Id: Ibfbe7e0cf0fbeabef602d70b20f75e1dd3bdf9c9
2013-02-15 11:06:11 -06:00
David Hadas
83d89228a9 minor development doc changes
Reviewed development saio and guidelines while installing
a new dev and test node.

Change-Id: I59238a4d43fa08a89be14f43681ad4693f1261dd
2013-02-09 11:11:05 +02:00
Michael Barton
c37954d935 saio truncate instead of dd (for reals)
Changing that complex dd to a functionally equivalent truncate that's easier
to read and modify.

DocImpact

Change-Id: I64404318364608e62a1d80f6a0550271eb4cd03a
2012-12-20 20:12:50 -08:00
Michael Barton
5d5b262be8 Revert "suggest fallocate instead of dd in saio"
Change-Id: I3742c4580d920e09de434e86b75b134c62f1732e
2012-12-20 13:35:45 -08:00
Michael Barton
e81ca8daf3 suggest fallocate instead of dd in saio
Change-Id: I4d5ba8e58216749c2858dc1c0b8225b3619a7287
2012-12-19 14:10:07 -08:00
Chuck Thier
4c6a354483 Add config option to turn eventlet debug on/off
By default, this will be turned off.  This will cause eventlet to not
print stack traces to stderr which can be very annoying on production
systems.  It is still recommended to turn it on for development or
debuging purposes.

DocImpact
Change-Id: I5e5b902d3d9ed85f784549e53f2ee2fc87cbe2e5
2012-12-10 10:22:09 -06:00
Michael Barton
c7e2e9a1f3 remove configobj from deps
Change-Id: Ie566e712ba0d3dca543e99efbdb576b19623e73c
2012-10-08 14:20:00 -07:00
Michael Barton
5e3e9a882d local WSGI Request and Response classes
This change replaces WebOb with a mostly compatible local library,
swift.common.swob.  Subtle changes to WebOb's API over the years have been a
huge headache.  Swift doesn't even run on the current version.

There are a few incompatibilities to simplify the implementation/interface:
 * It only implements the header properties we use.  More can be easily added.
 * Casts header values to str on assignment.
 * Response classes ("HTTPNotFound") are no longer subclasses, but partials
   on Response, so things like isinstance no longer work on them.
 * Unlike newer webob versions, will never return unicode objects.

Change-Id: I76617a0903ee2286b25a821b3c935c86ff95233f
2012-09-28 14:48:48 -07:00
Florian Hines
c0537ac6e0 Breakout search_devs & add get_builder() for reuse
This moves search_devs into RingBuilder to make it accessible to other utils
that need to search the builder. Along the same lines this also adds a
load() call to get a RingBuilder instance when working with the builder files.

- This adds python-mock >= 0.7 as a dependency for unittests. On Ubuntu
  10.04 you'll have to pip install it, on 12.04 you can apt-get install
  it. Fedora 17+ should be able to yum install it.
- new pep8 compliance
- Fixed a small issue (undefined var) in swift-ring-builder when remove was
called but failed to find a match.

Change-Id: I2e02684235aa2f4e901a00858ae037091594c545
2012-09-06 20:16:46 -05:00
Darrell Bishop
4a2ae2b460 Upating proxy-server StatsD logging.
Removed many StatsD logging calls in proxy-server and added
swift-informant-style catch-all logging in the proxy-logger middleware.
Many errors previously rolled into the "proxy-server.<type>.errors"
counter will now appear broken down by response code and with timing
data at: "proxy-server.<type>.<verb>.<status>.timing".  Also, bytes
transferred (sum of in + out) will be at:
"proxy-server.<type>.<verb>.<status>.xfer".  The proxy-logging
middleware can get its StatsD config from standard vars in [DEFAULT] or
from access_log_statsd_* config vars in its config section.

Similarly to Swift Informant, request methods ("verbs") are filtered
using the new proxy-logging config var, "log_statsd_valid_http_methods"
which defaults to GET, HEAD, POST, PUT, DELETE, and COPY.  Requests with
methods not in this list use "BAD_METHOD" for <verb> in the metric name.
To avoid user error, access_log_statsd_valid_http_methods is also
accepted.

Previously, proxy-server metrics used "Account", "Container", and
"Object" for the <type>, but these are now all lowercase.

Updated the admin guide's StatsD docs to reflect the above changes and
also include the "proxy-server.<type>.handoff_count" and
"proxy-server.<type>.handoff_all_count" metrics.

The proxy server now saves off the original req.method and proxy_logging
will use this if it can (both for request logging and as the "<verb>" in
the statsd timing metric).  This fixes bug 1025433.

Removed some stale access_log_* related code in proxy/server.py.  Also
removed the BaseApplication/Application distinction as it's no longer
necessary.

Fixed up the sample config files a bit (logging lines, mostly).

Fixed typo in SAIO development guide.

Got proxy_logging.py test coverage to 100%.

Fixed proxy_logging.py for PEP8 v1.3.2.

Enhanced test.unit.FakeLogger to track more calls to enable testing
StatsD metric calls.

Change-Id: I45d94cb76450be96d66fcfab56359bdfdc3a2576
2012-08-29 16:08:30 -07:00
gholt
c509ac2371 Added ability to disable fallocate
Change-Id: Id8872c581ed23378a8e14cbf3bf049b5c0d21577
2012-08-29 19:57:26 +00:00
Darrell Bishop
56bb00f7c9 A couple minor tweaks to the SAIO guide.
Added "-p" to mkdir cmds in a "set -e" script (/etc/rc.local).

/var/log/swift gets owned by the group "adm", but isn't group writeable
by default, so the find/rm in "resetswift" fails.  So I added a chmod to
add group-write for that directory tree.

Added commands for installing tox.

Change-Id: Ic141491272a83a9233cd6a6a70c3ef74411f3f51
2012-08-21 16:18:51 -07:00
Tong Li
d8c5c0c897 all in one setup rc.local needs -p for mkdir command
bug fixes for swift 1033089
Swift 1.6.0 All in One setup uses rc.local to create a set of
directories like the following:
   /var/cache/swift /var/cache/swift2 /var/cache/swift3
   /var/cache/swift4
   /var/run/swift
without -p for mkdir command, when some of these directories
already exist, the rc.local fails so that the next set of
command won't execute. This causes swift fail to start because
of /var/run/swift does not exist. Adding -p to both mkdir
command solves this problem.

Change-Id: Ia26c9a857723747008293c0631c9acc1b58eca7e
2012-08-06 09:11:07 -04:00
Florian Hines
0da1b4f82d Create and configure /var/cache/swift in SAIO
Fixes bug #1024477

Change-Id: Id09bbff6bb4a5d032727d3ca53b3276b4769d178
2012-07-23 16:44:06 -05:00
Chmouel Boudjnah
22572b506a Remove swift.common.client from here.
- It has been to its own gerrit project.
- direct_client should follow next.
- Implements blueprint clientbindings.

Change-Id: I3bb50c95eba81302bfec71cb7ce5288b85a41dc0
2012-06-07 16:36:49 +02:00
Michael Barton
7c98e7a625 Move proxy server logging to middleware.
Change-Id: I771c87207d4e1821e32c3424b341d182cc7ea7c0
2012-05-24 21:15:51 -07:00
Dan Dillinger
f7fdb9cf12 Implement unit_test config to disable syslog.
bug 701248
bug 819303

This change makes syslog optional, by providing a unit test
config with the ability to cause test/unit/__init__.py to
replace SysLogHandler with a fake logging class. The
default behavior is unchanged.

FakeLogger now inherits directly from Handler and mocks out
its API - this was a backward-compatibility issue.

Change-Id: I653dec148c4f6cf81759de03964c6a3290c1a290
2012-05-03 14:25:46 -04:00
Pete Zaitcev
93964ff0a5 Fix colorization of "hash" in SAIO doc.
Admittedly, this is ridiculously trivial, but the coloring of
"hash" always peeved me. Not so much that it existed, but that
I did not understand what caused it. After consulting our Sphinx
experts, it turned out that the code block was considered code
in Python and "hash" is a reserved word in Python, so it was
colored.

One option is to declare the code block "ini", which is what
it is, but that causes the block stand out just as badly, unless
we highlight every one of them. So, for now use "none".

Change-Id: Icee021fc30839eeb02cd3e232ff6be92ed32df52
2012-04-10 10:44:55 -06:00
gholt
1c349a387d Make tests work like they used to.
Change-Id: I081d853a90ad0efce5b7a20b52794ce3348a9617
2012-03-13 15:47:41 +00:00
Florian Hines
9e713e1078 Add instructions for pip install via easy_install.
Add a line item to easy_install pip since we now need to pip install
openstack.nose_plugin. Fixes bug 951019

Change-Id: Ibf4b8bb7feac2b8290db2f4c4665dacf8a881462
2012-03-09 20:49:06 +00:00
Maru Newby
e6ea310751 Add support for venv-based test run with tox.
* Adds tox config
   - based on the config from python-quantumclient and updated for
     test, pep8 and coverage execution as per nova's run_tests.sh.
 * Adds nosetests defaults in setup.cfg
 * Adds runtime dependencies in tools/pip-requires
   - dependencies were gathered by referencing the packages used in
     creation of a Swift All In One.  Versions were determined by
     checking the swift-core/trunk ppa or, failing that, the version
     available in lucid.
 * Adds test dependencies in tools/test-requires
 * Updates swift/common/middleware/formpost.py for pep8 compliance
 * Adds instructions for executing the tests with Tox to the
   developer_guidelines
 * Adds instructions for installing openstack.nose_plugin to
   developer_saio
 * Fixes bug 909177

Change-Id: I5407924d2181e9ab335aaf76bf30c8d40deccbb4
2012-03-07 22:44:34 -08:00
Derek Higgins
40dbefc9f4 Fixing some minor issues in SAIO on Fedora
Change-Id: I2abdcd1a3c7f2a54b200489efe57f8990bd94bbf
2012-03-05 15:30:43 +00:00
Jonathan Gonzalez V
38113e22fc Few fix in the SAIO doc:
- Fix the ubuntu isos urls
 - Removed trailing whitespaces
 - Correct the text that explain how to build the documentation

Change-Id: Ie19604d4586562dcef8c1d46addd14c15db2c442
2012-02-28 12:20:49 -03:00
Pete Zaitcev
6f7f95ffde Add Fedora to SAIO.
In the past we had a document at Wiki, called RHELInstructions,
but it was permanently out of date, and someone deleted it, in
order to stop the confusion.

Differences for Fedora as quite small and it makes sense to maintain
them inside the main document.

This patch also fixes a couple of obvious problems:
 - a missing period
 - unwanted leading space that breaks a paragraph

Still no idea how to remove color from [swift-hash].

Change-Id: I73cda027c9d9bf695cd640a6d55735d82fba0048
2012-02-22 08:19:37 -07:00
Daniele Valeriani
29e703327c ~/swift/trunk doesn't exist after git clone
Change-Id: I0b12a1bd1a119aeedda07fa6f7883ae6318ee4d6
2012-01-03 15:40:35 +00:00
Maru Newby
56645b9cb1 Updates swift docs to use correct release PPA.
Changes swift doc references from defunct ppa repo swift-core/ppa to
new swift-core/release repo.  Fixes lp bug 905608.

Change-Id: I58d85b22d8dbcff43b03eb256ad39e03dd208bfc
2011-12-16 20:22:18 -08:00
Chmouel Boudjnah
74752c8d26 Remove bazaar/lp reference and change it to github
Change-Id: Ic58d3023f41204568bff9a4a77c895937b9456d2
2011-11-01 08:18:51 +01:00
Chmouel Boudjnah
6170e66b42 Use account_autocreate for tempauth.
We don't need to create ourself account in tempauth as we can use
account_autocreate from proxy server.

Change-Id: I6d0a044be361a53faee15d15177a9b5bc95da19e
2011-10-06 11:35:43 -04:00