326 Commits

Author SHA1 Message Date
John Dickinson
3c82151c44 update object versions docs
Change-Id: I00bf2e20c2683d0440ca425461e0ab87197ee720
2012-08-07 16:26:38 -07:00
Tong Li
d8c5c0c897 all in one setup rc.local needs -p for mkdir command
bug fixes for swift 1033089
Swift 1.6.0 All in One setup uses rc.local to create a set of
directories like the following:
   /var/cache/swift /var/cache/swift2 /var/cache/swift3
   /var/cache/swift4
   /var/run/swift
without -p for mkdir command, when some of these directories
already exist, the rc.local fails so that the next set of
command won't execute. This causes swift fail to start because
of /var/run/swift does not exist. Adding -p to both mkdir
command solves this problem.

Change-Id: Ia26c9a857723747008293c0631c9acc1b58eca7e
2012-08-06 09:11:07 -04:00
gholt
a1a4d35362 Logging improvements: handoffs and thread locals
A warning log line is emitted whenever the proxy has to use a handoff
node. Monitoring these warnings can indicate a problem within your
cluster; however, you can disable these log lines by setting the
proxy conf's log_handoffs to false.

While working on this, I also noticed why many proxy log lines did
not have txn_id and client_ip -- subcoroutines. Now the logger thread
locals are copied to the subcoroutines.

Change-Id: Ibac086e1b985f566c068d083620287509de35da8
2012-07-30 00:50:13 +00:00
Florian Hines
0da1b4f82d Create and configure /var/cache/swift in SAIO
Fixes bug #1024477

Change-Id: Id09bbff6bb4a5d032727d3ca53b3276b4769d178
2012-07-23 16:44:06 -05:00
Jenkins
a9887c0e15 Merge "Move swift_auth middleware from keystone to swift." 2012-07-20 17:30:01 +00:00
Vincent Untz
faff4ae769 Forbid substrings based on a regexp in name_filter middleware
In comments from https://review.openstack.org/8798 it was raised that it
might make sense to forbid some substrings in the name_filter
middleware.

There is now a new forbidden_regexp option for the name_filter
middleware to specify which substrings to forbid. The default is
"/\./|/\.\./|/\.$|/\.\.$" (or in a non-regexp language: the /./ and /../
substrings as well as strings ending with /. or /..).

This can be useful for extra paranoia to avoid directory traversals
(bug 1005908), or for more general filtering.

Change-Id: I39bf2de45b9dc7d3ca4d350d24b3f2276e958a62
DocImpact: new forbidden_regexp option for the name_filter middleware
2012-07-19 14:13:47 +02:00
Florian Hines
5f72a8db4a Fix Dispersion report and swift-bench on saio
We're still using saio:11000 in a few spots so a few things
don't work out of the box on the saio. Fixes bug #1024561

Change-Id: I226de54c2785b0d0b681c8d0cc24260adbd3d663
2012-07-13 17:48:37 -05:00
Chmouel Boudjnah
afa4f70024 Move swift_auth middleware from keystone to swift.
- Rename it to keystoneauth for consistenties.
- Implements blueprint keystone-middleware.

Change-Id: I208fecdf3ee991694b4239f065032324d297fd35
2012-07-05 14:36:14 -04:00
Jenkins
501a3e4929 Merge "Remove ambiguity in memcache_servers documentation" 2012-07-03 16:37:24 +00:00
Jenkins
c16b70bb3d Merge "Remove hard coded man page paths." 2012-06-28 15:31:21 +00:00
Vincent Untz
1125368624 Remove ambiguity in memcache_servers documentation
The documentation could be understood like the following:
memcache_servers from memcache.conf is always used if set, even if
memcache_servers in proxy-server.conf is set.

This is clearly not the case, as proxy-server.conf has a higher priority
if it memcache_servers is set there.

Change-Id: I967c7e80796a0e296c5c65bd097df1669d16203e
2012-06-28 16:09:10 +02:00
Jenkins
57008e553a Merge "Patch for Swift Solaris (Illumos) compability." 2012-06-27 16:41:43 +00:00
Dan Prince
69fd05f439 Remove hard coded man page paths.
Fixes LP Bug #1016656.

Change-Id: I22455d27cb82830dabd4fb0677da26915a60eb85
2012-06-22 13:05:53 -04:00
Jenkins
cc73f5cbe0 Merge "Added config options for rate limiting of large object downloads." 2012-06-20 18:12:48 +00:00
Victor Rodionov
13e4de1899 Patch for Swift Solaris (Illumos) compability.
* Add new configuration option log_address.

Change-Id: I636bd4116687629c997b70a0d804b7ed4bc46032
2012-06-19 15:38:56 +04:00
Jenkins
c0e7c38c9f Merge "Make object server's caching more configurable." 2012-06-12 11:01:16 +00:00
Samuel Merritt
28cd9b2da8 Make object server's caching more configurable.
The object server had a constant KEEP_CACHE_SIZE = 5*1024*1024;
unauthenticated GET requests for files smaller than KEEP_CACHE_SIZE
would not evict the file from the kernel's buffer cache after it was
read from disk.

Now that hardcoded constant is a configuration parameter
("keep_cache_size"), and now there is also another parameter called
"keep_cache_private". If set, then both authenticated and
unauthenticated GET requests for small files will not evict the data
from the buffer cache. The default values are 5 MiB and False,
respectively, so the default behavior is the same.

Bonus: the "mb_per_sync" parameter is now documented in the deployment
guide.

Change-Id: I9a11dbe861f4c23535c6aa82a9111a6fe2db2a59
2012-06-07 17:12:11 -07:00
Greg Lange
6987d222de Added config options for rate limiting of large object downloads.
bug 786197

Change-Id: Idd73986ca4550b20dff340df4acd12f3bfc01075
2012-06-07 20:19:31 +00:00
Chmouel Boudjnah
22572b506a Remove swift.common.client from here.
- It has been to its own gerrit project.
- direct_client should follow next.
- Implements blueprint clientbindings.

Change-Id: I3bb50c95eba81302bfec71cb7ce5288b85a41dc0
2012-06-07 16:36:49 +02:00
John Dickinson
d668b27c09 fixed doc table format
Change-Id: I319de933ecfb1e3853e3064656968c36980ce5f5
2012-05-28 13:36:59 -05:00
Jenkins
676c338b7c Merge "Expand recon middleware support" 2012-05-25 05:12:35 +00:00
Michael Barton
7c98e7a625 Move proxy server logging to middleware.
Change-Id: I771c87207d4e1821e32c3424b341d182cc7ea7c0
2012-05-24 21:15:51 -07:00
Florian Hines
ccb6334c17 Expand recon middleware support
Expand recon middleware to include support for account and container
servers in addition to the existing object servers. Also add support
for retrieving recent information from auditors, replicators, and
updaters. In the case of certain checks (such as container auditors)
the stats returned are only for the most recent path processed.

The middleware has also been refactored and should now also handle
errors better in cases where stats are unavailable.

While new check's have been added the output from pre-existing
check's has not changed. This should allow existing 3rd party
utilities such as the Swift ZenPack to continue to function.

Change-Id: Ib9893a77b9b8a2f03179f2a73639bc4a6e264df7
2012-05-24 14:50:00 -05:00
Jenkins
a74cd3b01b Merge "Remove swift3 from here." 2012-05-24 15:30:17 +00:00
Jenkins
86ddaab942 Merge "!! Changed db_preallocation to False" 2012-05-24 15:28:34 +00:00
Chmouel Boudjnah
d02a73f4a9 Remove swift3 from here.
- Reference https://github.com/fujita/swift3 in associated_projects.
- Implements blueprint add-associated-projects-docs.

Change-Id: I48ef4c03449edf6ef4fda1a391228cacac7d2ac6
2012-05-24 10:21:42 +02:00
John Dickinson
1e90b61076 Re-add cname lookup and domain remap middleware
Revert "removed cname lookup middleware"

This reverts commit b47bcf19e41e862ca84d77a7b8843f836e084b6a.

Revert "removed domain remap middleware"

This reverts commit 317cf868bdf66dbc17c02d4ca383efafa5e2f229.

Change-Id: I260498d555c93b28896ace48a6f0e96701cbcc38
2012-05-21 20:02:04 -05:00
gholt
9eb797b099 !! Changed db_preallocation to False
Long explanation, but hopefully answers any questions.

We don't like changing the default behavior of Swift unless there's a
really good reason and, up until now, I've tried doing this with this
new db_preallocation setting.

For clusters with dedicated account/container servers that usually
have fewer disks overall but SSD for speed, having db_preallocation
on will gobble up disk space quite quickly and the fragmentation it's
designed to fight isn't that big a speed impact to SSDs anyway.

For clusters with account/container servers spread across all servers
along with object servers usually having standard disks for cost,
having db_preallocation off will cause very fragmented database files
impacting speed, sometimes dramatically.

Weighing these two negatives, it seems the second is the lesser evil.
The first can cause disks to fill up and disable the cluster. The
second will cause performance degradation, but the cluster will still
function.

Furthermore, if just one piece of code that touches all databases
runs with db_preallocation on, it's effectively on for the whole
cluster. We discovered this most recently when we finally configured
everything within the Swift codebase to have db_preallocation off,
only to find out Slogging didn't know about the new setting and so
ran with it on and starting filling up SSDs.

So that's why I'm proposing this change to the default behavior.

We will definitely need to post a prominent notice of this change
with the next release.

Change-Id: I48a43439264cff5d03c14ec8787f718ee44e78ea
2012-05-22 00:30:47 +00:00
gholt
1c3b75c291 Reverted the pulling out of various middleware:
RateLimit
StaticWeb
TempURL/FormPOST

Change-Id: I988e93e6f4aacb817a2e354d43a04e47516fdf88
2012-05-16 21:25:10 +00:00
Darrell Bishop
3d3ed34f44 Adding StatsD logging to Swift.
Documentation, including a list of metrics reported and their semantics,
is in the Admin Guide in a new section, "Reporting Metrics to StatsD".
An optional "metric prefix" may be configured which will be prepended to
every metric name sent to StatsD.

Here is the rationale for doing a deep integration like this versus only
sending metrics to StatsD in middleware.  It's the only way to report
some internal activities of Swift in a real-time manner. So to have one
way of reporting to StatsD and one place/style of configuration, even
some things (like, say, timing of PUT requests into the proxy-server)
which could be logged via middleware are consistently logged the same
way (deep integration via the logger delegate methods).

When log_statsd_host is configured, get_logger() injects a
swift.common.utils.StatsdClient object into the logger as
logger.statsd_client.  Then a set of delegate methods on LogAdapter
either pass through to the StatsdClient object or become no-ops. This
allows StatsD logging to look like:
    self.logger.increment('some.metric.here')
and do the right thing in all cases and with no messy conditional logic.

I wanted to use the pystatsd module for the StatsD client, but the
version on PyPi is lagging the git repo (and is missing both the prefix
functionality and timing_since() method).  So I wrote my
swift.common.utils.StatsdClient.  The interface is the same as
pystatsd.Client, but the code was written from scratch.  It's pretty
simple, and the tests I added cover it.  This also frees Swift from an
optional dependency on the pystatsd module, making this feature easier
to enable.

There's test coverage for the new code and all existing tests continue
to pass.

Refactored out _one_audit_pass() method in swift/account/auditor.py and
swift/container/auditor.py.

Fixed some misc. PEP8 violations.

Misc test cleanups and refactorings (particularly the way "fake logging"
is handled).

Change-Id: Ie968a9ae8771f59ee7591e2ae11999c44bfe33b2
2012-05-11 15:25:38 -07:00
Jenkins
e51cdbc8ba Merge "Implement unit_test config to disable syslog." 2012-05-10 15:47:54 +00:00
Samuel Merritt
bb509dd863 As-unique-as-possible partition replica placement.
This commit introduces a new algorithm for assigning partition
replicas to devices. Basically, the ring builder organizes the devices
into tiers (first zone, then IP/port, then device ID). When placing a
replica, the ring builder looks for the emptiest device (biggest
parts_wanted) in the furthest-away tier.

In the case where zone-count >= replica-count, the new algorithm will
give the same results as the one it replaces. Thus, no migration is
needed.

In the case where zone-count < replica-count, the new algorithm
behaves differently from the old algorithm. The new algorithm will
distribute things evenly at each tier so that the replication is as
high-quality as possible, given the circumstances. The old algorithm
would just crash, so again, no migration is needed.

Handoffs have also been updated to use the new algorithm. When
generating handoff nodes, first the ring looks for nodes in other
zones, then other ips/ports, then any other drive. The first handoff
nodes (the ones in other zones) will be the same as before; this
commit just extends the list of handoff nodes.

The proxy server and replicators have been altered to avoid looking at
the ring's replica count directly. Previously, with a replica count of
C, RingData.get_nodes() and RingData.get_part_nodes() would return
lists of length C, so some other code used the replica count when it
needed the number of nodes. If two of a partition's replicas are on
the same device (e.g. with 3 replicas, 2 devices), then that
assumption is no longer true. Fortunately, all the proxy server and
replicators really needed was the number of nodes returned, which they
already had. (Bonus: now the only code that mentions replica_count
directly is in the ring and the ring builder.)

Change-Id: Iba2929edfc6ece89791890d0635d4763d821a3aa
2012-05-09 15:56:06 -07:00
John Dickinson
b47bcf19e4 removed cname lookup middleware
The code has moved to https://github.com/notmyname/swift-cnamelookup.

For current users of cname lookup, this will require installing the new
package and changing the "use" line of the cname lookup conf section's
to:

[filter:cname_lookup]
use = egg:swift_cnamelookup#swift_cnamelookup

And then 'swift-init proxy reload'.

Change-Id: If622486ddb04a53251244c9840aa3cfe72168fc5
2012-05-09 14:00:51 -05:00
gholt
3f00c1a630 Pulled out Rate Limit middleware
Rate Limit middleware is now at
http://dpgoetz.github.com/swift-ratelimit/

For current users of Rate Limit, this will require installing the new
package and changing the "use" line of the ratelimit conf section to:

[filter:ratelimit]
use = egg:swiftratelimit#middleware

And then 'swift-init proxy reload'.

Change-Id: I2ab774e9cee9fba4103c1be4bea6d52d1adb29f7
2012-05-09 18:44:45 +00:00
John Dickinson
7dfbd785b0 removed domain remap middleware
The code has moved to https://github.com/notmyname/swift-domainremap.

For current users of domain remap, this will require installing the new
package and changing the "use" line of the domain remap conf section's
to:

[filter:domain_remap]
use = egg:swift_domainremap#swift_domainremap

And then 'swift-init proxy reload'.

Change-Id: I710caf9b991f9d37df36b826ae4338086d0ec36d
2012-05-08 21:30:35 -05:00
gholt
c0532a6ef2 Pulled out TempURL/FormPOST
TempURL/FormPOST is now at http://gholt.github.com/swift-tempurl/

For current users of TempURL/FormPOST, this will require installing the new
package and changing the "use" line of the tempurl and formpost conf section's
to:

[filter:tempurl]
use = egg:swifttempurl#tempurl

[filter:formpost]
use = egg:swifttempurl#formpost

And then 'swift-init proxy reload'.

Change-Id: I5bddf7f9e09ee07815530a41c46ff901fc21b447
2012-05-08 20:20:29 +00:00
Jenkins
8d2e7bd112 Merge "Pulled StaticWeb out to separate project" 2012-05-08 18:43:38 +00:00
gholt
7dde909621 Pulled StaticWeb out to separate project
StaticWeb is now at http://gholt.github.com/swift-staticweb/

For current users of StaticWeb, this will require installing the new
package and changing the "use" line of the staticweb filter conf
section to:

use = egg:swiftstaticweb#middleware

And then 'swift-init proxy reload'.

Change-Id: Iab32adb5927698a667c5c6d6a572c44ca23414eb
2012-05-05 00:40:33 +00:00
Greg Lange
8d2fe89a7d Added an internal client.
Refactored object expirer to use this client.

Change-Id: Ibeca6dba873f8b4a558ecf3ba6e8d23d36f545b0
2012-05-04 18:07:54 +00:00
gholt
f7c0f91da6 Added associated_projects.rst
In the interest of keeping the core Swift code repository less
complex, but still offering a quick way to find associated projects
that enhance or use Swift, I've added this new Associated Projects
page prominently to the Swift documentation.

This will allow much less barrier to entry on enhancing Swift as
other projects can work independently and will only need to wait on
the core Swift project for approval of minimal tweaks to the core
Swift code base.

It will also allow an easy central place to find cool associated
projects that otherwise might go unnoticed or even duplicated.

The plan is to move non-essential projects that are currently
embedded in the Swift repository out into their own projects with
links to them on this new page. This would include items such as
(just what I can think of right now): bin/swift command line tool and
clients, swift-bench, swift-dispersion, TempURL, FormPost, StaticWeb,
Rate Limiting, Swift3, Domain Remap, and CNAME Lookup.

After all that is done, those projects will be able to move forward
much more quickly and new developers for Swift itself will have much
less to learn and get confused about.

Change-Id: Ib8447d8bd480f0a3d8f0413137ccdba73a11dd91
2012-05-04 03:36:56 +00:00
Dan Dillinger
f7fdb9cf12 Implement unit_test config to disable syslog.
bug 701248
bug 819303

This change makes syslog optional, by providing a unit test
config with the ability to cause test/unit/__init__.py to
replace SysLogHandler with a fake logging class. The
default behavior is unchanged.

FakeLogger now inherits directly from Handler and mocks out
its API - this was a backward-compatibility issue.

Change-Id: I653dec148c4f6cf81759de03964c6a3290c1a290
2012-05-03 14:25:46 -04:00
gholt
5f18d987a0 Fixed tabs in the multi-node doc.
Change-Id: Idf2ea227b0078026eb2693ff7c0a24ec926bb6f3
2012-04-26 20:29:08 +00:00
John Dickinson
3053c53ef7 added docs and warnings about using SSL
Change-Id: I766b2da0a3769869b5b6c5565106e9032fea8c3e
2012-04-23 19:29:26 -05:00
Jenkins
ebfa3dd2c1 Merge "Adds links to docs.openstack.org" 2012-04-23 16:05:08 +00:00
Jenkins
069beddde4 Merge "Fixed grammar and improve docs." 2012-04-23 15:49:45 +00:00
annegentle
a6a810f233 Adds links to docs.openstack.org
In the side bar, links to past versions and docs.openstack.org

On the index.rst, links to wiki and docs.openstack.org

Change-Id: Icf33c6f396e1ab016fd86a56e61df3e063a1bae2
2012-04-23 10:48:24 -05:00
Jenkins
722127d6c0 Merge "updated docs for domain remap and cname lookup middleware" 2012-04-18 21:23:52 +00:00
Jenkins
dfbe362d90 Merge "Appending some new information to proxy, account and container config manpages. Adding object-expirer files and swift-orphans as well. Also making some minor modification on the other files to stop lithian from complaining during package creation checks. Adding name to Authors file." 2012-04-12 15:40:25 +00:00
Marcelo Martins
6a9d5f921d Appending some new information to proxy, account and container config manpages. Adding object-expirer files and swift-orphans as well. Also making some minor modification on the other files to stop lithian from complaining during package creation checks. Adding name to Authors file.
Change-Id: Id68231ced070b27d1f7fb5a230d345694632e7dd
2012-04-10 14:52:52 -05:00
Paul McMillan
92fbf44d10 Fixed grammar and improve docs.
Corrected its/it's mistakes, harmonized line wrapping within some docs
and clarified doc wording in several places.

Change-Id: Ib9ac6d5e859f770a702e1fad6de8d4abe0390b47
2012-04-10 12:27:14 -07:00