2392 Commits

Author SHA1 Message Date
Jenkins
44e3915564 Merge "Change in swift-drive-audit handling log rotation." 2013-02-21 02:41:37 +00:00
Chuck Thier
e88ff34685 Cleanup of file permissions
Mostly removed execute on non-executable files

Change-Id: Ibfbe7e0cf0fbeabef602d70b20f75e1dd3bdf9c9
2013-02-15 11:06:11 -06:00
Jenkins
b66232a997 Merge "Swift MemcacheRing (set) interface is incompatible fixes" 2013-02-14 20:30:22 +00:00
Jenkins
ce49777892 Merge "fix large objects with unicode" 2013-02-14 18:37:17 +00:00
Jenkins
70385d3af9 Merge "log message in exception level when path encoding errors happen." 2013-02-14 11:20:21 +00:00
David Goetz
3b341e375f fix large objects with unicode
Change-Id: I2a088e6cfe354d18cb2107a42bc626c2e7822e42
2013-02-13 12:05:25 -08:00
Tong Li
2267b030bf Swift MemcacheRing (set) interface is incompatible fixes
This patch fixes the Swift MemcacheRing set and set_multi
interface incompatible problem with python memcache. The fix
added two extra named parameters to both set and set_multi
method. When only time or timeout parameter is present, then one
of the value will be used. When both time and timeout are present,
the time parameter will be used.

Named parameter min_compress_len is added for pure compatibility
purposes. The current implementation ignores this parameter.

To make swift memcached methods all consistent cross the board,
method incr and decr have also been changed to include a new
named parameter time.

In future OpenStack releases, the named parameter timeout will be
removed, keep the named parameter timeout around for now is
to make sure that mismatched releases between client and server
will still work.

From now on, when a call is made to set, set_multi, decr, incr
by using timeout parametner, a warning message will be logged to
indicate the deprecation of the parameter.

Fixes: bug #1095730
Change-Id: I07af784a54d7d79395fc3265e74145f92f38a893
2013-02-13 13:56:04 -05:00
Jenkins
23f33b2069 Merge "Make statsd sample rate behave better." 2013-02-13 08:19:46 +00:00
Kun Huang
f4a09ebf47 log message in exception level when path encoding errors happen.
Original code is kind of except-pass. This will ignore the error
occurs in process of encoding path. And if such error happened
without any information in logs, developer have to debug in a
long distance from error stack. That's wasting time, so there
should be a logging message here.

Fixes: bug #1123048
Change-Id: I7941b524e936241fb8d62fad5ad90de3daffe5e7
2013-02-13 10:01:33 +08:00
Joe Gordon
45f0502b52 Fix spelling mistakes
git ls-files | misspellings -f -
Source: https://github.com/lyda/misspell-check

Change-Id: I4132e6a276e44e2a8985238358533d315ee8d9c4
2013-02-12 16:39:40 -08:00
jola-mirecka
902b66d3ae Change in swift-drive-audit handling log rotation.
Change supports kern.log rotation in order to avoid loss
of significant information.

There is a year change functionality added as kern.log
does not keep record of year.

There is also backwards function added which allows
reading logs from the back to the front, speeding up the
execution along with the unit test for it

Fixes Bug 1080682

Change-Id: I93436c405aff5625396514000cab774b66022dd0
2013-02-12 13:34:49 +00:00
Jenkins
af3bd46679 Merge "Add generated keystone documentation to misc.rst." 2013-02-12 12:03:21 +00:00
Jenkins
e4e55588fb Merge "Replaces Copyright statements for LLC with Foundation, removes date." 2013-02-12 06:25:45 +00:00
Jenkins
3df9229dae Merge "Use a doubled proxy-logging instead of each middleware handling it differently (if at all)" 2013-02-12 03:47:13 +00:00
annegentle
72428434f7 Replaces Copyright statements for LLC with Foundation, removes date.
Replaced GA code for cross-domain tracking.

Patchset addresses reviewer's comments
and follows new guidance from Foundation:
http://wiki.openstack.org/Documentation/Copyright

Adds current year to each Sphinx-built page.

Addresses only the docs copyright attribution, not code files.

Change-Id: Ib90fd1c92c8fafce2db821bc2b17cef1377cfc1e
2013-02-11 16:32:33 -06:00
Chmouel Boudjnah
1b86380e2e Add generated keystone documentation to misc.rst.
- Change a bit the formatting of the documention as well.
- Fix WARNING: Title underline too short. in misc.rst.

Change-Id: I2f4e36bcb5e01e984f0af0152bc5b3b9f7e942ce
2013-02-11 21:44:53 +01:00
Jenkins
89d4a51359 Merge "Made internal client handle failed requests better" 2013-02-11 19:47:02 +00:00
David Goetz
a622349eda Use a doubled proxy-logging instead of each middleware handling it
differently (if at all)

Adding a swift.source to wsgi pre_auth funcs and all middleware that makes
subrequests to proxy server.

NOTE: This change will result in a change in the number of proxy logs made for
staticweb, formpost, tempurl, and any other middleware that performs sub
requests (including swauth and SOS).

Please see docs for details.

DocImpact

Change-Id: I80cf2806add1c3d34054147e2515944be340455b
2013-02-11 09:22:25 -08:00
Jenkins
2dca894aae Merge "Fixed some swift-recon typos" 2013-02-09 22:32:35 +00:00
David Hadas
83d89228a9 minor development doc changes
Reviewed development saio and guidelines while installing
a new dev and test node.

Change-Id: I59238a4d43fa08a89be14f43681ad4693f1261dd
2013-02-09 11:11:05 +02:00
Jenkins
c0e8ad609b Merge "Allow change the endpoint_type when use swift-dispersion tools" 2013-02-08 23:55:03 +00:00
gholt
0640a61776 Fixed some swift-recon typos
Change-Id: Ibe4b59a90b82742b70b00353d22aedd972bbeb59
2013-02-08 20:19:48 +00:00
Jenkins
f7ded57cfc Merge "Deprecate is_admin feature." 2013-02-08 19:04:52 +00:00
Jenkins
9de643e453 Merge "Change the flag set to specify bulk delete and expand archives from a HTTP header to a query parameter." 2013-02-08 18:18:29 +00:00
Jenkins
f6b224e2da Merge "Fixed bug in domain_remap and cname_lookup middleware" 2013-02-07 22:58:15 +00:00
David Goetz
08c017418b Change the flag set to specify bulk delete and expand archives from a HTTP
header to a query parameter.

This is needed because query parameters show up in proxy logs and headers do
not. With this change it will be easy to determine from any log line that gets
created from the original request (of which there is currently none) that the
request was a bulk action.

Note: This is not backwards compatible with the previous method of setting a
header. Because the bulk middleware has not been included in an openstack swift
release this should be fine.

Change-Id: I0297fa2de9e491bf0b8c430c0781e2e12316ed4b
2013-02-07 11:11:32 -08:00
Jenkins
9f14161455 Merge "Fixes 'not in' operator usage" 2013-02-07 00:47:33 +00:00
Jenkins
946c7a5d41 Merge "Basic container quotas" 2013-02-06 02:25:55 +00:00
Michael Barton
24ef12027c Basic container quotas
Add a new middleware implementing some basic container quotas.

Quotas are subject to several limitations: eventual consistency, the timeliness
of the cached container_info (60 second ttl by default), and it’s unable to
reject chunked transfer uploads that exceed the quota (though once the quota
is exceeded, new chunked transfers will be refused).

However, they get most of the way to container quotas fairly inexpensively.

Quotas are set by adding meta values to the container, and are validated when
set:

  X-Container-Meta-Quota-Bytes: Maximum size of the container, in bytes.
  X-Container-Meta-Quota-Count: Maximum object count of the container.

DocImpact

Change-Id: I77cfbf6dc231a2e522bd67328e4c082424a93eee
2013-02-05 06:03:38 -08:00
Greg Lange
07d85d3ee8 Made internal client handle failed requests better
In two places, the internal client was not doing what it should with failed requests.

Change-Id: I49e566108fe920d85ca60fe76d24debe8ec7ffad
2013-02-04 19:59:38 +00:00
Darrell Bishop
ec084de189 Optimize the ring builder's _reassign_parts() method.
Another ring builder optimization.  Profiling revealed hotspots in many
calls to min() and list.sort() in _reassign_parts().  That method didn't
get exercised in my last optimization pass because that pass targeted a
rebalance where nothing really moved around.

This time, I wrote a script which created a fresh ring, added a bunch of
devices, did the initial balance, deleted some devices, balanced, and
added some more back in.

Results from homebrew Python 2.7.3 on OS X 10.8.2 Macbook Pro
(bare-metal):

 BEFORE:
 Using part-power = 18, adding 600 devices, removing 100, then adding 300 more...
 NOT Profiling to 'initial_balance.prof'
   wall-time delta: 131.33s
 NOT Profiling to 'deleting_200_rebalance.prof'
   wall-time delta: 25.67s
 NOT Profiling to 'first_rebalance.prof'
   wall-time delta: 62.00s

 AFTER:
 Using part-power = 18, adding 600 devices, removing 100, then adding 300 more...
 NOT Profiling to 'initial_balance.prof'
   wall-time delta: 28.04s
 NOT Profiling to 'deleting_200_rebalance.prof'
   wall-time delta: 9.35s
 NOT Profiling to 'first_rebalance.prof'
   wall-time delta: 16.41s

The driver script I used is available here:
https://gist.github.com/adb982aec6f0709f1273

Change-Id: I17e270acb12b5e4d4bbb1e34d8867dea90678961
2013-02-01 22:33:38 -08:00
Jenkins
62e71a2b1f Merge "Deterministic, repeatable serialization for rings." 2013-02-02 02:23:22 +00:00
Zhongyue Luo
c3d4a63021 Fixes 'not in' operator usage
Fixes bug 1110150

Change-Id: I6c48553564b1809a7ae5de9f778d93dccbe157ae
2013-02-02 09:55:33 +08:00
Jenkins
65baec39d2 Merge "Allow rebalance to take a seed." 2013-02-01 04:04:07 +00:00
Samuel Merritt
156cdc8edf Deterministic, repeatable serialization for rings.
The (account|container|object).ring.gz files contain, among other
things, a JSON-encoded dictionary. This change simply makes the JSON
serializer sort the keys of that dictionary so that two
Python-identical rings will result in two bytewise-identical ring
files. Also, to get repeatable compression, we lock down the timestamp
in the gzip output stream to a fixed value. (There's a timestamp value
in a gzip stream header; by default, gzip.GzipFile sticks time.time()
in there.)

This only works on Python 2.7; on 2.6, the 'mtime' argument to
gzip.GzipFile() is unsupported. Don't worry, serialization still works
on 2.6. It just doesn't always produce the same bytes for the same
ring.

Change-Id: Ide446413d0aeb78536883933fd0caf440b8f54ad
2013-01-31 16:55:10 -08:00
Jenkins
c0d497c421 Merge "Cleanup based on pyflakes." 2013-01-31 23:31:46 +00:00
Monty Taylor
abe70e8323 Cleanup based on pyflakes.
pyflakes itself can't be used in any automated gating way, because there are
two sets of false errors it raises. However, as an exercise, cleaning up the
'valid' ones uncovered three actual bugs. The other changes (mostly unused
variables) are included here for fun.

Command run: pyflakes swift | grep -v "undefined name '_'"

Change-Id: I18696bf047dedad1a9fdbde3463e214fba95f7c6
2013-02-01 07:50:17 +11:00
gholt
85529531d6 Remove tempauth allowed_sync_hosts conf option
Seems we missed these references when committing
357b12dc2ba7b19c66196a573ccb2489d2104b93

DocImpact

Change-Id: Ia226ce1d63e52769bc067d50ec4704cea4e11c5c
2013-01-31 18:30:10 +00:00
Mehdi Abaakouk
a1395ec672 Allow change the endpoint_type when use swift-dispersion tools
Fixes bug 1102319
DocImpact

Change-Id: I8fb0417ab9468e97ed01a6cb1e262630905e7f29
2013-01-31 16:10:37 +01:00
Jenkins
0dfd705b98 Merge "Added fallocate_reserve option" 2013-01-30 23:45:36 +00:00
Christopher MacGown
e189723fec Allow rebalance to take a seed.
Passing a seed into rebalance makes the rebalance deterministic
which allows us to generate identical rings across disparate
nodes without having to copy the ring files around.

Change-Id: Ie5ae46ac030e61284bc501fdef9d77eeb5243afd
2013-01-29 17:08:20 -08:00
Jenkins
c9b24df5d6 Merge "Add wsgify and split_path utilities to swob" 2013-01-29 23:28:25 +00:00
Jenkins
5c93cb5766 Merge "Fix keystoneauth with OPTIONS headers (eg: CORS)" 2013-01-29 23:26:01 +00:00
gholt
87a42ab9ca Added fallocate_reserve option
Some systems behave badly when they completely run out of space. To
alleviate this problem, you can set the fallocate_reserve conf value
to a number of bytes to "reserve" on each disk. When the disk free
space falls at or below this amount, fallocate calls will fail, even
if the underlying OS fallocate call would succeed. For example, a
fallocate_reserve of 5368709120 (5G) would make all fallocate calls
fail, even for zero-byte files, when the disk free space falls under
5G.

The default fallocate_reserve is 0, meaning "no reserve", and so the
software behaves exactly as it always has unless you set this conf
value to something non-zero.

Also fixed ring builder's search_devs doc bugs.

Related: To get rsync to do the same, see
https://github.com/rackspace/cloudfiles-rsync
Specifically, see this patch:
https://github.com/rackspace/cloudfiles-rsync/blob/master/debian/patches/limit-fs-fullness.diff

DocImpact

Change-Id: I8db176ae0ca5b41c9bcfeb7cb8abb31c2e614527
2013-01-29 20:07:26 +00:00
Jenkins
988b1bac40 Merge "Handle access_log_headers in proxy_logging" 2013-01-28 18:55:36 +00:00
Motonobu Ichimura
f28914c702 Handle access_log_headers in proxy_logging
* in etc/proxy-server.conf-sample, there is an access_log_headers directive in proxy_logging middleware.
   but current code doesn't handle it. This patch enables proxy_logging to handle access_log_headers.

Change-Id: I03337ccc4f48625af3f578ed543da992d3eec2bd
2013-01-28 10:20:31 +09:00
Florian Hines
00dbad0825 Add optional locking to swift-ring-builder
If invoked as 'swift-ring-builder-safe' the directory containing the builder
file provided will be locked (via lock_parent_directory()). This provides a
small safe guard against multiple instances of the swift-ring-builder (or
other utilities that observe this lock) from attempting to write to or read
the builder/ring files while operations are in progress.

This is particularly useful in environments where ring management has been
automated (via Chef or custom solutions) but the operator still occasionally
needs to manually interact with the ring.

DocImpact

Change-Id: Ia362744a8151a91bfb586d01da582906726852e6
2013-01-25 08:00:33 -08:00
Michael Barton
c45e435d1f Add wsgify and split_path utilities to swob
And refactor some of the code to use them.

Remove unused imports.

Change-Id: Ica479c10247fa85c740bb99cf7d1db7fbb1b2c80
2013-01-25 00:38:32 -08:00
David Goetz
2f663ff9a0 Bulk Requests: auto extract archive and bulk delete middleware.
Fix small problem in ratelimiting middleware.

DocImpact

Change-Id: Ide3e0b9f4887626c30cae0b97eb7e2237b1df3ed
2013-01-24 12:34:56 -08:00
Chmouel Boudjnah
0dc32d31c5 Fix keystoneauth with OPTIONS headers (eg: CORS)
- Fixes bug 1101143.

Change-Id: I2a82269e4aed2c8926e0aa736c38120784e25de2
2013-01-24 12:36:35 +01:00