141 Commits

Author SHA1 Message Date
David Goetz
40cbff9fd3 proxy to be able to deny request to invalid hostnames
Change-Id: I974f729da60e5ab9453daf9e52466b3e1af5c69b
2012-04-12 14:18:40 -07:00
Julien Danjou
9a423d0b78 Allow to specify auth_version in swift-dispersion tools
Change-Id: I080d531471d8ea57c69a918d4f6930441f1e69f6
Signed-off-by: Julien Danjou <julien.danjou@enovance.com>
2012-04-02 16:42:05 +02:00
gholt
0becfab629 Added option to disable SQLite db preallocation
Added option to disable SQLite db preallocation. This can be very
useful on pure ssd account/container servers where the extra space is
worth more than the lesser fragmentation.

Change-Id: I8fbb028a9b6143775b25b343e97896497a8b63a9
2012-03-28 21:45:18 +00:00
John Dickinson
156f27c921 object versioning
Object versioning in swift is implemented by setting a flag on the container
to tell swift to version all objects in the container. The flag is the
``X-Versions-Location`` header on the container, and its value is the
container where the versions are stored.

When data is ``PUT`` into a versioned container (a container with the
versioning flag turned on), the existing data in the file is redirected to a
new object and the data in the ``PUT`` request is saved as the data for the
versioned object. The new object name (for the previous version) is
``<versions_container>/<object_name>/<timestamp>``, where the timestamp is
generated by converting the ``Last-Modified`` header value of the current
version to a unix timestamp.

A ``GET`` to a versioned object will return the current version of the object
without having to do any request redirects or metadata lookups.

Change-Id: I4fcd723145e02bbb2ec1d3ad356713f5dea43b8b
2012-03-28 15:48:43 -05:00
gholt
ac3cc680de Add an optional delay to account reaping.
Normally, the reaper begins deleting account information for deleted
accounts immediately. With this patch you can set it to delay its
work. You set the delay_reaping value in the [account-reaper] section
of the account-server.conf. The value is in seconds; 2592000 = 30
days, for example.

Unfortunately, there are currently zero tests for the account-reaper.
This also needs fixing, but I thought I'd submit this delay patch
alone for consideration.

Change-Id: Ic077df9cdd95c5d3f8949dd3bbe9893cf24c6623
2012-03-21 15:26:07 +00:00
gholt
2c6de2ae52 Added optional max_containers_per_account restr...
Added optional max_containers_per_account restriction. If set to a
positive value and if a client tries to perform a container PUT when
at or above the max_containers_per_acount cap, a 403 Forbidden will
be returned with an explanatory message.

This only restricts the proxy server, not any of the background
processes that might need to create containers (replication, for
instance). Also, the container count is cached for the proxy's
recheck_account_existence number of seconds. For these reasons, a
given account could exceed this cap before the 403 Forbidden
responses kick in and therefore this feature should be considered a
"soft" limit.

You may also add accounts to the proxy's max_containers_whitelist
setting to have accounts that ignore this cap.

Change-Id: I74e8fb152de5e78d070ed30006ad4e53f82c8376
2012-03-15 19:41:31 +00:00
Eamonn O'Toole
cf1aa3c309 Adds name_check filter
Bug 926048.

Filter checks path for user-defined forbidden characters, and for
user-defined maximum length.

Includes changes to reflect gholt's latest comments to Patch Set 4
Also includes a change to a unit-test, renames another unit-test,
and removes one superfluous unit-test.

Added section to the example proxy config

Fixed-up unit test pep8 warnings

Changed error response code to 400 (Bad Request)

Change-Id: Iace719d6a3d00fb3dda1b9d0bc185b8c4cbc00ca
2012-03-08 17:23:58 +00:00
gholt
7141c52bd6 Remove old stats tools.
https://blueprints.launchpad.net/swift/+spec/remove-old-swift-stats-tools

Change-Id: Ibb57b3d71c7f18060d2d48a1d3754a7f1444624d
2012-03-06 17:57:16 +00:00
gholt
52ba08d67d Improvements to database replication.
Note: I'd like to get this released as soon as possible as it is a
data durability issue.

1) Orders nodes so that none get starved (see code and footnote).

2) New max_diffs setting that caps how long the replicator will spend
   trying to sync a given database per pass so the other databases
   don't get starved.

3) Replaces run_pause with the more standard "interval", which means
   the replicator won't pause unless it takes less than the interval
   set.

Change-Id: I986742229e65031df88f5251ca61746b7c8d2bde
2012-03-06 06:08:18 +00:00
Florian Hines
5e4127ae2a Add json output option to swift-dispersion-report
Add's the configuration file option "dump_json" or command line
options [-j|--dump-json] to have swift-dispersion-report output
the report in json format. This allows the dispersion report to
be more easily consumed elsewhere.

There's also a few pep8 fixes and removal of unused imports.

Change-Id: I2374311ccbef43e6bbae24665c9584e60f3da173
2012-02-29 04:24:54 +00:00
gholt
7fc1721d7d TempURL and FormPost Middleware
Change-Id: I8d2ce2abdfe3a44605c9441ad7b1abc6c77e282d
2012-01-10 21:57:52 +00:00
gholt
338be6a681 Added memcache.conf option
Change-Id: Idfcb97421a5c1476b776acef94c7b997dab55aa9
2012-01-10 02:01:47 +00:00
gholt
872420efdb Expiring Objects Support
Please see the doc/source/overview_expiring_objects.rst for
more detail.

Change-Id: I4ab49e731248cf62ce10001016e0c819cc531738
2011-11-01 15:49:00 +00:00
Florian Hines
bb8c4eab41 Add documentation for Swift Recon.
Change-Id: I37f4fb624bdc5b8bbf2e691d29aa6b15cd648aa8
2011-10-21 00:17:10 +00:00
gholt
8c24a70139 Merged from trunk 2011-06-30 21:52:09 +00:00
gholt
552fa38ed1 Doc updates 2011-06-22 15:05:23 +00:00
John Dickinson
b3a451c832 merged with trunk 2011-06-17 16:47:39 -05:00
gholt
1dca388dec Merge from trunk 2011-06-16 22:05:03 +00:00
John Dickinson
5490c514fe removed slogging references from docs 2011-06-15 10:06:31 -05:00
gholt
d2230e853d Merged from trunk 2011-06-14 22:20:23 +00:00
John Dickinson
5949217ce1 removed basic slogging code 2011-06-13 15:57:32 -05:00
gholt
024df7c0f2 Merged from trunk 2011-06-13 20:51:06 +00:00
gholt
156b6e0c92 Added account_autocreate mode where authorized requests to accounts that don't yet exist within Swift will cause those accounts to be automatically created. Also did a small refactor surrounding swift.common.utils.TRUE_VALUES. 2011-06-10 22:27:05 +00:00
David Goetz
a18acc554b merging to trunk 2011-06-10 09:47:13 -07:00
gholt
56bafed131 Merged from trunk 2011-06-09 21:30:14 +00:00
David Goetz
68e5719a81 fixing case thing and adding docs 2011-06-08 08:55:14 -07:00
gholt
41fcf63241 Changed post_as_copy to object_post_as_copy 2011-06-08 04:29:24 +00:00
gholt
7c9e542c02 Implemented object POST as COPY 2011-06-08 04:19:34 +00:00
gholt
145651e04f Updated container-server.conf-sample 2011-06-06 16:12:54 +00:00
gholt
c3cef9207b Adding account_autocreate mode and refactoring TRUE_VALUES 2011-06-05 23:22:35 +00:00
gholt
7ac6baf6d1 container-sync: Support HTTP proxy. 2011-06-03 02:28:21 +00:00
gholt
b4221114c5 Merged with deswauth 2011-06-03 00:11:32 +00:00
gholt
ce710a54a5 Merged with trunk 2011-06-01 20:43:10 +00:00
gholt
f68b6354e2 Renaming TestAuth to TempAuth because nose hates anything with the word test in it. 2011-05-26 02:24:12 +00:00
gholt
b24b063552 Created testauth WSGI middleware to replace Swauth as development auth service. 2011-05-26 01:19:03 +00:00
David Goetz
bc894f9efb update with trunk 2011-05-12 14:24:51 -07:00
gholt
d0d98ba96e Updated to use standard shebang and explicitly flush output; added deprecation warning to stats.conf 2011-05-12 00:14:02 +00:00
David Goetz
84981953b2 adding processable flag and fix unicode thing 2011-05-11 17:01:24 -07:00
David Goetz
a1321fb17d stats conf changes 2011-05-10 15:53:21 -07:00
David Goetz
e35f8bf5a7 replace file and unit tests are working 2011-05-09 14:18:20 -07:00
Jay Payne
6ea412d90b Adding section for container stats 2011-05-02 23:52:20 +00:00
gholt
0546c8d15f Merged from trunk 2011-04-15 23:11:34 +00:00
gholt
6c13001244 Rename swift-stats-* to swift-dispersion-* to avoid confusion with log stats stuff 2011-03-31 22:32:41 +00:00
gholt
40505e892d Added logging 2011-03-25 08:33:46 +00:00
gholt
a464e4aa6b Updated sample conf 2011-03-24 23:17:38 +00:00
gholt
8476480394 Merged from trunk 2011-03-24 21:35:53 +00:00
John Dickinson
0c5aacb424 added default support for content-disposition and allows x-object-manifest to be manipulated like any other object metadata header 2011-03-24 13:03:49 -05:00
gholt
d872d94411 Update from feedback; docs 2011-03-24 03:37:07 +00:00
John Dickinson
88ad83767b objects can now have arbitrary headers set in metadata that will be served back when they are fetched 2011-03-22 20:05:44 -05:00
gholt
6318b996b5 Merged from trunk 2011-03-18 15:07:03 +00:00