Commit Graph

99 Commits

Author SHA1 Message Date
Christian Schwede
28c75db0e7 Account quotas
Add a new middleware implementing account quotas.

This middleware blocks write requests (PUT, POST) if a given quota (in bytes)
is exceeded while DELETE requests are still allowed.

Quotas are stored in the x-account-meta-quota-bytes metadata entry.
Write requests to this metadata setting are only allowed for resellers.

Change-Id: I57fd7c6209f34cc79d4bab72d500d43ba2a62083
2013-03-08 14:31:35 +01:00
Sergey Lukjanov
7d5095c122 Support listing endpoints for an object.
Implements blueprint list-endpoints.

DocImpact: new middleware list_endpoints.

Change-Id: I0c4911ff726abd4cb8ce2b6245c99786ad46b410
2013-03-07 01:38:21 +04:00
David Goetz
5d73da158b Static Large Object Support
DocImpact

Change-Id: I7edaa5e44208ab451f7f7566b64bb571b8eea1f9
2013-03-01 16:46:10 -08:00
Chuck Thier
5d2b70b624 Updating classifiers status to stable and clean up
Change-Id: I72943a9575b873cef1bf193f197b390b544ed6bf
2013-02-27 15:08:46 -06:00
Michael Barton
24ef12027c Basic container quotas
Add a new middleware implementing some basic container quotas.

Quotas are subject to several limitations: eventual consistency, the timeliness
of the cached container_info (60 second ttl by default), and it’s unable to
reject chunked transfer uploads that exceed the quota (though once the quota
is exceeded, new chunked transfers will be refused).

However, they get most of the way to container quotas fairly inexpensively.

Quotas are set by adding meta values to the container, and are validated when
set:

  X-Container-Meta-Quota-Bytes: Maximum size of the container, in bytes.
  X-Container-Meta-Quota-Count: Maximum object count of the container.

DocImpact

Change-Id: I77cfbf6dc231a2e522bd67328e4c082424a93eee
2013-02-05 06:03:38 -08:00
David Goetz
2f663ff9a0 Bulk Requests: auto extract archive and bulk delete middleware.
Fix small problem in ratelimiting middleware.

DocImpact

Change-Id: Ide3e0b9f4887626c30cae0b97eb7e2237b1df3ed
2013-01-24 12:34:56 -08:00
Julien Danjou
dcff5ff516 Use install_requires in setup.py
Using install_requires makes sure that if some code is using Swift internal
and wants to require Swift, all its dependencies have been already pulled.
This is really useful on test automation for example, otherwise, code might
be failing because on of the module listed in pip-requires is not instaled.

Also, we change the pip-requires files to use >= rather than ==, so the
requirements are easily fulfiled with distribution provided packages.

Change-Id: I65814bcd8ce798da21a5c17b4d5916a23f59e962
Signed-off-by: Julien Danjou <julien@danjou.info>
2012-12-12 11:18:45 +01:00
Doug Hellmann
529f410605 Add OpenStack trove classifier for PyPI
Add trove classifier to have swift listed among the
other OpenStack-related projets on PyPI to reserve
the name, even though we won't release packages there.

Change-Id: I2c6cd2e37ea55d2bc2ddbba813ee2a190b1c90a9
Signed-off-by: Doug Hellmann <doug.hellmann@dreamhost.com>
2012-10-22 18:55:15 -04:00
Tom Fifield
9344a4a582 Random pep8 fixes!
This patch merely fixes a selection of files to the point where
pep8 1.3.3 is happy. Most of the errors are indentation related to
continued lines (E126, E127, E128), bracket positions (E124) and the
use of backslash (E502).

Patch 2 fixes David's comments regarding backslash and an odd comment
 - thanks David!

Change-Id: I4fbd77ecf5395743cb96acb95fa946c322c16560
2012-10-13 12:04:25 +11:00
Darrell Bishop
ed3b12d05c Can run swift-bench across multiple cores/servers.
You run one or more swift-bench-client processes like this:
    $ swift-bench-client 127.0.0.1 20001
    $ swift-bench-client 127.0.0.1 20002

Then you run swift-bench with a new option, --bench-clients (-b), which is
specified once for each swift-bench-client:
    $ swift-bench -b 127.0.0.1:20001 -b 127.0.0.1:20002

You get log lines from each client (interleaved) along with a final report
for all clients:
    127.0.0.1:20002 swift-bench-server 2012-08-25 22:44:06,148 INFO Auth version: 1.0
    127.0.0.1:20001 swift-bench-server 2012-08-25 22:44:06,148 INFO Auth version: 1.0
    127.0.0.1:20001 swift-bench-server 2012-08-25 22:44:12,249 INFO 83 PUTS [0 failures], 41.5/s
    127.0.0.1:20002 swift-bench-server 2012-08-25 22:44:14,430 INFO 74 PUTS [0 failures], 34.3/s
    ...
    127.0.0.1:20002 swift-bench-server 2012-08-25 22:45:18,942 INFO Auth version: 1.0
    127.0.0.1:20002 swift-bench-server 2012-08-25 22:45:20,946 INFO 238 DEL [2 failures], 118.9/s
    swift-bench 2012-08-25 22:45:27,549 INFO 2000 PUTS **FINAL** [0 failures], 56.8/s
    swift-bench 2012-08-25 22:45:27,550 INFO 30000 GETS **FINAL** [50 failures], 974.6/s
    swift-bench 2012-08-25 22:45:27,550 INFO 2000 DEL **FINAL** [20 failures], 237.1/s

The concurrency, PUT count, and GET count config settings are divided by
the number of bench_clients.  In other words, the same volume of work is
attempted (vs. not specifying --bench-clients), but it can now span
servers and CPU cores.

Benchmark containers are created (if use_proxy = yes) and deleted (if
delete = yes), with appropriate concurrency, in the initiating
swift-bench process, not any of the swift-bench-client processes.

Change-Id: Idbf31a23093244ab357a9bf77e6031257774f24a
2012-08-30 18:22:49 +00:00
Chmouel Boudjnah
afa4f70024 Move swift_auth middleware from keystone to swift.
- Rename it to keystoneauth for consistenties.
- Implements blueprint keystone-middleware.

Change-Id: I208fecdf3ee991694b4239f065032324d297fd35
2012-07-05 14:36:14 -04:00
Chmouel Boudjnah
22572b506a Remove swift.common.client from here.
- It has been to its own gerrit project.
- direct_client should follow next.
- Implements blueprint clientbindings.

Change-Id: I3bb50c95eba81302bfec71cb7ce5288b85a41dc0
2012-06-07 16:36:49 +02:00
Michael Barton
7c98e7a625 Move proxy server logging to middleware.
Change-Id: I771c87207d4e1821e32c3424b341d182cc7ea7c0
2012-05-24 21:15:51 -07:00
Chmouel Boudjnah
d02a73f4a9 Remove swift3 from here.
- Reference https://github.com/fujita/swift3 in associated_projects.
- Implements blueprint add-associated-projects-docs.

Change-Id: I48ef4c03449edf6ef4fda1a391228cacac7d2ac6
2012-05-24 10:21:42 +02:00
John Dickinson
1e90b61076 Re-add cname lookup and domain remap middleware
Revert "removed cname lookup middleware"

This reverts commit b47bcf19e4.

Revert "removed domain remap middleware"

This reverts commit 317cf868bdf66dbc17c02d4ca383efafa5e2f229.

Change-Id: I260498d555c93b28896ace48a6f0e96701cbcc38
2012-05-21 20:02:04 -05:00
gholt
1c3b75c291 Reverted the pulling out of various middleware:
RateLimit
StaticWeb
TempURL/FormPOST

Change-Id: I988e93e6f4aacb817a2e354d43a04e47516fdf88
2012-05-16 21:25:10 +00:00
John Dickinson
b47bcf19e4 removed cname lookup middleware
The code has moved to https://github.com/notmyname/swift-cnamelookup.

For current users of cname lookup, this will require installing the new
package and changing the "use" line of the cname lookup conf section's
to:

[filter:cname_lookup]
use = egg:swift_cnamelookup#swift_cnamelookup

And then 'swift-init proxy reload'.

Change-Id: If622486ddb04a53251244c9840aa3cfe72168fc5
2012-05-09 14:00:51 -05:00
gholt
3f00c1a630 Pulled out Rate Limit middleware
Rate Limit middleware is now at
http://dpgoetz.github.com/swift-ratelimit/

For current users of Rate Limit, this will require installing the new
package and changing the "use" line of the ratelimit conf section to:

[filter:ratelimit]
use = egg:swiftratelimit#middleware

And then 'swift-init proxy reload'.

Change-Id: I2ab774e9cee9fba4103c1be4bea6d52d1adb29f7
2012-05-09 18:44:45 +00:00
John Dickinson
7dfbd785b0 removed domain remap middleware
The code has moved to https://github.com/notmyname/swift-domainremap.

For current users of domain remap, this will require installing the new
package and changing the "use" line of the domain remap conf section's
to:

[filter:domain_remap]
use = egg:swift_domainremap#swift_domainremap

And then 'swift-init proxy reload'.

Change-Id: I710caf9b991f9d37df36b826ae4338086d0ec36d
2012-05-08 21:30:35 -05:00
gholt
c0532a6ef2 Pulled out TempURL/FormPOST
TempURL/FormPOST is now at http://gholt.github.com/swift-tempurl/

For current users of TempURL/FormPOST, this will require installing the new
package and changing the "use" line of the tempurl and formpost conf section's
to:

[filter:tempurl]
use = egg:swifttempurl#tempurl

[filter:formpost]
use = egg:swifttempurl#formpost

And then 'swift-init proxy reload'.

Change-Id: I5bddf7f9e09ee07815530a41c46ff901fc21b447
2012-05-08 20:20:29 +00:00
gholt
7dde909621 Pulled StaticWeb out to separate project
StaticWeb is now at http://gholt.github.com/swift-staticweb/

For current users of StaticWeb, this will require installing the new
package and changing the "use" line of the staticweb filter conf
section to:

use = egg:swiftstaticweb#middleware

And then 'swift-init proxy reload'.

Change-Id: Iab32adb5927698a667c5c6d6a572c44ca23414eb
2012-05-05 00:40:33 +00:00
John Dickinson
1ecf5ebba1 updated copyright date for all files
Change-Id: Ifd909d3561c2647770a7e0caa3cd91acd1b4f298
2012-03-19 13:45:34 -05:00
Eamonn O'Toole
cf1aa3c309 Adds name_check filter
Bug 926048.

Filter checks path for user-defined forbidden characters, and for
user-defined maximum length.

Includes changes to reflect gholt's latest comments to Patch Set 4
Also includes a change to a unit-test, renames another unit-test,
and removes one superfluous unit-test.

Added section to the example proxy config

Fixed-up unit test pep8 warnings

Changed error response code to 400 (Bad Request)

Change-Id: Iace719d6a3d00fb3dda1b9d0bc185b8c4cbc00ca
2012-03-08 17:23:58 +00:00
gholt
7141c52bd6 Remove old stats tools.
https://blueprints.launchpad.net/swift/+spec/remove-old-swift-stats-tools

Change-Id: Ibb57b3d71c7f18060d2d48a1d3754a7f1444624d
2012-03-06 17:57:16 +00:00
gholt
7fc1721d7d TempURL and FormPost Middleware
Change-Id: I8d2ce2abdfe3a44605c9441ad7b1abc6c77e282d
2012-01-10 21:57:52 +00:00
gholt
65dba1a7aa Added swift-orphans and swift-oldies.
Change-Id: I95210098556a22d7bd05f245ae387ee13041fa61
2011-12-29 19:19:41 +00:00
gholt
872420efdb Expiring Objects Support
Please see the doc/source/overview_expiring_objects.rst for
more detail.

Change-Id: I4ab49e731248cf62ce10001016e0c819cc531738
2011-11-01 15:49:00 +00:00
John Dickinson
e76bfa67d4 simplified setup.py 2011-08-19 10:58:18 -05:00
Florian Hines
4d0e4622de add binary's/recon.py to setup.py 2011-08-11 15:17:28 -05:00
gholt
8c24a70139 Merged from trunk 2011-06-30 21:52:09 +00:00
John Dickinson
65d79d18b5 merged with trunk 2011-06-15 14:05:28 -05:00
gholt
ba4f74d4fe Merge from trunk 2011-06-14 23:13:47 +00:00
John Dickinson
0eeeae664e renamed st to swift 2011-06-14 11:04:06 -05:00
John Dickinson
5949217ce1 removed basic slogging code 2011-06-13 15:57:32 -05:00
gholt
b4221114c5 Merged with deswauth 2011-06-03 00:11:32 +00:00
gholt
ce710a54a5 Merged with trunk 2011-06-01 20:43:10 +00:00
gholt
288ccb1703 Merged from trunk 2011-05-31 18:09:53 +00:00
Soren Hansen
2d1274c497 Add a __canonical_version__ attribute to the swift module. This is only the numbered part of the version string, no "-dev" or similar suffixes.
Add a couple of unit tests to make sure __version__ and __canonical_version__ are generated correctly and to ensure __canonical_version__ never accidentally has anything other than numbers and points in it.
2011-05-26 11:25:39 +02:00
gholt
f68b6354e2 Renaming TestAuth to TempAuth because nose hates anything with the word test in it. 2011-05-26 02:24:12 +00:00
gholt
3ee4a01100 Remove swauth; update references from swauth to testauth. 2011-05-26 02:17:42 +00:00
gholt
b24b063552 Created testauth WSGI middleware to replace Swauth as development auth service. 2011-05-26 01:19:03 +00:00
David Goetz
bc894f9efb update with trunk 2011-05-12 14:24:51 -07:00
Jay Payne
56d00c2f29 Added bin/swift-container-stats-logger 2011-05-02 23:49:44 +00:00
gholt
0546c8d15f Merged from trunk 2011-04-15 23:11:34 +00:00
gholt
6c13001244 Rename swift-stats-* to swift-dispersion-* to avoid confusion with log stats stuff 2011-03-31 22:32:41 +00:00
gholt
6318b996b5 Merged from trunk 2011-03-18 15:07:03 +00:00
gholt
39889d71dd Merge from trunk 2011-03-18 15:04:09 +00:00
gholt
34f82afc62 Fixes with removing DevAuth 2011-03-14 03:08:23 +00:00
gholt
bd22dbe712 Removing DevAuth 2011-03-14 02:56:37 +00:00
gholt
305e4b41f5 Initial work on container syncing 2011-02-23 23:26:05 -08:00