Rate limiting in OpenStack Object Storage is implemented as a pluggable middleware that you configure on the proxy server. Rate limiting is performed on requests that result in database writes to the account and container sqlite dbs. It uses memcached and is dependent on the proxy servers having highly synchronized time. The rate limits are limited by the accuracy of the proxy server clocks.

All configuration is optional. If no account or container limits are provided there will be no rate limiting. Available configuration options include: The container rate limits are linearly interpolated from the values given. A sample container rate limiting could be: container_ratelimit_100 = 100 container_ratelimit_200 = 50 container_ratelimit_500 = 20 This would result in:

Values for Rate Limiting with Sample Configuration Settings

Container Size	Rate Limit
0-99	No limiting
100	100
150	75
500	20
1000	20

Health Check provides a simple way to monitor if the swift proxy server is alive. If the proxy is access with the path /healthcheck, it will respond with “OK” in the body, which can be used by monitoring tools.

Domain Remap is middleware that translates container and account parts of a domain to path parameters that the proxy server understands.

CNAME Lookup is middleware that translates an unknown domain in the host header to something that ends with the configured storage_domain by looking up the given domain's CNAME record in DNS.

Allows the creation of URLs to provide temporary access to objects. For example, a website may wish to provide a link to download a large object in Swift, but the Swift account has no public access. The website can generate a URL that will provide GET access for a limited time to the resource. When the web browser user clicks on the link, the browser will download the object directly from Swift, obviating the need for the website to act as a proxy for the request. If the user were to share the link with all his friends, or accidentally post it on a forum, etc. the direct access would be limited to the expiration time set when the website created the link. To create such temporary URLs, first an X-Account-Meta-Temp-URL-Key header must be set on the Swift account. Then, an HMAC-SHA1 (RFC 2104) signature is generated using the HTTP method to allow (GET or PUT), the Unix timestamp the access should be allowed until, the full path to the object, and the key set on the account. For example, here is code generating the signature for a GET for 60 seconds on /v1/AUTH_account/container/object: import hmac from hashlib import sha1 from time import time method = 'GET' expires = int(time() + 60) path = '/v1/AUTH_account/container/object' key = 'mykey' hmac_body = '%s\n%s\n%s' % (method, expires, path) sig = hmac.new(key, hmac_body, sha1).hexdigest() Be certain to use the full path, from the /v1/ onward. Let's say the sig ends up equaling da39a3ee5e6b4b0d3255bfef95601890afd80709 and expires ends up 1323479485. Then, for example, the website could provide a link to: https://swift-cluster.example.com/v1/AUTH_account/container/object? temp_url_sig=da39a3ee5e6b4b0d3255bfef95601890afd80709& temp_url_expires=1323479485 Any alteration of the resource path or query arguments would result in 401 Unauthorized. Similarly, a PUT where GET was the allowed method would 401. HEAD is allowed if GET or PUT is allowed. Using this in combination with browser form post translation middleware could also allow direct-from-browser uploads to specific locations in Swift. Note that changing the X-Account-Meta-Temp-URL-Key will invalidate any previously generated temporary URLs within 60 seconds (the memcache time for the key). A script called swift-temp-url distributed with swift source code eases the temporary URL creation: $ bin/swift-temp-url GET 3600 /v1/AUTH_account/container/object mykey /v1/AUTH_account/container/object? temp_url_sig=5c4cc8886f36a9d0919d708ade98bf0cc71c9e91& temp_url_expires=1374497657 The path returned by the above command is prefixed with swift storage hostname.

Name Check is a filter that disallows any paths that contain defined forbidden characters or that exceed a defined length.

The swift-constraints section in swift.conf allows modification of internal limits within swift. These are advanced features for tuning the performance of the cluster and should be altered with caution.

Configuration options for swift-constraints in swift.conf

Option	Default	Description
max_file_size	5368709122	the largest "normal" object that can be saved in the cluster. This is also the limit on the size of each segment of a "large" object when using the large object manifest support. This value is set in bytes. Setting it to lower than 1MiB will cause some tests to fail. It is STRONGLY recommended to leave this value at the default (5 * 2**30 + 2).
max_meta_name_length	128	the max number of bytes in the utf8 encoding of the name portion of a metadata header.
max_meta_value_lenth	256	the max number of bytes in the utf8 encoding of a metadata value
max_meta_count	90	the max number of metadata keys that can be stored on a single account, container, or object
max_meta_overall_size	4096	the max number of bytes in the utf8 encoding of the metadata (keys + values)
max_object_name_length	1024	the max number of bytes in the utf8 encoding of an object name
container_listing_limit	10000	the default (and max) number of items returned for a container listing request
max_account_name_length	256	the max number of bytes in the utf8 encoding of an account name
max_container_name_length	256	the max number of bytes in the utf8 encoding of a container name

There is a swift-dispersion-report tool for measuring overall cluster health. This is accomplished by checking if a set of deliberately distributed containers and objects are currently in their proper places within the cluster. For instance, a common deployment has three replicas of each object. The health of that object can be measured by checking if each replica is in its proper place. If only 2 of the 3 is in place the object’s heath can be said to be at 66.66%, where 100% would be perfect. A single object’s health, especially an older object, usually reflects the health of that entire partition the object is in. If we make enough objects on a distinct percentage of the partitions in the cluster, we can get a pretty valid estimate of the overall cluster health. In practice, about 1% partition coverage seems to balance well between accuracy and the amount of time it takes to gather results. The first thing that needs to be done to provide this health value is create a new account solely for this usage. Next, we need to place the containers and objects throughout the system so that they are on distinct partitions. The swift-dispersion-populate tool does this by making up random container and object names until they fall on distinct partitions. Last, and repeatedly for the life of the cluster, we need to run the swift-dispersion-report tool to check the health of each of these containers and objects. These tools need direct access to the entire cluster and to the ring files (installing them on a proxy server will probably do). Both swift-dispersion-populate and swift-dispersion-report use the same configuration file, /etc/swift/dispersion.conf. Example dispersion.conf file: [dispersion] auth_url = http://localhost:8080/auth/v1.0 auth_user = test:tester auth_key = testing There are also options for the conf file for specifying the dispersion coverage (defaults to 1%), retries, concurrency, etc. though usually the defaults are fine. Once the configuration is in place, run swift-dispersion-populate to populate the containers and objects throughout the cluster. Now that those containers and objects are in place, you can run swift-dispersion-report to get a dispersion report, or the overall health of the cluster. Here is an example of a cluster in perfect health: $ swift-dispersion-report Queried 2621 containers for dispersion reporting, 19s, 0 retries 100.00% of container copies found (7863 of 7863) Sample represents 1.00% of the container partition space Queried 2619 objects for dispersion reporting, 7s, 0 retries 100.00% of object copies found (7857 of 7857) Sample represents 1.00% of the object partition space Now, deliberately double the weight of a device in the object ring (with replication turned off) and rerun the dispersion report to show what impact that has: $ swift-ring-builder object.builder set_weight d0 200 $ swift-ring-builder object.builder rebalance ... $ swift-dispersion-report Queried 2621 containers for dispersion reporting, 8s, 0 retries 100.00% of container copies found (7863 of 7863) Sample represents 1.00% of the container partition space Queried 2619 objects for dispersion reporting, 7s, 0 retries There were 1763 partitions missing one copy. 77.56% of object copies found (6094 of 7857) Sample represents 1.00% of the object partition space You can see the health of the objects in the cluster has gone down significantly. Of course, this test environment has just four devices, in a production environment with many devices the impact of one device change is much less. Next, run the replicators to get everything put back into place and then rerun the dispersion report: ... start object replicators and monitor logs until they're caught up ... $ swift-dispersion-report Queried 2621 containers for dispersion reporting, 17s, 0 retries 100.00% of container copies found (7863 of 7863) Sample represents 1.00% of the container partition space Queried 2619 objects for dispersion reporting, 7s, 0 retries 100.00% of object copies found (7857 of 7857) Sample represents 1.00% of the object partition space Alternatively, the dispersion report can also be output in json format. This allows it to be more easily consumed by third party utilities: $ swift-dispersion-report -j {"object": {"retries:": 0, "missing_two": 0, "copies_found": 7863, "missing_one": 0, "copies_expected": 7863, "pct_found": 100.0, "overlapping": 0, "missing_all": 0}, "container": {"retries:": 0, "missing_two": 0, "copies_found": 12534, "missing_one": 0, "copies_expected": 12534, "pct_found": 100.0, "overlapping": 15, "missing_all": 0}}

This feature is very similar to Dynamic Large Object (DLO) support in that it allows the user to upload many objects concurrently and afterwards download them as a single object. It is different in that it does not rely on eventually consistent container listings to do so. Instead, a user defined manifest of the object segments is used.

The container_quotas middleware implements simple quotas that can be imposed on swift containers by a user with the ability to set container metadata, most likely the account administrator. This can be useful for limiting the scope of containers that are delegated to non-admin users, exposed to formpost uploads, or just as a self-imposed sanity check. Any object PUT operations that exceed these quotas return a 413 response (request entity too large) with a descriptive body. Quotas are subject to several limitations: eventual consistency, the timeliness of the cached container_info (60 second ttl by default), and it's unable to reject chunked transfer uploads that exceed the quota (though once the quota is exceeded, new chunked transfers will be refused). Quotas are set by adding meta values to the container, and are validated when set: X-Container-Meta-Quota-Bytes: Maximum size of the container, in bytes. X-Container-Meta-Quota-Count: Maximum object count of the container.

The account_quotas middleware aims to block write requests (PUT, POST) if a given account quota (in bytes) is exceeded while DELETE requests are still allowed. The x-account-meta-quota-bytes metadata entry must be set to store and enable the quota. Write requests to this metadata entry are only permitted for resellers. There isn't any account quota limitation on a reseller account even if x-account-meta-quota-bytes is set. Any object PUT operations that exceed the quota return a 413 response (request entity too large) with a descriptive body. The following command uses an admin account that own the Reseller role to set a quota on the test account: $ swift -A http://127.0.0.1:8080/auth/v1.0 -U admin:admin -K admin \ --os-storage-url=http://127.0.0.1:8080/v1/AUTH_test post -m quota-bytes:10000 Here is the stat listing of an account where quota has been set: $ swift -A http://127.0.0.1:8080/auth/v1.0 -U test:tester -K testing stat Account: AUTH_test Containers: 0 Objects: 0 Bytes: 0 Meta Quota-Bytes: 10000 X-Timestamp: 1374075958.37454 X-Trans-Id: tx602634cf478546a39b1be-0051e6bc7a The command below removes the account quota: $ swift -A http://127.0.0.1:8080/auth/v1.0 -U admin:admin -K admin --os-storage-url=http://127.0.0.1:8080/v1/AUTH_test post -m quota-bytes:

Will delete multiple files from their account with a single request. Responds to DELETE requests with a header 'X-Bulk-Delete: true_value'. The body of the DELETE request will be a newline separated list of files to delete. The files listed must be URL encoded and in the form: /container_name/obj_name If all files were successfully deleted (or did not exist) will return an HTTPOk. If any files failed to delete will return an HTTPBadGateway. In both cases the response body is a json dictionary specifying in the number of files successfully deleted, not found, and a list of the files that failed.

The swift-drive-audit configuration items reference a script that can be run via cron to watch for bad drives. If errors are detected, it will unmount the bad drive, so that OpenStack Object Storage can work around it. It takes the following options:

The Form Post middleware provides the ability to upload objects to a cluster using an HTML form POST. The format of the form is: <![CDATA[ <form action="<swift-url>" method="POST" enctype="multipart/form-data"> <input type="hidden" name="redirect" value="<redirect-url>" /> <input type="hidden" name="max_file_size" value="<bytes>" /> <input type="hidden" name="max_file_count" value="<count>" /> <input type="hidden" name="expires" value="<unix-timestamp>" /> <input type="hidden" name="signature" value="<hmac>" /> <input type="file" name="file1" /><br /> <input type="submit" /> </form>]]> The swift-url is the URL to the Swift destination, such as: https://swift-cluster.example.com/v1/AUTH_account/container/object_prefix The name of each file uploaded will be appended to the swift-url given. So, you can upload directly to the root of container with a url like: https://swift-cluster.example.com/v1/AUTH_account/container/ Optionally, you can include an object prefix to better separate different users’ uploads, such as: https://swift-cluster.example.com/v1/AUTH_account/container/object_prefix Note the form method must be POST and the enctype must be set as “multipart/form-data”. The redirect attribute is the URL to redirect the browser to after the upload completes. The URL will have status and message query parameters added to it, indicating the HTTP status code for the upload (2xx is success) and a possible message for further information if there was an error (such as “max_file_size exceeded”). The max_file_size attribute must be included and indicates the largest single file upload that can be done, in bytes. The max_file_count attribute must be included and indicates the maximum number of files that can be uploaded with the form. Include additional <![CDATA[<input type="file" name="filexx"/>]]> attributes if desired. The expires attribute is the Unix timestamp before which the form must be submitted before it is invalidated. The signature attribute is the HMAC-SHA1 signature of the form. Here is sample code for computing the signature: import hmac from hashlib import sha1 from time import time path = '/v1/account/container/object_prefix' redirect = 'https://myserver.com/some-page' max_file_size = 104857600 max_file_count = 10 expires = int(time() + 600) key = 'mykey' hmac_body = '%s\n%s\n%s\n%s\n%s' % (path, redirect, max_file_size, max_file_count, expires) signature = hmac.new(key, hmac_body, sha1).hexdigest() The key is the value of the X-Account-Meta-Temp-URL-Key header on the account. Be certain to use the full path, from the /v1/ onward. The command line tool swift-form-signature may be used (mostly just when testing) to compute expires and signature. Also note that the file attributes must be after the other attributes in order to be processed correctly. If attributes come after the file, they won’t be sent with the subrequest (there is no way to parse all the attributes on the server-side without reading the whole thing into memory – to service many requests, some with large files, there just isn’t enough memory on the server, so attributes following the file are simply ignored).

When configured, the StaticWeb WSGI middleware serves container data as a static web site with index file and error file resolution and optional file listings. This mode is normally only active for anonymous requests.