24 Commits

Author SHA1 Message Date
Alistair Coles
71a4aea31a Update docs to discourage policy names being numbers
There are times when it is convenient to specify a policy by name or
by index (see Related-Change), but policy names can unfortunately
collide with indexes. Using a number as a policy name should at least
be discouraged.

Change-Id: I0cdd3b86b527d6656b7fb50c699e3c0cc566e732
Related-Change: Icf1517bd930c74e9552b88250a7b4019e0ab413e
2021-03-26 09:17:34 +00:00
Clay Gerrard
4601548dab Deprecate per-service auto_create_account_prefix
If we move it to constraints it's more globally accessible in our code,
but more importantly it's more obvious to ops that everything breaks if
you try to mis-configure different values per-service.

Change-Id: Ib8f7d08bc48da12be5671abe91a17ae2b49ecfee
2020-01-05 09:53:30 -06:00
Romain LE DISEZ
673fda7620 Configure diskfile per storage policy
With this commit, each storage policy can define the diskfile to use to
access objects. Selection of the diskfile is done in swift.conf.

Example:
    [storage-policy:0]
    name = gold
    policy_type = replication
    default = yes
    diskfile = egg:swift#replication.fs

The diskfile configuration item accepts the same format than middlewares
declaration: [[scheme:]egg_name#]entry_point
The egg_name is optional and default to "swift". The scheme is optional
and default to the only valid value "egg". The upstream entry points are
"replication.fs" and "erasure_coding.fs".

Co-Authored-By: Alexandre Lécuyer <alexandre.lecuyer@corp.ovh.com>
Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Change-Id: I070c21bc1eaf1c71ac0652cec9e813cadcc14851
2018-08-24 02:29:13 +00:00
junboli
0d07e3fdb1 Update and optimize documentation links
* Update URLs according to document migration
 * Update the dead and outdated links

Change-Id: Id92552f4a2d0fb79ddefc55a08636f2e7aeb07cb
2017-08-01 15:12:00 +01:00
Tim Burke
2c3ac543f4 Require that known-bad EC schemes be deprecated
We said we were going to do it, we've had two releases saying we'd do
it, we've even backported our saying it to Newton -- let's actually do
it.

Upgrade Consideration
=====================

Erasure-coded storage policies using isa_l_rs_vand and nparity >= 5 must
be configured as deprecated, preventing any new containers from being
created with such a policy. This configuration is known to harm data
durability. Any data in such policies should be migrated to a new
policy. See https://bugs.launchpad.net/swift/+bug/1639691 for more
information.

UpgradeImpact
Related-Change: I50159c9d19f2385d5f60112e9aaefa1a68098313
Change-Id: I8f9de0bec01032d9d9b58848e2a76ac92e65ab09
Closes-Bug: 1639691
2017-06-16 17:58:43 +00:00
Kota Tsuyuzaki
40ba7f6172 EC Fragment Duplication - Foundational Global EC Cluster Support
This patch enables efficent PUT/GET for global distributed cluster[1].

Problem:
Erasure coding has the capability to decrease the amout of actual stored
data less then replicated model. For example, ec_k=6, ec_m=3 parameter
can be 1.5x of the original data which is smaller than 3x replicated.
However, unlike replication, erasure coding requires availability of at
least some ec_k fragments of the total ec_k + ec_m fragments to service
read (e.g. 6 of 9 in the case above). As such, if we stored the
EC object into a swift cluster on 2 geographically distributed data
centers which have the same volume of disks, it is likely the fragments
will be stored evenly (about 4 and 5) so we still need to access a
faraway data center to decode the original object. In addition, if one
of the data centers was lost in a disaster, the stored objects will be
lost forever, and we have to cry a lot. To ensure highly durable
storage, you would think of making *more* parity fragments (e.g.
ec_k=6, ec_m=10), unfortunately this causes *significant* performance
degradation due to the cost of mathmetical caluculation for erasure
coding encode/decode.

How this resolves the problem:
EC Fragment Duplication extends on the initial solution to add *more*
fragments from which to rebuild an object similar to the solution
described above. The difference is making *copies* of encoded fragments.
With experimental results[1][2], employing small ec_k and ec_m shows
enough performance to store/retrieve objects.

On PUT:

- Encode incomming object with small ec_k and ec_m  <- faster!
- Make duplicated copies of the encoded fragments. The # of copies
  are determined by 'ec_duplication_factor' in swift.conf
- Store all fragments in Swift Global EC Cluster

The duplicated fragments increase pressure on existing requirements
when decoding objects in service to a read request.  All fragments are
stored with their X-Object-Sysmeta-Ec-Frag-Index.  In this change, the
X-Object-Sysmeta-Ec-Frag-Index represents the actual fragment index
encoded by PyECLib, there *will* be duplicates.  Anytime we must decode
the original object data, we must only consider the ec_k fragments as
unique according to their X-Object-Sysmeta-Ec-Frag-Index.  On decode no
duplicate X-Object-Sysmeta-Ec-Frag-Index may be used when decoding an
object, duplicate X-Object-Sysmeta-Ec-Frag-Index should be expected and
avoided if possible.

On GET:

This patch inclues following changes:
- Change GET Path to sort primary nodes grouping as subsets, so that
  each subset will includes unique fragments
- Change Reconstructor to be more aware of possibly duplicate fragments

For example, with this change, a policy could be configured such that

swift.conf:
ec_num_data_fragments = 2
ec_num_parity_fragments = 1
ec_duplication_factor = 2
(object ring must have 6 replicas)

At Object-Server:
node index (from object ring):  0 1 2 3 4 5 <- keep node index for
                                               reconstruct decision
X-Object-Sysmeta-Ec-Frag-Index: 0 1 2 0 1 2 <- each object keeps actual
                                               fragment index for
                                               backend (PyEClib)

Additional improvements to Global EC Cluster Support will require
features such as Composite Rings, and more efficient fragment
rebalance/reconstruction.

1: http://goo.gl/IYiNPk (Swift Design Spec Repository)
2: http://goo.gl/frgj6w (Slide Share for OpenStack Summit Tokyo)

Doc-Impact

Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Change-Id: Idd155401982a2c48110c30b480966a863f6bd305
2017-02-22 10:56:13 -08:00
Janie Richling
96a0e07753 Enable object body and metadata encryption
Adds encryption middlewares.

All object servers and proxy servers should be upgraded before
introducing encryption middleware.

Encryption middleware should be first introduced with the
encryption middleware disable_encryption option set to True.
Once all proxies have encryption middleware installed this
option may be set to False (the default).

Increases constraints.py:MAX_HEADER_COUNT by 4 to allow for
headers generated by encryption-related middleware.

Co-Authored-By: Tim Burke <tim.burke@gmail.com>
Co-Authored-By: Christian Cachin <cca@zurich.ibm.com>
Co-Authored-By: Mahati Chamarthy <mahati.chamarthy@gmail.com>
Co-Authored-By: Peter Chng <pchng@ca.ibm.com>
Co-Authored-By: Alistair Coles <alistair.coles@hpe.com>
Co-Authored-By: Jonathan Hinson <jlhinson@us.ibm.com>
Co-Authored-By: Hamdi Roumani <roumani@ca.ibm.com>

UpgradeImpact

Change-Id: Ie6db22697ceb1021baaa6bddcf8e41ae3acb5376
2016-06-30 23:31:15 -07:00
Nguyen Hung Phuong
edc413b85e Fix typos in Swift files
Change-Id: I39dbf55c094c42347b57ef67520abff9e6fc24bc
2016-04-07 13:33:20 +07:00
Christian Schwede
043fbca6d0 Remove Erasure Coding beta status from docs
This removes notes stating support for Erasure coding as beta. Questions
regarding the stability of EC are coming up regularly, and are often referring
to the docs that state EC as still in beta.

Besides this, a note marking statsd support as beta has been removed as well.

Change-Id: If4fb6a5c4cb741d42953db3cee8cb17a1d774e15
2016-03-04 14:27:23 +00:00
Christian Schwede
e46d6b1725 Add note on using printable chars for swift_hash_path_suffix/prefix
Using unprintable characters for swift_hash_path_prefix/suffix might lead to
hard problems when parsing of these values changes, for example due to newer
Python versions or changes in the parsers itself. Let's avoid this and add a
note that deployers should use printable strings for these values.

Change-Id: I976982b753b6af831ab91d7190f50f8f15bf73bf
2016-02-17 20:47:53 +01:00
Christian Berendt
b98c1c60f9 Remove executable flag from sample config file
swift.conf-sample should not be executbale

Change-Id: I960058e2af10b43efc6324a7534d9a23c810e39a
2016-02-09 12:57:24 +00:00
Ondřej Nový
16976a0f14 Changed EC backend from jerasure to liberasurecode in examples and docs
liberasurecode_rs_vand is build-in liberasurecode, so you don't need
another depedency libjerasure2.

liberasurecode_rs_vand is supported by pyeclib from 1.0.8
version, so bumping version up.

Closes-Bug: #1534325
Change-Id: If2d96875694df8fd48c5278395859aaa165cb566
2016-02-02 23:08:11 +01:00
Catherine Northcott
211758f8cb Add support for storage policies to have more than one name
This patch alters storage_policy.py to allow storage policies
to have multiple names. Now users are able to add a number of
human-readable aliases for storage policies. Policies now have
a .name (the default name), .aliases (a string of comma
seperated aliases), and .aliases_list (a list of all human
readable names). Policies will always have an .aliases value
if no aliases are set it will contain the default name.
The policy docs and tests have been updated to reflect changes
and policy.get_policy_info has been altered to display the
name and aliases

Change-Id: I02967ca8d7c790595e5ee551581196aa64552eea
2015-12-10 14:47:23 +13:00
Charles Hsu
f0d51882b9 Add extra_header_count to document and config.
Change-Id: Iec86b488d71553c295afe7098822ce2046df9546
2015-08-05 22:09:40 +08:00
Christian Schwede
edfca861b6 Increase httplib._MAXHEADERS
Python 2.7.9+ and 3.2.6+ limits the number of maximum headers in httplib to 100
[1,2,3]. This setting is too low for Swift.

By default the maximum number of allowed headers depends on the number of max
allowed metadata settings plus a default value of 32 for regular http headers.
If for some reason this is not enough (custom middleware for example) it can be
increased with the extra_header_count constraint.

[1] https://bugs.python.org/issue16037
[2] https://hg.python.org/cpython/raw-file/15c95b7d81dc/Misc/NEWS
[3] https://hg.python.org/cpython/raw-file/v3.2.6/Misc/NEWS

Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Co-Authored-By: Matthew Oliver <matt@oliver.net.au>
Co-Authored-By: Thomas Herve <therve@redhat.com>

Change-Id: I388fd697ec88476024b0e9f1ae75ba35ff765282
2015-06-26 14:35:40 +00:00
Clay Gerrard
4aba2fbb25 Check if REST API version is valid
Swift doesn't check if the used API version is valid. Currently there
is only one valid REST API version, but that might change in the
future.

This patch enforces "v1" or "v1.0" as the version string when accessing
account, containers and objects.

The list of accepted version strings can be manually overridden using a
comma-separated list in swift.conf to make this backward-compatible.
The constraint loader has been modified slightly to accept strings as
well as integers.

Any request to an account, container, and object which does not
provide the correct version string will get a 400 BadRequest response.

The allowed api versions are by default excluded from /info.

Co-Authored-By: Christian Schwede <christian.schwede@enovance.com>
Co-Authored-By: John Dickinson <me@not.mn>

Closes Bug #1437442

Change-Id: I5ab6e236544378abf2eab562ab759513d09bc256
2015-04-14 16:00:37 -07:00
Tushar Gohad
ed54066288 Add support for policy types, 'erasure_coding' policy
This patch extends the StoragePolicy class for non-replication storage
policies, the first one being "erasure coding".

Changes:

 - Add 'policy_type' support to BaseStoragePolicy class
 - Disallow direct instantiation of BaseStoragePolicy class

 - Subclass BaseStoragePolicy

   - "StoragePolicy":
     . Replication policy, default
     . policy_type = 'replication'

   - "ECStoragePolicy":
     . Erasure Coding policy
     . policy_type = 'erasure_coding'
     . Private member variables
       ec_type (EC backend),
       ec_num_data_fragments (number of fragments original
         data split into after erasure coding operation),
       ec_num_parity_fragments (number of parity fragments
       generated during erasure coding)
     . Private methods
       EC specific attributes and ring validator methods.

 - Swift will use PyECLib, a Python Erasure Coding library, for
   erasure coding operations. PyECLib is already an approved
   OpenStack core requirement.
   (https://bitbucket.org/kmgreen2/pyeclib/)

 - Add test cases for
   - 'policy_type' StoragePolicy member
   - policy_type == 'erasure_coding'

DocImpact

Co-Authored-By: Alistair Coles <alistair.coles@hp.com>
Co-Authored-By: Thiago da Silva <thiago@redhat.com>
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Co-Authored-By: Paul Luse <paul.e.luse@intel.com>
Co-Authored-By: Samuel Merritt <sam@swiftstack.com>
Co-Authored-By: Christian Schwede <christian.schwede@enovance.com>
Co-Authored-By: Yuan Zhou <yuan.zhou@intel.com>
Change-Id: Ie0e09796e3ec45d3e656fb7540d0e5a5709b8386
Implements: blueprint ec-proxy-work
2015-04-13 22:57:42 -07:00
Paul Luse
46c68aebd1 Add Storage Policy Support
The basic idea here is to replace the use of a single object ring in
the Application class with a collection of object rings. The
collection includes not only the Ring object itself but the policy
name associated with it, the filename for the .gz and any other
metadata associated with the policy that may be needed. When
containers are created, a policy (thus a specific obj ring) is
selected allowing apps to specify policy at container creation time
and leverage policies simply by using different containers for object
operations.

The policy collection is based off of info in the swift.conf file.
The format of the sections in the .conf file is as follows:

swift.conf format:

    [storage-policy:0]
    name = chicken

    [storage-policy:1]
    name = turkey
    default = yes

With the above format:

- Policy 0 will always be used for access to existing containers
  without the policy specified. The ring name for policy 0 is always
  'object', assuring backwards compatiblity. The parser will always
  create a policy 0 even if not specified

- The policy with 'default=yes' is the one used for new container
  creation. This allows the admin to specify which policy is used without
  forcing the application to add the metadata.

This commit simply introduces storage policies and the loading
thereof; nobody's using it yet. That will follow in subsequent
commits.

Expose storage policies in /info

DocImpact
Implements: blueprint storage-policies
Change-Id: Ica05f41ecf3adb3648cc9182f11f1c8c5c678985
2014-06-18 17:31:37 -07:00
anc
58fe2f256f Add tests and comments re constraints in /info
Add test to check that only the expected keys are
reported by proxy in /info, and add comments to
raise awareness that default constraints will be
automatically published by proxy in response to /info
requests.

Change-Id: Ia5f6339b06cdc2e1dc960d1f75562a2505530202
2014-04-09 11:59:24 +01:00
Florent Flament
865243c167 Fix Error 400 Header Line Too Long
Fix Error 400 Header Line Too Long when using Identity v3 PKI Tokens

Uses swift.conf max_header_size option to set wsgi.MAX_HEADER_LINE,
allowing the operator to customize this parameter.

The default value has been let to 8192 to avoid unexpected
configuration change on deployed platforms. The max_header_size option
has to be increased (for example to 16384), to accomodate for large
Identity v3 PKI tokens, including more than 7 catalog entries.

The default max header line size of 8192 is exceeded in the following
scenario:
- Auth tokens generated by Keystone v3 API include the catalog.
- Keystone's catalog contains more than 7 services.

Similar fixes have been merged in other projects.

Change-Id: Ia838b18331f57dfd02b9f71d4523d4059f38e600
Closes-Bug: 1190149
2014-01-17 12:36:30 +00:00
Kun Huang
5c8785aaee Add max_header_size to swift.conf-sample and relative UT
1. Add explanation of MAX_HEADER_SIZE into swift.conf-sample as same as
other settings in swift.conf. Especially point out the default size of
header line in eventlet is 8192 which is the main reason why we set 8192
for MAX_HEADER_SIZE in swift.

2. Add some unit tests to check valid settings in swift.conf. Test cases
in test_constraints use /etc/swift/swift.conf if exists, and if any
wrong settings are in it (MAX_META_VALE > MAX_META_OVERALL_SIZE), swift's
unit test must fail. These new unit tests is used in this case.

Change-Id: I7bb21951d46050163c1b7bceac8d49302b9209f7
2013-06-19 23:45:38 +08:00
David Hadas
04a3ba43ae Fixing /etc/swift.conf-sample to include
swift_hash_path_prefix

Change-Id: I60f5f3d4083937a03ecb7ed531185c617ea08016
2013-04-22 19:32:45 +03:00
John Dickinson
a2ac5efaa6 swift constraints are now settable via config
Change previously hard-coded constants into config variables. This
allows deployers to tune their cluster more specifically based on
their needs. For example, a deployment that uses direct swift access
for public content may need to set a larger header value constraint to
allow for the full object name to be represented in the Content-
Disposition header (which browsers check to determine the name of a
downloaded object).

The new settings are set in the [swift-constraints] section of
/etc/swift/swift.conf. Comments were also added to this config file.

Cleaned up swift/common/constraints.py to pass pep8 1.3.3

Funtional tests now require constraints to be defined in /etc/test.conf or in
/etc/swift/swift.conf (in the case of running the functional tests against a
local swift cluster). To have any hope of tests passing, the defined
constraints must match the constraints on the tested cluster.

Removed a ton of "magic numbers" in both unit and functional tests.

Change-Id: Ie4588e052fd158314ddca6cd8fca9bc793311465
2012-09-07 11:18:42 -07:00
David Goetz
c99c976881 Refactor SWIFT_HASH_PATH_SUFFIX to be in a config file 2010-10-14 15:58:44 -07:00