This reverts commit 7760f41c3ce436cb23b4b8425db3749a3da33d32
Change-Id: I95e57a2563784a8cd5e995cc826afeac0eadbe62
Signed-off-by: Peter Portante <peter.portante@redhat.com>
Place all the methods related to on-disk layout and / or configuration
into a new common module that can be shared by the various modules
using the same on-disk layout.
Change-Id: I27ffd4665d5115ffdde649c48a4d18e12017e6a9
Signed-off-by: Peter Portante <peter.portante@redhat.com>
except x,y: was deprected and is removed in Python 3.x.
Use "except x as y:" instead which works in any Python
version >= 2.6.
Change-Id: I7008c74b807340f3457d3a0c8bd0b83f23169d14
We also fix up any other pep8 failures that snuck in from merges along
the way.
Change-Id: I4ea984780ac2eac458c98fe181684eef4e04beaf
Signed-off-by: Peter Portante <peter.portante@redhat.com>
Address all the "hacking" lines that are flagged, and all the modules
that just have one item flagged.
Change-Id: I372a4bdf9c7748f73e38c4fd55e5954f1afade5b
Signed-off-by: Peter Portante <peter.portante@redhat.com>
Certain old clients that shall not be named do not like some of the
recent changes to the xml output for container listings. This fixes
those changes. If we want to change any of the xml output, then we
should do it in another API version. I'm not sure if any of these
changes in the XML were put in any documentation, but if they were, they
should be changed back.
DocImpact
Change-Id: I983332ca9d620a9a539be26bc6655df9b2ef87ed
The account, container, and object servers all do the same thing at
the top of GET/PUT/etc.: they split the path, validate that the first
two components are a good (device, partition) pair, and return a 400
if they're not. The object server already had a module-local helper
function for this, but the account and container servers just had
duplicate boilerplate code. Now it's all in one common helper.
Change-Id: I9a20d37fc9e1a68b10149a7aa78cb9691fc04ea9
There were a few different places where we had some repeated code to
figure out what format an account or container listing response should
be in (text, JSON, or XML). Now that's been pulled into a single
function.
As part of this, you can now raise HTTPException subclasses in proxy
controllers instead of laboriously plumbing error responses all the
way back up to swift.proxy.server.Application.handle_request(). This
lets us avoid certain ugly patterns, like the one where a method
returns a tuple of (x, y, z, error) and the caller has to see if it
got (value, value, value, None) or (None, None, None, errorvalue). Now
we can just raise the error.
Change-Id: I316873df289160d526487ad116f6fbb9a575e3de
Have json and xml use common record cleanup code.
Do a somewhat better job of parsing extensions from content-types.
Use a real XML serializer.
Change-Id: I10e14dffd1da590b4fd180b4d33ab5de862e2b55
This lets us get rid of some really repetitive exception conversion
code, like everybody that called common.utils.get_param() had to catch
a UnicodeDecodeError and turn that into returning HTTPBadRequest. Now
get_param() just raises HTTPBadRequest directly, and the __call__
methods in the account/container/object servers catch and return
it. All that "except UnicodeDecodeError" stuff goes away.
Refactored the path splitting and device validation in the object
server too.
There are other things that can benefit from this as well, but this
patch is big enough.
Change-Id: I2be96ef757d04bfd6af180cd9c92393c841db21f
Based on this http://en.wikipedia.org/wiki/ISO_8601#Time_zone_designators.
A isofomt of time need a 'Z' for UTC time zone, or none for local time zone
request on GET /<version>/<account>/<container>?format=json still output a
UTC time without 'Z'
fixes bug #1169287
Change-Id: Ib599b5ec8fd223878ec18df7c1ec8d952fc2630a
Remove logic of allowed methods list from object, container and account
servers. Instead of it add replicator decorator to utils and use new
decorator for REPLICATE methods in object/account/container servers.
This decorator mark method as special for usfor use only by the
replication.
If the option replication_server is not used, then this mechanism is not
enabled. If the replicaton_server option is set (not None) then the
respective server is a replicator (option value is True) and should use
ONLY the methods marked for replication server using the decorator, or
it is a normal server type and should NOT use methods marked for the
replication server.
Change-Id: I1041b31413cd0c39000317cc57a8c27816e1dfe8
A HEAD request to /v1/a[/c]?format=%FF would result in a 503 since
there was an unhandled UnicodeDecodeError. Now it doesn't.
bug 1190395
Change-Id: I4d8ec677092617391fe154a25f22d919536a72a5
For HEAD and GET requests on both containers and accounts APIs we now
perform parameter checking before we access the database. The unit
tests were updated to show that the parameter checking is performed
before accesses are performed.
Change-Id: Ieb753316cdccabf45022e3d83522d87d34aa6b0e
Signed-off-by: Peter Portante <peter.portante@redhat.com>
The results that are being parsed are really JSON, so using a JSON parser is
more semantically correct; in addition using eval anywhere can set a bad
example.
Change-Id: Idcd55400b2571aba0f2f377cf66a3cbf4d3af960
Support separate replication ip address:
- Added new function in utils. This function provides ability
to select separate IP address for replication service.
- Db_replicator and object replicators were changed.
Replication process uses new function now.
Replication network parameters:
- Replication network fields (replication_ip, replication_port)
support was added to device dictionary in swift-ring-builder script.
- Changes were made to support new fields in search, show and set_info
functions.
Implementation of replication servers:
- Separate replication servers use the same code as normal replication
servers, but with replication_server parameter = True. When using a
separate replication network, the non-replication servers set
replication_server = False. When there is no separate replication
network (the default case), replication_server is not included in the config.
DocImpact
Change-Id: Ie9af5bdcdf9241c355e36053ca4adfe49dc35bd0
Implements: blueprint dedicated-replication-network
Enhance internally logged messages to report referer and user-agent.
Pass the referering URL and METHOD between internal servers (when
known), and set the user-agent to be the server type (obj-server,
container-server, proxy-server, obj-updater, obj-replicator,
container-updater, direct-client, etc.) with the process PID. In
conjunction with the transaction ID, it helps to track down which PID
from a given system was responsible for initiating the request and
what that server was working on to make this request.
This has been helpful in tracking down interactions between object,
container and account servers.
We also take things a bit further performaing a bit of refactoring to
consolidate calls to transfer_headers() now that we have a helper
method for constructing them.
Finally we performed further changes to avoid header key duplication
due to string literal header key values and the various objects
representing headers for requests and responses. See below for more
details.
====
Header Keys
There seems to be a bit of a problem with the case of the various
string literals used for header keys and the interchangable way
standard Python dictionaries, HeaderKeyDict() and HeaderEnvironProxy()
objects are used.
If one is not careful, a header object of some sort (one that does not
normalize its keys, and that is not necessarily a dictionary) can be
constructed containing header keys which differ only by the case of
their string literals. E.g.:
{ 'x-trans-id': '1234', 'X-Trans-Id': '5678' }
Such an object, when passed to http_connect() will result in an
on-the-wire header where the key values are merged together, comma
separated, that looks something like:
HTTP_X_TRANS_ID: 1234,5678
For some headers in some contexts, this is behavior is desirable. For
example, one can also use a list of tuples which enumerate the multiple
values a single header should have.
However, in almost all of the contexts used in the code base, this is
not desirable.
This behavior arises from a combination of factors:
1. Header strings are not constants and different lower-case and
title-case header strings values are used interchangably in the
code at times
It might be worth the effort to make a pass through the code to
stop using string literals and use constants instead, but there
are plusses and minuses to doing that, so this was not attempted
in this effort
2. HeaderEnvironProxy() objects report their keys in ".title()"
case, but normalize all other key references to the form
expected by the Request class's environ field
swob.Request.headers fields are HeaderEnvironProxy() objects.
3. HeaderKeyDict() objects report their keys in ".lower()" case,
and normalize all other key references to ".lower()" case
swob.Response.headers fields are HeaderKeyDict() objects.
Depending on which object is used and how it is used, one can end up
with such a mismatch.
This commit takes the following steps as a (PROPOSED) solution:
1. Change HeaderKeyDict() to normalize using ".title()" case to
match HeaderEnvironProxy()
2. Replace standard python dictionary objects with HeaderKeyDict()
objects where possible
This gives us an object that normalizes key references to avoid
fixing the code to normalize the string literals.
3. Fix up a few places to use title case string literals to match
the new defaults
Change-Id: Ied56a1df83ffac793ee85e796424d7d20f18f469
Signed-off-by: Peter Portante <peter.portante@redhat.com>
When the number of account/container or container/object replicas are
different, Swift had a few misbehaviors. This commit fixes them.
* On an object PUT/POST/DELETE, if there were 3 object replicas and
only 2 container replicas, then only 2 requests would be made to
object servers. Now, 3 requests will be made, but the third won't
have any X-Container-* headers in it.
* On an object PUT/POST/DELETE, if there were 3 object replicas and 4
container replicas, then only 3/4 container servers would receive
immediate updates; the fourth would be ignored. Now one of the
object servers will receive multiple (comma-separated) values in the
X-Container-* headers and it will attempt to contact both of them.
One side effect is that multiple async_pendings may be written for
updates to the same object. They'll have differing timestamps,
though, so all but the newest will be deleted unread. To trigger
this behavior, you have to have more container replicas than object
replicas, 2 or more of the container servers must be down, and the
headers sent to one object server must reference 2 or more down
container servers; it's unlikely enough and the consequences are so
minor that it didn't seem worth fixing.
The situation with account/containers is analogous, only without the
async_pendings.
Change-Id: I98bc2de93fb6b2346d6de1d764213d7563653e8d
The container and account servers should respond with 406 if the Accept header
isn't satisfiable. This behavior is defined in RFC 2616 section 14.1.
Change-Id: I8a67ccafe33dc70ef4f7794686a54fbc8581f4dc
This change replaces WebOb with a mostly compatible local library,
swift.common.swob. Subtle changes to WebOb's API over the years have been a
huge headache. Swift doesn't even run on the current version.
There are a few incompatibilities to simplify the implementation/interface:
* It only implements the header properties we use. More can be easily added.
* Casts header values to str on assignment.
* Response classes ("HTTPNotFound") are no longer subclasses, but partials
on Response, so things like isinstance no longer work on them.
* Unlike newer webob versions, will never return unicode objects.
Change-Id: I76617a0903ee2286b25a821b3c935c86ff95233f
The WebOb 1.2 support patch introduced a regression where an unknown
format= query used to just default back to plain but now returns a
400. This will break anyone that uses format=text for instance, or
anything else "invalid".
Arguably, it makes the most sense to 400 in the case of an invalid
format requested, but since it is also a backwards compatibility
break, I recommend we keep the previous behavior. Retaining the
previous behavior seems less damaging than enforcing the new
behavior.
Change-Id: I6db015b33a6f3ab239b8cb4a3562ebdba1f76590
Based on PatchSet 3 of https://review.openstack.org/#/c/7569/ , make them to pass all funcional tests with both webob 1.x and 1.2.
The additional following compatibility issues were addressed:
- Until patch for range header issue is merged into official webob release, testRangedGetsWithLWSinHeader() should skip test against webob 1.2
(49c175aec2)
- common.constraints.check_utf8() can accept both utf8 str and unicode.
- To convert unicode to utf-8 str if necessary.
- Making proxy_logging can handle invalid utf-8 str
bug 888371
bug 959881
blueprint webob-support
Change-Id: I00e5fd04cd1653259606a4ffdd4926db3c84c496
This fixes 500 server crashes caused by requests such as:
curl -X__init__ "http://your-swift-object-server:6000/sda1/p/a/c/o"
Fixes bug 1005903
Change-Id: I6c0ad39a29e07ce5f46b0fdbd11a53a9a1010a04
Fixes bug 798268.
Python datetime's isoformat() uses %Y-%m-%dT%H:%M:%S.%f format, but
the miliseconds part is not included when it's zero.
As consequence the compliant ISO 8601 format was not consistent
when performing a GET request over a container (listing objects info).
Change-Id: Ifed3f0adf3eaca47304c142615169bd3f1901631
added try/except around the accept header matcher
to catch the assertionerror that was being thrown
and return a more meaningful message.
Change-Id: I64184be0a40f8696f8e7e3801763d555ec2526dd
Updated eventlet.TimeoutError (deprecated) references to
Timeout and, more importantly, updated many except Exception
clauses to except (Exception, Timeout).
Change-Id: Ib089265551bd20b94c00ea84f11140ccd795d301