59 Commits

Author SHA1 Message Date
Peter Portante
9411a24ba7 Revert "Refactor common/utils methods to common/ondisk"
This reverts commit 7760f41c3ce436cb23b4b8425db3749a3da33d32

Change-Id: I95e57a2563784a8cd5e995cc826afeac0eadbe62
Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-10-07 17:18:09 -04:00
ZhiQiang Fan
f72704fc82 Change OpenStack LLC to Foundation
Change-Id: I7c3df47c31759dbeb3105f8883e2688ada848d58
Closes-bug: #1214176
2013-09-20 01:02:31 +08:00
Peter Portante
7760f41c3c Refactor common/utils methods to common/ondisk
Place all the methods related to on-disk layout and / or configuration
into a new common module that can be shared by the various modules
using the same on-disk layout.

Change-Id: I27ffd4665d5115ffdde649c48a4d18e12017e6a9
Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-09-17 17:32:04 -04:00
Dirk Mueller
3d36a76156 Use Python 3.x compatible except construct
except x,y: was deprected and is removed in Python 3.x.
Use "except x as y:" instead which works in any Python
version >= 2.6.

Change-Id: I7008c74b807340f3457d3a0c8bd0b83f23169d14
2013-09-07 10:50:54 +02:00
Peter Portante
22451b22cb Pep8 final two unit test modules and enforce (12 of 12)
We also fix up any other pep8 failures that snuck in from merges along
the way.

Change-Id: I4ea984780ac2eac458c98fe181684eef4e04beaf
Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-09-04 23:35:46 -04:00
Peter Portante
14037b46e6 Flake8 container test_server.py (11 of 12)
Change-Id: Ie06ed2955838556d5399a49ba3599dfbca4f7512
Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-09-04 22:25:55 -04:00
Jenkins
f2ca932b07 Merge "Fix changes in XML that broke old clients" 2013-09-03 21:10:53 +00:00
Peter Portante
c067abd21e Pep8 unit test modules for hacking and one liners (4 of 12)
Address all the "hacking" lines that are flagged, and all the modules
that just have one item flagged.

Change-Id: I372a4bdf9c7748f73e38c4fd55e5954f1afade5b
Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-09-01 15:12:39 -04:00
Chuck Thier
fc9cd6bf5a Fix changes in XML that broke old clients
Certain old clients that shall not be named do not like some of the
recent changes to the xml output for container listings.  This fixes
those changes.  If we want to change any of the xml output, then we
should do it in another API version.  I'm not sure if any of these
changes in the XML were put in any documentation, but if they were, they
should be changed back.

DocImpact

Change-Id: I983332ca9d620a9a539be26bc6655df9b2ef87ed
2013-08-30 20:09:26 +00:00
Samuel Merritt
61a8a9af18 Refactor path splitting and validation.
The account, container, and object servers all do the same thing at
the top of GET/PUT/etc.: they split the path, validate that the first
two components are a good (device, partition) pair, and return a 400
if they're not. The object server already had a module-local helper
function for this, but the account and container servers just had
duplicate boilerplate code. Now it's all in one common helper.

Change-Id: I9a20d37fc9e1a68b10149a7aa78cb9691fc04ea9
2013-08-27 09:39:02 -07:00
Samuel Merritt
a4f371439b Refactor how we pick listings' content type.
There were a few different places where we had some repeated code to
figure out what format an account or container listing response should
be in (text, JSON, or XML). Now that's been pulled into a single
function.

As part of this, you can now raise HTTPException subclasses in proxy
controllers instead of laboriously plumbing error responses all the
way back up to swift.proxy.server.Application.handle_request(). This
lets us avoid certain ugly patterns, like the one where a method
returns a tuple of (x, y, z, error) and the caller has to see if it
got (value, value, value, None) or (None, None, None, errorvalue). Now
we can just raise the error.

Change-Id: I316873df289160d526487ad116f6fbb9a575e3de
2013-08-16 15:45:45 -07:00
Michael Barton
53345da70e some container serialization cleanup
Have json and xml use common record cleanup code.
Do a somewhat better job of parsing extensions from content-types.
Use a real XML serializer.

Change-Id: I10e14dffd1da590b4fd180b4d33ab5de862e2b55
2013-07-29 17:35:08 -07:00
Samuel Merritt
df7fc9658b Catch swob responses that are raised.
This lets us get rid of some really repetitive exception conversion
code, like everybody that called common.utils.get_param() had to catch
a UnicodeDecodeError and turn that into returning HTTPBadRequest. Now
get_param() just raises HTTPBadRequest directly, and the __call__
methods in the account/container/object servers catch and return
it. All that "except UnicodeDecodeError" stuff goes away.

Refactored the path splitting and device validation in the object
server too.

There are other things that can benefit from this as well, but this
patch is big enough.

Change-Id: I2be96ef757d04bfd6af180cd9c92393c841db21f
2013-07-24 16:59:45 -07:00
Kun Huang
7cd01a63d3 Add 'Z' into isoformat for UTC time
Based on this http://en.wikipedia.org/wiki/ISO_8601#Time_zone_designators.
A isofomt of time need a 'Z' for UTC time zone, or none for local time zone

request on GET /<version>/<account>/<container>?format=json still output a
UTC time without 'Z'

fixes bug #1169287
Change-Id: Ib599b5ec8fd223878ec18df7c1ec8d952fc2630a
2013-07-20 09:52:56 +08:00
Vladimir Vechkanov
bc08215f83 Move replication allow method to decorators
Remove logic of allowed methods list from object, container and account
servers. Instead of it add replicator decorator to utils and use new
decorator for REPLICATE methods in object/account/container servers.
This decorator mark method as special for usfor use only by the
replication.

If the option replication_server is not used, then this mechanism is not
enabled. If the replicaton_server option is set (not None) then the
respective server is a replicator (option value is True) and should use
ONLY the methods marked for replication server using the decorator, or
it is a normal server type and should NOT use methods marked for the
replication server.

Change-Id: I1041b31413cd0c39000317cc57a8c27816e1dfe8
2013-07-12 11:38:17 +04:00
Jenkins
0d24b3e822 Merge "More xml fixes" 2013-06-14 22:37:27 +00:00
gholt
e9d1199887 More xml fixes
subdir elements now have their names quoteattr

Change-Id: I37faf164e339deacb6555db94196135debe22a6c
2013-06-14 16:18:20 +00:00
Samuel Merritt
aff39d75bb Fix 503 on account/container HEAD w/invalid format.
A HEAD request to /v1/a[/c]?format=%FF would result in a 503 since
there was an unhandled UnicodeDecodeError. Now it doesn't.

bug 1190395

Change-Id: I4d8ec677092617391fe154a25f22d919536a72a5
2013-06-12 15:55:47 -07:00
Peter Portante
8d6869a6cc Move parameter checking before disk accesses
For HEAD and GET requests on both containers and accounts APIs we now
perform parameter checking before we access the database. The unit
tests were updated to show that the parameter checking is performed
before accesses are performed.

Change-Id: Ieb753316cdccabf45022e3d83522d87d34aa6b0e
Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-05-29 00:48:53 -04:00
Alex Gaynor
b147c9e221 Made tests use a JSON parser in liue of eval.
The results that are being parsed are really JSON, so using a JSON parser is
more semantically correct; in addition using eval anywhere can set a bad
example.

Change-Id: Idcd55400b2571aba0f2f377cf66a3cbf4d3af960
2013-05-28 07:26:46 -07:00
Sergey Kraynev
ea7858176b Implementation of replication servers
Support separate replication ip address:
- Added new function in utils. This function provides ability
  to select separate IP address for replication service.
- Db_replicator and object replicators were changed.
  Replication process uses new function now.

Replication network parameters:
- Replication network fields (replication_ip, replication_port)
  support was added to device dictionary in swift-ring-builder script.
- Changes were made to support new fields in search, show and set_info
  functions.

Implementation of replication servers:
- Separate replication servers use the same code as normal replication
  servers, but with replication_server parameter = True.  When using a
  separate replication network, the non-replication servers set
  replication_server = False.  When there is no separate replication
  network (the default case), replication_server is not included in the config.

DocImpact
Change-Id: Ie9af5bdcdf9241c355e36053ca4adfe49dc35bd0
Implements: blueprint dedicated-replication-network
2013-04-21 18:14:42 -04:00
Peter Portante
8825c9c74a Enhance log msg to report referer and user-agent
Enhance internally logged messages to report referer and user-agent.

Pass the referering URL and METHOD between internal servers (when
known), and set the user-agent to be the server type (obj-server,
container-server, proxy-server, obj-updater, obj-replicator,
container-updater, direct-client, etc.) with the process PID. In
conjunction with the transaction ID, it helps to track down which PID
from a given system was responsible for initiating the request and
what that server was working on to make this request.

This has been helpful in tracking down interactions between object,
container and account servers.

We also take things a bit further performaing a bit of refactoring to
consolidate calls to transfer_headers() now that we have a helper
method for constructing them.

Finally we performed further changes to avoid header key duplication
due to string literal header key values and the various objects
representing headers for requests and responses. See below for more
details.

====

Header Keys

There seems to be a bit of a problem with the case of the various
string literals used for header keys and the interchangable way
standard Python dictionaries, HeaderKeyDict() and HeaderEnvironProxy()
objects are used.

If one is not careful, a header object of some sort (one that does not
normalize its keys, and that is not necessarily a dictionary) can be
constructed containing header keys which differ only by the case of
their string literals. E.g.:

   { 'x-trans-id': '1234', 'X-Trans-Id': '5678' }

Such an object, when passed to http_connect() will result in an
on-the-wire header where the key values are merged together, comma
separated, that looks something like:

   HTTP_X_TRANS_ID: 1234,5678

For some headers in some contexts, this is behavior is desirable. For
example, one can also use a list of tuples which enumerate the multiple
values a single header should have.

However, in almost all of the contexts used in the code base, this is
not desirable.

This behavior arises from a combination of factors:

   1. Header strings are not constants and different lower-case and
      title-case header strings values are used interchangably in the
      code at times

      It might be worth the effort to make a pass through the code to
      stop using string literals and use constants instead, but there
      are plusses and minuses to doing that, so this was not attempted
      in this effort

   2. HeaderEnvironProxy() objects report their keys in ".title()"
      case, but normalize all other key references to the form
      expected by the Request class's environ field

      swob.Request.headers fields are HeaderEnvironProxy() objects.

   3. HeaderKeyDict() objects report their keys in ".lower()" case,
      and normalize all other key references to ".lower()" case

      swob.Response.headers fields are HeaderKeyDict() objects.

Depending on which object is used and how it is used, one can end up
with such a mismatch.

This commit takes the following steps as a (PROPOSED) solution:

   1. Change HeaderKeyDict() to normalize using ".title()" case to
      match HeaderEnvironProxy()

   2. Replace standard python dictionary objects with HeaderKeyDict()
      objects where possible

      This gives us an object that normalizes key references to avoid
      fixing the code to normalize the string literals.

   3. Fix up a few places to use title case string literals to match
      the new defaults

Change-Id: Ied56a1df83ffac793ee85e796424d7d20f18f469
Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-05-13 17:39:02 +00:00
Greg Lange
44f00a23c1 fixed some minor things in tests that pyflakes complained about
Change-Id: Ifeab56a964630bcf941e932fcbe39e6572e62975
2013-03-26 20:42:26 +00:00
Leah Klearman
60489cdedd additional unit test coverage for container/server.py
Change-Id: Id510cceb88f062f921450a5d25d7c97b7ca5943b
2013-03-22 19:49:47 -07:00
Samuel Merritt
6ff644b945 Allow for multiple X-(Account|Container)-* headers.
When the number of account/container or container/object replicas are
different, Swift had a few misbehaviors. This commit fixes them.

* On an object PUT/POST/DELETE, if there were 3 object replicas and
  only 2 container replicas, then only 2 requests would be made to
  object servers. Now, 3 requests will be made, but the third won't
  have any X-Container-* headers in it.

* On an object PUT/POST/DELETE, if there were 3 object replicas and 4
  container replicas, then only 3/4 container servers would receive
  immediate updates; the fourth would be ignored. Now one of the
  object servers will receive multiple (comma-separated) values in the
  X-Container-* headers and it will attempt to contact both of them.

  One side effect is that multiple async_pendings may be written for
  updates to the same object. They'll have differing timestamps,
  though, so all but the newest will be deleted unread. To trigger
  this behavior, you have to have more container replicas than object
  replicas, 2 or more of the container servers must be down, and the
  headers sent to one object server must reference 2 or more down
  container servers; it's unlikely enough and the consequences are so
  minor that it didn't seem worth fixing.

The situation with account/containers is analogous, only without the
async_pendings.

Change-Id: I98bc2de93fb6b2346d6de1d764213d7563653e8d
2013-01-14 12:38:46 -08:00
Michael Barton
064ee2b583 406 if we can't satisfy Accept
The container and account servers should respond with 406 if the Accept header
isn't satisfiable.  This behavior is defined in RFC 2616 section 14.1.

Change-Id: I8a67ccafe33dc70ef4f7794686a54fbc8581f4dc
2012-12-03 11:42:37 -08:00
Michael Barton
0d1b42007b Set content-type on account/container head
Change-Id: Ib54fa5adb7539bebfbd6644064be9d7f4d7af9db
2012-11-05 12:59:16 -08:00
Michael Barton
5e3e9a882d local WSGI Request and Response classes
This change replaces WebOb with a mostly compatible local library,
swift.common.swob.  Subtle changes to WebOb's API over the years have been a
huge headache.  Swift doesn't even run on the current version.

There are a few incompatibilities to simplify the implementation/interface:
 * It only implements the header properties we use.  More can be easily added.
 * Casts header values to str on assignment.
 * Response classes ("HTTPNotFound") are no longer subclasses, but partials
   on Response, so things like isinstance no longer work on them.
 * Unlike newer webob versions, will never return unicode objects.

Change-Id: I76617a0903ee2286b25a821b3c935c86ff95233f
2012-09-28 14:48:48 -07:00
gholt
de9b81baee Fixes regression with format=somethingelse
The WebOb 1.2 support patch introduced a regression where an unknown
format= query used to just default back to plain but now returns a
400. This will break anyone that uses format=text for instance, or
anything else "invalid".

Arguably, it makes the most sense to 400 in the case of an invalid
format requested, but since it is also a backwards compatibility
break, I recommend we keep the previous behavior. Retaining the
previous behavior seems less damaging than enforcing the new
behavior.

Change-Id: I6db015b33a6f3ab239b8cb4a3562ebdba1f76590
2012-07-31 02:26:39 +00:00
Iryoung Jeong
de4d23c2a5 Adapt Swift for WebOb 1.2
Based on PatchSet 3 of https://review.openstack.org/#/c/7569/ , make them to pass all funcional tests with both webob 1.x and 1.2.

The additional following compatibility issues were addressed:
 - Until patch for range header issue is merged into official webob release, testRangedGetsWithLWSinHeader() should skip test against webob 1.2
(49c175aec2)

 - common.constraints.check_utf8() can accept both utf8 str and unicode.

 - To convert unicode to utf-8 str if necessary.

 - Making proxy_logging can handle invalid utf-8 str

bug 888371
bug 959881

blueprint webob-support

Change-Id: I00e5fd04cd1653259606a4ffdd4926db3c84c496
2012-07-15 00:03:01 +09:00
Ionuț Arțăriși
9f5a6bba1a only allow methods which implement HTTP verbs to be called remotely
This fixes 500 server crashes caused by requests such as:

curl -X__init__ "http://your-swift-object-server:6000/sda1/p/a/c/o"

Fixes bug 1005903

Change-Id: I6c0ad39a29e07ce5f46b0fdbd11a53a9a1010a04
2012-06-04 17:46:17 +02:00
John Dickinson
1ecf5ebba1 updated copyright date for all files
Change-Id: Ifd909d3561c2647770a7e0caa3cd91acd1b4f298
2012-03-19 13:45:34 -05:00
Juan J. Martinez
bb3bfe1dbd Consistent timestamp formatting for last_modified.
Fixes bug 798268.

Python datetime's isoformat() uses %Y-%m-%dT%H:%M:%S.%f format, but
the miliseconds part is not included when it's zero.

As consequence the compliant ISO 8601 format was not consistent
when performing a GET request over a container (listing objects info).

Change-Id: Ifed3f0adf3eaca47304c142615169bd3f1901631
2012-02-02 11:54:15 +00:00
Scott Simpson
dfb9a9f0a3 fixes bug lp 891247
added try/except around the accept header matcher
to catch the assertionerror that was being thrown
and return a more meaningful message.

Change-Id: I64184be0a40f8696f8e7e3801763d555ec2526dd
2011-11-23 11:50:09 -06:00
gholt
dd839048f1 Updated TimeoutError and except Exception refs...
Updated eventlet.TimeoutError (deprecated) references to
Timeout and, more importantly, updated many except Exception
clauses to except (Exception, Timeout).

Change-Id: Ib089265551bd20b94c00ea84f11140ccd795d301
2011-11-07 16:26:03 +00:00
gholt
872420efdb Expiring Objects Support
Please see the doc/source/overview_expiring_objects.rst for
more detail.

Change-Id: I4ab49e731248cf62ce10001016e0c819cc531738
2011-11-01 15:49:00 +00:00
David Goetz
d234461344 merge up to trunk, rollback xml 2011-07-22 12:58:22 -07:00
David Goetz
9be62bd9cd rolling back xml changes bzr merge -r319..318 2011-07-22 10:54:54 -07:00
David Goetz
065318a251 rolling back xml changes bzr merge -r323..322 2011-07-22 10:53:25 -07:00
gholt
498edf3e34 Reset container sync points when the sync-to changes 2011-07-14 20:07:45 +00:00
gholt
7b06c9855c Update to <?xml version="1.1" 2011-07-13 20:03:04 +00:00
gholt
082b324bc3 Update to comply with XML 1.1: No NULLs in names allowed; control chars are converted to entities (ex: &#x4;) 2011-06-17 00:57:00 +00:00
gholt
f68232bf89 Fixed account and container listings to return with charset=utf-8 instead of charset=utf8 2011-06-10 18:36:02 +00:00
Conrad Weidenkeller
68ec704cbc Modified Unit tests for XML change 2011-05-19 06:59:47 +00:00
gholt
cc00bd40e0 Fix tests to cleanup their /tmp dirs 2011-01-24 17:12:38 -08:00
Clay Gerrard
e64e21dd01 pep8 2011-01-19 16:19:43 -06:00
Clay Gerrard
105315dfc4 obsolete PATH_TO_TEST_XFS 2011-01-19 14:18:37 -06:00
Anne Gentle
8823427161 Changed copyright notices on py files and the single rst file with a copyright notice 2011-01-04 17:34:43 -06:00
Michael Barton
5d564a98b0 support text/xml 2010-11-02 16:04:15 +00:00
Michael Barton
cefcd568cd more tests 2010-10-29 10:28:19 +00:00