Commit Graph

3288 Commits

Author SHA1 Message Date
e0147e60d8 Add a unit test to verify proxy logging fields
Also bring unit test coverage to 100% (well, at least every line is
reported as "covered").

Change-Id: I659d0c02008368897b1307a7a5c9aaba73b80588
Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-11-27 16:19:51 -05:00
7ea00f30bc Merge "Reorganize SLO unit tests" 2013-11-27 21:09:41 +00:00
3c7c355120 Use TCP_NODELAY for created sockets.
Mark Seger at HP has been looking at small objects, 1 and 2 KB size,
and with Rick Jones' help noticed that TCP protocol traces showed
effects from the Nagel algorithm client-to-server and
server-to-client.

This patch just addresses our WSGI server responses, but does not
address out-bound connections from the various servers.

Change-Id: I11f86df1f56fba1c6ab6084dc1f580c395f072dc
Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-11-27 14:07:02 -05:00
4d6a9bfee1 Merge "Some functional tests for static large objects" 2013-11-27 18:20:51 +00:00
8e7925dcd1 Merge "Add HTML reporting for test branch coverage." 2013-11-27 18:00:00 +00:00
a1f8b190dc Merge "Fix bug in obj updater run_once()." 2013-11-27 17:52:49 +00:00
39032c359f Add HTML reporting for test branch coverage.
When including branch coverage results, also generate HTML reports into
a "cover" subdirectory under the directory in which .unittests resides
(i.e. known location at the top of the swift tree).

Change-Id: I493d74f38755f7bf0d7043052585efb27840b238
2013-11-27 12:07:42 -05:00
d8e46eba47 Merge "slightly less early quorum" 2013-11-27 15:59:37 +00:00
0ba071f27c Fix bug in obj updater run_once().
The "not" in front of the ismount() call got accidentally dropped in a
recent change.  This patch adds it back along with a few more tests.

Note that this bug only showed up on an SAIO during probe tests because
I used actually-mounted (virtual) "disks".  So keep that in mind when
building SAIOs for development/testing.

Change-Id: Ia193f3c4b73203605954036863575c22ddab6b03
2013-11-27 00:37:02 -05:00
fa8eef4b3f Merge "bare excepts, as is proper" 2013-11-27 01:57:28 +00:00
2ce0e62c31 Merge "add bare except to catch errors" 2013-11-26 22:57:52 +00:00
71d1805683 bare excepts, as is proper
Change-Id: Ifd28f6f14a781a67644315690491888161a7250c
2013-11-26 14:39:30 -08:00
9248c91f6d Merge "Use upstream patched Pool.get" 2013-11-26 21:47:31 +00:00
cdb6cd830a add bare except to catch errors
Change-Id: Ibe78912cf923591bddd6a8cf0e683cd028c9c4e8
2013-11-26 14:03:30 -05:00
6e313e957d Fix for memcache middleware configuration
The documentation rightly said to use "memcache_max_connections", but
the code was looking for "max_connections", and only looking for it in
proxy-server.conf, not in memcache.conf as a fall back.

This commit brings the code coverage for the memcache middleware to
100%.

Closes-Bug: 1252893
Change-Id: I6ea64baa2f961a09d60b977b40d5baf842449ece
Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-11-26 18:03:33 +00:00
9e25d38611 Merge "Add a DebugLogger for wsgi server tests" 2013-11-26 15:09:14 +00:00
c15f98f3b3 Merge "allow bare excepts in flake8" 2013-11-26 08:26:30 +00:00
7207926cff slightly less early quorum
The early quorum change has maybe added a little bit too much
eventual to the consistency of requests in Swift, and users can
sometimes get unexpected
results.

This change gives us a knob to turn in finding the right balance,
by adding a timeout where pending requests can finish after quorum
is achieved.

Change-Id: Ife91aaa8653e75b01313bbcf19072181739e932c
2013-11-25 21:25:55 +00:00
fa1f7d9420 Use upstream patched Pool.get
This works around an eventlet bug in eventlet 0.9.16.
This version properly keeps track of pool size accounting, and
therefore doesn't let the pool grow without bound. This patched
version is the result of commit
f5e5b2bda7b442f0262ee1084deefcc5a1cc0694 in eventlet and is
documented at https://bitbucket.org/eventlet/eventlet/issue/91

This patch includes full test coverage of the back-ported code, even
when the actually-installed eventlet is newer.

This fixes bug #1254119

Change-Id: I075bb5e40e08571d52fe17fcc3fa0e25be5befed
2013-11-25 15:56:38 -05:00
91deed871b Use a more portable errno in tests.
ELIBBAD doesn't exist on OS X. The exact value we use here isn't
important, so use something more portable.

Change-Id: Id03dc1773f416a94bbd14ad31b2b2a70f16b9a51
2013-11-24 09:11:04 -06:00
5989849512 Merge "Per device replication_lock" 2013-11-22 23:04:33 +00:00
20c445cc60 Merge "add an "inline" query parameter to tempurl" 2013-11-22 22:38:22 +00:00
901bd044c3 Merge "Remove obsolete future imports" 2013-11-22 22:25:13 +00:00
14c98c43be Merge "Get retry." 2013-11-22 21:50:19 +00:00
2cc46d70bf Merge "Simple fix for proxy-logging empty field handling" 2013-11-22 21:50:12 +00:00
c859ebf5ce Per device replication_lock
New replication_one_per_device (True by default)
that restricts incoming REPLICATION requests to
one per device, replication_currency allowing.

Also has replication_lock_timeout (15 by default)
to control how long a request will wait to obtain
a replication device lock before giving up.

This should be very useful in that you can be
assured any concurrent REPLICATION requests are
each writing to distinct devices. If you have 100
devices on a server, you can set
replication_concurrency to 100 and be confident
that, even if 100 replication requests were
executing concurrently, they'd each be writing to
separate devices. Before, all 100 could end up
writing to the same device, bringing it to a
horrible crawl.

NOTE: This is only for ssync replication. The
current default rsync replication still has the
potentially horrible behavior.

Change-Id: I36e99a3d7e100699c76db6d3a4846514537ff685
2013-11-22 21:40:29 +00:00
09e9e48c79 Merge "Fixed locale test in the presence of LANGUAGE" 2013-11-22 21:25:09 +00:00
f5648638ee Get retry.
If a source times out on read try another one of them with a
modified range.  There had to be a lot of moved around code
to get this working but it should all make sense.

Change-Id: Ieaf045690a8823927a6f38098a95b37a4d4adb70
2013-11-22 21:03:11 +00:00
b5b0b78fc7 Remove obsolete future imports
The with statement has been standard since Python 2.5, so we can get
rid of these imports.

Change-Id: I280971c3d8c01e94cc2c17cacaedcbe9d9c8a3c3
2013-11-22 12:23:58 -08:00
4cb5e2f456 Simple fix for proxy-logging empty field handling
Change-Id: Ia135575bd30a0bc04a2105291e68a6d18c7a3047
Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-11-22 14:46:38 -05:00
4ed1c8473f Handle optional arguments for run_forever()
All the other daemons do this, and since the out deamon wrapper
scripts pass all the command line options through directly, seems
simple enough to handle them by ignoring.

This is also applied to run_once().

Change-Id: I1df83bdf78f0dc3d911019f67f78301967b5da72
Closes-Bug: #1253891
Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-11-22 14:39:30 -05:00
c850580566 Merge "Fix DB locked error on commit" 2013-11-22 19:17:53 +00:00
b9efe1cd46 add an "inline" query parameter to tempurl
Giving the inline query parameter will cause the tempurl
response to be given a "Content-Disposition: inline" header,
regardless of other query parameters or metadata. This allows
easy in-line viewing, eg in browsers.

DocImpact

Change-Id: Icd5c544d6a749d4f58e8a921968f4e432a2185db
2013-11-22 09:46:40 -08:00
98e5371013 Merge "Do not format messages before they are logged" 2013-11-22 09:55:05 +00:00
62b693d133 Merge "Use POST in bulk-delete" 2013-11-22 09:11:05 +00:00
34eb76fc57 Merge "Import filter and app into namespace correctly" 2013-11-22 04:27:27 +00:00
a410730a2b Do not format messages before they are logged
Change-Id: Ia645c9eca47b7f404d9b987f68a96b4744031e9d
Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-11-21 22:34:36 -05:00
6b189e8ea9 Merge "Fix test to work with mock 0.8.0" 2013-11-22 03:08:36 +00:00
c39cef2046 Merge "Fix UnboundLocalError on container PUT" 2013-11-22 02:22:42 +00:00
34c91b377a Merge "Fix quarantine, error counts in audit logs" 2013-11-22 02:22:06 +00:00
62b87fca20 Fixed locale test in the presence of LANGUAGE
According to GNU documents, the priority order of language variables
is LANGUAGE, LC_ALL, LC_*, LANG. Therefore, if LANGUAGE is set, it
overrides the LC_ALL setting from the test. An empty value is ignored,
and setting it to empty is easier to deal with than just deleting the
variable.

Also fixed the Google translate fail esperanto grammar.

Fixes bug 1235058

Change-Id: Ic97b90dfc21997e19cc473250794a9b3c526beb5
2013-11-21 15:16:10 -08:00
15085349a2 Merge "Fix probe test" 2013-11-21 21:20:29 +00:00
700479bd67 Fix DB locked error on commit
This bug was introduced in ef7f9e27; while moving timeout for execute
to the cursor wrapper, commit was moved as well; however commit is
purely a connection method, only execute is passed on to a cursor.
Added unit tests to check both methods for correct timeouts.

This manifested in a test failure as:
ERROR __call__ error with POST /sdb1/418/AUTH_d1c4b610b16a48de83219c696261009c/TestContainer-tempest-1572414684 :
Traceback (most recent call last):
  File "/opt/stack/new/swift/swift/container/server.py", line 486, in __call__
    res = method(req)
  File "/opt/stack/new/swift/swift/common/utils.py", line 1915, in wrapped
    return func(*a, **kw)
  File "/opt/stack/new/swift/swift/common/utils.py", line 687, in _timing_stats
    resp = func(ctrl, *args, **kwargs)
  File "/opt/stack/new/swift/swift/container/server.py", line 464, in POST
    broker.update_metadata(metadata)
  File "/opt/stack/new/swift/swift/common/db.py", line 677, in update_metadata
    conn.commit()
OperationalError: database is locked (txn: tx5065394f288740e69fcec-00528e184e)

Change-Id: I269b133fac53d4792d21b62f801cc0c0ccf337ea
2013-11-21 12:02:33 -08:00
4f6d89ab51 Import filter and app into namespace correctly
The module used to simply "import swift", and then reference
the classes:

    swift.common.middleware.catch_errors.CatchErrorsMiddleware
    swift.proxy.server.Application

in order to very that the WSGI services loaded the proper filters and
apps.

However, those references only happen to work, as the WSGI loading
would properly import the rest of the path so that the namespace
reference would be okay. If the WSGI configuration were to change, or
if the behavior of WSGI broke, instead of of seeing the actual failure
condition, a module attribute error would result instead:

    AttributeError: 'module' object has no attribute 'middleware'

The referenced names are now properly imported with this change to
avoid misleading error conditions.

Change-Id: Ifff4271bc5be1136bf17e4e5b291b01033d608db
Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-11-21 13:11:02 -05:00
a73d5e4921 Merge "Handle COPY verb in account quota middleware" 2013-11-21 10:21:43 +00:00
9e80fd45a0 Add a DebugLogger for wsgi server tests
Change-Id: Ifd2528be443ba3879bf4921f6c5f4ef31f29044b
2013-11-21 01:35:58 -08:00
068b52cc9c Merge "Replace httplib with bufferedhttp in sphinx docs" 2013-11-21 08:37:17 +00:00
a015c3efb4 Merge "minor fix to unittest fake error" 2013-11-21 08:37:13 +00:00
b6727dbb49 Merge "Use utils.ismount in place of os.path.ismount" 2013-11-21 08:37:10 +00:00
37220086d7 Merge "improve docs in etc/dispersion.conf-sample" 2013-11-21 03:02:08 +00:00