3288 Commits

Author SHA1 Message Date
Peter Portante
e0147e60d8 Add a unit test to verify proxy logging fields
Also bring unit test coverage to 100% (well, at least every line is
reported as "covered").

Change-Id: I659d0c02008368897b1307a7a5c9aaba73b80588
Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-11-27 16:19:51 -05:00
Jenkins
7ea00f30bc Merge "Reorganize SLO unit tests" 2013-11-27 21:09:41 +00:00
Peter Portante
3c7c355120 Use TCP_NODELAY for created sockets.
Mark Seger at HP has been looking at small objects, 1 and 2 KB size,
and with Rick Jones' help noticed that TCP protocol traces showed
effects from the Nagel algorithm client-to-server and
server-to-client.

This patch just addresses our WSGI server responses, but does not
address out-bound connections from the various servers.

Change-Id: I11f86df1f56fba1c6ab6084dc1f580c395f072dc
Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-11-27 14:07:02 -05:00
Jenkins
4d6a9bfee1 Merge "Some functional tests for static large objects" 2013-11-27 18:20:51 +00:00
Jenkins
8e7925dcd1 Merge "Add HTML reporting for test branch coverage." 2013-11-27 18:00:00 +00:00
Jenkins
a1f8b190dc Merge "Fix bug in obj updater run_once()." 2013-11-27 17:52:49 +00:00
Darrell Bishop
39032c359f Add HTML reporting for test branch coverage.
When including branch coverage results, also generate HTML reports into
a "cover" subdirectory under the directory in which .unittests resides
(i.e. known location at the top of the swift tree).

Change-Id: I493d74f38755f7bf0d7043052585efb27840b238
2013-11-27 12:07:42 -05:00
Jenkins
d8e46eba47 Merge "slightly less early quorum" 2013-11-27 15:59:37 +00:00
Darrell Bishop
0ba071f27c Fix bug in obj updater run_once().
The "not" in front of the ismount() call got accidentally dropped in a
recent change.  This patch adds it back along with a few more tests.

Note that this bug only showed up on an SAIO during probe tests because
I used actually-mounted (virtual) "disks".  So keep that in mind when
building SAIOs for development/testing.

Change-Id: Ia193f3c4b73203605954036863575c22ddab6b03
2013-11-27 00:37:02 -05:00
Jenkins
fa8eef4b3f Merge "bare excepts, as is proper" 2013-11-27 01:57:28 +00:00
Jenkins
2ce0e62c31 Merge "add bare except to catch errors" 2013-11-26 22:57:52 +00:00
John Dickinson
71d1805683 bare excepts, as is proper
Change-Id: Ifd28f6f14a781a67644315690491888161a7250c
2013-11-26 14:39:30 -08:00
Jenkins
9248c91f6d Merge "Use upstream patched Pool.get" 2013-11-26 21:47:31 +00:00
John Dickinson
cdb6cd830a add bare except to catch errors
Change-Id: Ibe78912cf923591bddd6a8cf0e683cd028c9c4e8
2013-11-26 14:03:30 -05:00
Peter Portante
6e313e957d Fix for memcache middleware configuration
The documentation rightly said to use "memcache_max_connections", but
the code was looking for "max_connections", and only looking for it in
proxy-server.conf, not in memcache.conf as a fall back.

This commit brings the code coverage for the memcache middleware to
100%.

Closes-Bug: 1252893
Change-Id: I6ea64baa2f961a09d60b977b40d5baf842449ece
Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-11-26 18:03:33 +00:00
Jenkins
9e25d38611 Merge "Add a DebugLogger for wsgi server tests" 2013-11-26 15:09:14 +00:00
Jenkins
c15f98f3b3 Merge "allow bare excepts in flake8" 2013-11-26 08:26:30 +00:00
Michael Barton
7207926cff slightly less early quorum
The early quorum change has maybe added a little bit too much
eventual to the consistency of requests in Swift, and users can
sometimes get unexpected
results.

This change gives us a knob to turn in finding the right balance,
by adding a timeout where pending requests can finish after quorum
is achieved.

Change-Id: Ife91aaa8653e75b01313bbcf19072181739e932c
2013-11-25 21:25:55 +00:00
John Dickinson
fa1f7d9420 Use upstream patched Pool.get
This works around an eventlet bug in eventlet 0.9.16.
This version properly keeps track of pool size accounting, and
therefore doesn't let the pool grow without bound. This patched
version is the result of commit
f5e5b2bda7b442f0262ee1084deefcc5a1cc0694 in eventlet and is
documented at https://bitbucket.org/eventlet/eventlet/issue/91

This patch includes full test coverage of the back-ported code, even
when the actually-installed eventlet is newer.

This fixes bug #1254119

Change-Id: I075bb5e40e08571d52fe17fcc3fa0e25be5befed
2013-11-25 15:56:38 -05:00
Alex Gaynor
91deed871b Use a more portable errno in tests.
ELIBBAD doesn't exist on OS X. The exact value we use here isn't
important, so use something more portable.

Change-Id: Id03dc1773f416a94bbd14ad31b2b2a70f16b9a51
2013-11-24 09:11:04 -06:00
Jenkins
5989849512 Merge "Per device replication_lock" 2013-11-22 23:04:33 +00:00
Jenkins
20c445cc60 Merge "add an "inline" query parameter to tempurl" 2013-11-22 22:38:22 +00:00
Jenkins
901bd044c3 Merge "Remove obsolete future imports" 2013-11-22 22:25:13 +00:00
Jenkins
14c98c43be Merge "Get retry." 2013-11-22 21:50:19 +00:00
Jenkins
2cc46d70bf Merge "Simple fix for proxy-logging empty field handling" 2013-11-22 21:50:12 +00:00
gholt
c859ebf5ce Per device replication_lock
New replication_one_per_device (True by default)
that restricts incoming REPLICATION requests to
one per device, replication_currency allowing.

Also has replication_lock_timeout (15 by default)
to control how long a request will wait to obtain
a replication device lock before giving up.

This should be very useful in that you can be
assured any concurrent REPLICATION requests are
each writing to distinct devices. If you have 100
devices on a server, you can set
replication_concurrency to 100 and be confident
that, even if 100 replication requests were
executing concurrently, they'd each be writing to
separate devices. Before, all 100 could end up
writing to the same device, bringing it to a
horrible crawl.

NOTE: This is only for ssync replication. The
current default rsync replication still has the
potentially horrible behavior.

Change-Id: I36e99a3d7e100699c76db6d3a4846514537ff685
2013-11-22 21:40:29 +00:00
Jenkins
09e9e48c79 Merge "Fixed locale test in the presence of LANGUAGE" 2013-11-22 21:25:09 +00:00
David Goetz
f5648638ee Get retry.
If a source times out on read try another one of them with a
modified range.  There had to be a lot of moved around code
to get this working but it should all make sense.

Change-Id: Ieaf045690a8823927a6f38098a95b37a4d4adb70
2013-11-22 21:03:11 +00:00
Samuel Merritt
b5b0b78fc7 Remove obsolete future imports
The with statement has been standard since Python 2.5, so we can get
rid of these imports.

Change-Id: I280971c3d8c01e94cc2c17cacaedcbe9d9c8a3c3
2013-11-22 12:23:58 -08:00
Peter Portante
4cb5e2f456 Simple fix for proxy-logging empty field handling
Change-Id: Ia135575bd30a0bc04a2105291e68a6d18c7a3047
Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-11-22 14:46:38 -05:00
Peter Portante
4ed1c8473f Handle optional arguments for run_forever()
All the other daemons do this, and since the out deamon wrapper
scripts pass all the command line options through directly, seems
simple enough to handle them by ignoring.

This is also applied to run_once().

Change-Id: I1df83bdf78f0dc3d911019f67f78301967b5da72
Closes-Bug: #1253891
Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-11-22 14:39:30 -05:00
Jenkins
c850580566 Merge "Fix DB locked error on commit" 2013-11-22 19:17:53 +00:00
John Dickinson
b9efe1cd46 add an "inline" query parameter to tempurl
Giving the inline query parameter will cause the tempurl
response to be given a "Content-Disposition: inline" header,
regardless of other query parameters or metadata. This allows
easy in-line viewing, eg in browsers.

DocImpact

Change-Id: Icd5c544d6a749d4f58e8a921968f4e432a2185db
2013-11-22 09:46:40 -08:00
Jenkins
98e5371013 Merge "Do not format messages before they are logged" 2013-11-22 09:55:05 +00:00
Jenkins
62b693d133 Merge "Use POST in bulk-delete" 2013-11-22 09:11:05 +00:00
Jenkins
34eb76fc57 Merge "Import filter and app into namespace correctly" 2013-11-22 04:27:27 +00:00
Peter Portante
a410730a2b Do not format messages before they are logged
Change-Id: Ia645c9eca47b7f404d9b987f68a96b4744031e9d
Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-11-21 22:34:36 -05:00
Jenkins
6b189e8ea9 Merge "Fix test to work with mock 0.8.0" 2013-11-22 03:08:36 +00:00
Jenkins
c39cef2046 Merge "Fix UnboundLocalError on container PUT" 2013-11-22 02:22:42 +00:00
Jenkins
34c91b377a Merge "Fix quarantine, error counts in audit logs" 2013-11-22 02:22:06 +00:00
Steven Lang
62b87fca20 Fixed locale test in the presence of LANGUAGE
According to GNU documents, the priority order of language variables
is LANGUAGE, LC_ALL, LC_*, LANG. Therefore, if LANGUAGE is set, it
overrides the LC_ALL setting from the test. An empty value is ignored,
and setting it to empty is easier to deal with than just deleting the
variable.

Also fixed the Google translate fail esperanto grammar.

Fixes bug 1235058

Change-Id: Ic97b90dfc21997e19cc473250794a9b3c526beb5
2013-11-21 15:16:10 -08:00
Jenkins
15085349a2 Merge "Fix probe test" 2013-11-21 21:20:29 +00:00
Steven Lang
700479bd67 Fix DB locked error on commit
This bug was introduced in ef7f9e27; while moving timeout for execute
to the cursor wrapper, commit was moved as well; however commit is
purely a connection method, only execute is passed on to a cursor.
Added unit tests to check both methods for correct timeouts.

This manifested in a test failure as:
ERROR __call__ error with POST /sdb1/418/AUTH_d1c4b610b16a48de83219c696261009c/TestContainer-tempest-1572414684 :
Traceback (most recent call last):
  File "/opt/stack/new/swift/swift/container/server.py", line 486, in __call__
    res = method(req)
  File "/opt/stack/new/swift/swift/common/utils.py", line 1915, in wrapped
    return func(*a, **kw)
  File "/opt/stack/new/swift/swift/common/utils.py", line 687, in _timing_stats
    resp = func(ctrl, *args, **kwargs)
  File "/opt/stack/new/swift/swift/container/server.py", line 464, in POST
    broker.update_metadata(metadata)
  File "/opt/stack/new/swift/swift/common/db.py", line 677, in update_metadata
    conn.commit()
OperationalError: database is locked (txn: tx5065394f288740e69fcec-00528e184e)

Change-Id: I269b133fac53d4792d21b62f801cc0c0ccf337ea
2013-11-21 12:02:33 -08:00
Peter Portante
4f6d89ab51 Import filter and app into namespace correctly
The module used to simply "import swift", and then reference
the classes:

    swift.common.middleware.catch_errors.CatchErrorsMiddleware
    swift.proxy.server.Application

in order to very that the WSGI services loaded the proper filters and
apps.

However, those references only happen to work, as the WSGI loading
would properly import the rest of the path so that the namespace
reference would be okay. If the WSGI configuration were to change, or
if the behavior of WSGI broke, instead of of seeing the actual failure
condition, a module attribute error would result instead:

    AttributeError: 'module' object has no attribute 'middleware'

The referenced names are now properly imported with this change to
avoid misleading error conditions.

Change-Id: Ifff4271bc5be1136bf17e4e5b291b01033d608db
Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-11-21 13:11:02 -05:00
Jenkins
a73d5e4921 Merge "Handle COPY verb in account quota middleware" 2013-11-21 10:21:43 +00:00
Clay Gerrard
9e80fd45a0 Add a DebugLogger for wsgi server tests
Change-Id: Ifd2528be443ba3879bf4921f6c5f4ef31f29044b
2013-11-21 01:35:58 -08:00
Jenkins
068b52cc9c Merge "Replace httplib with bufferedhttp in sphinx docs" 2013-11-21 08:37:17 +00:00
Jenkins
a015c3efb4 Merge "minor fix to unittest fake error" 2013-11-21 08:37:13 +00:00
Jenkins
b6727dbb49 Merge "Use utils.ismount in place of os.path.ismount" 2013-11-21 08:37:10 +00:00
Jenkins
37220086d7 Merge "improve docs in etc/dispersion.conf-sample" 2013-11-21 03:02:08 +00:00