4841 Commits

Author SHA1 Message Date
Jenkins
e354f897c3 Merge "Add container sync probe test to SAIO default set" 2015-08-25 20:25:01 +00:00
Jenkins
daf448f490 Merge "Fix 500 for bogus Range request to 0-byte object." 2015-08-25 13:07:57 +00:00
Jenkins
90a1d650b6 Merge "pep8: Fix usage of the l10n _('...') function" 2015-08-25 11:04:08 +00:00
Jenkins
ca172887a6 Merge "Fix use of delimiter in account listings" 2015-08-25 11:03:59 +00:00
Jenkins
f7c235513c Merge "Follow up patch to fix a multiline import NITPIC" 2015-08-25 10:28:47 +00:00
Jenkins
d703a532cb Merge "Use correct Storage-Policy header for REPLICATE requests" 2015-08-25 08:30:57 +00:00
Jenkins
75664055b2 Merge "Minor cleanup handoff mode warnings" 2015-08-25 08:11:00 +00:00
Jenkins
e8b0461a34 Merge "New troubleshooting case in documentation." 2015-08-25 08:10:55 +00:00
Jenkins
3576176167 Merge "Fix typo of a comment in replicator" 2015-08-25 05:49:21 +00:00
Jenkins
3ce602d7fb Merge "Quorum on durable response is too low" 2015-08-25 04:20:08 +00:00
Matthew Oliver
6a35d479e8 Follow up patch to fix a multiline import NITPIC
This change cleans up test/unit/obj/test_replicator.py's imports
to use only 1 version of multiline import syntaxes (' \' vs '()').
I don't really mind which, but we should be consistant, at least
in the same file.

This is a follow up for patch 215857.

Change-Id: Ie2d328c25865b19092c493981a803ee246a9d7a5
2015-08-25 11:31:59 +10:00
Hisashi Osanai
edde5584af Fix typo of a comment in replicator
The typo was introduced by patch 138342 (sorry) so I fix it.

Change-Id: Id5126802d281ef7ee9be128bd2152c0d2584160e
2015-08-25 07:52:18 +09:00
Tim Burke
6151c6c45d Fix use of delimiter in account listings
Previously, account listings that used the delimiter query param could
omit some containers if they ended with the character that follows the
delimiter.

See If196e3075612b121ef8da4a9128167d00a248c27 for the corresponding fix
for container listings.

Change-Id: I57fcb97e51f653f5f4e306a632fcb3a0fb148c4e
2015-08-24 14:33:57 -07:00
Jenkins
e196fc5561 Merge "Fix the missing SLO state on fast-post" 2015-08-24 20:22:11 +00:00
Clay Gerrard
a38f63e1c6 Use correct Storage-Policy header for REPLICATE requests
Under some concurrency the object-replicator could potentially send the
wrong X-Backed-Storage-Policy-Index header to it's partner nodes during
replication if there were multiple storage policies on the same node
because of a race where multiple jobs being processed concurrently would
mutate some shared state on the ObjectReplicator instance.

Instead of using shared stated on the ObjectReplicator instance when
mutating the default headers send with REPLICATION requests each job
will copy them into a local where they can safely be updated.

Change-Id: I5522db57af7e308b1f9d4181f14ea14e386a71fd
2015-08-24 11:20:02 -07:00
Jenkins
2d41ff7b45 Merge "Enable Object Replicator's failure count in recon" 2015-08-24 07:32:08 +00:00
Kota Tsuyuzaki
4500ff340f Fix the missing SLO state on fast-post
When using fast-post and POST (i.e. metadata update) is requested to
a SLO manifest files, current Swift drops the 'X-Static-Large-Object'
header from the existing metadata. It results in breaking the SLO
state because the manifest missing the 'X-Static-Large-Object' metadata
will be maintained as a normal files.

This patch fixes object-server to keep the existing
'X-Static-Large-Object' flag and then keep the SLO state.

Change-Id: Ib1eb569071372c322dd105c52baeeb094003291e
Closes-bug: #1453807
2015-08-23 04:38:33 -07:00
Eran Rom
69e7424d3c Add container sync probe test to SAIO default set
SAIO Configuration and documentation changes enabling to run the
container sync probe test by default

Change-Id: Iccf59533d0d4fe72549d318339ab125d04dde006
Related-Bug: #1476623
2015-08-22 22:50:50 +03:00
Clay Gerrard
8b1df9918b Minor cleanup handoff mode warnings
* message is a little clearer
  * test is a little stronger

Change-Id: I745cde7f4a46dafc80ab42d39e6ccc92aa3b746e
2015-08-21 18:43:41 -07:00
Carlos Cavanna
a1ceab5a92 New troubleshooting case in documentation.
Added a new troubleshooting case for the "First Contribution to Swift"
documentation page.

Change-Id: I182ba702b49b28409fe56becae93326e5f63dcd0
2015-08-21 14:14:31 -04:00
Pradeep Kumar Singh
ab163702de Emit warning log in object replicator
When the object-replicator encounters handoffs_first and
handoff_delete options as enabled it should emit a log
warning indicating that it should be changed back to the
default before the next "normal" rebalance.

Closes-Bug: #1457262

Change-Id: If9dc2796c18ed3cf13da920831e2d5c2ae9f12a0
2015-08-21 02:47:04 +00:00
Samuel Merritt
be66aa8e76 Fix 500 for bogus Range request to 0-byte object.
The proxy was trying to pop a byterange off a Range header that didn't
contain syntactically-valid byteranges. This worked about as well as
you'd expect. Now we detect the bogus value and remove the header
entirely.

Change-Id: I24b92f900d33ec79880c7db2870378489d5a6810
2015-08-20 13:24:38 -07:00
Jenkins
ef8f14f2ba Merge "Add OpenStack release names to changelog" 2015-08-20 16:48:04 +00:00
Samuel Merritt
47dc31940d Add OpenStack release names to changelog
I can never remember the mapping of real version numbers to the
OpenStack names.

Change-Id: Ib7c5ae4ff2a33018364698eb34f8df1622719fab
2015-08-19 19:34:29 -07:00
Victor Stinner
c7eb589c6c pep8: Fix usage of the l10n _('...') function
Fix the pep8 warning H702 "Formatting operation should be outside of
localization method call".

For the logger, pass parameters as indexed parameters instead of using
the string str%args operator, the logger is more reliable in case of
formatting error.

Change-Id: If418bc155f6a6c0a00f63e3d87ebe4addf4aae55
2015-08-19 17:12:51 -07:00
Victor Stinner
7bea148d2f pep8: replace deprecated calls to assert_()
The TestCase.assert_() has been deprecated in Python 2.7. Replace it
with assertTrue() or even better methods (assertIn, assertNotIn,
assertIsInstance) which provide better error messages.

Change-Id: I21c730351470031a2dabe5238693095eabdb8964
2015-08-19 12:05:01 -07:00
Victor Stinner
8aaacbf88d pep8: Fix hacking H232 warnings (octal)
Fix warnings "H232: Python 3.x incompatible octal 000001234 should be
written as 0o1234".

Change-Id: I9a7bbb034357783885ac3e18fe1e9e32a5951616
2015-08-19 12:05:01 -07:00
Jenkins
a19443de3b Merge "move global statement up a few lines" 2015-08-19 14:33:41 +00:00
Jenkins
2a1558a907 Merge "Keep user id and project id in subrequests env" 2015-08-19 10:06:22 +00:00
Jenkins
1f0efee6ec Merge "Doc instructions for post rebase steps" 2015-08-19 02:01:00 +00:00
Jenkins
6444f9b16b Merge "Make swift-ring-builder filename usage more consistent" 2015-08-19 01:20:57 +00:00
Jenkins
446a2b3606 Merge "Add a probetest for HUP/reload" 2015-08-18 18:15:14 +00:00
Clément Contini
d6267c1f95 Keep user id and project id in subrequests env
Keep HTTP_X_USER_ID and HTTP_X_PROJECT_ID to be available as
user_id and project_id in storage.objects.outgoing.bytes in
ceilometer when downloading a multipart object.

Change-Id: I0f4734f021e5d6e84d48ed9bebeb321d7a9590ad
Closes-Bug: #1477283
2015-08-18 16:31:51 +00:00
John Dickinson
eaa006464c move global statement up a few lines
Change-Id: I190d2d530c6c0525d988cc88b0965b48a291fffb
2015-08-17 22:13:42 -07:00
Hamdi Roumani
1a81cda8b7 Doc instructions for post rebase steps
Improve the 'first_contribution_swift' by adding instructions for how to
rebuild swift following a rebase.

Change-Id: If5c91dc4e1e8d1712bbd8b326c675967fb4b8c15
2015-08-18 00:03:10 -04:00
Hisashi Osanai
79ba4a8598 Enable Object Replicator's failure count in recon
This patch makes the count of object replication failure in recon.
And "failure_nodes" is added to Account Replicator and
Container Replicator.

Recon shows the count of object repliction failure as follows:
$ curl http://<ip>:<port>/recon/replication/object
{
    "replication_last": 1416334368.60865,
    "replication_stats": {
        "attempted": 13346,
        "failure": 870,
	"failure_nodes": {
            "192.168.0.1": {"sdb1": 3},
            "192.168.0.2": {"sdb1": 851,
                            "sdc1": 1,
                            "sdd1": 8},
            "192.168.0.3": {"sdb1": 3,
                            "sdc1": 4}
	},
        "hashmatch": 0,
        "remove": 0,
        "rsync": 0,
        "start": 1416354240.9761429,
        "success": 1908
    },
    "replication_time": 2316.5563162644703,
    "object_replication_last": 1416334368.60865,
    "object_replication_time": 2316.5563162644703
}

Note that 'object_replication_last' and 'object_replication_time' are
considered to be transitional and will be removed in the subsequent
releases. Use 'replication_last' and 'replication_time' instead.

Additionaly this patch adds the count in swift-recon and it will be
showed as follows:
$ swift-recon object -r
========================================================================
=======
--> Starting reconnaissance on 4 hosts
========================================================================
=======
[2014-11-27 16:14:09] Checking on replication
[replication_failure] low: 0, high: 0, avg: 0.0, total: 0, Failed: 0.0%,
no_result: 0, reported: 4
[replication_success] low: 3, high: 3, avg: 3.0, total: 12,
Failed: 0.0%, no_result: 0, reported: 4
[replication_time] low: 0, high: 0, avg: 0.0, total: 0, Failed: 0.0%,
no_result: 0, reported: 4
[replication_attempted] low: 1, high: 1, avg: 1.0, total: 4,
Failed: 0.0%, no_result: 0, reported: 4
Oldest completion was 2014-11-27 16:09:45 (4 minutes ago) by
192.168.0.4:6002.
Most recent completion was 2014-11-27 16:14:19 (-10 seconds ago) by
192.168.0.1:6002.
========================================================================
=======

In case there is a cluster which has servers, a server runs with this
patch and the other servers run without this patch. If swift-recon
executes on the server which runs with this patch, there are unnecessary
information on the output such as [failure], [success] and [attempted].
Because other servers which run without this patch are not able to
send a response with information that this patch needs.
Therefore once you apply this patch, you also apply this patch to other
servers before you execute swift-recon.

DocImpact
Change-Id: Iecd33655ae2568482833131f422679996c374d78
Co-Authored-By: Kenichiro Matsuda <matsuda_kenichi@jp.fujitsu.com>
Co-Authored-By: Brian Cline <bcline@softlayer.com>
Implements: blueprint enable-object-replication-failure-in-recon
2015-08-18 11:40:02 +09:00
Jenkins
617c6b0107 Merge "Time synchronization check in recon." 2015-08-18 01:21:22 +00:00
Christian Schwede
eeb0fa40a1 Make swift-ring-builder filename usage more consistent
Sometimes the given argument is internally altered and another filename is used
without a note to the operator. Even worse, a given .ring.gz filename is
sometimes written out as builder file, without updating the corresponding
.builder file.

There is already a method to parse the given argv and return the name of the
builder and ring file. However, it's rarely used and no warning is given to the
user if it is altered. This patch uses the already parsed builder and ring file
name instead of argv[1], and also adds a note to the user if the used filename
is differently to the one given as argument.

Closes-Bug: 1482096
Change-Id: I2f8ef23aeab8b07caaa799f7dcd57e684b4b2ad2
2015-08-17 16:30:49 -05:00
Bill Huber
b75d2a4e37 Quorum on durable response is too low
Increase the .durable quorum from 2 to "parity + 1" to guarantee
that we will never fail to rebuild an object.  Otherwise, with
low durable responses back (< parity + 1), the putter objects
return with failed attribute set to true, thereby failing the
rebuild of fragments for an object.

Change-Id: I80d666f61273e589d0990baa78fd657b3470785d
Closes-Bug: 1484565
2015-08-17 16:05:56 -05:00
Kai Zhang
fa35e38c9f Fix some minor typos
Fixed some typos in function name and comments.

Change-Id: Ida76ab4b331a51b71e57650702acc136e66ba4b2
2015-08-14 16:49:41 -07:00
Jenkins
a2c0e6f687 Merge "Adding bandit for security static analysis testing in swift" 2015-08-12 20:55:16 +00:00
Jenkins
37469babac Merge "Test that get_hashes ignores only removed hash dir" 2015-08-12 16:06:46 +00:00
Jenkins
810af1b73f Merge "Fix shebang of commands" 2015-08-12 11:12:35 +00:00
Samuel Merritt
7064706b27 Fix 500 in versioned writes with bad Destination
When this code lived in the proxy, it was protected by an "except
HTTPException" clause in proxy.Application.handle_request(). When it
moved to middleware, it lost that, and then things like
constraints.check_name_format that raised HTTPException would cause
500s. The HTTPException would make it all the way out to catch_errors
and get translated to a 500.

This commit just wraps a couple try/excepts around the bits in
versioned writes that can raise HTTPException. I tried to make it use
wsgify so I could get that for free, but that wound up being a real
pain because env/start_response are plumbed through pretty much the
whole versioned-writes middleware.

Closes-Bug: 1483705

Change-Id: Ife165bf709e64f313ed07c779b21914045e51f25
2015-08-12 04:35:20 +00:00
Jenkins
eb62f1b379 Merge "Speed up reaper for a big account delete and some better error handling" 2015-08-11 11:22:19 +00:00
Jenkins
f644eaef7d Merge "test/unit: Replace python print operator with print function (pep H233, py33)" 2015-08-11 09:49:50 +00:00
Jenkins
1894914577 Merge "pep8 fix: assertEquals -> assertEqual" 2015-08-11 08:38:03 +00:00
Jenkins
57791b6cd2 Merge "+Document method to avoid rsync filling root drive" 2015-08-11 08:27:17 +00:00
Jenkins
0279411c58 Merge "versioned writes middleware" 2015-08-10 17:37:49 +00:00
Jenkins
4e717eae06 Merge "Handle removed suffix dirs the same as empty suffix dirs" 2015-08-10 17:17:06 +00:00