6880 Commits

Author SHA1 Message Date
Kota Tsuyuzaki
4ddd4c8a63 Update ubuntu version to be correct target
- saio describes both 14.04 and 16.04 procedure
- currently we're testing on 16.04 (xenial) envrionment on the gate

Remaining task (probably another work):
- review the installation guide which adjusts to the ubuntu 14.04 LTS

Change-Id: Id690a1deabeb24bfc1af3ba3a3019794fe4b8eb9
2017-09-12 02:21:41 +09:00
Jenkins
834b733d29 Merge "Remove vestigate HUDSON_PUBLISH_DOCS reference" 2017-09-06 20:11:26 +00:00
Jenkins
9fbda9946e Merge "Unify 'Content-Type' header key name in API reference" 2017-09-06 19:07:33 +00:00
Jenkins
7e5a39a5bd Merge "Add explanation about Content-Length header limit with DLO to docs" 2017-09-06 18:32:15 +00:00
Kazuhiro MIYAHARA
9fd9141f7d Unify 'Content-Type' header key name in API reference
In current API reference, 'Content-Type' header key name is used in response
parameter tables except Container GET table. On the other hand, 'Content_Type'
is used in Container GET table.

This patch fixes the 'Content_Type' in Container GET table to 'Content-Type'.

Change-Id: Idf242477dd089202635b69b85b0c19739e0c6321
2017-09-06 11:06:26 +00:00
Jenkins
2b790efc8b Merge "Fix bytes and name API reference of Container GET" 2017-09-06 07:52:34 +00:00
Kazuhiro MIYAHARA
a2f557e921 Add explanation about Content-Length header limit with DLO to docs
If a number of DLO segments is larger than container listing limit,
Content-Length header will not be included in GET or HEAD response.
However, this fact is not explained in document of large objects.

This patch add explanation about this fact to the document.

Change-Id: Ia45fad05797f38fa8b6b0ed917b4f9d7fb337149
Closes-Bug: 1680219
2017-09-06 07:18:45 +00:00
Kazuhiro MIYAHARA
275da4c18e Fix bytes and name API reference of Container GET
In current API reference, 'bytes' and 'name' descriptions of Container GET are
shared with Account GET. However, the descriptions are not correct for
Container GET.

This patch separate descriptions for Container GET and descriptions for Account
GET and fix descriptions for Container GET.

Change-Id: Ibedd08c5d9ebe145caf567edbd9757d7bc83b96d
2017-09-06 04:16:40 +00:00
Jenkins
6d1a3c4e29 Merge "doc migration: update the doc link address[1/3]" 2017-09-06 02:58:15 +00:00
Jenkins
f40339bcc0 Merge "Stop clearing req.acl in DLO" 2017-09-06 01:02:33 +00:00
junboli
6998d804ff doc migration: update the doc link address[1/3]
Update the doc link brought by the swift doc migration.
Although we had some effort to fix these before, it still left lots
of bad doc link, I separate these changes into 3 patches aim to fix
all of these, this is the 1st patch for doc/manpages.

Change-Id: Ib49696706e61bbd36ae56b15b1d94aa4ce84531c
2017-09-05 19:13:47 +00:00
Kazuhiro MIYAHARA
ddec5f5373 Update Container-Sync Docs for internal_client.conf
At patch [1], container sync daemon is updated to use internal client.
However, the "Configuring Container Sync" section [2] of swift docs doesn't
updated for this change.

This patch adds an explanation about the change.

[1]: https://review.openstack.org/#/c/452773/
[2]: http://docs.openstack.org/developer/swift/overview_container_sync.html#configuring-container-sync

Change-Id: Ia1aabeab250cbe27414d7877138ad38708f5ce3f
Closes-Bug: 1446721
2017-09-05 07:26:11 +00:00
Monty Taylor
03c54d505f
Remove vestigate HUDSON_PUBLISH_DOCS reference
This isn't actually used (and in swift is commented out already)
and is a leftover from a thing we did about seven years ago.

Change-Id: I9889bcfd29054f14679ae7430b077ad3afb25b98
2017-09-02 13:09:17 -05:00
Jenkins
6da17e9923 Merge "Cleanup test tempdirs in tearDown" 2017-09-01 16:56:42 +00:00
Alistair Coles
0e51ac09ad Cleanup test tempdirs in tearDown
Stop leaking tmp dirs in unit tests

Change-Id: I606e9deeedc7c52a85d270b3cef7dfba13b4f0d3
2017-09-01 15:34:30 +01:00
Tim Burke
8ed6e663d5 Stop clearing req.acl in DLO
This has been pure misdirection since DLO was pulled out of the proxy
server; req.acl is always set by the object controller before calling
swift.authorize.

See TestDlo.test_dlo_referer_on_segment_container in
test/functional/test_dlo.py for proof that we still enforce segment
container ACLs properly.

Change-Id: I5321b5537787fcea126065243f1c4bd5e601181e
Related-Change: Ib3b3830c246816dd549fc74be98b4bc651e7bace
2017-08-31 23:57:04 +00:00
Tim Burke
9f5aa3bbd7 api-ref: update docs links
Change-Id: I83da881f82faf340d0e394c79f7e9d4df7f34b04
2017-08-31 15:57:13 -07:00
Jenkins
add2a94cd8 Merge "Make X-Backend-Replication consistent for HEAD" 2017-08-29 17:42:34 +00:00
Jenkins
9c3ef2dff8 Merge "Small readability change to test_keystoneauth" 2017-08-28 10:51:45 +00:00
Sachin Patil
fb07863155 Small readability change to test_keystoneauth
Changed assertTrue to assertIsNone for better readability

Change-Id: Ibc83c32ccfb5165abf897a15cf2692414a7f95c3
Signed-off-by: Sachin Patil <psachin@redhat.com>
2017-08-28 10:34:49 +00:00
Jenkins
8ca5bf2364 Merge "Add probe test for ssync of unexpired metadata to an expired object" 2017-08-26 02:13:03 +00:00
Jenkins
92282339f9 Merge "Fix DirectClientExceptions with utf-8 encoded paths" 2017-08-26 02:12:55 +00:00
Alistair Coles
e109c7800f Add probe test for ssync of unexpired metadata to an expired object
Verify that metadata can be sync'd to a frag that has missed a POST
and consequently that frag appears to be expired, when in fact the
POST removed the X-Delete-At header.

Tests the fix added by the Related-Change.

Related-Bug: #1683689
Related-Change: I919994ead2b20dbb6c5671c208823e8b7f513715
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>

Change-Id: I9af9fc26098893db4043cc9a8d05d772772d4259
2017-08-25 15:36:00 -07:00
Jenkins
da37aaf610 Merge "Tolerate swiftclient *not* mutatinng args" 2017-08-25 22:23:31 +00:00
Jenkins
4b6e8bc7ec Merge "Add OpenSuse SAIO build instructions" 2017-08-25 22:23:22 +00:00
Clay Gerrard
2e0ca543e8 Make X-Backend-Replication consistent for HEAD
Current an X-Backend-Replication GET request for an expired object will
still open the expired datafile and return the object, but HEAD request
with the same headers will 404.  This can lead to some bad assumptions
in probetests and other places where we make direct HEAD requests

N.B. Because SSYNC replication does not make any HEAD requests this
change is immaterial to the correctness of the consistency engine.

Related-Change-Id: I7f90b732c3268cb852b64f17555c631d668044a8
Change-Id: Idc01970b37d1b77e1d48f9c4f4979f63ee771093
2017-08-25 15:08:21 -07:00
Clay Gerrard
87eaaebd67 Fix DirectClientExceptions with utf-8 encoded paths
Because the direct_client module uses the buffered_http module it's
requests are already robust to receiving either unicode or utf-8 paths.

The DirectClientException message however encodes the given path with
the device key from a ring node - which having come from a json
de-serialized ring will be a unicode type.  Despite the device key
almost always being only ascii characters; python string interpolation
with any unicode type will always force all binary strings to be
converted to unicode - which will raise an error if any byte strings
includes non-ascii characters.

To maintain robustness in DirectClientException, when the provided path
is not already unicode we decode the bytes as utf-8 before mixing them
with the other unicode strings and then normalize everything back to a
quoted utf-8 byte string.

Change-Id: I162d2e093a3110856d6e1d513de3c7919079c9e7
2017-08-25 14:35:11 -07:00
Tim Burke
00ca1ce6fe Tolerate swiftclient *not* mutatinng args
Change-Id: If82fe9e1d2da8c5122881f34dfbaaa7944c66265
Related-Change: Ia1638c216eff9db6fbe416bc0570c27cfdcfe730
2017-08-25 12:27:41 -07:00
OpenStack Proposal Bot
412898a34d Imported Translations from Zanata
For more information about this automatic import see:
https://docs.openstack.org/i18n/latest/reviewing-translation-import.html

Change-Id: Ibd1ec2b671f10482daf20e7f53fa92b9527a828f
2017-08-25 07:41:28 +00:00
Gábor Antal
87340e5f29 Use more specific asserts in test/unit/account tests
I changed asserts with more specific assert methods.
e.g.: from assertTrue(sth == None) to assertIsNone(*) or
assertTrue(isinstance(inst, type)) to assertIsInstace(inst, type) or
assertTrue(not sth) to assertFalse(sth).

The code gets more readable, and a better description will be shown on fail.

Change-Id: Icdbf3c63fe8dd6db1129023885655a9f7032d4a7
2017-08-23 17:22:54 +02:00
Jenkins
daea9932d4 Merge "Socket errors don't warrant tracebacks when talking to memcached" 2017-08-23 14:55:43 +00:00
Jenkins
33830ddecf Merge "Drop reconstructor stats when worker has no devices" 2017-08-23 04:17:55 +00:00
Tim Burke
73929539cc Socket errors don't warrant tracebacks when talking to memcached
Currently, timeouts when talking to memcached cause log lines like

   ERROR:root:Timeout connecting to memcached: 192.168.23.62:11211

Meanwhile, socket errors (which you'd expect to be about as common
as timeouts) cause log lines like

   ERROR:root:Error talking to memcached: 192.168.23.70:11211
   Traceback (most recent call last):
     File "/usr/lib/pymodules/python2.7/swift/common/memcached.py", line 293, in set
       (key, flags, timeout, len(value), value))
     File "/usr/lib/python2.7/dist-packages/eventlet/greenio/base.py", line 390, in sendall
       tail = self.send(data, flags)
     File "/usr/lib/python2.7/dist-packages/eventlet/greenio/base.py", line 384, in send
       return self._send_loop(self.fd.send, data, flags)
     File "/usr/lib/python2.7/dist-packages/eventlet/greenio/base.py", line 371, in _send_loop
       return send_method(data, *args)
   error: [Errno 32] Broken pipe

... or ...

   ERROR:root:Error connecting to memcached: 192.168.23.73:11211
   Traceback (most recent call last):
     File "/opt/ss/lib/python2.7/site-packages/swift/common/memcached.py", line 239, in _get_conns
       fp, sock = self._client_cache[server].get()
     File "/opt/ss/lib/python2.7/site-packages/swift/common/memcached.py", line 135, in get
       fp, sock = self.create()
     File "/opt/ss/lib/python2.7/site-packages/swift/common/memcached.py", line 128, in create
       sock.connect((host, int(port)))
     File "/opt/ss/lib/python2.7/site-packages/eventlet/greenio/base.py", line 237, in connect
       while not socket_connect(fd, address):
     File "/opt/ss/lib/python2.7/site-packages/eventlet/greenio/base.py", line 39, in socket_connect
       raise socket.error(err, errno.errorcode[err])
   error: [Errno 101] ENETUNREACH

... which seems excessive. Now, socket errors are logged as normal
errors, without the tracebacks.

Change-Id: I71a2c4786c0406dbc43b829cad5a0c7e2c45de21
2017-08-22 20:42:51 -07:00
Jenkins
8c790e6605 Merge "Fix more X-Delete-At timing issues" 2017-08-22 16:15:04 +00:00
Jenkins
fcf318c421 Merge "Make test assertion more holistic" 2017-08-22 16:10:04 +00:00
Jenkins
b0142d0cd2 Merge "Retrieve encryption root secret from Barbican" 2017-08-21 21:19:09 +00:00
Clay Gerrard
63ca3a74ef Drop reconstructor stats when worker has no devices
If you're watching (new) node's reconstruction_last time to ensure a
cycle finishes since the last ring rebalance you won't ever see
reconstructors with no devices drop recon stats.

Change-Id: I84c07fc6841119b00d1a74078fe53f4ce637187b
2017-08-21 17:50:10 +01:00
26a2361f6b Update reno for stable/pike
Change-Id: Ic59d3886b5fc0af10ea53d6d825a4f566402e367
2017-08-21 16:19:22 +00:00
Jenkins
1f751a7979 Merge "headers_to_account_info include per policy stats" 2017-08-18 23:56:34 +00:00
Jenkins
33b07c5fc5 Merge "Add a jitter for check of time synchronization." 2017-08-18 23:55:47 +00:00
Jenkins
fb4c5aacab Merge "authors/changelog for 2.15.1" 2017-08-18 23:55:02 +00:00
Jenkins
251b10dcd3 Merge "Reduced complexity of _response_iter() method" 2017-08-18 23:48:40 +00:00
Tim Burke
72ed76d84b Make test assertion more holistic
Previously, introducing an error would yield an accurate failure, but
without much context, like

Traceback (most recent call last):
  File "/vagrant/swift/test/unit/proxy/controllers/test_base.py", line 635, in test_headers_to_account_info_storage_policies
    resp['storage_policies'][0]['bytes'], 1002)
KeyError: 'bytes'

Now, we'll have a much larger picture of what's going on:

Traceback (most recent call last):
  File "/vagrant/swift/test/unit/proxy/controllers/test_base.py", line 636, in test_headers_to_account_info_storage_policies
    'bytes': 0},
AssertionError: {0: {'object_count': 13, 'container_count': 120}, 1:
{'object_count': 10, 'conta [truncated]... != {0: {'object_count': 13,
'container_count': 120, 'bytes': 1002}, 1: {'object_cou [truncated]...
- {0: {'container_count': 120, 'object_count': 13},
+ {0: {'bytes': 1002, 'container_count': 120, 'object_count': 13},
?      +++++++++++++++

-  1: {'container_count': 20, 'object_count': 10}}
+  1: {'bytes': 0, 'container_count': 20, 'object_count': 10}}
?      ++++++++++++

Related-Change: I800266d15aabcc7b6e0234de3c9b965b5c15a623
Change-Id: I9a4e5d62674c467bca4bc70d5150d918d48aac42
2017-08-18 17:34:50 +00:00
John Dickinson
56e081c0a1 authors/changelog for 2.15.1
Change-Id: Id5f5467a50119d521fead28faf751c621ce1e4ee
2017-08-18 10:01:52 -07:00
Jenkins
164074325f Merge "Inconsistencies in endpoint definitions" 2017-08-18 07:54:24 +00:00
Jenkins
c9acb0a717 Merge "functests: Only convert headers that should be ints to be ints" 2017-08-18 07:54:17 +00:00
Jenkins
f6b78f6de8 Merge "Clarify usage of replicator and reconstructor override options" 2017-08-18 03:08:06 +00:00
Jenkins
16bc798f0e Merge "Use more specific asserts in test/unit/account tests" 2017-08-18 03:07:59 +00:00
Jenkins
cb1c438006 Merge "Add functests for disallowed COPYs into a versioned container" 2017-08-18 03:04:11 +00:00
Tim Burke
15e339da1f Fix more X-Delete-At timing issues
Seen in a gate:

======================================================================
2017-08-17 23:10:29.662540 | FAIL: test_GET_but_expired
(test.unit.obj.test_server.TestObjectController)
2017-08-17 23:10:29.662577 |
----------------------------------------------------------------------
2017-08-17 23:10:29.662600 | Traceback (most recent call last):
2017-08-17 23:10:29.662651 |   File "/home/jenkins/workspace/gate-cross-swift-python27-ubuntu-xenial/test/unit/obj/test_server.py", line 5754, in test_GET_but_expired
2017-08-17 23:10:29.662677 |     self.assertEqual(resp.status_int, 201)
2017-08-17 23:10:29.662697 | AssertionError: 400 != 201
2017-08-17 23:10:29.662729 | -------------------- >> begin captured stdout << ---------------------
2017-08-17 23:10:29.662769 | test INFO: None - - [17/Aug/2017:23:08:45 +0000] "PUT /sda1/p/a/c/o" 201 - "-" "-" "-" 0.0072 "-" 6413 -
2017-08-17 23:10:29.662811 | test INFO: None - - [17/Aug/2017:23:08:45 +0000] "GET /sda1/p/a/c/o" 200 4 "-" "-" "-" 0.0011 "-" 6413 -
2017-08-17 23:10:29.662852 | test INFO: None - - [17/Aug/2017:23:08:45 +0000] "PUT /sda1/p/a/c/o" 400 19 "-" "-" "-" 0.0004 "-" 6413 -
2017-08-17 23:10:29.662865 |
2017-08-17 23:10:29.662896 | --------------------- >> end captured stdout << ----------------------
2017-08-17 23:10:29.662925 |     '400 != 201' = '%s != %s' % (safe_repr(400), safe_repr(201))
2017-08-17 23:10:29.662956 |     '400 != 201' = self._formatMessage('400 != 201', '400 != 201')
2017-08-17 23:10:29.662981 | >>  raise self.failureException('400 != 201')

Change-Id: I643be9af8f054f33897dd74071027a739eaa2c5c
Related-Change: I10d3b9fcbefff3c415a92fa284a1ea1eda458581
Related-Bug: #1597520
2017-08-18 01:12:39 +00:00