Seen in prod:
Traceback (most recent call last):
File ".../swift/proxy/controllers/obj.py", line 416, in _get_conn_response
self.app.node_timeout, not final_phase)
File ".../swift/proxy/controllers/obj.py", line 1645, in await_response
self.resp = self.conn.getexpect()
File ".../swift/common/bufferedhttp.py", line 229, in getexpect
response = BufferedHTTPResponse(self.sock, **kwargs)
File ".../swift/common/bufferedhttp.py", line 64, in __init__
self._real_socket = sock.fd._sock
AttributeError: 'NoneType' object has no attribute 'fd'
Change-Id: I35b3660f954bcf91aee2698b2c9cc10d5447abc1
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Recent versions of py27 [1] have begun raising InvalidURL if you try to
include non-ASCII characters in the request path. This was observed
recently in the periodic checks of stable/ocata and stable/pike. In
particular, we would spin up some in-process servers in
test.unit.proxy.test_server.TestSocketObjectVersions and do a container
listing with a prefix param that included raw (unquoted) UTF-8. This
query string would pass unmolested through the proxy, tripping the
InvalidURL error when bufferedhttp called putrequest.
More recent versions of Swift would not exhibit this particular failure,
as the listing_formats middleware would force a decoding/re-encoding of
the query string for account and container requests. However, object
requests with errant query strings would likely be able to trip the same
error.
Swift on py3 should not exhibit this behavior, as we so
thoroughly re-write the request line to avoid hitting
https://bugs.python.org/issue33973.
Now, always parse and re-encode the query string in bufferedhttp. This
prevents any errors on object requests and cleans up any callers that
might use bufferedhttp directly.
[1] Anything after https://github.com/python/cpython/commit/bb8071a;
see https://bugs.python.org/issue30458
Closes-Bug: 1843816
Change-Id: I73f84b96f164e6fc5d3cb890355871c26ed271a6
Related-Change: Id3ce37aa0402e2d8dd5784ce329d7cb4fbaf700d
Related-Change: Ie648f5c04d4415f3b620fb196fa567ce7575d522
I've at least tried it out with a py3 proxy, and it seems to work out
OK. I haven't tried killing the socket and verifying that it's actualy
dead, but getting a hold of _real_close *seems like* what we want?
At least the three (!!) tests pass.
Change-Id: Ic08c26185d63a36a5422793d81f621e0698fa572
Recently out gate started blowing up intermittently with a strange
case of ports mixed up. Sometimes a functional tests tries to
authorize on a port that's clearly an object server port, and
the like. As it turns out, eventlet developers added an unavoidable
SO_REUSEPORT into listen(), which makes listen(("localhost",0)
to reuse ports.
There's an issue about it:
https://github.com/eventlet/eventlet/issues/411
This patch is working around the problem while eventlet people
consider the issue.
Change-Id: I67522909f96495a6a30e1acdb79835dce2189549
a1c32702, 736cf54a, and 38787d0f remove uses of `simplejson` from
various parts of Swift in favor of the standard libary `json`
module (introduced in Python 2.6). This commit performs the remaining
`simplejson` to `json` replacements, removes two comments highlighting
quirks of simplejson with respect to Unicode, and removes the references
to it in setup documentation and requirements.txt.
There were a lot of places where we were importing json from
swift.common.utils, which is less intuitive than a direct `import json`,
so that replacement is made as well.
(And in two more tiny drive-bys, we add some pretty-indenting to an XML
fragment and use `super` rather than naming a base class explicitly.)
Change-Id: I769e88dda7f76ce15cf7ce930dc1874d24f9498a
assertEquals is deprecated in py3, replacing it.
Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8
Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
As documented in linked bug report, tempauth unit tests
were seen to fail on a system where simplejson was
installed but without the speedups extension. This
is because the tempauth account acl validation checks
that values are type str, but without the speedups
extension the json parser is returning unicode objects.
Fix is to have the acl validator tolerate those objects
being unicode or str.
Also change common/bufferedhttp.py to coerce ring device
to type str when constructing a path, in order to avoid
a UnicodeDecodeError when httplib sends a message that
has non-ascii header values.
Change-Id: I01524282cbaa25dc4b6dfa09f3f4723516cdba99
Closes-Bug: 1425776
On a loopback device (e.g., when proxy-server and object-server are on
same node), PUTs in the range 64-200K may experience a delay due to the
effect of Nagel interacting with the loopback MTU of 64K.
This effect has been directly seen by Mark Seger and Rick Jones on a
proxy-server to object-server PUT. However, you could expect to see a
similar effect on replication via ssync if the object being replicated
is on a different drive on the same node.
A prior change [1] related to Nagel set TCP_NODELAY on responses. This change
sets it on all outgoing connections.
[1] I11f86df1f56fba1c6ab6084dc1f580c395f072dc
Change-Id: Ife8885a42b289a5eb4ac7e4698f8889858bc8b7e
Closes-bug: 1408622
except x,y: was deprected and is removed in Python 3.x.
Use "except x as y:" instead which works in any Python
version >= 2.6.
Change-Id: I7008c74b807340f3457d3a0c8bd0b83f23169d14
Updated eventlet.TimeoutError (deprecated) references to
Timeout and, more importantly, updated many except Exception
clauses to except (Exception, Timeout).
Change-Id: Ib089265551bd20b94c00ea84f11140ccd795d301
