783f16035a
When an object server was handling concurrent GET or POST requests from very fast clients, it would starve other connected clients. The greenthreads responsible for servicing the fast clients would hog the processor and only rarely yield to another greenthread. The reason this happens for GET requests is found in eventlet.greenio.GreenSocket, in the send() method. When you call .send(data) on a GreenSocket, it immediately calls .send(data) on its underlying real socket (socket._socketobject). If the real socket accepts all the data, then GreenSocket.send() returns without yielding to another greenthread. Only if the real socket failed to accept all the data (either .send(data) < len(data) or by raising EWOULDBLOCK) does the GreenSocket yield control. Under most workloads, this isn't a problem. The TCP connection to client X can only consume data so quickly, and therefore the greenthread serving client X will frequently encounter a full socket buffer and yield control, so no clients starve. However, when there's a lot of contention for a single object from a large number of fast clients (e.g. on a LAN connected w/10Gb Ethernet), then one winds up in a situation where reading from the disk is slower than writing to the network, and so full socket buffers become rare, and therefore so do context switches. The end result is that many clients time out waiting for data. The situation for PUT requests is analogous; GreenSocket.recv() seldom encounters EWOULDBLOCK, so greenthreads seldom yield. This patch calls eventlet.sleep() to yield control after each chunk, preventing any one greenthread's IO from blocking the hub for very long. This code has the flaw that it will greenthread-switch twice when a send() or recv() does block, but since there isn't a way to find out if a switch occurred or not, there's no way to avoid it. Since greenlet switches are quite fast (faster than system calls, which the object server does a lot of), this shouldn't have a significant performance impact. Change-Id: I8549adfb4a198739b80979236c27b76df607eebf
Swift ----- A distributed object store that was originally developed as the basis for Rackspace's Cloud Files. To build documentation run `python setup.py build_sphinx`, and then browse to /doc/build/html/index.html. The best place to get started is the "SAIO - Swift All In One", which will walk you through setting up a development cluster of Swift in a VM. For more information, visit us at http://launchpad.net/swift, or come hang out on our IRC channel, #openstack on freenode. -- Swift Development Team