openstack/oslo.db
Code Issues Proposed changes

Increase the default max_overflow value

Browse Source 
It turned out that distribution of WSGI requests between forks is not
even when eventlet is used, which can cause pool timeout issues under
load, while there are idle workers with open DB connections, which
should have served these HTTP requests.

The default max_overflow value should be increased to allow DB oriented
services (mostly APIs) handle spikes in number of concurrent requests
trying to use a DB.

At the same time, the default number of greenlets is decreased:

 I65b40b9906b75146a0085bbe168f1e6bcae82f21

which effectively causes a particular worker to stop accepting new
requests it has no resources (DB connections) to handle and allow
other forks accept()'ing on the same FD to proceed.

Testing was performed using this script:

  https://gist.github.com/zzzeek/c69138fd0d0b3e553a1f

With 100 greenthreads in the pool, 50 DB connections allowed overflow
no pool timeout issues were seen with up to 500 concurrent requests
done by ab, while current default values (1000/10) could not handle
even 100 concurrent requests.

See this ML thread for details:

  http://lists.openstack.org/pipermail/openstack-dev/2015-December/082717.html

DocImpact

This change potentially increases the number of connections open to the
RDBMS server. If you start seeing "too many connections" errors, please
check these settings and adjust them appropriately:

https://dev.mysql.com/doc/refman/5.5/en/server-system-variables.html#sysvar_max_connections
http://www.postgresql.org/docs/current/static/runtime-config-connection.html#GUC-MAX-CONNECTIONS

Closes-Bug: #1535375

Change-Id: I2e9c2a71d8231e0dfbefc6293ad319e1e459beec
This commit is contained in:
Roman Podoliaka 2016-01-18 19:03:16 +02:00
parent 8717f08d8f
commit db5ecf69c8
2 changed files with 26 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
oslo_db/options.py
View File

@ -89,6 +89,7 @@ database_opts = [
group='DATABASE')],
help='Interval between retries of opening a SQL connection.'),
cfg.IntOpt('max_overflow',
default=50,
deprecated_opts=[cfg.DeprecatedOpt('sql_max_overflow',
group='DEFAULT'),
cfg.DeprecatedOpt('sqlalchemy_max_overflow',

25
releasenotes/notes/increase-default-max-overflow-0af787268807f926.yaml Normal file
View File

@ -0,0 +1,25 @@
---
upgrade:
- |
The default value of ``max_overflow`` config option
has been increased from 10 to 50 in order to allow
OpenStack services heavily using DBs to better handle
spikes of concurrent requests and lower the probability
of getting a pool timeout issue.
This change potentially leads to increasing of the number
of open connections to an RDBMS server. Depending on the
configuration, you may see "too many connections" errors
in logs of OpenStack services / RDBMS server. The max limit of
connections can be set by the means of these config options:
http://dev.mysql.com/doc/refman/5.7/en/server-system-variables.html#sysvar_max_connections
http://www.postgresql.org/docs/current/static/runtime-config-connection.html#GUC-MAX-CONNECTIONS
For details, please see the following LP:
https://bugs.launchpad.net/oslo.db/+bug/1535375
and the ML thread:
http://lists.openstack.org/pipermail/openstack-dev/2015-December/082717.html