opendev/elastic-recheck
Code Issues Proposed changes
Classify tempest-devstack failures using ElasticSearch
345 Commits 2 Branches 0 Tags 14 MiB
Python 53.3%
JavaScript 38.1%
HTML 7.5%
Makefile 0.5%
CSS 0.4%
Other 0.2%
d3dc41644d
Go to file
Matthew Treinish d3dc41644d Add --noirc to tox run job 
This commit adds the --noirc option to the tox job for running the
bot for testing. This way when testing the bot it won't leave irc
messages.

Change-Id: I923072261bc03022a3c133e1aa425e528be77770
2014-01-24 12:25:05 -05:00
doc/source Document adding bug signatures to e-r. 2013-12-12 12:34:54 -08:00
elastic_recheck Merge "add tests for loading the queries" 2014-01-24 17:03:17 +00:00
queries Expand DNS failures to catch all "errno" string output 2014-01-24 11:42:44 -05:00
web Add per job classification rate to uncategorized.html 2014-01-22 12:33:34 -08:00
.coveragerc Apply Cookiecutter to the repo. 2013-09-23 15:27:39 -07:00
.gitignore Apply Cookiecutter to the repo. 2013-09-23 15:27:39 -07:00
.gitreview Apply Cookiecutter to the repo. 2013-09-23 15:27:39 -07:00
.testr.conf Apply Cookiecutter to the repo. 2013-09-23 15:27:39 -07:00
babel.cfg Apply Cookiecutter to the repo. 2013-09-23 15:27:39 -07:00
CONTRIBUTING.rst Apply Cookiecutter to the repo. 2013-09-23 15:27:39 -07:00
elasticRecheck.conf.sample move queries.yaml into a queries subdir 2013-12-02 11:43:00 -05:00
LICENSE Apply Cookiecutter to the repo. 2013-09-23 15:27:39 -07:00
MANIFEST.in Apply Cookiecutter to the repo. 2013-09-23 15:27:39 -07:00
README.rst Add some documentation on wildcard limitations in queries 2014-01-07 11:57:38 -08:00
recheckwatchbot.yaml Make bot.py behave like a daemon 2013-09-18 17:45:12 -04:00
requirements.txt add uncategorized failure generation code 2014-01-17 09:35:40 -05:00
setup.cfg add uncategorized failure generation code 2014-01-17 09:35:40 -05:00
setup.py Apply Cookiecutter to the repo. 2013-09-23 15:27:39 -07:00
test-requirements.txt moving readiness checks into stream 2014-01-13 20:00:23 -05:00
tox.ini Add --noirc to tox run job 2014-01-24 12:25:05 -05:00

README.rst

elastic-recheck

"Use ElasticSearch to classify OpenStack gate failures"

Idea

Identifying the specific bug that is causing a transient error in the gate is very hard. Just identifying which tempest test failed is not enough because a single bug can potentially cause multiple tempest tests to fail. If we can find a fingerprint for a specific bug using logs, then we can use ElasticSearch to automatically detect any occurrences of the bug.

Using these fingerprints elastic-recheck can:

  • Search ElasticSearch for all occurrences of a bug.
  • Identify bug trends such as: when it started, is the bug fixed, is it getting worse, etc.
  • Classify bug failures in real time and report back to gerrit if we find a match, so a patch author knows why the test failed.

queries/

All queries are stored in separate yaml files in a queries directory at the top of the elastic-recheck code base. The format of these files is ######.yaml (where ###### is the launchpad bug number), the yaml should have a query keyword which is the query text for elastic search.

Guidelines for good queries

  • After a bug is resolved and has no more hits in elasticsearch, we should flag it with a resolved_at keyword. This will let us keep some memory of past bugs, and see if they come back.
  • Queries should get as close as possible to fingerprinting the root cause
  • Queries should not return any hits for successful jobs, this is a sign the query isn't specific enough

In order to support rapidly added queries, it's considered socially acceptable to +A changes that only add 1 new bug query, and to even self approve those changes by core reviewers.

Adding Bug Signatures

Most transient bugs seen in gate are not bugs in tempest associated with a specific tempest test failure, but rather some sort of issue further down the stack that can cause many tempest tests to fail.

  1. Given a transient bug that is seen during the gate, go through the logs (logs.openstack.org) and try to find a log that is associated with the failure. The closer to the root cause the better.

  2. Go to logstash.openstack.org and create an elastic search query to find the log message from step 1. To see the possible fields to search on click on an entry. Lucene query syntax is available at http://lucene.apache.org/core/4_0_0/queryparser/org/apache/lucene/queryparser/classic/package-summary.html#package_description

    Note that wildcard analysis is disabled by default in ElasticSearch so while a query in logstash might work with wildcards, it will not work in elastic-recheck. See the ElasticSearch documentation for more information on wildcard analysis: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-query-string-query.html#_wildcards

  3. Add a comment to the bug with the query you identified and a link to the logstash url for that query search.

  4. Add the query to elastic-recheck/queries/BUGNUMBER.yaml and push the patch up for review. https://git.openstack.org/cgit/openstack-infra/elastic-recheck/tree/queries

Future Work

  • Move config files into a separate directory
  • Make unit tests robust
  • Add debug mode flag
  • Expand gating testing
  • Cleanup and document code better
  • Add ability to check if any resolved bugs return
  • Move away from polling ElasticSearch to discover if its ready or not
  • Add nightly job to propose a patch to remove bug queries that return no hits -- Bug hasn't been seen in 2 weeks and must be closed