opendev/elastic-recheck
Code Issues Proposed changes
Classify tempest-devstack failures using ElasticSearch
210 Commits 2 Branches 0 Tags 14 MiB
Python 53.3%
JavaScript 38.1%
HTML 7.5%
Makefile 0.5%
CSS 0.4%
Other 0.2%
d14cde9dec
Go to file
Davanum Srinivas d14cde9dec Ability to Soft Delete Stale Bugs from elastic-recheck 
Add a resolved_at attribute in the query yaml files
that can be used to mark when  a bug has been
fixed or does not occur any more. This can help us
re-enable bugs quickly when we see them again.

Change-Id: I7af7ce9417eec5ff9ecc2487a920ff9d1286a714
2013-12-10 16:52:16 -05:00
doc/source Apply Cookiecutter to the repo. 2013-09-23 15:27:39 -07:00
elastic_recheck Ability to Soft Delete Stale Bugs from elastic-recheck 2013-12-10 16:52:16 -05:00
queries Add e-r query for nova-network bug 1258848 2013-12-10 08:07:51 -08:00
web set y axis on failure rate graph 2013-12-06 17:40:31 -05:00
.coveragerc Apply Cookiecutter to the repo. 2013-09-23 15:27:39 -07:00
.gitignore Apply Cookiecutter to the repo. 2013-09-23 15:27:39 -07:00
.gitreview Apply Cookiecutter to the repo. 2013-09-23 15:27:39 -07:00
.testr.conf Apply Cookiecutter to the repo. 2013-09-23 15:27:39 -07:00
babel.cfg Apply Cookiecutter to the repo. 2013-09-23 15:27:39 -07:00
CONTRIBUTING.rst Apply Cookiecutter to the repo. 2013-09-23 15:27:39 -07:00
elasticRecheck.conf.sample move queries.yaml into a queries subdir 2013-12-02 11:43:00 -05:00
LICENSE Apply Cookiecutter to the repo. 2013-09-23 15:27:39 -07:00
MANIFEST.in Apply Cookiecutter to the repo. 2013-09-23 15:27:39 -07:00
README.rst wrapping the README.rst file to 80 cols 2013-12-02 11:43:51 -05:00
recheckwatchbot.yaml Make bot.py behave like a daemon 2013-09-18 17:45:12 -04:00
requirements.txt Make pid file configurable 2013-09-30 10:29:32 -07:00
setup.cfg add support for installing the web dashboard 2013-12-03 10:41:21 -08:00
setup.py Apply Cookiecutter to the repo. 2013-09-23 15:27:39 -07:00
test-requirements.txt Add mox fixture to base TestCase 2013-10-01 18:05:33 -04:00
tox.ini Fix E122,E126,E128 items in codebase 2013-12-02 11:43:51 -05:00

README.rst

elastic-recheck

"Classify tempest-devstack failures using ElasticSearch"

Idea

When a tempest job failure is detected, by monitoring gerrit (using gerritlib), a collection of logstash queries will be run on the failed job to detect what the bug was.

Eventually this can be tied into the rechecker tool and launchpad

queries/

All queries are stored in separate yaml files in a queries directory at the top of the elastic_recheck code base. The format of these files is ######.yaml (where ###### is the bug number), the yaml should have a query keyword which is the query text for elastic search.

Guidelines for good queries

  • After a bug is resolved and has no more hits in elasticsearch, we should flag it with a resolved_at keyword. This will let us keep some memory of past bugs, and see if they come back. (Note: this is a forward looking statement, sorting out resolved_at will come in the future)
  • Queries should get as close as possible to fingerprinting the root cause
  • Queries should not return any hits for successful jobs, this is a sign the query isn't specific enough

In order to support rapidly added queries, it's considered socially acceptable to +A changes that only add 1 new bug query, and to even self approve those changes by core reviewers.

Future Work

  • Move config files into a separate directory
  • Make unit tests robust
  • Add debug mode flag
  • Expand gating testing
  • Cleanup and document code better
  • Sort out resolved_at stamping to remove active bugs
  • Move away from polling ElasticSearch to discover if its ready or not
  • Add nightly job to propose a patch to remove bug queries that return no hits -- Bug hasn't been seen in 2 weeks and must be closed
  • implement resolved_at in loader

Main Dependencies

  • gerritlib
  • pyelasticsearch