# How to get this going quickly...

1. Add the query to queries.yml
2. Add an associated local sova query to sova-pattens.yml
3. Add a blob of text from the error you would like to test against to errors.txt

Note: the softlinks
errors.txt        ---> samples/errors-testing.err
queries.yml       ---> src/data/queries.yml
sova-patterns.yml ---> src/data/sova-patterns.yml

4. Execute tox
5. Submit patch
6. GREAT SUCCESS!


# queries

Hosts reusable log queries which are built into a single queries.json file.

## Query database structure

Queries are defined using the data model from src/model.py which builds
a JSON Validation schema, making easy to validate the file.

One example of file can be seen at [queries-example.yml](https://opendev.org/openstack/tripleo-ci-health-queries/raw/branch/master/src/data/queries-example.yml)

Both [elastic-search](https://www.elastic.co/guide/en/elasticsearch/reference/current/term-level-queries.html) and [artcl](https://opendev.org/openstack/ansible-role-collect-logs) can make use of `regex` searches.

Pattern is supposed to be an exact string match and if multiple are present
we could easily convert them into a regex or logstash expression that uses
logical `AND`.

### Pattern

On elastic-rechheck queries we have cases with multiple entries used on
patterns, like `message:foo AND message:bar`. This is why we also allow
a list of strings.

### Categories

A query can have only one category out of a determined list of possible
values, currently `infra` and `code` are allowed. These can be used to
list found matches in section, making them easier to read.

### Tags

Tags are also used to build the logstash queries. List of known values
already used inside elastic-recheck queries:

```yaml
tags:
  - console
  - console.html
  - devstack-gate-setup-host.txt
  - grenade.sh.txt
  - job-output.txt
  - screen-c-api.txt
  - screen-c-bak.txt
  - screen-n-cpu.txt
  - screen-n-sch.txt
  - screen-q-agt.txt
  - syslog.txt
```

When logstash query is build `OR` is used between multiple tags.

### Uncovered cases:

We do not currently support the exclusions like below (2/93 found):
```yaml
query: >-
  message:"RESULT_TIMED_OUT: [untrusted : git.openstack.org/openstack/tempest/playbooks/devstack-tempest.yaml@master]" AND
  tags:"console" AND NOT
  (build_name:"tempest-all" OR
   build_name:"tempest-slow" OR
   build_name:"tempest-slow-py3")

query2: >-
  (message: "FAILED with status: 137" OR
  message: "FAILED with status: 143" OR
  message: "RUN END RESULT_TIMED_OUT") AND
  NOT message:"POST-RUN END RESULT_TIMED_OUT" AND
  tags: "console"
```

To allow us to cover for corner cases not covered byt the generic format,
we could have an optional `logstash` key that mentions the query. When this
would be present, we woudl avoid building the logstash query ourselves and
just use it.

## Disable queries per backend

To avoid using a particular query on a particular backend we can make use of
``skip: ['er', 'artcl']``.


## Parameters in a query

| Key           | Required | Type                     | Sova/ER      | Description |
| ------------- |:--------:| ------------------------:|-------------:|------------:|
| id            | Required | string, unique           | Sova         | This id will appear in sova.log when this bug is encountered. |
| pattern       | Optional | string / list of strings | ER (,Sova)   | This used as it is by Elastic recheck to form logstash queries like  ``` message:foo ``` or ``` message:foo AND message:bar ``` If not present Elastic Recheck skips this entry |
| regex         | Optional | string / list of strings | Sova         | Sova searches for this regex in the log files. If not present the pattern string is converted to regex (escaping special characters) and used by sova. |
| tags          | Optional | string / list of strings | ER           | Elastic recheck uses this to form logstash queries like ``` AND (tags:"console.html" OR tags:"job-output.txt")``` |
| url           | Optional | string                   | ER           | Launchpad bug URL. If present the bug number and details are displayed on ER dashboard. If not the graph is titled Unknow/private bug |
| suppress-graph| Optional | bool                     | ER           | Decides whether or not to display a graph for this bug in Elastic recheck dashboard  |

## Adding a new item in queries.yml

To add a new query (optional: from a bug)

- Add a query item in [src/data/queries.yml](https://opendev.org/openstack/tripleo-ci-health-queries/src/branch/master/src/data/queries.yml)
    - id : unique ID
    - pattern: The error message(s) to look for.
    - regex: If it is a complicated error message you can add a regex for sova to use.
    - tags: tag(s) for logstash query
    - url: Launchpad Bug URL
    - suppress-graph: To display/suppress graph for the bug in ER

To move an existing sova regex from sova-patterns.json to queries.yml

- Choose an item from ``` regexes ``` in output/sova-patterns.json. This is the file that sova currently uses.
  (Do not look at patterns for now)
- If the regex feels still relevant, vote for it here https://docs.google.com/spreadsheets/d/16rqgaSSoQrYNjsI4q0YInJxrOOJL3xyP3t3D6ZpluFY/edit#gid=846893892
- Add a query entry in src/data/queries.yml
    - id: `name` of the regexes item you chose.
    - pattern: If you want this error to be shown in ER dashboard add a string here corresponding to the regex
    - regex: `regex` of the regexes item you chose.

- Add a sample string matching the pattern (or regex if you have a special regex) in samples/errors-testing.err.
As the part of check job a task calls sova to check this file and makes sure there is a match for all the regexes
  in queries.yml.
- Run tox. If it passes it should create files in output/elastic-recheck and update sova-pattern-generated.
- Commit all these files.