kolla-ansible/doc/source/reference/logging-and-monitoring/prometheus-guide.rst

Prometheus - Monitoring System & Time Series Database

Overview

Kolla can deploy a full working Prometheus setup in either a all-in-one or multinode setup.

Preparation and deployment

To enable Prometheus, modify the configuration file /etc/kolla/globals.yml and change the following:

enable_prometheus: "yes"

Note: This will deploy Prometheus version 2.x. Any potentially existing Prometheus 1.x instances deployed by previous Kolla Ansible releases will conflict with current version and should be manually stopped and/or removed. If you would like to stay with version 1.x, set the enable_prometheus variable to no.

In order to remove leftover volume containing Prometheus 1.x data, execute:

docker volume rm prometheus

on all hosts wherever Prometheus was previously deployed.

Extending the default command line options

It is possible to extend the default command line options for Prometheus by using a custom variable. As an example, to set query timeout to 1 minute and data retention size to 30 gigabytes:

prometheus_cmdline_extras: "--query.timeout=1m --storage.tsdb.retention.size=30GB"

Extending prometheus.cfg

If you want to add extra targets to scrape, you can extend the default prometheus.yml config file by placing additional configs in {{ node_custom_config }}/prometheus/prometheus.yml.d. These should have the same format as prometheus.yml. These additional configs are merged so that any list items are extended. For example, if using the default value for node_custom_config, you could add additional targets to scape by defining /etc/kolla/config/prometheus/prometheus.yml.d/10-custom.yml containing the following:

scrape_configs:
  - job_name: custom
    static_configs:
      - targets:
        - '10.0.0.111:1234'
  - job_name: custom-template
    static_configs:
      - targets:
{% for host in groups['prometheus'] %}
        - '{{ hostvars[host]['ansible_' + hostvars[host]['api_interface']]['ipv4']['address'] }}:{{ 3456 }}'
{% endfor %}

The jobs, custom, and custom_template would be appended to the default list of scrape_configs in the final prometheus.yml. To customize on a per host basis, files can also be placed in {{ node_custom_config }}/prometheus/<inventory_hostname>/prometheus.yml.d where, inventory_hostname is one of the hosts in your inventory. These will be merged with any files in {{ node_custom_config }}/prometheus/prometheus.yml.d, so in order to override a list value instead of extending it, you will need to make sure that no files in {{ node_custom_config }}/prometheus/prometheus.yml.d set a key with an equivalent hierarchical path.

Extra files

Sometimes it is necessary to reference additional files from within prometheus.yml, for example, when defining file service discovery configuration. To enable you to do this, kolla-ansible will resursively discover any files in {{ node_custom_config }}/prometheus/extras and template them. The templated output is then copied to /etc/prometheus/extras within the container on startup. For example to configure ipmi_exporter, using the default value for node_custom_config, you could create the following files:

• /etc/kolla/config/prometheus/prometheus.yml.d/ipmi-exporter.yml:

  ---
  scrape_configs:
  - job_name: ipmi
    params:
      module: ["default"]
      scrape_interval: 1m
      scrape_timeout: 30s
      metrics_path: /ipmi
      scheme: http
      file_sd_configs:
        - files:
            - /etc/prometheus/extras/file_sd/ipmi-exporter-targets.yml
      refresh_interval: 5m
      relabel_configs:
        - source_labels: [__address__]
          separator: ;
          regex: (.*)
          target_label: __param_target
          replacement: ${1}
          action: replace
        - source_labels: [__param_target]
          separator: ;
          regex: (.*)
          target_label: instance
          replacement: ${1}
          action: replace
        - separator: ;
          regex: .*
          target_label: __address__
          replacement: "{{ ipmi_exporter_listen_address }}:9290"
          action: replace

  where ipmi_exporter_listen_address is a variable containing the IP address of the node where the exporter is running.

• /etc/kolla/config/prometheus/extras/file_sd/ipmi-exporter-targets.yml:

  ---
  - targets:
    - 192.168.1.1
  labels:
      job: ipmi_exporter