144 lines
5.0 KiB
ReStructuredText
144 lines
5.0 KiB
ReStructuredText
.. _prometheus-guide:
|
|
|
|
=====================================================
|
|
Prometheus - Monitoring System & Time Series Database
|
|
=====================================================
|
|
|
|
Overview
|
|
~~~~~~~~
|
|
|
|
Kolla can deploy a full working Prometheus setup in either a **all-in-one** or
|
|
**multinode** setup.
|
|
|
|
Preparation and deployment
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
To enable Prometheus, modify the configuration file ``/etc/kolla/globals.yml``
|
|
and change the following:
|
|
|
|
.. code-block:: yaml
|
|
|
|
enable_prometheus: "yes"
|
|
|
|
Note: This will deploy Prometheus version 2.x. Any potentially existing
|
|
Prometheus 1.x instances deployed by previous Kolla Ansible releases will
|
|
conflict with current version and should be manually stopped and/or removed.
|
|
If you would like to stay with version 1.x, set the ``enable_prometheus``
|
|
variable to ``no``.
|
|
|
|
In order to remove leftover volume containing Prometheus 1.x data, execute:
|
|
|
|
.. code-block:: console
|
|
|
|
docker volume rm prometheus
|
|
|
|
on all hosts wherever Prometheus was previously deployed.
|
|
|
|
Extending the default command line options
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
It is possible to extend the default command line options for Prometheus by
|
|
using a custom variable. As an example, to set query timeout to 1 minute
|
|
and data retention size to 30 gigabytes:
|
|
|
|
.. code-block:: yaml
|
|
|
|
prometheus_cmdline_extras: "--query.timeout=1m --storage.tsdb.retention.size=30GB"
|
|
|
|
Extending prometheus.cfg
|
|
~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
If you want to add extra targets to scrape, you can extend the default
|
|
``prometheus.yml`` config file by placing additional configs in
|
|
``{{ node_custom_config }}/prometheus/prometheus.yml.d``. These should have the
|
|
same format as ``prometheus.yml``. These additional configs are merged so
|
|
that any list items are extended. For example, if using the default value for
|
|
``node_custom_config``, you could add additional targets to scape by defining
|
|
``/etc/kolla/config/prometheus/prometheus.yml.d/10-custom.yml`` containing the
|
|
following:
|
|
|
|
.. code-block:: jinja
|
|
|
|
scrape_configs:
|
|
- job_name: custom
|
|
static_configs:
|
|
- targets:
|
|
- '10.0.0.111:1234'
|
|
- job_name: custom-template
|
|
static_configs:
|
|
- targets:
|
|
{% for host in groups['prometheus'] %}
|
|
- '{{ hostvars[host]['ansible_' + hostvars[host]['api_interface']]['ipv4']['address'] }}:{{ 3456 }}'
|
|
{% endfor %}
|
|
|
|
The jobs, ``custom``, and ``custom_template`` would be appended to the default
|
|
list of ``scrape_configs`` in the final ``prometheus.yml``. To customize on a per
|
|
host basis, files can also be placed in
|
|
``{{ node_custom_config }}/prometheus/<inventory_hostname>/prometheus.yml.d``
|
|
where, ``inventory_hostname`` is one of the hosts in your inventory. These
|
|
will be merged with any files in ``{{ node_custom_config }}/prometheus/prometheus.yml.d``,
|
|
so in order to override a list value instead of extending it, you will need to make
|
|
sure that no files in ``{{ node_custom_config }}/prometheus/prometheus.yml.d``
|
|
set a key with an equivalent hierarchical path.
|
|
|
|
Extra files
|
|
~~~~~~~~~~~
|
|
|
|
Sometimes it is necessary to reference additional files from within
|
|
``prometheus.yml``, for example, when defining file service discovery
|
|
configuration. To enable you to do this, kolla-ansible will resursively
|
|
discover any files in ``{{ node_custom_config }}/prometheus/extras`` and
|
|
template them. The templated output is then copied to
|
|
``/etc/prometheus/extras`` within the container on startup. For example to
|
|
configure `ipmi_exporter <https://github.com/soundcloud/ipmi_exporter>`_, using
|
|
the default value for ``node_custom_config``, you could create the following
|
|
files:
|
|
|
|
- ``/etc/kolla/config/prometheus/prometheus.yml.d/ipmi-exporter.yml``:
|
|
|
|
.. code-block:: jinja
|
|
|
|
---
|
|
scrape_configs:
|
|
- job_name: ipmi
|
|
params:
|
|
module: ["default"]
|
|
scrape_interval: 1m
|
|
scrape_timeout: 30s
|
|
metrics_path: /ipmi
|
|
scheme: http
|
|
file_sd_configs:
|
|
- files:
|
|
- /etc/prometheus/extras/file_sd/ipmi-exporter-targets.yml
|
|
refresh_interval: 5m
|
|
relabel_configs:
|
|
- source_labels: [__address__]
|
|
separator: ;
|
|
regex: (.*)
|
|
target_label: __param_target
|
|
replacement: ${1}
|
|
action: replace
|
|
- source_labels: [__param_target]
|
|
separator: ;
|
|
regex: (.*)
|
|
target_label: instance
|
|
replacement: ${1}
|
|
action: replace
|
|
- separator: ;
|
|
regex: .*
|
|
target_label: __address__
|
|
replacement: "{{ ipmi_exporter_listen_address }}:9290"
|
|
action: replace
|
|
|
|
where ``ipmi_exporter_listen_address`` is a variable containing the IP address of
|
|
the node where the exporter is running.
|
|
|
|
- ``/etc/kolla/config/prometheus/extras/file_sd/ipmi-exporter-targets.yml``:
|
|
.. code-block:: yaml
|
|
|
|
---
|
|
- targets:
|
|
- 192.168.1.1
|
|
labels:
|
|
job: ipmi_exporter
|