In Kolla, an OpenStack project is created to store logs and metrics
harvested from the control plane by Monasca. This commit enables
the Monasca Datasource in the Grafana organisation which maps to
this OpenStack control plane project. What this means in practice
is that if a user logs into Monasca Grafana, and has access to the
the control plane project, they will immediately be able to create
dashboards using data from Monasca which has been gathered from the
control plane.
Support to enable creation of this datasource for other OpenStack
projects can be added in a separate commit.
Partially-Implements: blueprint monasca-grafana
Change-Id: I03e741ddb1c582b7280c64637ed3e3683df6419b
Some Monasca services support sending StatsD metrics to
allow monitoring those services. This commit connects
these services to the StatsD service provided by the
Monasca Agent.
Partially-Implements: blueprint monasca-roles
Change-Id: I1da376384a31b89fea1b8a6f907aea35282909a4
The Monasca Grafana fork allows users to log into Grafana with their
OpenStack user credentials and see metrics associated with their
OpenStack project. The long term goal is to enable Keystone support
in upstream Grafana, but this work seems to have stalled.
Partially-Implements: blueprint monasca-grafana
Change-Id: Icc04613b2571c094ae23b66d0bcc38b58c0ee4e1
This changes allows the user to configure a Monasca database
which may be different from the default database.
Partially-Implements: blueprint monasca-roles
Change-Id: Ia905190b8037ecb1782a758c0b65581fe9024bf6
Before this change the HAProxy configuration for Monasca printed:
"Copying over haproxy haproxy config".
This change fixes the message.
TrivialFix
Change-Id: I2823775f6fa5b0be919b567f311c671f1469d19f
Make it easy to override the Keystone endpoints to support deploying
stand-alone Monasca which can integrate with an externally provided
Keystone instance.
Partially-Implements: blueprint monasca-roles
Change-Id: I9ae3b243c792ef88075702b47b62f164a1705c2e
The Monasca Agent collects metrics and in this change is deployed
across the control plane. These metrics are collected into an OpenStack
project. It supports configuring a small number of plugins, which can
be extended in later commits. It also makes the Monasca Agent credentials
available to other roles, such as the common role to allow forwarding
logs to Monasca.
Partially-Implements: blueprint monasca-roles
Change-Id: I76b34fc5e1c76407a45fcf272268d5798b473ca2
A small number of services set the recurse flag when they create
their config directory. This can change permission of files within
the directory, which are later set back to the original state. The
side effect is that the service is then restarted, even though the
net change to the config files amounts to nothing. The expected
behaviour is that a service only restarts if the config *has*
changed. This patch fixes this issue.
Change-Id: Ib6f1ca7b416247f8d455fb25892f4a3b27de03ba
Closes-Bug: 1800480
The format for the Logstash Kafka output plugin worker count is
integer in Logstash 2 and string in Logstash 5. Since we have downgraded
Logstash we need to switch formats here.
Change-Id: I98113dda05bbb44410916c27d7d3bcd59a034a98
Partially-Implements: blueprint monasca-roles
To build the Monasca images, they need to be built from source, however,
this doesn't include Logstash. Logstash does not differentiate between
binary or source install types, so we can just use the default.
Partially-Implements: blueprint monasca-roles
Change-Id: I7eb355f25864f2b993780cd2b000c8bcd13f78e8
* Convert all log levels to lowercase
* Make number of threads configurable
* Improve naming
* Update path and log options for Logstash 2.4
* Change worker count from string to integer for the Kafka output plugin supplied
with Logstash 2.4.
Partially-Implements: blueprint monasca-roles
Change-Id: I9379969ee639dff0b525d7a1a03df59bcf18b98b
* Make number of threads configurable
* Improve naming
* Update path and log options for Logstash 2.4
* Use the correct logs directory
Partially-Implements: blueprint monasca-roles
Change-Id: I09388de4d31740f2786393c957201bd037572226
Increase the maximum payload size so that it slightly exceeds the default
maximum chunk size (8MB) in Fluentd. This prevents the Monasca Log API
dropping logs when under heavy load.
Partially-Implements: blueprint monasca-roles
Change-Id: I13020abde7de1c92acc07352466b814b4a6ce5bc
If the config option is not specified Monasca Notification defaults
to YAML config parsing and fails to load.
Partially-Implements: blueprint monasca-roles
Change-Id: Id85d9765df525b34d43a77a9e4045e0de19aef99
Jira, Slack and possiblly other plugins allow custom templates
for defining the format of notifications. This change lets you
provide these in a templates folder which is copied into the
monasca-notification container.
Partially-Implements: blueprint monasca-roles
Change-Id: Ibc5ba3944d51f6c8ffc8bdc9ed60f43dd91ca7e0
Monasca is not yet compatible with InfluxDB > 1.1.10, which means
that the official Ansible modules for managing InfluxDB don't work [1].
We therefore fall back to manual commands to register the database
and a default retention policy.
[1] https://github.com/influxdata/influxdb-python
#influxdb-pre-v110-users
Partially-Implements: blueprint monasca-roles
Change-Id: I59ceda1e7a6e945b13872089011045db04548b94
This is required for upcoming log query support and it also
causes an error in the Keystone middleware if it's missing.
Partially-Implements: blueprint monasca-roles
Change-Id: I2bcb32bc0c079c799d2b0e45a97b454d38896986
On a single node deployment, the Monasca persister can
limit the rate at which Monasca can persist metrics to
InfluxDB. Increasing the thread count can remove this
bottle neck.
Partially-Implements: blueprint monasca-roles
Change-Id: I763a5ae6aa8c8ab3bf766ab5b58c386da74a188b
The Monasca Persister reads metrics from Kafka and stores them
in a configurable time series database.
Change-Id: I8166b32bfb1583098ab8318a5f38d25bddb81e89
Partially-Implements: blueprint monasca-roles
The Monasca Notification engine generates alerts such as Slack
notifications from alerts.
Change-Id: I84861d5feefe6b6f38acc4dd71e94c386d40b562
Partially-Implements: blueprint monasca-roles
Monasca Thresh is a Storm topology which generates alerts from
metric streams according to alarms defined via the Monasca API.
This change runs the thresholder in local mode, which means that
the log output for the topology is directed to stdout and the
topology is restarted if the container is restarted. A future
change will improve the log collection and introduce a better
way of the checking the topology is running for multi-node
clusters.
Change-Id: I063dca5eead15f3cec009df62f0fc5d857dd4bb0
Partially-Implements: blueprint monasca-roles
Having all services in one giant haproxy file makes altering
configuration for a service both painful and dangerous. Each service
should be configured with a simple set of variables and rendered with a
single unified template.
Available are two new templates:
* haproxy_single_service_listen.cfg.j2: close to the original style, but
only one service per file
* haproxy_single_service_split.cfg.j2: using the newer haproxy syntax
for separated frontend and backend
For now the default will be the single listen block, for ease of
transition.
Change-Id: I6e237438fbc0aa3c89a3c8bd706a53b74e71904b
The log metrics service generates metrics from log messages
which allows further analysis and alerting to be performed
on them. Basic configuration is provided so that metrics
are generated for high level warning logs such as error, or
warning.
Change-Id: I45cc17817c716296451f620f304c0b1108162a56
Partially-Implements: blueprint monasca-roles
This option doesn't actually do anything and a bug to remove
it from the Monasca API config file has been raised.
Partially-Implements: blueprint monasca-roles
Change-Id: I7ec1786b5828ab0135ca86ec040f83a6f4c78d9f
Alembic uses the monasca-agent config file for the DB
credentials. These are therefore no longer required.
Partially-Implements: blueprint monasca-roles
Change-Id: Id076e7a0b838888314159dd9e76697f89adecb5e
Monasca defaults to using the public endpoint to register
Monasca endpoints which is bad practice and doesn't work on
some deployments. This change uses the admin interface by
default.
Change-Id: Ia9c5e041e71867fe72aad43e1344cd2107652d7d
Co-Authored-By: Mark Goddard <mark@stackhpc.com>
Closes-Bug: #1787610
Option auth_uri from group keystone_authtoken is deprecated[1].
Use option www_authenticate_uri from group keystone_authtoken.
[1]https://review.openstack.org/#/c/508522/
Co-Authored-By: confi-surya <singh.surya64mnnit@gmail.com>
Change-Id: Ifd8527d404f1df807ae8196eac2b3849911ddc26
Closes-Bug: #1761907
