Those loglevels can build up over time and create unnecessary high metrics cardinality.
Change-Id: Ib1a03772d0bd58758430b37b4f2f67126cf86fa3
Closes-bug: #1906796
When the internal VIP is moved in the event of a failure of the active
controller, OpenStack services can become unresponsive as they try to
talk with MariaDB using connections from the SQLAlchemy pool.
It has been argued that OpenStack doesn't really need to use connection
pooling with MariaDB [1]. This commit reduces the use of connection
pooling via two configuration options:
- max_pool_size is set to 1 to allow only a single connection in the
pool (it is not possible to disable connection pooling entirely via
oslo.db, and max_pool_size = 0 means unlimited pool size)
- lower connection_recycle_time from the default of one hour to 10
seconds, which means the single connection in the pool will be
recreated regularly
These settings have shown better reactivity of the system in the event
of a failover.
[1] http://lists.openstack.org/pipermail/openstack-dev/2015-April/061808.html
Change-Id: Ib6a62d4428db9b95569314084090472870417f3d
Closes-Bug: #1896635
It was found to be useless in [1].
It is one of distro_python_version usages.
Note Freezer and Horizon still use python_path (and hence
distro_python_version) for different purposes.
[1] https://review.opendev.org/675822
Change-Id: I6d6d9fdf4c28cb2b686d548955108c994b685bb1
Partially-Implements: blueprint drop-distro-python-version
This updates the Elasticsearch template used by Monasca to
persist logs so that is uses the 'new' string types [1]. As
an aside it helps to make the template more clear; full text
search for log messages, and keyword searches for everything
else.
[1] https://www.elastic.co/blog/strings-are-dead-long-live-strings
Closes-Bug: #1892376
Change-Id: I0cd6bf22d4695d88d93241da4364d170d8d8c80e
This patch introduces a global keep alive timeout value for services
that leverage httpd + wsgi to handle http/https requests. The default
value is one minute.
Change-Id: Icf7cb0baf86b428a60a7e9bbed642999711865cd
Partially-Implements: blueprint add-ssl-internal-network
Switch to the Confluent Kafka client in all remaining Python based
Monasca services. This should allow us to later un-pin the Kafka
messaging version for Monasca.
Change-Id: I42bc78ffe304ba21c448c2e08b025e93a70ddb44
I9b6bf5b6690f4b4b3445e7d15a40e45dd42d2e84 was updated to use the original
config file name during review, but the config file was not renamed
accordingly. The result is that an empty config file is written out.
TrivialFix
Change-Id: I5d0384b38ddb38133e5e11df85d8cf76f4044a64
The Monasca Log API has been removed and in this change we switch
to using the unified API. If dedicated log APIs are required then
this can be supported through configuration. Out of the box the
Monasca API is used for both logs and metrics which is envisaged to
work for most use cases.
In order to use the unified API for logs, we need to disable the
legacy Kafka client. We also rename the Monasca API config file
to remove a warning about using the old style name.
Depends-On: https://review.opendev.org/#/c/728638
Change-Id: I9b6bf5b6690f4b4b3445e7d15a40e45dd42d2e84
Monasca deployment fails on master due to an invalid variable reference
(monasca_log_dir) in the config.json for monasca API and monasca log
API.
This change fixes the issue by correcting the variable definition.
Change-Id: I2ec497fa430c2f301dca6a7653ac988e49007469
Closes-Bug: #1864181
The use of default(omit) is for module parameters, not templates. We
define a default value for openstack_cacert, so it should never be
undefined anyway.
Change-Id: Idfa73097ca168c76559dc4f3aa8bb30b7113ab28
Currently the WSGI configuration for binary images uses python2.7
site-packages in some places. This change uses distro_python_version to
select the correct python path.
Change-Id: Id5f3f0ede106498b9264942fa0399d7c7862c122
Partially-Implements: blueprint python-3
Include a reference to the globally configured Certificate Authority to
all services. Services use the CA to verify HTTPs connections.
Change-Id: I38da931cdd7ff46cce1994763b5c713652b096cc
Partially-Implements: blueprint support-trusted-ca-certificate-file
Currently we don't put global Apache error logs into /var/log/kolla,
this change adds statements that redirect those logs there.
Adapted the logfile names to catch into openstack wsgi logging fluentd
input config and existing logrotate cron entries.
Change-Id: I21216e688a1993239e3e81411a4e8b6f13e138c2
Introduce kolla_address filter.
Introduce put_address_in_context filter.
Add AF config to vars.
Address contexts:
- raw (default): <ADDR>
- memcache: inet6:[<ADDR>]
- url: [<ADDR>]
Other changes:
globals.yml - mention just IP in comment
prechecks/port_checks (api_intf) - kolla_address handles validation
3x interface conditional (swift configs: replication/storage)
2x interface variable definition with hostname
(haproxy listens; api intf)
1x interface variable definition with hostname with bifrost exclusion
(baremetal pre-install /etc/hosts; api intf)
neutron's ml2 'overlay_ip_version' set to 6 for IPv6 on tunnel network
basic multinode source CI job for IPv6
prechecks for rabbitmq and qdrouterd use proper NSS database now
MariaDB Galera Cluster WSREP SST mariabackup workaround
(socat and IPv6)
Ceph naming workaround in CI
TODO: probably needs documenting
RabbitMQ IPv6-only proto_dist
Ceph ms switch to IPv6 mode
Remove neutron-server ml2_type_vxlan/vxlan_group setting
as it is not used (let's avoid any confusion)
and could break setups without proper multicast routing
if it started working (also IPv4-only)
haproxy upgrade checks for slaves based on ipv6 addresses
TODO:
ovs-dpdk grabs ipv4 network address (w/ prefix len / submask)
not supported, invalid by default because neutron_external has no address
No idea whether ovs-dpdk works at all atm.
ml2 for xenapi
Xen is not supported too well.
This would require working with XenAPI facts.
rp_filter setting
This would require meddling with ip6tables (there is no sysctl param).
By default nothing is dropped.
Unlikely we really need it.
ironic dnsmasq is configured IPv4-only
dnsmasq needs DHCPv6 options and testing in vivo.
KNOWN ISSUES (beyond us):
One cannot use IPv6 address to reference the image for docker like we
currently do, see: https://github.com/moby/moby/issues/39033
(docker_registry; docker API 400 - invalid reference format)
workaround: use hostname/FQDN
RabbitMQ may fail to bind to IPv6 if hostname resolves also to IPv4.
This is due to old RabbitMQ versions available in images.
IPv4 is preferred by default and may fail in the IPv6-only scenario.
This should be no problem in real life as IPv6-only is indeed IPv6-only.
Also, when new RabbitMQ (3.7.16/3.8+) makes it into images, this will
no longer be relevant as we supply all the necessary config.
See: https://github.com/rabbitmq/rabbitmq-server/pull/1982
For reliable runs, at least Ansible 2.8 is required (2.8.5 confirmed
to work well). Older Ansible versions are known to miss IPv6 addresses
in interface facts. This may affect redeploys, reconfigures and
upgrades which run after VIP address is assigned.
See: https://github.com/ansible/ansible/issues/63227
Bifrost Train does not support IPv6 deployments.
See: https://storyboard.openstack.org/#!/story/2006689
Change-Id: Ia34e6916ea4f99e9522cd2ddde03a0a4776f7e2c
Implements: blueprint ipv6-control-plane
Signed-off-by: Radosław Piliszek <radoslaw.piliszek@gmail.com>
Monasca log transformer currently throws exceptions on encountering a
non-UTC time offset (+0000):
"""
"exception": "Invalid format: \"2019-08-08 17:39:45 +0100\" is malformed at \" +0100\"",
"config_parsers":"yyyy-MM-dd HH:mm:ss +0000,ISO8601"}
"""
This fix allows logstash to interpret any valid ISO8601 offset.
Change-Id: Id70c3dd9cdcf681e955931f18a054e19cc284c0a
Closes-Bug: #1839597
A user may want to define and use Logstash patterns. This
commit adds support to copy them into the Monasca Log
Transformer container. In the future support could be
added for other Logstash containers.
Change-Id: Id8cde14af6dc7f49714f6b1cb878882d0048d293
This prevents the container's root filesystem from filling up.
Change-Id: Icc5a08c82312d6688edf2ef36562967ac94e8ac9
Depends-On: https://review.opendev.org/#/c/674779
Closes-Bug: #1839149
This plugin is useful for monitoring host clock synchronisation with
an NTP reference. If the delta becomes too large, the metrics from
this plugin can be used to trigger an alarm.
Change-Id: Id1fe6d7c823f8404c19c81ccdeb8b311bcb46e47
Change I0ca38f2cc7d63b9b47eedb304ba7b00a94816f9a removed the roles
middleware from the example paste pipeline.
Change-Id: Ie9a3b0fef395aaf414407f6bae1ac4bca158240d
When using the the default domain name there are issues authenticating
with Keystone. For example, you can only log in on the second attempt
and the Monasca datasource fails to authenticate. Switching to the
default domain id resolves these issues.
Change-Id: I2cb4b2608c74dd853c97e4fc27078930bc72fdf8
backport: stein
If I deploy monasca by setting enable_monasca to true, the monasca_notification
restarts with the following error:
ERROR:__main__:MissingRequiredSource: /var/lib/kolla/config_files/notification_templates/* file is not found
These templates are optional, so we need to mark this directory as optional in
config.json.
Change-Id: Ia2dd835daa7ab1153617cc92f17c2d8d498c73e0
Closes-Bug: #1823726
By parsing the creation_time timestamp in Logstash, Elasticsearch
can parse it correctly. This closes a bug where the creation_time
timestamp was shown as a date shortly after the epoch (1970) when
viewed in Kibana.
Closes-Bug: #1816585
Change-Id: I00decfe94607845ef0eae9bec631a0e729aac3fa
Use <project>_install_type instead of kolla_install_type
to set python_path. For example, general kolla_install_type
is 'binary', but user wants to deploy Horizon from 'source'.
Horizon templates still use python_path=/usr/share/openstack-dashboard,
it is wrong.
Change-Id: Ide6a24e17b1f8ab6506aa5e53f70693706830418
Some Monasca services support sending StatsD metrics to
allow monitoring those services. This commit connects
these services to the StatsD service provided by the
Monasca Agent.
Partially-Implements: blueprint monasca-roles
Change-Id: I1da376384a31b89fea1b8a6f907aea35282909a4
The Monasca Grafana fork allows users to log into Grafana with their
OpenStack user credentials and see metrics associated with their
OpenStack project. The long term goal is to enable Keystone support
in upstream Grafana, but this work seems to have stalled.
Partially-Implements: blueprint monasca-grafana
Change-Id: Icc04613b2571c094ae23b66d0bcc38b58c0ee4e1
This changes allows the user to configure a Monasca database
which may be different from the default database.
Partially-Implements: blueprint monasca-roles
Change-Id: Ia905190b8037ecb1782a758c0b65581fe9024bf6
Make it easy to override the Keystone endpoints to support deploying
stand-alone Monasca which can integrate with an externally provided
Keystone instance.
Partially-Implements: blueprint monasca-roles
Change-Id: I9ae3b243c792ef88075702b47b62f164a1705c2e
The Monasca Agent collects metrics and in this change is deployed
across the control plane. These metrics are collected into an OpenStack
project. It supports configuring a small number of plugins, which can
be extended in later commits. It also makes the Monasca Agent credentials
available to other roles, such as the common role to allow forwarding
logs to Monasca.
Partially-Implements: blueprint monasca-roles
Change-Id: I76b34fc5e1c76407a45fcf272268d5798b473ca2
The format for the Logstash Kafka output plugin worker count is
integer in Logstash 2 and string in Logstash 5. Since we have downgraded
Logstash we need to switch formats here.
Change-Id: I98113dda05bbb44410916c27d7d3bcd59a034a98
Partially-Implements: blueprint monasca-roles
* Convert all log levels to lowercase
* Make number of threads configurable
* Improve naming
* Update path and log options for Logstash 2.4
* Change worker count from string to integer for the Kafka output plugin supplied
with Logstash 2.4.
Partially-Implements: blueprint monasca-roles
Change-Id: I9379969ee639dff0b525d7a1a03df59bcf18b98b
* Make number of threads configurable
* Improve naming
* Update path and log options for Logstash 2.4
* Use the correct logs directory
Partially-Implements: blueprint monasca-roles
Change-Id: I09388de4d31740f2786393c957201bd037572226
Increase the maximum payload size so that it slightly exceeds the default
maximum chunk size (8MB) in Fluentd. This prevents the Monasca Log API
dropping logs when under heavy load.
Partially-Implements: blueprint monasca-roles
Change-Id: I13020abde7de1c92acc07352466b814b4a6ce5bc
If the config option is not specified Monasca Notification defaults
to YAML config parsing and fails to load.
Partially-Implements: blueprint monasca-roles
Change-Id: Id85d9765df525b34d43a77a9e4045e0de19aef99
Jira, Slack and possiblly other plugins allow custom templates
for defining the format of notifications. This change lets you
provide these in a templates folder which is copied into the
monasca-notification container.
Partially-Implements: blueprint monasca-roles
Change-Id: Ibc5ba3944d51f6c8ffc8bdc9ed60f43dd91ca7e0
This is required for upcoming log query support and it also
causes an error in the Keystone middleware if it's missing.
Partially-Implements: blueprint monasca-roles
Change-Id: I2bcb32bc0c079c799d2b0e45a97b454d38896986
On a single node deployment, the Monasca persister can
limit the rate at which Monasca can persist metrics to
InfluxDB. Increasing the thread count can remove this
bottle neck.
Partially-Implements: blueprint monasca-roles
Change-Id: I763a5ae6aa8c8ab3bf766ab5b58c386da74a188b
The Monasca Persister reads metrics from Kafka and stores them
in a configurable time series database.
Change-Id: I8166b32bfb1583098ab8318a5f38d25bddb81e89
Partially-Implements: blueprint monasca-roles
The Monasca Notification engine generates alerts such as Slack
notifications from alerts.
Change-Id: I84861d5feefe6b6f38acc4dd71e94c386d40b562
Partially-Implements: blueprint monasca-roles
