Fluentd has a default timeout of 5s for flushing data to ElasticSearch.
If there is a significant backlog of unsent log messages, this timeout
can be exceeded, resulting in Fluentd failing to make further progress.
Raise the default timeout to 60s.
This patch adopts the configuration parameters previously proposed by
Krzysztof Klimonda.
Closes-Bug: #1983031
Closes-Bug: #1896611
Change-Id: I1aaab654a5a0752fccef2cfb8cc0bde4a0ee2562
Adds a new configuration file that provides fluentd with an appropiate regex to match with OpenvSwitch logs in both default files.
The regex is segmented with variable as to isolate the relevant parts of each log message.
Closes-Bug: #1965815
Signed-off-by: Juan Pablo Suazo <jsuazo@whitestack.com>
Change-Id: Ife83c50c048d517a5c8a5dee588f8f7846fcee00
This project [1] can provide a one-stop solution to log collection,
cleaning, indexing, analysis, alarm, visualization, report generation
and other needs, which involves helping operator or maintainer to
quickly solve retrieve problems, grasp the operational health of the
platform, and improve the level of platform management.
[1] https://wiki.openstack.org/wiki/Venus
Change-Id: If3562bbed6181002b76831bab54f863041c5a885
After we remove the problematic, legacy "parser" plugin, the
config only needs the new, modern syntax.
This removes warnings that were logged.
Depends-On: https://review.opendev.org/c/openstack/kolla/+/823071
Change-Id: I33aa06de6ce88288769a8291ebd59e78e07cdbaf
To satisfy the needs of the modern parser plugin.
Needed-By: https://review.opendev.org/c/openstack/kolla/+/823071
Co-Authored-By: Dr. Jens Harbott <harbott@osism.tech>
Change-Id: I2b748d0544c14bebefe8c62aa5aafaaa5371ce53
We stopped using this file in Queens
(https://review.opendev.org/494635), but the file was not removed at
that time.
Change-Id: Ibe5fb291e7c39965f5c4ff5ee4ea0bb1f8e6e9c2
Closes-Bug: #1840158
This change adds the dnsmasq.log for the ironic-dnsmasq container and
also enables more verbose logging when debug logging enabled.
This can be triggered globbaly via 'openstack_logging_debug' or per
service via 'ironic_logging_debug' or 'neutron_logging_debug'.
Change-Id: I0e6b089beb88827effbcc365625eb2df902f5470
Signed-off-by: Maksim Malchuk <maksim.malchuk@gmail.com>
This patch adds support for integrating Prometheus with Fluentd.
This can be used to extract useful information about the status
of Fluentd, such as output buffer capacity and logging rate,
and also to extract metrics from logs via custom Fluentd
configuration. More information can be found here in [1].
[1] https://docs.fluentd.org/monitoring-fluentd/monitoring-prometheus
Change-Id: I233d6dd744848ef1f1589a462dbf272ed0f3aaae
By default, Ansible injects a variable for every fact, prefixed with
ansible_. This can result in a large number of variables for each host,
which at scale can incur a performance penalty. Ansible provides a
configuration option [0] that can be set to False to prevent this
injection of facts. In this case, facts should be referenced via
ansible_facts.<fact>.
This change updates all references to Ansible facts within Kolla Ansible
from using individual fact variables to using the items in the
ansible_facts dictionary. This allows users to disable fact variable
injection in their Ansible configuration, which may provide some
performance improvement.
This change disables fact variable injection in the ansible
configuration used in CI, to catch any attempts to use the injected
variables.
[0] https://docs.ansible.com/ansible/latest/reference_appendices/config.html#inject-facts-as-vars
Change-Id: I7e9d5c9b8b9164d4aee3abb4e37c8f28d98ff5d1
Partially-Implements: blueprint performance-improvements
We really want elasticsearch or monasca to catch all logs,
to providd the required centrailsed logging.
While these appears to make little material difference,
it should make it harder for logs to not get caught by
any of the outputs we have configured.
TrivialFix
Change-Id: I3bb74dcdc3cbe78cd1e1657f44e2a0af9d6508ef
When using elasticsearch 7 with fluentd, you seem to get a lot
of warnings in the docker logs output that look like:
[types removal] Specifying types in bulk requests is deprecated.
The docs suggest adding suppress_type_name to stop these warnings,
and that seems to work without affecting any functionality.
Further info here:
https://github.com/uken/fluent-plugin-elasticsearch/issues/785
Closes-Bug: #1930856
Change-Id: I45be67df3717f78d78bcdc7df69600ab8681922f
Currently the logs tagged with infra.mariadb.xinetd flow into
elasticsearch with no hostname or programname attach, thus making
navigating the logs very hard.
The quick fix is renaming the tag to infra.mariadb-xinetd, which is just
enought to ensure the logs are processed correctly with the existing
filters.
TrivialFix
Change-Id: Icd72206de7c1f701bdf35c8fb3b128ef2dbe29a8
Currently when elasticsearch log output is enabled there are lots
of warnings going into elasticsearch about type being deprecated
and needing to move to @type. This change stops those warnings.
TrivialFix
Change-Id: Ideac1925cb764ad0d7d8416f56d5e4a993c6d8b6
This is a follow up on the change with the following ID:
I337f42e174393f68b43e876ef075a74c887a5314
TrivialFix
Change-Id: Ibb67811d7b086ef9ef4c695ae589171af0c4d657
This change allows a user to forward control plane logs
directly to Elasticsearch from Fluentd, rather than via
the Monasca Log API when Monasca is enabled. The Monasca
Log API can continue to handle tenant logs.
For many use cases this is simpler, reduces resource
consumption and helps to decouple control plane logging
services from tenant logging services.
It may not always be desired, so is optional and off by
default.
Change-Id: I195e8e4b73ca8f573737355908eb30a3ef13b0d6
There are a few issues fixed here:
- The Barbican API service doesn't set a log file, so all the Barbican API
service logs go to loadwsgi.py.log by default.
- The logs in loadwsgi.py.log are not ingested properly by Fluentd.
- uWSGI logs go to barbican-api.log. This would normally be used as the log
file for the Barbican API service logs.
This patch makes the following changes to address the above issues:
- All uWSGI logs (from the Emperor and Vassals) go to barbican_api_uwsgi_access.log
Although these logs aren't strictly all access logs, this follows the existing
pattern for WSGI logs.
- The Barbican API service logs are written to barbican-api.log instead of
loadwsgi.py.log. This follows the pattern used by other OpenStack services.
- Fluentd is configured to parse the Barbican API service logs as it would with
other OpenStack Python services.
Change-Id: I6d03fa8c81c52b6f061514a836bbd15bb6639aaf
Closes-Bug: #1891343
Add TLS support for backend Neutron API Server communication using
HAProxy to perform TLS termination. When used in conjunction with
enabling TLS for service API endpoints, network communication will be
encrypted end to end, from client through HAProxy to the Neutron
service.
Change-Id: Ib333a1f1bd12491df72a9e52d961161210e2d330
Partially-Implements: blueprint add-ssl-internal-network
The goal for this push request is to normalize the construction and use
of internal, external, and admin URLs. While extending Kolla-ansible
to enable a more flexible method to manage external URLs, we noticed
that the same URL was constructed multiple times in different parts
of the code. This can make it difficult for people that want to work
with these URLs and create inconsistencies in a large code base with
time. Therefore, we are proposing here the use of
"single Kolla-ansible variable" per endpoint URL, which facilitates
for people that are interested in overriding/extending these URLs.
As an example, we extended Kolla-ansible to facilitate the "override"
of public (external) URLs with the following standard
"<component/serviceName>.<companyBaseUrl>".
Therefore, the "NAT/redirect" in the SSL termination system (HAproxy,
HTTPD or some other) is done via the service name, and not by the port.
This allows operators to easily and automatically create more friendly
URL names. To develop this feature, we first applied this patch that
we are sending now to the community. We did that to reduce the surface
of changes in Kolla-ansible.
Another example is the integration of Kolla-ansible and Consul, which
we also implemented internally, and also requires URLs changes.
Therefore, this PR is essential to reduce code duplicity, and to
facility users/developers to work/customize the services URLs.
Change-Id: I73d483e01476e779a5155b2e18dd5ea25f514e93
Signed-off-by: Rafael Weingärtner <rafael@apache.org>
A "@type copy" statement is already present at the beginning of each
match element, so extra "type copy" are not needed. They are causing the
following warnings in fluentd logs:
[warn]: parameter 'type' in <match syslog.local0.**>
[warn]: parameter 'type' in <match syslog.local1.**>
This commit also harmonizes indentation of the Monasca config block.
Change-Id: I779c2b942d007acbdd43d999f2fc0cdc131d431f
Related-Bug: #1885873
Time format in Ruby Time.strptime is not accepting padding flags,
therefore we need to remove them for the Fluentd to be able
to parse MariaDB xinetd logs properly.
Change-Id: Iabfa9afdcad505106a5580eb2d058273ee5f7c1f
Closes-Bug: #1886002
In Fluentd v0.12, both the in memory and file buffer chunk size default
to 8MB. In v1.0 the file buffer defaults to 256MB. This can exceed the
Monasca Log or Unified API maximum chunk size which is set to 10MB.
This can result in logs being rejected and filling the local buffer
on disk.
Change-Id: I9c495773db726a3c5cd94b819dff4141737a1d6e
Closes-Bug: #1885885
Co-Authored-By: Sebastian Luna Valero <sebastian.luna.valero@gmail.com>
Resolve trivial syntax error in Fluentd output config for Monasca.
Change-Id: I20b37bb83a76bfabb1126925a1b4f1f59767b7a3
Co-Authored-By: Sebastian Luna Valero <sebastian.luna.valero@gmail.com>
Closes-Bug: #1885873
Currently there is no way to configure a CA certificate bundle file for
fluentd to Elasticsearch communication. This change adds a new variable,
'fluentd_elasticsearch_cacert' with a default value set to the value of
'openstack_cacert.
Closes-Bug: #1885109
Change-Id: I5bbf55a4dd4ccce9fa2635cee720139c088268e3
The Monasca Log API has been removed and in this change we switch
to using the unified API. If dedicated log APIs are required then
this can be supported through configuration. Out of the box the
Monasca API is used for both logs and metrics which is envisaged to
work for most use cases.
In order to use the unified API for logs, we need to disable the
legacy Kafka client. We also rename the Monasca API config file
to remove a warning about using the old style name.
Depends-On: https://review.opendev.org/#/c/728638
Change-Id: I9b6bf5b6690f4b4b3445e7d15a40e45dd42d2e84
