This updates the helm tests for the fluent-logging chart to make
them more robust in being able to check for indexes defined in the
chart. This is done by calculating the combined flush interval
for both fluentbit and fluentd, and sleeping for at least one
flush cycle to ensure all functional indexes have received logged
events.
Then, the test determines what indexes should exist by checking
all Elasticsearch output configuration entries, determining
whether to use the default logstash-* index or the logstash_prefix
configuration value if it exists. For each of these indexes, the
test checks whether the indexes have successful hits (ie: there
have been successful entries into these indexes)
Change-Id: I36ed7b707491e92da6ac4b422936a1d65c92e0ac
This updates the kubernetes plugin test for fluent-logging to
search across all indices instead of the default logstash-* index
to account for custom indexes created for the events tagged with
the kubernetes plugin.
This also makes the search pattern for the tag more flexible to
account for any arbitrary number of prefixes and/or suffixes
added to the 'kube' tag as a result of any processing done in
fluentd.
Change-Id: Ib1a431cc8b2ca2cc143a8c8337b87f54f56d1029
This updates the helm tests and the elasticsearch template job.
This changes the tests to conditionally check whether the
template job is enabled and the templates key is not empty, and
uses the result to determine whether to test for the existence
of those templates (to account for situations where the job is
disabled).
This updates the job to also check whether there are templates
defined in additional to checking whether the job itself is
enabled.
Change-Id: I14cedeb8d8a4444a73ea974426c3b0f136d1b698
This updates the fluent-logging chart to support the creation of
an arbitrary number of templates for elasticsearch. This allows
for the definition of multiple index mappings driven via the
chart's values. This provides flexibility in determining specific
structures for indexes that may differ between log types.
This also moves to define these mappings via json instead of XML.
As gotpl can convert yaml directly to json, and elasticsearch can
ingest json directly for index creation, we no longer need an XML
helper function to generate the required configuration. This helps
reduce the number of helper functions we need to maintain
Change-Id: I3c85fb9a1e700eb1592d96f83e632172d0eb2681
This moves Elasticsearch and Kibana to use the latest version
(6.2.2), as the images we were using are no longer supported with
the 6.x release. There was a change in the doc reference in the
log entries that prevented the previous ES version from indexing
those entries, resulting in a busted gate. Moving Kibana to 6.2.2
was required to match major/minor versions with Elasticsearch
The Elasticsearch version change also required changing config file
locations, changing the entrypoint used for launching the service,
changing the running user for the elasticsearch service, and
updated the ES tests as some of the API responses changed between
versions
This also required updating the elasticsearch template job as the
mapping definition entries changed between versions
Change-Id: Ia4cd9a66851754a1bb8f225c7e24513c43568e93
Fluent-logging stores logs on a elasticsearch by default. Elasticsearch
stores all fields as tokens by default, but some fields shoud be stored
for purposes such as retrieval, without splitting. Mapping in elasticsearch
is used to define a property of fields and Template can defines the mapping
for an index. fluent-logging use it to define the index structure. Specific
index type can be defined on the value file.
Change-Id: Id597111f478fcddf709b36d2db9ac5a5d6d8206f
Run elasticsearch behind apache as a reverse proxy to supply basic
auth for elasticsearch, as xpack requires a suscription to support
security for elasticsearch
Change-Id: I72d06ed9cd2179ead86ddc67db33c68a1e40c437
This introduces an initial helm chart for fluent logging.
It provides a functional fluent-bit and fluentd deployment to
use in conjunction with elasticsearch and kibana to consume
and aggregate logs from all resource types in a cluster.
It can deliver logs to kafka for external tools to consume.
This PS moves fluent-logging chart from osh-addons, osh to
osh-infra repo.
previous ps(addon): https://review.openstack.org/#/c/507023/
previous ps(osh): https://review.openstack.org/#/c/514622/
Specification: https://review.openstack.org/#/c/505491/
Partially implements: blueprint osh-logging-framework
Change-Id: I72e580aa3a197550060fc07af8396a7c8368d40b
