8 Commits

Author SHA1 Message Date
Duncan Martin Walker
81ab7008e1 Exposed beat logging levels
The elastic beats are currently deployed with a default logging level of
info, which can produce large amounts of data, particularly in the
case that beats are monitoring one another's logs, and then logging this fact.
This commit exposes the configuration, allowing it to be set by the
variable log_level. The default level is unchanged.

Change-Id: Idbf715fc99450003920a45323123fcba919f2a83
2020-02-18 10:09:48 +00:00
Sam Choraria
81eb58f1e8 Allow beat processors to be defined through configuration data
Processors for each of the beats are currently defined per template and
can be configured directly via template or once deployed. This change
allows processors for all or a subset of beats to be configured through
role data or via an overlay inventory.

Change-Id: I71fc96611082555b43fd0f505219c42c890464ef
2020-02-10 18:34:48 +00:00
Zuul
f73c2d05dc Merge "Exposed config for logstash elasticsearch endpoints" 2020-02-10 17:07:45 +00:00
Duncan Martin Walker
c30ad4c447 Increase logstash output connection timeout for beats
When using Metricbeat in an environment where only the system module
is enabled, the configuration reports system metrics to logstash every
60s. However, the current connection timout is shorter than this,
leading to the connection resetting and re-opening repeatedly due to
i/o timeouts. This commit increases the timeout to 90s, such that the
connection stays open and errors aren't reported to the metricbeat
logs.

Change-Id: Ib30f983df71c0b0381bfa23d9d2c7f1220184a4b
2020-02-06 10:49:55 +00:00
Duncan Martin Walker
b21034c4b8 Exposed config for logstash elasticsearch endpoints
The Elasticsearch endpoints in use by logstash are currently hardcoded
to localhost in both the logstash monitoring config and the default
logstash pipeline template. This commit allows this to be configured,
such that (multiple) remote endpoints can be specified for the case that
logstash is not co-located with elasticsearch. The default behaviour is
unchanged.

Change-Id: Id871f201168ff882a5b37677747c3484f908298f
2020-02-05 16:41:24 +00:00
Duncan Martin Walker
a192fb3129 Elastic beat index template configuration
Templates for the beat config files have been updated to allow more
configuration options to be passed through to the associated
Elasticsearch index templates. In particular, one can optionally
specify values of index_template_max_docvalue_search to be set at
the creation of the beat index template. This can prevent shard failure
errors when viewing output in Kibana relating to "Trying to retrieve
too many docvalue_fields". Any similar config options can in future
be passed into the template via the elastic_beat_settings object.

Change-Id: Ic9136c8e063bbd231ed280bb446661b251879407
2020-01-23 17:23:48 +00:00
Georgina Shippey
68664a9dc1 Config updates for elk 7.x
Updated ELK config files to elk 7.x reference samples, bringing over
existing customisation from elk_metrics_6x.

Removed deprecated use of --pipeline in elastic_beat_setup/tasks/main.yml,
--pipeline is no longer a valid cli argument.

Updated logstash-pipelines and removed the dynamic insertion of the date into
index names. This function is now done with the new ILM feature in elasticsearch
rather than logstash.

Installation of each beat creates an ILM policy for that beat and this patch
does not change the default policy. It is possible that the default policy
will exhaust the available storage and future work needs to be done to address
this.

The non-beat elements of the logstash pipeline (syslog, collectd and others)
are not yet updated to be compatible with ILM.

Change-Id: I735b64c2b7b93e23562f35266134a176a00af1b7
2019-08-05 07:47:35 +00:00
Georgina Shippey
5e96844123 Duplicate of elk_metrics_6x to elk_metrics_7x
Change-Id: I92a894e31f725a20c684165f93dd4c34b9c8b450
2019-07-10 17:52:49 +01:00