The elastic beats are currently deployed with a default logging level of
info, which can produce large amounts of data, particularly in the
case that beats are monitoring one another's logs, and then logging this fact.
This commit exposes the configuration, allowing it to be set by the
variable log_level. The default level is unchanged.
Change-Id: Idbf715fc99450003920a45323123fcba919f2a83
Processors for each of the beats are currently defined per template and
can be configured directly via template or once deployed. This change
allows processors for all or a subset of beats to be configured through
role data or via an overlay inventory.
Change-Id: I71fc96611082555b43fd0f505219c42c890464ef
When using Metricbeat in an environment where only the system module
is enabled, the configuration reports system metrics to logstash every
60s. However, the current connection timout is shorter than this,
leading to the connection resetting and re-opening repeatedly due to
i/o timeouts. This commit increases the timeout to 90s, such that the
connection stays open and errors aren't reported to the metricbeat
logs.
Change-Id: Ib30f983df71c0b0381bfa23d9d2c7f1220184a4b
The Elasticsearch endpoints in use by logstash are currently hardcoded
to localhost in both the logstash monitoring config and the default
logstash pipeline template. This commit allows this to be configured,
such that (multiple) remote endpoints can be specified for the case that
logstash is not co-located with elasticsearch. The default behaviour is
unchanged.
Change-Id: Id871f201168ff882a5b37677747c3484f908298f
Templates for the beat config files have been updated to allow more
configuration options to be passed through to the associated
Elasticsearch index templates. In particular, one can optionally
specify values of index_template_max_docvalue_search to be set at
the creation of the beat index template. This can prevent shard failure
errors when viewing output in Kibana relating to "Trying to retrieve
too many docvalue_fields". Any similar config options can in future
be passed into the template via the elastic_beat_settings object.
Change-Id: Ic9136c8e063bbd231ed280bb446661b251879407
Updated ELK config files to elk 7.x reference samples, bringing over
existing customisation from elk_metrics_6x.
Removed deprecated use of --pipeline in elastic_beat_setup/tasks/main.yml,
--pipeline is no longer a valid cli argument.
Updated logstash-pipelines and removed the dynamic insertion of the date into
index names. This function is now done with the new ILM feature in elasticsearch
rather than logstash.
Installation of each beat creates an ILM policy for that beat and this patch
does not change the default policy. It is possible that the default policy
will exhaust the available storage and future work needs to be done to address
this.
The non-beat elements of the logstash pipeline (syslog, collectd and others)
are not yet updated to be compatible with ILM.
Change-Id: I735b64c2b7b93e23562f35266134a176a00af1b7
