5 Commits

Author SHA1 Message Date
Sam Choraria
81eb58f1e8 Allow beat processors to be defined through configuration data
Processors for each of the beats are currently defined per template and
can be configured directly via template or once deployed. This change
allows processors for all or a subset of beats to be configured through
role data or via an overlay inventory.

Change-Id: I71fc96611082555b43fd0f505219c42c890464ef
2020-02-10 18:34:48 +00:00
Duncan Martin Walker
a192fb3129 Elastic beat index template configuration
Templates for the beat config files have been updated to allow more
configuration options to be passed through to the associated
Elasticsearch index templates. In particular, one can optionally
specify values of index_template_max_docvalue_search to be set at
the creation of the beat index template. This can prevent shard failure
errors when viewing output in Kibana relating to "Trying to retrieve
too many docvalue_fields". Any similar config options can in future
be passed into the template via the elastic_beat_settings object.

Change-Id: Ic9136c8e063bbd231ed280bb446661b251879407
2020-01-23 17:23:48 +00:00
Georgina Shippey
f89dd344c3 Improvements to ILM
Elasticsearch 7 introduced Index Lifecycle Management, my aim is to eventually
replace curator with this.

This update allows for a default ILM policy to be configured for all beats.
The default policy is quite basic: an index will be written to for 15 days or
until the index reaches 30GB in size, then it will remain in the hot stage
(no performance degradation) for another 15 days, after which it will be
deleted.

This still may lead to a situation where the disk space on log nodes could be
filled.

ILM policies can be configured for each beat by overriding the defaults
that are in the role for each beat.

ILM is set up for beats when undergoing an update (elk_package_state="latest").
During ILM set up elasticsearch creates an ILM ready template for the beat and
uses the ILM policy we provision to nodes for the initial ILM policy.
Subsequent ILM policy updates use ES APIs. New ILM policy files are not
provisioned to nodes outside of using the beat upgrade flags, so the policy
file on the node may fall out of step.

Change-Id: I2c5c3abd4bb65075f2377227cbbfe31b68b0dc38
2019-10-09 13:13:56 +00:00
Georgina Shippey
68664a9dc1 Config updates for elk 7.x
Updated ELK config files to elk 7.x reference samples, bringing over
existing customisation from elk_metrics_6x.

Removed deprecated use of --pipeline in elastic_beat_setup/tasks/main.yml,
--pipeline is no longer a valid cli argument.

Updated logstash-pipelines and removed the dynamic insertion of the date into
index names. This function is now done with the new ILM feature in elasticsearch
rather than logstash.

Installation of each beat creates an ILM policy for that beat and this patch
does not change the default policy. It is possible that the default policy
will exhaust the available storage and future work needs to be done to address
this.

The non-beat elements of the logstash pipeline (syslog, collectd and others)
are not yet updated to be compatible with ILM.

Change-Id: I735b64c2b7b93e23562f35266134a176a00af1b7
2019-08-05 07:47:35 +00:00
Georgina Shippey
5e96844123 Duplicate of elk_metrics_6x to elk_metrics_7x
Change-Id: I92a894e31f725a20c684165f93dd4c34b9c8b450
2019-07-10 17:52:49 +01:00