14 Commits

Author SHA1 Message Date
Kevin Carter
3a69a1c43d Update elk_6x for 6.5.x
This change updates the roles / playbooks to begin using Elasticsearch
relesae 6.5.x. Core to this change is the conversion of the journalbeat
role from custom compiled go, to simple package install which was made
possible by the folks at elastic within this release. Because of the
conversion the "beats-community" playbook has been removed given its now
empty.

A change to the bootstrap script was made allowing it to parse an OS id
with a "-" in it, like "opensuse-tumbleweed".

Change-Id: Ic9b80234d6a6ce876bff885f3223874602d55dd6
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2018-11-27 18:56:22 +00:00
Bjoern Teipel
7c819e05dc Fixes related to installation on Trusty
As the role based macro links are pointing to unreachable destination
and are not required, they will be removed.
Additionally the timesouts are increased for API commands to ES, along
with minor changes around the upstart system manager

Change-Id: I2572bce230af2fd43261c9b0bf903bfd9655959e
2018-10-22 18:12:42 +00:00
Zuul
27ef77f601 Merge "Add tags for beats setup tasks" 2018-09-25 15:24:03 +00:00
Jonathan Rosser
ac46b2be6a Fix journalbeat installation for mixed environments
The previous code would terminate the play immediatley if any hosts
in the environment did not have a journal directory. This change runs
the journalbeat install role selectively on hosts that have the journal
directory, and skips hosts that do not.

In addition a legacy task to stop the play after uninstallation is removed,
this functionality is currently broken.

Change-Id: I412e3594c4b2292caafafb580bb4ede9ccfd3944
2018-09-25 12:34:21 +01:00
Jonathan Rosser
8cf20bfea2 Add tags for beats setup tasks
Previously the beat setup tasks were tagged with 'setup' but the include
statements were not, so the tasks were always skipped when using '--tags
setup'. This change adds tags to the includes so that the tasks are executed
as expected

Change-Id: If16069cd273d84a22b229b8140e5a8d56eed86d1
2018-09-25 12:30:22 +01:00
Guilherme Steinmüller
7430f6c8d5 Add variable to define a beat service state
This patch aims to provide the user a way to enable/disable
beats by overriding {beatname}_service_state variable accordingly
to the beats that the users wants to be receiving data.

There are some use cases that users just wants a subset of the
beats provided, mostly to avoid unecessary use of bandwidth
with data that woudn't be used. So the way that this patch proposes
this use case is just enable/disable after install, keeping the service
installed in case of the users needs it.

Change-Id: I2251095d7fcfc48a239fe9d4984269503cc835da
2018-09-20 16:27:20 +00:00
Kevin Carter
0d4a4a92c7
Converg the logstash pipelines and enhance memory backed queues
The multi-logstash pipeline setup, while amazingly fast, was crashing
and causing index errors when under high load for a long period of time.
Because of the crashing behavior and the fact that the folks from
Elastic describe multi-pipeline queues to be "beta" at this time the
logstash pipelines have been converted back into a single pipeline.

The memory backed queue options are now limited by a ram disk (tmpfs)
which will ensure that a burst within the queue does not cause OOM
issues and ensures a highly performant deployment and limiting memory
usage at the same time. Memory backed queues will be enabled when the
underlying system is using "rotational" media as detected by ansible
facts. This will ensure a fast and consistent experience across all
deployment types.

Pipeline/ml/template/dashboard setup has been added to the beat
configurations which will ensure beats are properly configured even
when running in an isolated deployment and outside of normal operations
where beats are generally configured on the first data node.

Change-Id: Ie3c775f98b14f71bcbed05db9cb1c5aa46d9c436
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2018-09-16 23:44:58 -05:00
Mohammed Naser
4933114a94 Refactor templates to use a single macro template file
It looks like when using a normal include, Ansible tries to include
all sorts of extra things, further breaking the environment by trying
to resolve unaccessible variables.

This patch refactors all of those includes to macros therefore including
a single file and also decreasing the memory usage to avoid copying
over the entire context.

Change-Id: Ie8733c7d52b1fc5bde484855988bddf6a06dbe00
2018-09-02 20:04:28 +00:00
Kevin Carter
e4c84aa28d
Add Redhat to the ELK deployment capabilities
Change-Id: Id34e046a546f8d0878843596f53e400165e37c6e
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2018-08-13 18:59:57 -05:00
Kevin Carter
bf6a8d85e7
Add SUSE support
This change adds SUSE 42.3 support to the elastic telemetry solutions.

Change-Id: Ibe93ea0d1ead9e7fe6da16d89989cfe5ade0f43e
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2018-08-12 23:53:23 -05:00
Kevin Carter
45df59ed7e
move the bulk to templates into the new roles
This change will help with organization throughout the stack.

Change-Id: I2ad865db534ae1d377bbdecd4b421ee0fc802536
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2018-08-12 22:19:19 -05:00
Kevin Carter
79c3a3cf93
Add trusty support to the project
This change adds Ubuntu 14.04 support to the project.

Change-Id: I20695e19409b63c6e1def4ccf8929c6d52be647e
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2018-08-11 00:00:03 -05:00
Kevin Carter
6aa88dd7b7
Add bionic job to elk_metrics_6x testing
Change-Id: I67bbfa116c45a82eb8b5bc191d19d203493f0b00
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2018-08-09 00:44:32 -05:00
Kevin Carter
3a0b3d2cde
Convert playbooks into roles
This change adds the scaffolding required to get multi-distro support
running in the roles. The change breaks up our playbooks converting all
of the tasks into various roles with internal dependencies. While this
will improve execution time, the change is being done to reduce boiler
plate and to allow us to build on the pattern used in OSA to provide
multi-distro capabilities.

A side effect of this change is a major improvement in idempotency. The
playbooks should now be 100% idempotent.

All of the templates have been left in the main playbook directory. This
was done to help ease the transition. In a future PR the template
structure will be moved into the roles where it needs to be.

The main variable files has been left intact. This file will be carved
up into role defaults in a future PR.

Change-Id: I938a10564128ce4078fa12edcf614dcdbd684b25
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2018-08-09 00:41:05 -05:00