20 Commits

Author SHA1 Message Date
Kevin Carter
aabf90d1a4 Update heartbeat config for the latest stable release
Change-Id: I0db06c07ac9320c5db927f23e32fdb8194e5106b
Signed-off-by: Kevin Carter <kevin@cloudnull.com>
2019-02-27 06:26:13 +00:00
Kevin Carter
7491b6df8e Update the embedded-ansible-setup process to be configurable
This change allows the embedded ansible process to be configurable by
the end user.
  * Python requirements and ansible roles will all now be user
    configurable.
  * Setup is now a local only playbook. This playbook replaces the bash
    commands we were rerunning when the `bootstrap-embedded-ansible.sh`
    script was executed.
  * Embedded ansible version is now 2.7.5 as default.
  * Deprecation warnings have been resolved.
  * Tests impacted by this change have been updated.

Change-Id: I4303c44e249cda31457a4f05a681e298d225a8b7
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2019-01-04 11:46:19 -06:00
Kevin Carter
5586d8a80f Convert template setup to a role
This change reduces code throughout the playbooks thereby speeding up
the task execution.
  * A new role named `elastic_beat_setup` was created to
    facilitate template setup as needed.
  * Beats retention policies are now defined on the elastic-logstash
    nodes instead of on all target hosts. This method will speed-up
    deployments on massive installations while streamlining all deployments.
  * Kibana variable assumptions have been fixed. This will allow for
    deployments without Kibana to be accomplished.

Change-Id: I36343264042e81dfcb68bad0f6c3a503e525eceb
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2019-01-02 20:38:47 -06:00
Kevin Carter
9a896aa81a Fence options before casting to json
These options could be "undefined" which is an object and not json
serializable. This change ensures if an option is undefined it defaults
to an empty set which will allow the option to be json serialized.

Change-Id: I1a81bafa441aa6400bfbec50d57e56df4d09bda3
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2018-12-20 17:38:18 -06:00
Kevin Carter
eb3bcb8daa Add local facts for template creations
The template creates can take a lot of time when dealing with large data
sets. This change makes it so template create will only happen on a
greenfield installation or when upgrading. A simple rerun of the
playbooks will not trigger template creations which will allow deployers
the ability to better change or modify deploymens without having to
worry about extended runtimes due to template interactions.

Change-Id: Ia9b77277553fbdbe0444737f39ec3de75f07cc0f
2018-12-19 17:59:21 -06:00
Kevin Carter
3a69a1c43d Update elk_6x for 6.5.x
This change updates the roles / playbooks to begin using Elasticsearch
relesae 6.5.x. Core to this change is the conversion of the journalbeat
role from custom compiled go, to simple package install which was made
possible by the folks at elastic within this release. Because of the
conversion the "beats-community" playbook has been removed given its now
empty.

A change to the bootstrap script was made allowing it to parse an OS id
with a "-" in it, like "opensuse-tumbleweed".

Change-Id: Ic9b80234d6a6ce876bff885f3223874602d55dd6
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2018-11-27 18:56:22 +00:00
Bjoern Teipel
7c819e05dc Fixes related to installation on Trusty
As the role based macro links are pointing to unreachable destination
and are not required, they will be removed.
Additionally the timesouts are increased for API commands to ES, along
with minor changes around the upstart system manager

Change-Id: I2572bce230af2fd43261c9b0bf903bfd9655959e
2018-10-22 18:12:42 +00:00
Zuul
b2c9cf4221 Merge "Add host metadata to core beats output" 2018-09-25 15:59:17 +00:00
Jonathan Rosser
1a48236ced Add host metadata to core beats output
This change adds fields such as host OS, version and platform to the
core beats output, giving extra query/filter capabilities.

Change-Id: Iff61bb4402eaa45b8f1c134a6a39cebe6613cbf3
2018-09-25 13:34:18 +01:00
Jonathan Rosser
8cf20bfea2 Add tags for beats setup tasks
Previously the beat setup tasks were tagged with 'setup' but the include
statements were not, so the tasks were always skipped when using '--tags
setup'. This change adds tags to the includes so that the tasks are executed
as expected

Change-Id: If16069cd273d84a22b229b8140e5a8d56eed86d1
2018-09-25 12:30:22 +01:00
Mohammed Naser
12c9687437 Add ARM64 support
This patch adds support for ARM64 beats.  Unfortunately, Elastic does
not publish any packages, so this points at local builds.  Also, it
looks like Packetbeat fails to builds so for now we just don't do
anything about it on ARM.

Change-Id: I1889ce51f1a4c13c311165b8b76dde7c71ecfa2d
2018-09-23 16:17:04 +00:00
Guilherme Steinmüller
7430f6c8d5 Add variable to define a beat service state
This patch aims to provide the user a way to enable/disable
beats by overriding {beatname}_service_state variable accordingly
to the beats that the users wants to be receiving data.

There are some use cases that users just wants a subset of the
beats provided, mostly to avoid unecessary use of bandwidth
with data that woudn't be used. So the way that this patch proposes
this use case is just enable/disable after install, keeping the service
installed in case of the users needs it.

Change-Id: I2251095d7fcfc48a239fe9d4984269503cc835da
2018-09-20 16:27:20 +00:00
Kevin Carter
0d4a4a92c7
Converg the logstash pipelines and enhance memory backed queues
The multi-logstash pipeline setup, while amazingly fast, was crashing
and causing index errors when under high load for a long period of time.
Because of the crashing behavior and the fact that the folks from
Elastic describe multi-pipeline queues to be "beta" at this time the
logstash pipelines have been converted back into a single pipeline.

The memory backed queue options are now limited by a ram disk (tmpfs)
which will ensure that a burst within the queue does not cause OOM
issues and ensures a highly performant deployment and limiting memory
usage at the same time. Memory backed queues will be enabled when the
underlying system is using "rotational" media as detected by ansible
facts. This will ensure a fast and consistent experience across all
deployment types.

Pipeline/ml/template/dashboard setup has been added to the beat
configurations which will ensure beats are properly configured even
when running in an isolated deployment and outside of normal operations
where beats are generally configured on the first data node.

Change-Id: Ie3c775f98b14f71bcbed05db9cb1c5aa46d9c436
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2018-09-16 23:44:58 -05:00
Mohammed Naser
4933114a94 Refactor templates to use a single macro template file
It looks like when using a normal include, Ansible tries to include
all sorts of extra things, further breaking the environment by trying
to resolve unaccessible variables.

This patch refactors all of those includes to macros therefore including
a single file and also decreasing the memory usage to avoid copying
over the entire context.

Change-Id: Ie8733c7d52b1fc5bde484855988bddf6a06dbe00
2018-09-02 20:04:28 +00:00
Kevin Carter
e4c84aa28d
Add Redhat to the ELK deployment capabilities
Change-Id: Id34e046a546f8d0878843596f53e400165e37c6e
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2018-08-13 18:59:57 -05:00
Kevin Carter
bf6a8d85e7
Add SUSE support
This change adds SUSE 42.3 support to the elastic telemetry solutions.

Change-Id: Ibe93ea0d1ead9e7fe6da16d89989cfe5ade0f43e
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2018-08-12 23:53:23 -05:00
Kevin Carter
45df59ed7e
move the bulk to templates into the new roles
This change will help with organization throughout the stack.

Change-Id: I2ad865db534ae1d377bbdecd4b421ee0fc802536
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2018-08-12 22:19:19 -05:00
Kevin Carter
79c3a3cf93
Add trusty support to the project
This change adds Ubuntu 14.04 support to the project.

Change-Id: I20695e19409b63c6e1def4ccf8929c6d52be647e
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2018-08-11 00:00:03 -05:00
Kevin Carter
6aa88dd7b7
Add bionic job to elk_metrics_6x testing
Change-Id: I67bbfa116c45a82eb8b5bc191d19d203493f0b00
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2018-08-09 00:44:32 -05:00
Kevin Carter
3a0b3d2cde
Convert playbooks into roles
This change adds the scaffolding required to get multi-distro support
running in the roles. The change breaks up our playbooks converting all
of the tasks into various roles with internal dependencies. While this
will improve execution time, the change is being done to reduce boiler
plate and to allow us to build on the pattern used in OSA to provide
multi-distro capabilities.

A side effect of this change is a major improvement in idempotency. The
playbooks should now be 100% idempotent.

All of the templates have been left in the main playbook directory. This
was done to help ease the transition. In a future PR the template
structure will be moved into the roles where it needs to be.

The main variable files has been left intact. This file will be carved
up into role defaults in a future PR.

Change-Id: I938a10564128ce4078fa12edcf614dcdbd684b25
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2018-08-09 00:41:05 -05:00