This change allows the embedded ansible process to be configurable by
the end user.
* Python requirements and ansible roles will all now be user
configurable.
* Setup is now a local only playbook. This playbook replaces the bash
commands we were rerunning when the `bootstrap-embedded-ansible.sh`
script was executed.
* Embedded ansible version is now 2.7.5 as default.
* Deprecation warnings have been resolved.
* Tests impacted by this change have been updated.
Change-Id: I4303c44e249cda31457a4f05a681e298d225a8b7
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
This change reduces code throughout the playbooks thereby speeding up
the task execution.
* A new role named `elastic_beat_setup` was created to
facilitate template setup as needed.
* Beats retention policies are now defined on the elastic-logstash
nodes instead of on all target hosts. This method will speed-up
deployments on massive installations while streamlining all deployments.
* Kibana variable assumptions have been fixed. This will allow for
deployments without Kibana to be accomplished.
Change-Id: I36343264042e81dfcb68bad0f6c3a503e525eceb
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
These options could be "undefined" which is an object and not json
serializable. This change ensures if an option is undefined it defaults
to an empty set which will allow the option to be json serialized.
Change-Id: I1a81bafa441aa6400bfbec50d57e56df4d09bda3
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
The template creates can take a lot of time when dealing with large data
sets. This change makes it so template create will only happen on a
greenfield installation or when upgrading. A simple rerun of the
playbooks will not trigger template creations which will allow deployers
the ability to better change or modify deploymens without having to
worry about extended runtimes due to template interactions.
Change-Id: Ia9b77277553fbdbe0444737f39ec3de75f07cc0f
This change updates the roles / playbooks to begin using Elasticsearch
relesae 6.5.x. Core to this change is the conversion of the journalbeat
role from custom compiled go, to simple package install which was made
possible by the folks at elastic within this release. Because of the
conversion the "beats-community" playbook has been removed given its now
empty.
A change to the bootstrap script was made allowing it to parse an OS id
with a "-" in it, like "opensuse-tumbleweed".
Change-Id: Ic9b80234d6a6ce876bff885f3223874602d55dd6
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
As the role based macro links are pointing to unreachable destination
and are not required, they will be removed.
Additionally the timesouts are increased for API commands to ES, along
with minor changes around the upstart system manager
Change-Id: I2572bce230af2fd43261c9b0bf903bfd9655959e
This change adds fields such as host OS, version and platform to the
core beats output, giving extra query/filter capabilities.
Change-Id: Iff61bb4402eaa45b8f1c134a6a39cebe6613cbf3
Previously the beat setup tasks were tagged with 'setup' but the include
statements were not, so the tasks were always skipped when using '--tags
setup'. This change adds tags to the includes so that the tasks are executed
as expected
Change-Id: If16069cd273d84a22b229b8140e5a8d56eed86d1
This patch adds support for ARM64 beats. Unfortunately, Elastic does
not publish any packages, so this points at local builds. Also, it
looks like Packetbeat fails to builds so for now we just don't do
anything about it on ARM.
Change-Id: I1889ce51f1a4c13c311165b8b76dde7c71ecfa2d
This patch aims to provide the user a way to enable/disable
beats by overriding {beatname}_service_state variable accordingly
to the beats that the users wants to be receiving data.
There are some use cases that users just wants a subset of the
beats provided, mostly to avoid unecessary use of bandwidth
with data that woudn't be used. So the way that this patch proposes
this use case is just enable/disable after install, keeping the service
installed in case of the users needs it.
Change-Id: I2251095d7fcfc48a239fe9d4984269503cc835da
The multi-logstash pipeline setup, while amazingly fast, was crashing
and causing index errors when under high load for a long period of time.
Because of the crashing behavior and the fact that the folks from
Elastic describe multi-pipeline queues to be "beta" at this time the
logstash pipelines have been converted back into a single pipeline.
The memory backed queue options are now limited by a ram disk (tmpfs)
which will ensure that a burst within the queue does not cause OOM
issues and ensures a highly performant deployment and limiting memory
usage at the same time. Memory backed queues will be enabled when the
underlying system is using "rotational" media as detected by ansible
facts. This will ensure a fast and consistent experience across all
deployment types.
Pipeline/ml/template/dashboard setup has been added to the beat
configurations which will ensure beats are properly configured even
when running in an isolated deployment and outside of normal operations
where beats are generally configured on the first data node.
Change-Id: Ie3c775f98b14f71bcbed05db9cb1c5aa46d9c436
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
It looks like when using a normal include, Ansible tries to include
all sorts of extra things, further breaking the environment by trying
to resolve unaccessible variables.
This patch refactors all of those includes to macros therefore including
a single file and also decreasing the memory usage to avoid copying
over the entire context.
Change-Id: Ie8733c7d52b1fc5bde484855988bddf6a06dbe00
This change adds SUSE 42.3 support to the elastic telemetry solutions.
Change-Id: Ibe93ea0d1ead9e7fe6da16d89989cfe5ade0f43e
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
This change will help with organization throughout the stack.
Change-Id: I2ad865db534ae1d377bbdecd4b421ee0fc802536
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
This change adds Ubuntu 14.04 support to the project.
Change-Id: I20695e19409b63c6e1def4ccf8929c6d52be647e
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
This change adds the scaffolding required to get multi-distro support
running in the roles. The change breaks up our playbooks converting all
of the tasks into various roles with internal dependencies. While this
will improve execution time, the change is being done to reduce boiler
plate and to allow us to build on the pattern used in OSA to provide
multi-distro capabilities.
A side effect of this change is a major improvement in idempotency. The
playbooks should now be 100% idempotent.
All of the templates have been left in the main playbook directory. This
was done to help ease the transition. In a future PR the template
structure will be moved into the roles where it needs to be.
The main variable files has been left intact. This file will be carved
up into role defaults in a future PR.
Change-Id: I938a10564128ce4078fa12edcf614dcdbd684b25
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>