This change removes the {h,q}_mem options in favor of a new variable which
clearly states the upper and lower limits for a given deployment. This change
also makes these options a lot more conservative by default which will allow
the deployment to better run on shared infra.
Change-Id: I169f457198c11edc4881a04df65312f6c4f67feb
Signed-off-by: Kevin Carter <kevin@cloudnull.com>
This change will allow logstash to ingest metrics from collectd. New
options have been added to enable the deployment and configure it.
Change-Id: I995c0db69fc68d5f5bcae27ce16956876368e2a8
Signed-off-by: cloudnull <kevin@cloudnull.com>
This change makes it possible for users to set the `elastic_heap_size_default`
value. Before this change, the option was unreachable due to a series of facts
ganerated template values. The options `elastic_heap_size` or `logstash_heap_size`
have also been exposed giving deployers the ability to define service specific
heap sizes as needed.
Change-Id: Ida3a57fdcff388f8e4bb3f325b787205a6183970
Signed-off-by: Kevin Carter <kevin@cloudnull.com>
The queue path within logstash may be a symlink which will fail to mount
as tempfs. To ensure queue path can be tempfs, a readlink command is
used to fetch the true path, which will be used in a mount when nessisary.
Change-Id: I5fe6bf311e0621c98766ae458371b5f11f89a61f
Signed-off-by: Kevin Carter <kevin@cloudnull.com>
This change allows the embedded ansible process to be configurable by
the end user.
* Python requirements and ansible roles will all now be user
configurable.
* Setup is now a local only playbook. This playbook replaces the bash
commands we were rerunning when the `bootstrap-embedded-ansible.sh`
script was executed.
* Embedded ansible version is now 2.7.5 as default.
* Deprecation warnings have been resolved.
* Tests impacted by this change have been updated.
Change-Id: I4303c44e249cda31457a4f05a681e298d225a8b7
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
The findmnt command is printing the fsroot as [/dir] at the end
of the source device and that is causing as a parsing error to
occur when it is assumed that only the device string is returned leading
to an error looking up an invalid device.
Change-Id: If95f8e0ed8154ad0277972159afac9f967b79c8f
These changes mirror systemd tunables for elasticsearch and are needed
to ensure any OS without systemd (like Ubuntu 14.04) has the same
capabilities and OS's with systemd. This also adds a specific sysctl
file to use when making sysctl changes. This will ensure we're not
subjecting our deployment to other changes from other sources, like an
OSA playbook run.
Change-Id: Ic0e0bc0f93a12298c1e2f634cf5a1b4c6be2995e
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
The logstash and elasticsearch performance can be improved by using
async index options, pulling back the refresh interval, and by not
fingerprinting every document.
* Async translog allows elasticsearch to using run fsync in the
background instead of blocking
* the refresh interval will now be 5x the number of replicas with a cap
of 30. This integer is representitive of the seconds between index
refresh calls which greatly lowers the load generated across the
cluster.
* All documents were fingerprinted before writting to the cluster. This
was a costly operation as elasticsearch will do a forward lookup on all
documents with a preset ID resulting in 100's, if not 1000's, of extra
reads. The purpose of the fingerprint function is to limit repeading
writes so to keep some of this functionality the fingerprint function is
now only added to documents with messages.
* G1 garbage collection is now enabled by default when the heap size is
> 6GiB. Early versions of elasticsearch did not recommend this setting
however its since stabalized in recent releases.
* JVM options have been moved into the elasticsearch and logstash roles
allowing these tasks to trigger service restarts when changes are made.
Change-Id: I805129b207ad4db182ae6e59b6ec78eb3e246b54
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
* We don't need to create the containers as they are created during the
initial run.
* Remove quoting in favor of {% raw %} blocks
Change-Id: Ied696ad0882169d523a60a900788e7c2ba1d3fa3
Presently the node role assignment is only automatic. Auto selection
makes the assumption every node is identical however in many deployments
a deployer may want to assign node roles to specific hardware thereby
optimizing resources and improving general performance. This change
adds and documents the ability to set the node roles within an ansible
inventory.
Change-Id: I22a2b636cb1441f17e575439b55ca64f9c7b0336
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
The multi-logstash pipeline setup, while amazingly fast, was crashing
and causing index errors when under high load for a long period of time.
Because of the crashing behavior and the fact that the folks from
Elastic describe multi-pipeline queues to be "beta" at this time the
logstash pipelines have been converted back into a single pipeline.
The memory backed queue options are now limited by a ram disk (tmpfs)
which will ensure that a burst within the queue does not cause OOM
issues and ensures a highly performant deployment and limiting memory
usage at the same time. Memory backed queues will be enabled when the
underlying system is using "rotational" media as detected by ansible
facts. This will ensure a fast and consistent experience across all
deployment types.
Pipeline/ml/template/dashboard setup has been added to the beat
configurations which will ensure beats are properly configured even
when running in an isolated deployment and outside of normal operations
where beats are generally configured on the first data node.
Change-Id: Ie3c775f98b14f71bcbed05db9cb1c5aa46d9c436
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
The list of elasticsearch hosts was being randomized too much which
results in the a performance issue. This change reduces the entropy and
ensures that the list of hosts is correctly ordered such that localhost
is always used first and other nodes in the cluster will be used as a
fall back.
Change-Id: Ifb551a6e01b5c0e1f62c1466a3d5b344a3c5da97
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
The apache2 monitoring process requires a couple interactions to deploy
successfully. This change will ensure that if the apache2 monitoring
fails, in any way, it does not block the deployment.
Change-Id: Ibe35197a1c65f4abe9e4870c07ee15f37f9a58ab
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
The current setup was using processor cores from ansible facts which in
a multi-core, single socket system could result in 1. Using the
processor count will return the logical processor count giving us a more
performant setup when the compute power is present.
Change-Id: Ia5b63d45691f58e848d05cc4a4e5f353b993a347
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
This change adds SUSE 42.3 support to the elastic telemetry solutions.
Change-Id: Ibe93ea0d1ead9e7fe6da16d89989cfe5ade0f43e
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
This change will help with organization throughout the stack.
Change-Id: I2ad865db534ae1d377bbdecd4b421ee0fc802536
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
The new option logstash_syslog_input_enabled has been added which will
allow users to enable a direct syslog input. When enabled, messages will
be processed via logstash and sent directly to elasticsearch.
Change-Id: Icb7712ecb8aae3d7f99df80ae1c5cd647a15ce83
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
This change adds Ubuntu 14.04 support to the project.
Change-Id: I20695e19409b63c6e1def4ccf8929c6d52be647e
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
This change adds the scaffolding required to get multi-distro support
running in the roles. The change breaks up our playbooks converting all
of the tasks into various roles with internal dependencies. While this
will improve execution time, the change is being done to reduce boiler
plate and to allow us to build on the pattern used in OSA to provide
multi-distro capabilities.
A side effect of this change is a major improvement in idempotency. The
playbooks should now be 100% idempotent.
All of the templates have been left in the main playbook directory. This
was done to help ease the transition. In a future PR the template
structure will be moved into the roles where it needs to be.
The main variable files has been left intact. This file will be carved
up into role defaults in a future PR.
Change-Id: I938a10564128ce4078fa12edcf614dcdbd684b25
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>