15 Commits

Author SHA1 Message Date
Kevin Carter
3a0b3d2cde
Convert playbooks into roles
This change adds the scaffolding required to get multi-distro support
running in the roles. The change breaks up our playbooks converting all
of the tasks into various roles with internal dependencies. While this
will improve execution time, the change is being done to reduce boiler
plate and to allow us to build on the pattern used in OSA to provide
multi-distro capabilities.

A side effect of this change is a major improvement in idempotency. The
playbooks should now be 100% idempotent.

All of the templates have been left in the main playbook directory. This
was done to help ease the transition. In a future PR the template
structure will be moved into the roles where it needs to be.

The main variable files has been left intact. This file will be carved
up into role defaults in a future PR.

Change-Id: I938a10564128ce4078fa12edcf614dcdbd684b25
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2018-08-09 00:41:05 -05:00
Jonathan Rosser
eeb2e5a853 Make index retention calculation pluggable
This allows alternate algorithms to be developed and enables the
same tasks to be included in a test to verify the planned retention
does not exceed the cluster storage capacity.

Change-Id: Ie3d80d6cfad16b946ccd790859bc7cd92b90fdef
2018-07-26 21:47:44 +01:00
Kevin Carter
d6c0fc9613
Add x-pack monitoring by default
This change adds xpack monitoring by default for kibana, logstash, and
elasticsearch.

Change-Id: I03c8b8390d33bb996201671aae481415bfc1e691
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2018-07-23 15:29:13 -05:00
Kevin Carter
e0f77d531a Add systemd configs and update playbook uniformity
Systemd overrides have been added to the service unit files for all
beats and services. All of the playbooks have been updated to make them
look and feel uniform.

This also sets handlers within the playbooks so that we're improving the
idempotence.

Change-Id: I2dd3183dae4bfddc607cc74f9dfb7af115b80abc
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2018-07-18 21:29:39 +00:00
Kevin Carter
2f1bd5d2ea
Collect facts by default
To install curator facts are required, this change ensures we gather
them.

Change-Id: I510692095cdf8ecb5806a43c714b7bbbace47022
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2018-07-16 21:09:10 -05:00
Kevin Carter
b6a9a6fc7a Add dynamic retention policies to curator
The curator retention policies will now query the storage nodes within
a given deployment and set a suitable index retention policy based on
the total amount of storage each index is assumed to produce every day.
To ensure we're minimizing the storage required and optimizing search
performance several actions are now being taken:

* Indexes will be shrunk after a quarter of their retention time.
* Indexes will be deleted should they exceed the retention time.

Change-Id: I8bf548620b5404d25deaadba8fda93452ef64fa0
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2018-07-12 17:03:40 +00:00
Kevin Carter
c0e4e29cb2
Add the virtualenv package when installing currator
Change-Id: I7e1a4891d43e44b048a97364d80a8dd2920f12d1
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2018-06-20 17:02:03 -05:00
Zuul
2e8a6b0da3 Merge "Break out setup and always tag facts" 2018-06-15 23:18:22 +00:00
Kevin Carter
fb7aed401b Break out setup and always tag facts
This change breaks out the basic install and setup playbooks into
different playbooks. This is done to ensure an operator can easily rerun
any part of the playbook as they upgrade, change, or modify a setup over
the lifetime of the deployment.

Change-Id: I66c53fcb21880c950ea3fee202e7d2224dfdff3a
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2018-06-15 17:24:01 -05:00
Kevin Carter
cf78fd744d Add retries to all tasks with external deps
The tasks with external deps should all have a retry so to cater for
cases of intermittent service interruptions. This change adds retires
and tags to of the tasks with external deps.

Change-Id: Icd0a8143347bcbffc2a1e0a02da92091b59ed483
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2018-06-14 19:31:09 -05:00
Kevin Carter
2207df4622 Add task to install requirements and change pathing
To run the curator playbook pip is required. This change installs the
package `python-virtualenv` to resolve the dependency and installs the
curator application into a virtualenv within /opt. This will keep the
host clean by isolating the dependencies and ensure that we can easily
clean up should we ever need to.

Change-Id: If872bb30281948d79a3f79a0e3a39e8a457c6fdb
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2018-06-14 15:35:47 +00:00
Jonathan Rosser
31c5b833ed Bypass pip constraints when installing Elastic Curator
The elasticsearch-curator package requires certifi 2018.4.16 but this
is constrained to version 2018.1.18. Use isolated mode to allow
installation of the upstream version.

Change-Id: I4ba387afab35230e70b45fab42b6bfaf9ffeba19
2018-06-05 12:56:24 +01:00
Jonathan Rosser
dc43a27d4a Allow deploy time env vars when deploying ELK stack
Change-Id: I029b2a88c4f04f9d2a67763a35db9e52a672fc8a
2018-05-16 12:34:04 +01:00
Kevin Carter
846a90d025 Tune down the collection intervals and default retention policy
At present we're collecting too much info by default. We're seeing
+500GB on a <50 node environment in just two weeks. While we dont expect
the data set to grow much larger given the use of curator, this change
lowers the default collection intervals of the various beats and updates
the retention / detection policies so we're not storing too much
information.

To correct a unicode problem with py2 the host index loops have been
updated.

Curator has also been updated to run everyday.

Change-Id: Ic202eb19806d1b805fa314d3d8bde05b286740e0
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2018-05-11 16:04:14 +00:00
Kevin Carter
0c41b0fd70
Add curator and dynamic shard counts
Curator has been added to automatically maintain the cluster with
sensible defaults when it pertains to data retention.

The index counts have been modified such that they're determined by the
size of the initial cluster. While these shard counts can be modified
post deployment by reindexing the data, it's not something being done at
this time.

Depends-On: https://review.openstack.org/c/565807
Change-Id: I249d715ae5241ab57c4117b14377e4d07cb6e984
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2018-05-02 12:11:30 -05:00