22 Commits

Author SHA1 Message Date
cloudnull
0a0a4a0880
Add the ability to enable or disable rollups / indexes
This change creates a new option to enable or disbale rollup jobs. This
is also providing the default basic index patterns for kibana index
patterns and elastic indexes.

Change-Id: I60e96a2cdbe27de760b54c4d9d43bcde4d09bbf5
Signed-off-by: cloudnull <kevin@cloudnull.com>
2019-02-11 23:14:51 -06:00
Kevin Carter
b1232aead5
Add pretasks to exit quick when needed
The journalbeat playbook uses conditionals to know when to deploy the
journalbeat collector. This change makes it so the playbook simply exits
when the journal is not found or the environment being deployed is not
using systemd. This change will result in faster deployments in mixed
environments as the role will no longer need to iterate over its
conditional.

Change-Id: I581b61902723f54237623036566a83c9be79210e
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2019-01-03 09:05:32 -06:00
Kevin Carter
5586d8a80f Convert template setup to a role
This change reduces code throughout the playbooks thereby speeding up
the task execution.
  * A new role named `elastic_beat_setup` was created to
    facilitate template setup as needed.
  * Beats retention policies are now defined on the elastic-logstash
    nodes instead of on all target hosts. This method will speed-up
    deployments on massive installations while streamlining all deployments.
  * Kibana variable assumptions have been fixed. This will allow for
    deployments without Kibana to be accomplished.

Change-Id: I36343264042e81dfcb68bad0f6c3a503e525eceb
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2019-01-02 20:38:47 -06:00
Jonathan Rosser
ac46b2be6a Fix journalbeat installation for mixed environments
The previous code would terminate the play immediatley if any hosts
in the environment did not have a journal directory. This change runs
the journalbeat install role selectively on hosts that have the journal
directory, and skips hosts that do not.

In addition a legacy task to stop the play after uninstallation is removed,
this functionality is currently broken.

Change-Id: I412e3594c4b2292caafafb580bb4ede9ccfd3944
2018-09-25 12:34:21 +01:00
Kevin Carter
57754a4346 Make journalbeat install detect if it should install
When running on a system where there is no journal or no systemd skip
the journalbeat playbooks.

Change-Id: I92c804e8eb2ab2f9b86eca09fc51d19be66c7190
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2018-08-16 04:45:53 +00:00
Kevin Carter
3a0b3d2cde
Convert playbooks into roles
This change adds the scaffolding required to get multi-distro support
running in the roles. The change breaks up our playbooks converting all
of the tasks into various roles with internal dependencies. While this
will improve execution time, the change is being done to reduce boiler
plate and to allow us to build on the pattern used in OSA to provide
multi-distro capabilities.

A side effect of this change is a major improvement in idempotency. The
playbooks should now be 100% idempotent.

All of the templates have been left in the main playbook directory. This
was done to help ease the transition. In a future PR the template
structure will be moved into the roles where it needs to be.

The main variable files has been left intact. This file will be carved
up into role defaults in a future PR.

Change-Id: I938a10564128ce4078fa12edcf614dcdbd684b25
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2018-08-09 00:41:05 -05:00
Kevin Carter
47aff12e6c Add functional tests to elk_metrics_6x
Change-Id: Ia997aa0a9b9388be0e8f72b633b3b9cc25b4429c
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2018-07-20 18:14:22 +00:00
Kevin Carter
e0f77d531a Add systemd configs and update playbook uniformity
Systemd overrides have been added to the service unit files for all
beats and services. All of the playbooks have been updated to make them
look and feel uniform.

This also sets handlers within the playbooks so that we're improving the
idempotence.

Change-Id: I2dd3183dae4bfddc607cc74f9dfb7af115b80abc
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2018-07-18 21:29:39 +00:00
Kevin Carter
553fd7d30b Correct log path and set handler
The journalbeat process was restarting every playbook run which was not
required. This change moves the restart process to a handler which will
ensure we're not restarting the services when it's not required.

Change-Id: I4c0082d04d99c71c902ae39ee5ad9efc5074889f
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2018-07-06 20:23:58 +00:00
Zuul
1123cf7d47 Merge "Use openstack_version to determine Journalbeat hosts" 2018-07-06 16:21:31 +00:00
Jonathan Rosser
b3fb995cd5 Only pass journal dirs that are present to Journalbeat
Change-Id: I3e65a0a06c452632847a55effd2ce4d0c3cb4ac0
2018-07-06 16:33:54 +01:00
Jonathan Rosser
dd3042bcf2 Use openstack_version to determine Journalbeat hosts
Systemd journals are linked from the containers down to the hosts
starting in Rocky. Prior to this Journalbeat must also be on all
containers.

Change-Id: Iaaef19e76c40ba9c1ad58164c20da46766abeee6
2018-07-05 21:48:18 +01:00
Jonathan Rosser
dbc1ed3af3 Add an 'always' tag to cluster layout task include
The templates which generate the configuration for the various
components require access to the facts defining the cluster layout.

Running the setup*.yml playbooks with --tags config would not include
common_task_data_node_hosts.yml and templating would fail due to
undefined vars.

This change applys an 'always' tag to all uses of the include.

Change-Id: I6c55a60aef54f9bb26eb9a2af08315265daa82d9
2018-06-20 15:03:20 +00:00
Zuul
2e8a6b0da3 Merge "Break out setup and always tag facts" 2018-06-15 23:18:22 +00:00
Kevin Carter
fb7aed401b Break out setup and always tag facts
This change breaks out the basic install and setup playbooks into
different playbooks. This is done to ensure an operator can easily rerun
any part of the playbook as they upgrade, change, or modify a setup over
the lifetime of the deployment.

Change-Id: I66c53fcb21880c950ea3fee202e7d2224dfdff3a
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2018-06-15 17:24:01 -05:00
Kevin Carter
cf78fd744d Add retries to all tasks with external deps
The tasks with external deps should all have a retry so to cater for
cases of intermittent service interruptions. This change adds retires
and tags to of the tasks with external deps.

Change-Id: Icd0a8143347bcbffc2a1e0a02da92091b59ed483
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2018-06-14 19:31:09 -05:00
Kevin Carter
bc2937d9c9
Use elasticsearch coordinator nodes as smart LBs
Elasticsearch can be used as a smart load balancer for all traffic
which will remove the requirement for a VIP and move the cluster to a
mesh topology. All of the Kibana nodes will now run elasticsearch as
cordonator.

* Kibana will now connect to elasticsearch on localhost.
* All of the beats have been setup to use use the new mesh topology.
* jvm memory management has been updated to reflect the additional
  services.

More on node assigments can be found here:
* https://www.elastic.co/guide/en/elasticsearch/reference/6.2/modules-node.html#modules-node

* The readme has been updated to reflect these changes.

Change-Id: I769e0251072f5dbde56fcce7753236d37d5c3b19
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2018-06-13 23:37:48 -05:00
Jonathan Rosser
ebf309a9e7 Use environment: to manipulate PATH and GOPATH env vars
Change-Id: I71567994b4e479d118a0f402b2120da23b4fbd4d
2018-05-16 17:57:20 +01:00
Jonathan Rosser
dc43a27d4a Allow deploy time env vars when deploying ELK stack
Change-Id: I029b2a88c4f04f9d2a67763a35db9e52a672fc8a
2018-05-16 12:34:04 +01:00
Kevin Carter
846a90d025 Tune down the collection intervals and default retention policy
At present we're collecting too much info by default. We're seeing
+500GB on a <50 node environment in just two weeks. While we dont expect
the data set to grow much larger given the use of curator, this change
lowers the default collection intervals of the various beats and updates
the retention / detection policies so we're not storing too much
information.

To correct a unicode problem with py2 the host index loops have been
updated.

Curator has also been updated to run everyday.

Change-Id: Ic202eb19806d1b805fa314d3d8bde05b286740e0
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2018-05-11 16:04:14 +00:00
Kevin Carter
0c41b0fd70
Add curator and dynamic shard counts
Curator has been added to automatically maintain the cluster with
sensible defaults when it pertains to data retention.

The index counts have been modified such that they're determined by the
size of the initial cluster. While these shard counts can be modified
post deployment by reindexing the data, it's not something being done at
this time.

Depends-On: https://review.openstack.org/c/565807
Change-Id: I249d715ae5241ab57c4117b14377e4d07cb6e984
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2018-05-02 12:11:30 -05:00
Kevin Carter
85f5175bc6
Add journalbeat
Journalbeat is a community beat which allows journals to be directly
shipped to logstash. This beat has been setup to start using the common
systemd role and will only be executed on systems where systemd is
present.

Change-Id: I8e911b83e28c82dd2e19dc4a044b1dd3e75ebf77
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2018-04-29 00:45:34 -05:00