This change creates a new option to enable or disbale rollup jobs. This
is also providing the default basic index patterns for kibana index
patterns and elastic indexes.
Change-Id: I60e96a2cdbe27de760b54c4d9d43bcde4d09bbf5
Signed-off-by: cloudnull <kevin@cloudnull.com>
The journalbeat playbook uses conditionals to know when to deploy the
journalbeat collector. This change makes it so the playbook simply exits
when the journal is not found or the environment being deployed is not
using systemd. This change will result in faster deployments in mixed
environments as the role will no longer need to iterate over its
conditional.
Change-Id: I581b61902723f54237623036566a83c9be79210e
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
This change reduces code throughout the playbooks thereby speeding up
the task execution.
* A new role named `elastic_beat_setup` was created to
facilitate template setup as needed.
* Beats retention policies are now defined on the elastic-logstash
nodes instead of on all target hosts. This method will speed-up
deployments on massive installations while streamlining all deployments.
* Kibana variable assumptions have been fixed. This will allow for
deployments without Kibana to be accomplished.
Change-Id: I36343264042e81dfcb68bad0f6c3a503e525eceb
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
The previous code would terminate the play immediatley if any hosts
in the environment did not have a journal directory. This change runs
the journalbeat install role selectively on hosts that have the journal
directory, and skips hosts that do not.
In addition a legacy task to stop the play after uninstallation is removed,
this functionality is currently broken.
Change-Id: I412e3594c4b2292caafafb580bb4ede9ccfd3944
When running on a system where there is no journal or no systemd skip
the journalbeat playbooks.
Change-Id: I92c804e8eb2ab2f9b86eca09fc51d19be66c7190
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
This change adds the scaffolding required to get multi-distro support
running in the roles. The change breaks up our playbooks converting all
of the tasks into various roles with internal dependencies. While this
will improve execution time, the change is being done to reduce boiler
plate and to allow us to build on the pattern used in OSA to provide
multi-distro capabilities.
A side effect of this change is a major improvement in idempotency. The
playbooks should now be 100% idempotent.
All of the templates have been left in the main playbook directory. This
was done to help ease the transition. In a future PR the template
structure will be moved into the roles where it needs to be.
The main variable files has been left intact. This file will be carved
up into role defaults in a future PR.
Change-Id: I938a10564128ce4078fa12edcf614dcdbd684b25
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
Systemd overrides have been added to the service unit files for all
beats and services. All of the playbooks have been updated to make them
look and feel uniform.
This also sets handlers within the playbooks so that we're improving the
idempotence.
Change-Id: I2dd3183dae4bfddc607cc74f9dfb7af115b80abc
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
The journalbeat process was restarting every playbook run which was not
required. This change moves the restart process to a handler which will
ensure we're not restarting the services when it's not required.
Change-Id: I4c0082d04d99c71c902ae39ee5ad9efc5074889f
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
Systemd journals are linked from the containers down to the hosts
starting in Rocky. Prior to this Journalbeat must also be on all
containers.
Change-Id: Iaaef19e76c40ba9c1ad58164c20da46766abeee6
The templates which generate the configuration for the various
components require access to the facts defining the cluster layout.
Running the setup*.yml playbooks with --tags config would not include
common_task_data_node_hosts.yml and templating would fail due to
undefined vars.
This change applys an 'always' tag to all uses of the include.
Change-Id: I6c55a60aef54f9bb26eb9a2af08315265daa82d9
This change breaks out the basic install and setup playbooks into
different playbooks. This is done to ensure an operator can easily rerun
any part of the playbook as they upgrade, change, or modify a setup over
the lifetime of the deployment.
Change-Id: I66c53fcb21880c950ea3fee202e7d2224dfdff3a
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
The tasks with external deps should all have a retry so to cater for
cases of intermittent service interruptions. This change adds retires
and tags to of the tasks with external deps.
Change-Id: Icd0a8143347bcbffc2a1e0a02da92091b59ed483
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
Elasticsearch can be used as a smart load balancer for all traffic
which will remove the requirement for a VIP and move the cluster to a
mesh topology. All of the Kibana nodes will now run elasticsearch as
cordonator.
* Kibana will now connect to elasticsearch on localhost.
* All of the beats have been setup to use use the new mesh topology.
* jvm memory management has been updated to reflect the additional
services.
More on node assigments can be found here:
* https://www.elastic.co/guide/en/elasticsearch/reference/6.2/modules-node.html#modules-node
* The readme has been updated to reflect these changes.
Change-Id: I769e0251072f5dbde56fcce7753236d37d5c3b19
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
At present we're collecting too much info by default. We're seeing
+500GB on a <50 node environment in just two weeks. While we dont expect
the data set to grow much larger given the use of curator, this change
lowers the default collection intervals of the various beats and updates
the retention / detection policies so we're not storing too much
information.
To correct a unicode problem with py2 the host index loops have been
updated.
Curator has also been updated to run everyday.
Change-Id: Ic202eb19806d1b805fa314d3d8bde05b286740e0
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
Curator has been added to automatically maintain the cluster with
sensible defaults when it pertains to data retention.
The index counts have been modified such that they're determined by the
size of the initial cluster. While these shard counts can be modified
post deployment by reindexing the data, it's not something being done at
this time.
Depends-On: https://review.openstack.org/c/565807
Change-Id: I249d715ae5241ab57c4117b14377e4d07cb6e984
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
Journalbeat is a community beat which allows journals to be directly
shipped to logstash. This beat has been setup to start using the common
systemd role and will only be executed on systems where systemd is
present.
Change-Id: I8e911b83e28c82dd2e19dc4a044b1dd3e75ebf77
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>