390314e18b
With the option to deploy grafana the following changes allow a user to automatically connect ELK and Grafana. Change-Id: Ic8e64a31d860940c6863f46ce558908d5ef8f8e7 Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
158 lines
4.7 KiB
ReStructuredText
158 lines
4.7 KiB
ReStructuredText
Install ELK with beats to gather metrics
|
|
########################################
|
|
:tags: openstack, ansible
|
|
|
|
About this repository
|
|
---------------------
|
|
|
|
This set of playbooks will deploy elk cluster (Elasticsearch, Logstash, Kibana)
|
|
with topbeat to gather metrics from hosts metrics to the ELK cluster.
|
|
|
|
**These playbooks require Ansible 2.4+.**
|
|
|
|
OpenStack-Ansible Integration
|
|
-----------------------------
|
|
|
|
These playbooks can be used as standalone inventory or as an integrated part of
|
|
an OpenStack-Ansible deployment. For a simple example of standalone inventory,
|
|
see ``inventory.example.yml``.
|
|
|
|
Optional | Load balancer VIP address
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
|
|
In order to use multi-node elasticsearch a loadbalancer is required. Haproxy can
|
|
provide the load balancer functionality needed. The option
|
|
`internal_lb_vip_address` is used as the endpoint (virtual IP address) services
|
|
like Kibana will use when connecting to elasticsearch. If this option is
|
|
omitted, the first node in the elasticsearch cluster will be used.
|
|
|
|
Optional | configure haproxy endpoints
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
|
|
Edit the `/etc/openstack_deploy/user_variables.yml` file and add fiel following
|
|
lines
|
|
|
|
.. code-block:: yaml
|
|
|
|
haproxy_extra_services:
|
|
- service:
|
|
haproxy_service_name: kibana
|
|
haproxy_ssl: False
|
|
haproxy_backend_nodes: "{{ groups['kibana'] | default([]) }}"
|
|
haproxy_port: 81 # This is set using the "kibana_nginx_port" variable
|
|
haproxy_balance_type: tcp
|
|
- service:
|
|
haproxy_service_name: elastic-logstash
|
|
haproxy_ssl: False
|
|
haproxy_backend_nodes: "{{ groups['elastic-logstash'] | default([]) }}"
|
|
haproxy_port: 5044 # This is set using the "logstash_beat_input_port" variable
|
|
haproxy_balance_type: tcp
|
|
- service:
|
|
haproxy_service_name: elastic-logstash
|
|
haproxy_ssl: False
|
|
haproxy_backend_nodes: "{{ groups['elastic-logstash'] | default([]) }}"
|
|
haproxy_port: 9201 # This is set using the "elastic_hap_port" variable
|
|
haproxy_check_port: 9200 # This is set using the "elastic_port" variable
|
|
haproxy_backend_port: 9200 # This is set using the "elastic_port" variable
|
|
haproxy_balance_type: tcp
|
|
|
|
Optional | run the haproxy-install playbook
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
|
|
.. code-block:: bash
|
|
|
|
cd /opt/openstack-ansible/playbooks/
|
|
openstack-ansible haproxy-install.yml --tags=haproxy-service-config
|
|
|
|
Deployment Process
|
|
------------------
|
|
|
|
Clone the elk-osa repo
|
|
|
|
.. code-block:: bash
|
|
|
|
cd /opt
|
|
git clone https://github.com/openstack/openstack-ansible-ops
|
|
|
|
Copy the env.d file into place
|
|
|
|
.. code-block:: bash
|
|
|
|
cd /opt/openstack-ansible-ops/elk_metrics_6x
|
|
cp env.d/elk.yml /etc/openstack_deploy/env.d/
|
|
|
|
Copy the conf.d file into place
|
|
|
|
.. code-block:: bash
|
|
|
|
cp conf.d/elk.yml /etc/openstack_deploy/conf.d/
|
|
|
|
In **elk.yml**, list your logging hosts under elastic-logstash_hosts to create
|
|
the elasticsearch cluster in multiple containers and one logging host under
|
|
kibana_hosts to create the kibana container
|
|
|
|
.. code-block:: bash
|
|
|
|
vi /etc/openstack_deploy/conf.d/elk.yml
|
|
|
|
Create the containers
|
|
|
|
.. code-block:: bash
|
|
|
|
cd /opt/openstack-ansible-playbooks
|
|
openstack-ansible lxc-containers-create.yml -e 'container_group=elastic-logstash:kibana'
|
|
|
|
install master/data elasticsearch nodes on the elastic-logstash containers
|
|
|
|
.. code-block:: bash
|
|
|
|
cd /opt/openstack-ansible-ops/elk_metrics_6x
|
|
openstack-ansible installElastic.yml
|
|
|
|
Install Logstash on all the elastic containers
|
|
|
|
.. code-block:: bash
|
|
|
|
cd /opt/openstack-ansible-ops/elk_metrics_6x
|
|
openstack-ansible installLogstash.yml
|
|
|
|
Install Kibana, nginx reverse proxy and metricbeat on the kibana container
|
|
|
|
.. code-block:: bash
|
|
|
|
cd /opt/openstack-ansible-ops/elk_metrics_6x
|
|
openstack-ansible installKibana.yml
|
|
|
|
Install Metricbeat everywhere to start shipping metrics to our logstash
|
|
instances
|
|
|
|
.. code-block:: bash
|
|
|
|
cd /opt/openstack-ansible-ops/elk_metrics_6x
|
|
openstack-ansible installMetricbeat.yml
|
|
|
|
|
|
Adding Grafana visualizations
|
|
-----------------------------
|
|
|
|
See the grafana directory for more information on how to deploy grafana. Once
|
|
When deploying grafana, source the variable file from ELK in order to
|
|
automatically connect grafana to the Elasticsearch datastore and import
|
|
dashboards. Including the variable file is as simple as adding
|
|
``-e @../elk_metrics_6x/vars/variables.yml`` to the grafana playbook
|
|
run.
|
|
|
|
Included dashboards
|
|
|
|
* https://grafana.com/dashboards/5569
|
|
* https://grafana.com/dashboards/5566
|
|
|
|
Trouble shooting
|
|
----------------
|
|
|
|
If everything goes bad, you can clean up with the following command
|
|
|
|
.. code-block:: bash
|
|
|
|
openstack-ansible lxc-containers-destroy.yml --limit=kibana:elastic-logstash_all
|