17 Commits

Author SHA1 Message Date
Swapnil Kulkarni (coolsvap)
b1f3aa0c1c Update Heka dockerfile for formatting
Change-Id: Ib5cab0200f6e8fbea3de89624067771e92024b2f
Partial-Bug:#1569417
2016-04-12 21:27:54 +05:30
Michal Rostecki
2069d7562c Create and chown /etc/heka directory
This directory is needed for kolla-mesos to save config files
there.

Change-Id: Ia9810061d370bc600dfffeb33a4d4402c423e3fc
Closes-Bug: 1567437
2016-04-12 08:06:21 +02:00
Serguei Bezverkhi
8ec7811037 Changes horizon log location to kolla_logs volume
Changes location of Horizon log, they will be stored on common log volume
kolla_logs.

Change-Id: Ie9d56999a83efd05ab7c3dcb00b4dc42c9bce8f8
Closes-Bug: 1560250
2016-03-22 18:26:18 -04:00
Éric Lemoine
491aff0b88 Make Heka send logs to Elasticsearch
This patch includes changes relative to integrating Heka with
Elasticsearch and Kibana.

The main change is the addition of an Heka ElasticSearchOutput plugin
to make Heka send the logs it collects to Elasticsearch.

Since Logstash is not used the enable_elk deploy variable is renamed
to enable_central_logging.

If enable_central_logging is false then Elasticsearch and Kibana are
not started, and Heka won't attempt to send logs to Elasticsearch.

By default enable_central_logging is set to false. If
enable_central_logging is set to true after deployment then the Heka
container needs to be recreated (for Heka to get the new
configuration).

The Kibana configuration used property names that are deprecated in
Kibana 4.2. This is changed to use non-deprecated property names.

Previously logs read from files and from Syslog had a different Type
in Heka. This is changed to always use "log" for the Type. In this
way just one index instead of two is used in Elasticsearch, making
things easier to the user on the visualization side.

The HAProxy configuration is changed to add entries for Kibana.
Kibana server is now accessible via the internal VIP, and also via
the external VIP if there's one configured.

The HAProxy configuration is changed to add an entry for
Elasticsearch. So Elasticsearch is now accessible via the internal
VIP. Heka uses that channel for communicating with Elasticsearch.

Note that currently the Heka logs include "Plugin
elasticsearch_output" errors when Heka starts. This occurs when Heka
starts processing logs while Elasticsearch is not yet started. These
are transient errors that go away when Elasticsearch is ready. And
with buffering enabled on the ElasticSearchOuput plugin logs will be
buffered and then retransmitted when Elasticsearch is ready.

Change-Id: I6ff7a4f0ad04c4c666e174693a35ff49914280bb
Implements: blueprint central-logging-service
2016-03-05 11:51:49 +01:00
Éric Lemoine
95cf542f8d Fix parsing of RabbitMQ logs
Currently Heka fails to parse the RabbitMQ logs. There are two
problems:

1. The rabbit-sasl.log file is ignored but the file_match expression
   does not match.
2. The delimiter used in the RegexSplitter makes Heka stop on the
   very first log entry. '\n\n(=[^=]+====' (with two \n's) is
   a better delimiter. deliver_incomplete_final is used to get the
   final log entry.

TrivialFix

Change-Id: I94720340d5b2d6fd5d7641b9ff58733f6cd882ee
2016-02-26 14:50:32 +01:00
Jenkins
2b0a070ee5 Merge "Fix Swift logging" 2016-02-22 21:49:42 +00:00
Jenkins
7c9ef860e8 Merge "Change Heka sudoers file mode bits" 2016-02-22 18:00:46 +00:00
Éric Lemoine
e6a9b9627a Fix Swift logging
Swift uses Syslog, but it uses a custom log format.  So this commit
adds a specific Heka decoder for Swift.

It also increases the log level from "warning" to "info" to make
Swift more verbose.  Note that "info" is the default log level in
Swift.

And it disables the Heka configuration for Swift when "enable_swift"
is set to "no".  This prevents Heka from creating 15 empty Swift log
files in the logs volume.

Partially implements: blueprint heka

Change-Id: If7a7d0707e71be2957178e2d45b5de51b788232e
2016-02-22 08:53:20 -08:00
Éric Lemoine
c0e46f0c5f Run yum clean all in Heka Dockerfile
This follows up on a review comment from Steven Dake.  See
https://review.openstack.org/#/c/275840/14/docker/heka/Dockerfile.j2@21

Closes-Bug: #1547535
Partially implements: blueprint heka

Change-Id: I4f5094145d8f5bbc04fe23e64c6a39719775bae4
2016-02-22 17:43:51 +01:00
Éric Lemoine
2069aaa2e3 Create "heka" group and add "heka" user to it
TrivialFix
Partially implements: blueprint heka

Change-Id: I55053251596394685fd632748b42ba23ecfbeccb
2016-02-22 14:05:30 +01:00
Éric Lemoine
891fae2085 Change Heka sudoers file mode bits
This follows up on a review comment from Steven Dake. See
https://review.openstack.org/#/c/275840/14/docker/heka/Dockerfile.j2@21

Closes-Bug: #1547536
Partially implements: blueprint heka

Change-Id: Ie85b64be139257f7da7cecd82a7971d7550d27d4
2016-02-22 03:03:39 -08:00
Éric Lemoine
b16724e09a Add Heka log decoder for Keystone/Apache
Partially implements: blueprint heka
Change-Id: I562dddcfa75f3a814874f7735d735416d9aa7803
2016-02-19 21:49:21 +00:00
Éric Lemoine
7fb0971fb6 Add Heka log decoder for MariaDB
In the future the Heka Lua sandboxes will be moved out of Kolla, and
installed in the Heka container using a deb or rpm package.

Partially implements: blueprint heka
Change-Id: Iad1e5177d239410a88546fc4ed8c23b9e46beef3
2016-02-19 21:49:21 +00:00
Éric Lemoine
884c302451 Add Heka log decoder for OpenStack
In the future the Heka Lua sandboxes will be moved out of Kolla, and
installed in the Heka container using a deb or rpm package.

Partially implements: blueprint heka
Change-Id: I70fd4af920aa2dfb1c8ef9ce8bf6e0c28507ec3e
2016-02-19 20:32:32 +00:00
Éric Lemoine
dfe1f3bac2 Add Heka log decoder for RabbitMQ
In the future the Heka Lua sandboxes will be moved out of Kolla, and
installed in the Heka container using a deb or rpm package.

Partially implements: blueprint heka
Change-Id: I1aa27048d22f1a5ed385180696d320af9a961389
2016-02-19 20:32:17 +00:00
Éric Lemoine
9ef1be1c38 Add Heka log decoder infrastructure
In the future the Heka Lua sandboxes will be moved out of Kolla, and
installed in the Heka container using a deb or rpm package.

Partially implements: blueprint heka
Change-Id: I34cc80f62ddbca8ee330c971f58fee8686e245e6
2016-02-19 20:28:12 +00:00
Éric Lemoine
ee95dc5c27 Add a Dockerfile for Heka
Partially implements: blueprint heka
Change-Id: Iafe0dba1333f1f84bc5956890c3b7a0c2a45cef3
2016-02-19 19:23:49 +00:00