Heka logs almost done. When testing, The collected logs can copy
to the /tmp/logs folder, which will be handle by the gate. This
will be very useful for our gate debug.
TrivialFix
Change-Id: I06ca8ebd13933bfd15cb2fb5e53fc3038b17b8c3
The code in docker/base/start.sh that waits for the log socket does
not work because it includes a bad "space" character after "-S".
This patch changes that character to a real "space" character. It
also sets the SKIP_LOG_SETUP envvar for the Heka container.
TrivialFix
Change-Id: I17a3005df5bbd57678350fc77030e44b5ad6acae
This patch includes changes relative to integrating Heka with
Elasticsearch and Kibana.
The main change is the addition of an Heka ElasticSearchOutput plugin
to make Heka send the logs it collects to Elasticsearch.
Since Logstash is not used the enable_elk deploy variable is renamed
to enable_central_logging.
If enable_central_logging is false then Elasticsearch and Kibana are
not started, and Heka won't attempt to send logs to Elasticsearch.
By default enable_central_logging is set to false. If
enable_central_logging is set to true after deployment then the Heka
container needs to be recreated (for Heka to get the new
configuration).
The Kibana configuration used property names that are deprecated in
Kibana 4.2. This is changed to use non-deprecated property names.
Previously logs read from files and from Syslog had a different Type
in Heka. This is changed to always use "log" for the Type. In this
way just one index instead of two is used in Elasticsearch, making
things easier to the user on the visualization side.
The HAProxy configuration is changed to add entries for Kibana.
Kibana server is now accessible via the internal VIP, and also via
the external VIP if there's one configured.
The HAProxy configuration is changed to add an entry for
Elasticsearch. So Elasticsearch is now accessible via the internal
VIP. Heka uses that channel for communicating with Elasticsearch.
Note that currently the Heka logs include "Plugin
elasticsearch_output" errors when Heka starts. This occurs when Heka
starts processing logs while Elasticsearch is not yet started. These
are transient errors that go away when Elasticsearch is ready. And
with buffering enabled on the ElasticSearchOuput plugin logs will be
buffered and then retransmitted when Elasticsearch is ready.
Change-Id: I6ff7a4f0ad04c4c666e174693a35ff49914280bb
Implements: blueprint central-logging-service
The storage hostname is used in many conf files all over the place and
should be registered upon each run for every service
TrivialFix
Change-Id: I1d5a20cb6d51cee9f529cf2e1fb144158f7718a9
Due to poor planning on our variable names we have a situation where
we have "internal_address" which must be a VIP, but "external_address"
which should be a DNS name. Now with two vips "external_vip_address"
is a new variable.
This corrects that issue by deprecating kolla_internal_address and
replacing it with 4 nicely named variables.
kolla_internal_vip_address
kolla_internal_fqdn
kolla_external_vip_address
kolla_external_fqdn
The default behaviour will remain the same, and the way the variable
inheritance is setup the kolla_internal_address variable can still be
set in globals.yml and propogate out to these 4 new variables like it
normally would, but all reference to kolla_internal_address has been
completely removed.
Change-Id: I4556dcdbf4d91a8d2751981ef9c64bad44a719e5
Partially-Implements: blueprint ssl-kolla
Currently Heka fails to parse the RabbitMQ logs. There are two
problems:
1. The rabbit-sasl.log file is ignored but the file_match expression
does not match.
2. The delimiter used in the RegexSplitter makes Heka stop on the
very first log entry. '\n\n(=[^=]+====' (with two \n's) is
a better delimiter. deliver_incomplete_final is used to get the
final log entry.
TrivialFix
Change-Id: I94720340d5b2d6fd5d7641b9ff58733f6cd882ee
HAProxy: change to use option forwardfor to pass origin IP address
to backend via X-Forwarded-For header
Keystone: Apache does the audit logs for keystone. Change the
LogFormat to display the passed address instead of the connection
address which is that of the load balancer.
Nova, Cinder, Glance: these services can make use of the address
passed in X-Forwarded-For. With this setting the API logs for
these services include the client IP address.
Change-Id: Ia861ecc11a7c7d463d0366586926d1a842853f69
Closes-Bug: #1548935
This bootstrap was non-idempotent. This patch follows the style
first implemented with nova to make this idempotent.
TrivialFix
Change-Id: Id04e59c5274a7d8a5bffd3ce018f3bbb84839d75
Swift uses Syslog, but it uses a custom log format. So this commit
adds a specific Heka decoder for Swift.
It also increases the log level from "warning" to "info" to make
Swift more verbose. Note that "info" is the default log level in
Swift.
And it disables the Heka configuration for Swift when "enable_swift"
is set to "no". This prevents Heka from creating 15 empty Swift log
files in the logs volume.
Partially implements: blueprint heka
Change-Id: If7a7d0707e71be2957178e2d45b5de51b788232e
The Ansible logs are currently not collected by Heka. This can be
done later, with an Ansible-specific decoder for Heka.
Partially implements: blueprint heka
Change-Id: I8d3ba4edb527f61c0a8234024b4be953c6e6c565
This one is simple, simply start the services with new images!
Change-Id: Ia29cfa136856e8413dac1edee38a55d9f5a45c3e
Partially-Implements: blueprint upgrade-kolla
The extend_start.sh script for rsyslog is removed as it is no longer
needed. Docker no longer binds to /dev/log or /run/kolla/log
Closes-Bug: #1544545
Change-Id: Ic0a323a26ee4e9e15baf4598285844a8a4955f23
To allow for TLS to protect the service endpoints, the protocol
in the URLs for the endpoints will be either http or https.
This patch removes the hardcoded values of http and replaces them
with variables that can be adjusted accordingly in future patches.
Change-Id: Ibca6f8aac09c65115d1ac9957410e7f81ac7671e
Partially-implements: blueprint ssl-kolla
Docker 1.10 has broken the gate and this patch will correct that
breakage.
The issue comes with rsyslog. Due to a commit in Docker 1.10 [1] we
must change the way we get the log socket for rsyslog. The /dev/
folder will no longer populate as we used it. So instead we simply
make a new socket in a path we control and share that to the correct
location in the containers.
Additionally, adjust the gate for new Docker daemon.
[1] https://github.com/docker/docker/pull/16639
Partially-Implements: blueprint kolla-upgrade
Change-Id: I881a2ecdf6d7b35991e1d38a3f3e60d022d6577f
This change is needed for clarity. We have a kolla-ansible script.
We have a kolla-mesos repo. We plan to have a kolla-ansible repo.
Already we have had far too much confusion about whether we are
talking about the container or the project. Naming this kolla-toolbox
eliminates all of that confusion and its probably a bit more accurate
of a name too.
Closes-Bug: #1541053
Change-Id: I8fd1f49d5a22b36ede5b10f46b9fe02ddda9007e
After introduction of pull action and turing every main.yml into
{{action}}.yml we lost ability to perform upgrade
Change-Id: Ie9fa2cd083b061033abc733fba53d54f9c55e393
Fixes-Bug: #1538210
