This moves the default workers defined for Nagios back to 4, as
reducing the number to one has made Nagios deployments in our jobs
inconsistent
Change-Id: I288554c4f9d57714b3d70a241942b2e4fd334500
This reduces the maximum concurrent checks Nagios will execute to
prevent process sprawl on the host. This also reduces the number
of default workers to a single worker, to prevent Nagios from
forking off multiple processes that then execute service checks
and commands in parallel
Change-Id: I0d8445a265740b4a2491bdfd739cb0f27955f06d
This updates the Prometheus pod container status alerts. This
ensures there are alerts defined for ImagePullBackOff,
ErrImagePull, and CreateContainerConfigError errors.
This also updates the Nagios service checks to include correct
checks for those alerts
Change-Id: I91544e7dff8c6aac8c79cd8aa7d8f7bc03adaa9a
This updates the Nagios chart configuration to not use syslog for
logging, removes the logging of notifications, and drastically
increases the number of concurrent checks executed.
This also removes the hostPath for Nagios logs, as it seems to add
no value over what's already reported to the console. Finally, as
Nagios's log file has the potential to grow very rapidly while the
service has no means to disable logging to disk, this adds a
readiness probe that both checks whether Nagios's endpoint is
being served and clears out the log file by redirecting the
no-op commands output to the nagios log file.
Change-Id: I81151c48ef4e0b7877f595c271f55b8fd479e8c1
This updates the Elasticsearch health status expressions used in
Prometheus, Nagios and Grafana. The previous Prometheus rule
defined for Elasticsearch health checked for a status that was
> 0 to trigger an alarm for a green health status. The correct
returned values are: 1 for green, 0 for both red and yellow. This
changes the expression to use arithmetic operators to give us a
result that maps to: 2 for green, 1 for yellow, 0 for red.
This also updates the Elasticsearch dashboard in Grafana to add a
new mapping for the updated 2g,1y,0r scale.
Finally, this also updates the Nagios service check to be a bit
more verbose in its output.
For reference, see:
https://github.com/justwatchcom/elasticsearch_exporter/issues/120
Change-Id: I6ef2a7c308c6ebfdb693b46127a285bceb6ba872
This fixes the Nagios volume mount for the Elasticsearch query
file. Previously, the check for adding the volumemount to the
pod definition was incorrect. This fixes the conditional check,
and also adds the same conditional check to the configuration
secret
This adds a simple check to the monitoring and multinode jobs to
validate the resulting json gets mounted into the pod successfully
Change-Id: I2af289ccc4e1cff1669cb5e6e829514781b14dd3
This removes unused pod-etc-apache volumes from the charts that
use an apache sidecar container as a reverse proxy.
Change-Id: Ibafff3b53f9d3c20f5aed30d40ee6470cb515a8a
This PS implements the helm toolkit function to generate the
Egress in kubernetes network policy manifest based on overrideable values.
It also enbale the K8s network policy at Osh-infra gate.
Change-Id: Icbe2a18c98dba795d15398dcdcac64228f6a7b4c
This updates the Nagios image tag to include the updated plugin
for querying Elasticsearch for alerting on logged events
Change-Id: Idd61d82463b79baab0e94c20b32da1dc6a8b3634
This moves to update the host used for the ceph health checks, as
we should be checking the ceph-mgr service directly for ceph
metrics instead of trying to curl the host directly.
This also changes the ceph_health_check to use the base-os
hostgroup instead of the placeholder ceph-mgr host group, as we're
just executing a simple check against the ceph-mgr service.
This also adds default configuration values for the
max_concurrent_checks (60) and check_workers (4) values instead
of leaving them at the defaults Nagios uses (0 and # cores,
respectively)
Change-Id: Ib4072fcd545d8c05d5e9e4a93085a8330be6dfe0
This updates the Nagios image to use a tag that includes a fix for
the service discovery mechanism used for updating host checks.
After moving the Nagios chart to either run in shared or host PID
namespaces, the service discovery mechanism no longer worked due
to the plugin attempting to restart PID 1 instead of determining
the appropriate PID to restart.
For reference, see:
https://review.gerrithub.io/#/c/att-comdev/nagios/+/432205/
Change-Id: Ie01c3a93dd109a9dc99cfac5d27991583546605a
This adds session affinity to Nagios's ingress. This allows for
the use of cookies for Nagios's session affinity
Change-Id: I6054a92f644dc533dd06d35a2541fb44d46cba88
The ceph_health check in Nagios incorrectly sets the warning and
error level to 0. The ceph_health_status metric's value of 0
indicates the cluster is healthy, while 1 indicates a warning and
2 indicates an error state. The Nagios check for ceph_health is
updated to reflect these values
Change-Id: Iffe80f1c34f6edee6370dd7e707e5f55f83f1ec1
This moves Nagios to run as child processes of either
the pause container or use the hosts init system (for k8s <1.10)
to prevent defunct process sprawl
Change-Id: I6a93d446577674b0b012f9567d5e6a5794ebc44b
This removes the checks for Nagios to query Elasticsearch for
logged events. The current plugin in the image is resulting in
unstable behavior, and should be removed until this plugins been
improved
Change-Id: If1bdd954956f063ac1eebbb94d1128df8b8d2695
This updates the Nagios image to include an update to the
Elasticsearch plugin that adds the appropriate headers to the
request sent to Elasticsearch. As Elasticsearch >=6.0 no longer
tries to determine the request data type, we need to explicitly
tell Elasticsearch the request body is JSON. Since we use
Elasticsearch 5.6.4 as default, this change will make the
deprecation warnings for the 6.0 breaking change go away.
Change-Id: I0dbd8859ca8d0bd0893832b4edd92742e575598b
This patch set implements the helm toolkit function to generate a
kubernetes network policy manifest based on overrideable values.
This also adds a chart that shuts down all the ingress and egress
traffics in the namespace. This can be used to ensure the
whitelisted network policy works as intended.
Additionally, implementation is done for some infrastructure charts.
Change-Id: I78e87ef3276e948ae4dd2eb462b4b8012251c8c8
Co-Authored-By: Mike Pham <tp6510@att.com>
Signed-off-by: Tin Lam <tin@irrational.io>
Fixing opebstack API monitors
Adding additional neutron services monitors
Adding new Pod CrashLoopBaackOff status check
Adding new Host readiness check
Updated the nagios image reference(https://review.gerrithub.io/c/att-comdev/nagios/+/420590 - Pending)
This updated image provides a mechanism for querying Elasticsearch
with the goal of triggering alerts based on specified applications
and log levels.
Finally, this moves the endpoints resulting from the authenticated
endpoint lookups required for Nagios to the nagios secret instead
of handled via plain text environment variables
Change-Id: I517d8e6e6e8fa1d359382be8a131a8e45bf243e2
This PS adds the ability to attach a release uuid to pods and rc
objects as desired. A follow up ps will add the ability to add arbitary
annotations to the same objects.
Change-Id: Iceedba457a03387f6fc44eb763a00fd57f9d84a5
Signed-off-by: Pete Birley <pete@port.direct>
This PS updates the nagios chart to use the public endpoint for
ldap in the apache config.
Change-Id: Ia7c1881a15fda3100fb006e9cf1d06d22dcd6a8d
Signed-off-by: Pete Birley <pete@port.direct>
This updates the osh-infra charts to use a secret for their
configuration files instead of a configmap, allowing for the
storage of sensitive information
Change-Id: Ia32587162288df0b297c45fd43b55cef381cb064
This adds authentication to Prometheus with an apache reverse
proxy, similar to elasticsearch, kibana and nagios. This adds an
admin user and password via htpasswd along with adding ldap
support.
This required modifying the grafana chart to configure the
prometheus datasource's basic auth credentials in the data sources
provisioning configuration file by checking whether basic auth is
enabled and injecting the username/password defined in the
corresponding endpoint definition.
This also modifies the nagios chart to use the authenticated
endpoint for prometheus, which is required for nagios to
successfully query the prometheus endpoint for its service
checking mechanism
Change-Id: Ia4ccc3c44a89b2c56594be1f4cc28ac07169bf8c
In most cases, the ingress controller's nodeSelector key and value
are "node-role.kubernetes.io/ingress" and "true".
Using quote to treat the nodeSelector value as a string.
Change-Id: Ie1745629b90795e4d888d85f35565e6d6350e09b
This changes the ordering of the configmap annotations for kibana,
as older versions of helm require the configmap with the values
template definition for the apache proxy to be listed last. This
was addressed in the elasticsearch-client template but missed in
kibana.
This also adds the configmap hash annotations to the nagios chart
as they were previously missing. It also places them in the
correct order as above
Change-Id: I13befe8684d975f310f2723c5172b8a0f9f365d6
This proposes defining the apache proxy hosts entirely via values
templates. While complicated on its face, this gives flexibility
by allowing the ability to define the desired authentication
mechanism via values templates. These options can range from
using http basic auth for development purposes to defining more
complex ldap configurations without a need to modify the chart
directly
Change-Id: Ief1b6890444ff90cc9c0ca872087af74836c0771
Signed-off-by: Pete Birley <pete@port.direct>
This updates the Nagios image tag to include a version that fixes
the service discovery bug that resulted in duplicate host group
entries. The duplicate host group entries would prevent Nagios
from restarting, resulting in the service never coming back up
when duplicate host groups were identified and added
Change-Id: I555c525e47deffd95eeb5a7276c00cf044e61e3a
This updates the TLS secret templates to include the backend
service in the dict supplied to the manifest template, as it is
required for the TLS secret to render correctly.
This also removes the readiness probe from the nagios container in
the deployment for the nagios chart, as it wasn't functioning as
intended due to the port not being available for the probe
Change-Id: Iabcfd40c74938e0497d08ffeeebc98ab722fa660
Adds support for TLS on overriden fqdns for public endpoints for
the services that have them in openstack-helm-infra. Currently this
implementation is limited, in that it does not provide support for
dynamically loading CAs into the containers, or specifying them manually
via configuration. As a result only well known or CA's added manually
to containers will be recognised.
Change-Id: I4ab4bbe24b6544b64cd365467e8efb2a421ac3f4
This adds missing readiness probes to the following charts in
openstack-helm-infra: elasticsearch, fluent-logging, kibana,
nagios, prometheus-kube-state-metrics, prometheus-node-exporter,
and prometheus-openstack-exporter
Change-Id: I6a2635b08667c31eadb1b05ba848c658935a17e5
This updates the ordering of the basic auth providers in the
elasticsearch and nagios chart to check the file provider first
before going out to check the configured ldap server.
Change-Id: I47ff8a1c7b2cefa8425914c5d4d7a76aa8d43216
Signed-off-by: Steve Wilkerson <wilkers.steve@gmail.com>
This PS moves to use the current ga version for kubernetes daemonsets,
additionally any remaining deployments that were using the
`extensions/v1beta1` have been updated to `apps/v1`.
Story: 2002205
Task: 21735
Change-Id: If9703162dc472af1e6096bf2b9062802fd5ce8ab
Signed-off-by: Pete Birley <pete@port.direct>
This adds a basic check for capacity utilization for persistent
volume claims. To accomplish this, it adds a basic alerting rule
to prometheus that triggers after a persistent volume's usage
exceeds 80%, and triggers 5 minutes after that state has been
reached. In addition, there is a service check added to the
nagios chart that will query Prometheus to check if the alarm
for that threshhold is firing for any of the volume claims.
Change-Id: I862c860ac479a715733202f679bb151885d7aa7c
This PS simply moves functions within the chart to their correct location.
Change-Id: Ia3d693713903d226a864dcdcf9884dee67f07d2b
Signed-off-by: Pete Birley <pete@port.direct>
This adds the ability to drive the CGI configuration for
nagios via values, similar to the other nagios configuration
entities
Change-Id: I8e9de21d141e0a87cdda11c4a778abec210277f3
This adds an apache reverse proxy to the nagios chart, similar
to elasticsearch and kibana. It also adds authentication to
nagios via ldap
Change-Id: I7b17703b5d4c1e041691ffceb984a9f5951cbeb9
