This moves to update the host used for the ceph health checks, as
we should be checking the ceph-mgr service directly for ceph
metrics instead of trying to curl the host directly.
This also changes the ceph_health_check to use the base-os
hostgroup instead of the placeholder ceph-mgr host group, as we're
just executing a simple check against the ceph-mgr service.
This also adds default configuration values for the
max_concurrent_checks (60) and check_workers (4) values instead
of leaving them at the defaults Nagios uses (0 and # cores,
respectively)
Change-Id: Ib4072fcd545d8c05d5e9e4a93085a8330be6dfe0
This updates the Nagios image to use a tag that includes a fix for
the service discovery mechanism used for updating host checks.
After moving the Nagios chart to either run in shared or host PID
namespaces, the service discovery mechanism no longer worked due
to the plugin attempting to restart PID 1 instead of determining
the appropriate PID to restart.
For reference, see:
https://review.gerrithub.io/#/c/att-comdev/nagios/+/432205/
Change-Id: Ie01c3a93dd109a9dc99cfac5d27991583546605a
This adds session affinity to Nagios's ingress. This allows for
the use of cookies for Nagios's session affinity
Change-Id: I6054a92f644dc533dd06d35a2541fb44d46cba88
The ceph_health check in Nagios incorrectly sets the warning and
error level to 0. The ceph_health_status metric's value of 0
indicates the cluster is healthy, while 1 indicates a warning and
2 indicates an error state. The Nagios check for ceph_health is
updated to reflect these values
Change-Id: Iffe80f1c34f6edee6370dd7e707e5f55f83f1ec1
This moves Nagios to run as child processes of either
the pause container or use the hosts init system (for k8s <1.10)
to prevent defunct process sprawl
Change-Id: I6a93d446577674b0b012f9567d5e6a5794ebc44b
This removes the checks for Nagios to query Elasticsearch for
logged events. The current plugin in the image is resulting in
unstable behavior, and should be removed until this plugins been
improved
Change-Id: If1bdd954956f063ac1eebbb94d1128df8b8d2695
This updates the Nagios image to include an update to the
Elasticsearch plugin that adds the appropriate headers to the
request sent to Elasticsearch. As Elasticsearch >=6.0 no longer
tries to determine the request data type, we need to explicitly
tell Elasticsearch the request body is JSON. Since we use
Elasticsearch 5.6.4 as default, this change will make the
deprecation warnings for the 6.0 breaking change go away.
Change-Id: I0dbd8859ca8d0bd0893832b4edd92742e575598b
This patch set implements the helm toolkit function to generate a
kubernetes network policy manifest based on overrideable values.
This also adds a chart that shuts down all the ingress and egress
traffics in the namespace. This can be used to ensure the
whitelisted network policy works as intended.
Additionally, implementation is done for some infrastructure charts.
Change-Id: I78e87ef3276e948ae4dd2eb462b4b8012251c8c8
Co-Authored-By: Mike Pham <tp6510@att.com>
Signed-off-by: Tin Lam <tin@irrational.io>
Fixing opebstack API monitors
Adding additional neutron services monitors
Adding new Pod CrashLoopBaackOff status check
Adding new Host readiness check
Updated the nagios image reference(https://review.gerrithub.io/c/att-comdev/nagios/+/420590 - Pending)
This updated image provides a mechanism for querying Elasticsearch
with the goal of triggering alerts based on specified applications
and log levels.
Finally, this moves the endpoints resulting from the authenticated
endpoint lookups required for Nagios to the nagios secret instead
of handled via plain text environment variables
Change-Id: I517d8e6e6e8fa1d359382be8a131a8e45bf243e2
This PS adds the ability to attach a release uuid to pods and rc
objects as desired. A follow up ps will add the ability to add arbitary
annotations to the same objects.
Change-Id: Iceedba457a03387f6fc44eb763a00fd57f9d84a5
Signed-off-by: Pete Birley <pete@port.direct>
This PS updates the nagios chart to use the public endpoint for
ldap in the apache config.
Change-Id: Ia7c1881a15fda3100fb006e9cf1d06d22dcd6a8d
Signed-off-by: Pete Birley <pete@port.direct>
This updates the osh-infra charts to use a secret for their
configuration files instead of a configmap, allowing for the
storage of sensitive information
Change-Id: Ia32587162288df0b297c45fd43b55cef381cb064
This adds authentication to Prometheus with an apache reverse
proxy, similar to elasticsearch, kibana and nagios. This adds an
admin user and password via htpasswd along with adding ldap
support.
This required modifying the grafana chart to configure the
prometheus datasource's basic auth credentials in the data sources
provisioning configuration file by checking whether basic auth is
enabled and injecting the username/password defined in the
corresponding endpoint definition.
This also modifies the nagios chart to use the authenticated
endpoint for prometheus, which is required for nagios to
successfully query the prometheus endpoint for its service
checking mechanism
Change-Id: Ia4ccc3c44a89b2c56594be1f4cc28ac07169bf8c
In most cases, the ingress controller's nodeSelector key and value
are "node-role.kubernetes.io/ingress" and "true".
Using quote to treat the nodeSelector value as a string.
Change-Id: Ie1745629b90795e4d888d85f35565e6d6350e09b
This changes the ordering of the configmap annotations for kibana,
as older versions of helm require the configmap with the values
template definition for the apache proxy to be listed last. This
was addressed in the elasticsearch-client template but missed in
kibana.
This also adds the configmap hash annotations to the nagios chart
as they were previously missing. It also places them in the
correct order as above
Change-Id: I13befe8684d975f310f2723c5172b8a0f9f365d6
This proposes defining the apache proxy hosts entirely via values
templates. While complicated on its face, this gives flexibility
by allowing the ability to define the desired authentication
mechanism via values templates. These options can range from
using http basic auth for development purposes to defining more
complex ldap configurations without a need to modify the chart
directly
Change-Id: Ief1b6890444ff90cc9c0ca872087af74836c0771
Signed-off-by: Pete Birley <pete@port.direct>
This updates the Nagios image tag to include a version that fixes
the service discovery bug that resulted in duplicate host group
entries. The duplicate host group entries would prevent Nagios
from restarting, resulting in the service never coming back up
when duplicate host groups were identified and added
Change-Id: I555c525e47deffd95eeb5a7276c00cf044e61e3a
This updates the TLS secret templates to include the backend
service in the dict supplied to the manifest template, as it is
required for the TLS secret to render correctly.
This also removes the readiness probe from the nagios container in
the deployment for the nagios chart, as it wasn't functioning as
intended due to the port not being available for the probe
Change-Id: Iabcfd40c74938e0497d08ffeeebc98ab722fa660
Adds support for TLS on overriden fqdns for public endpoints for
the services that have them in openstack-helm-infra. Currently this
implementation is limited, in that it does not provide support for
dynamically loading CAs into the containers, or specifying them manually
via configuration. As a result only well known or CA's added manually
to containers will be recognised.
Change-Id: I4ab4bbe24b6544b64cd365467e8efb2a421ac3f4
This adds missing readiness probes to the following charts in
openstack-helm-infra: elasticsearch, fluent-logging, kibana,
nagios, prometheus-kube-state-metrics, prometheus-node-exporter,
and prometheus-openstack-exporter
Change-Id: I6a2635b08667c31eadb1b05ba848c658935a17e5
This updates the ordering of the basic auth providers in the
elasticsearch and nagios chart to check the file provider first
before going out to check the configured ldap server.
Change-Id: I47ff8a1c7b2cefa8425914c5d4d7a76aa8d43216
Signed-off-by: Steve Wilkerson <wilkers.steve@gmail.com>
This PS moves to use the current ga version for kubernetes daemonsets,
additionally any remaining deployments that were using the
`extensions/v1beta1` have been updated to `apps/v1`.
Story: 2002205
Task: 21735
Change-Id: If9703162dc472af1e6096bf2b9062802fd5ce8ab
Signed-off-by: Pete Birley <pete@port.direct>
This adds a basic check for capacity utilization for persistent
volume claims. To accomplish this, it adds a basic alerting rule
to prometheus that triggers after a persistent volume's usage
exceeds 80%, and triggers 5 minutes after that state has been
reached. In addition, there is a service check added to the
nagios chart that will query Prometheus to check if the alarm
for that threshhold is firing for any of the volume claims.
Change-Id: I862c860ac479a715733202f679bb151885d7aa7c
This PS simply moves functions within the chart to their correct location.
Change-Id: Ia3d693713903d226a864dcdcf9884dee67f07d2b
Signed-off-by: Pete Birley <pete@port.direct>
This adds the ability to drive the CGI configuration for
nagios via values, similar to the other nagios configuration
entities
Change-Id: I8e9de21d141e0a87cdda11c4a778abec210277f3
This adds an apache reverse proxy to the nagios chart, similar
to elasticsearch and kibana. It also adds authentication to
nagios via ldap
Change-Id: I7b17703b5d4c1e041691ffceb984a9f5951cbeb9
This ps proposes adding a common template for the image_repo_sync
jobs for consumption by the charts
Change-Id: I48476d1e4fd94bd1b08b13b46983e3d999f8d8ca
This moves all relevant charts in osh-infra to use the htk manifest
template for ingresses, bringing them in line with the charts in
openstack-helm
Change-Id: Ic9c3cc6f0051fa66b6f88ec2b2725698b36ce824
This ps adds more granular node selectors for the charts in osh
infra to match what is currently done in osh
Change-Id: I8957a95053b9fb3ea329fd37ff049cd223a7695d
This PS simplify the logic for dyanmicly merging the image management
depenencies into pod deps when active.
Change-Id: I0cf6c93173bc5fbce697ac15be8697d3b1326d0a
This ps updates the nagios chart to use endpoint port lookups for
port configuration, bringing it in line with the other charts
Change-Id: I500b4741d50132f6c316ded660981e2af8b71e7a
This adds the nagios chart to osh-infra to provide additional
monitoring functionality. It uses helper functions to consume
yaml definitions for services, commands, hosts and hostgroups
to generate the required configurations for those entities in
nagios's configuration
Change-Id: I6238bb8cb1e5c8dc48594ddea50693f3e7b0a176
