48 Commits

Author SHA1 Message Date
Steven Fitzpatrick
6de864110e Elasticsearch S3 Update
This change updates how the Elasticsearch chart handles
S3 configuration and snapshot repository registration.

This allows for
  - Multiple snapshot destinations to be configued
  - Repositories to use a specific placement target
  - Management of multiple account credentials

Change-Id: I12de918adc5964a4ded46f6f6cd3fa94c7235112
2021-04-06 15:12:34 +00:00
KHIYANI, RAHUL (rk0850)
9cfb1f8509 Add missing security-context for elasticsearch-data and elasticsearch-master
This also implements security-context template to add readOnly-fs flag

Change-Id: Iaeea66dad34a2616c0620eafacc53574ed79a7b5
2020-07-16 19:54:43 +00:00
Andrii Ostapenko
824f168efc Undo octal-values restriction together with corresponding code
Unrestrict octal values rule since benefits of file modes readability
exceed possible issues with yaml 1.2 adoption in future k8s versions.
These issues will be addressed when/if they occur.

Also ensure osh-infra is a required project for lint job, that matters
when running job against another project.

Change-Id: Ic5e327cf40c4b09c90738baff56419a6cef132da
Signed-off-by: Andrii Ostapenko <andrii.ostapenko@att.com>
2020-07-07 15:42:53 +00:00
Andrii Ostapenko
83e27e600c Enable key-duplicates and octal-values yamllint checks
With corresponding code changes.

Change-Id: I11cde8971b3effbb6eb2b69a7d31ecf12140434e
2020-06-17 13:14:30 -05:00
Andrii Ostapenko
dfb32ccf60 Enable yamllint rules for templates
- braces
- brackets
- colons
- commas
- comments
- comments-indentation
- document-start
- hyphens
- indentation

With corresponding code changes.

Also idempotency fix for lint script.

Change-Id: Ibe5281cbb4ad7970e92f3d1f921abb1efc89dc3b
2020-06-17 13:13:53 -05:00
diwakar thyagaraj
36fe912df0 Enable Apparmor to Elasticsearch Completed Pods
Change-Id: I52e07c585c50817706e64b8e2f26f73c25587da7
Signed-off-by: diwakar thyagaraj <diwakar.chitoor.thyagaraj@att.com>
2020-05-26 16:09:57 +00:00
Zuul
e53d28718d Merge "Remove OSH Authors copyright" 2020-05-12 20:00:38 +00:00
Steven Fitzpatrick
95e3c21df4 Settings for Remote Elasticsearch Clusters
This change adds a new Deployment to the Elasticsearch chart to add a
set of "gateway" nodes to the Elasticsearch cluster. These nodes will
facilitate Elasticsearch remote cluster, for features such as cross
cluster search.

Co-Authored-By: David Smith <ds3330@att.com>
Change-Id: Ic4ac988a922a12addce3c65e0ef4099d46bbc784
2020-05-08 13:07:54 -05:00
Gage Hugo
d14d826b26 Remove OSH Authors copyright
The current copyright refers to a non-existent group
"openstack helm authors" with often out-of-date references that
are confusing when adding a new file to the repo.

This change removes all references to this copyright by the
non-existent group and any blank lines underneath.

Change-Id: I1882738cf9757c5350a8533876fd37b5920b5235
2020-05-07 02:11:15 +00:00
Steve Wilkerson
03580ec90a Elasticsearch: Make node selectors more granular
This updates the Elasticsearch chart to make the values keys used
for defining node selectors for the various elasticsearch
components more granular

Change-Id: Ic1ac343b1d6ee48fc7cb456afe4cd9588c4aa13b
Signed-off-by: Steve Wilkerson <sw5822@att.com>
2019-12-10 15:55:38 -06:00
Steve Wilkerson
fd7067649a Elasticsearch: Remove unnecessary rbac definitions
This removes the cluster role definition from the Elasticsearch
component templates, as these are not needed for the service to
function correctly.

Change-Id: I671272affbed8984a47121187024e4b831937123
Signed-off-by: Steve Wilkerson <sw5822@att.com>
2019-12-03 09:06:13 -06:00
Steve Wilkerson
2d3c9575ff Elasticsearch/Kibana: Update version to 7.1.0
This updates the Elasticsearch and Kibana charts to deploy
version 7.1.0. This move required significant changes to both
charts, including: changing elasticsearch masters to a statefulset
to utilize reliable dns names for the discovery process, config
updates to reflect deprecated/updated/removed values, use the
kibana saved objects api for managing index patterns and setting
the default index, and updating the elasticsearch entrypoint
scripts to reflect the use of elastic-keystore for storing s3
credentials instead of defining them in the configuration file

Change-Id: I270d905f266fc15492e47d8376714ba80603e66d
Signed-off-by: Steve Wilkerson <sw5822@att.com>
2019-12-03 07:43:29 -06:00
Steve Wilkerson
91178c31bf Elasticsearch: Add data node specific entrypoint override
This updates the Elasticsearch chart to include a specific start
script for the Elasticsearch data nodes that includes a trap on
signals that removes a data node from allocation eligible nodes
before shutting down.  This results in all shards being moved from
a node on shut down to alleviate issues with planned down nodes,
such as during upgrade scenarios

Change-Id: I22f4957f90e4113831a8ddf48691cb14f811c1e5
2019-05-30 10:25:03 -05:00
Steve Wilkerson
36d03133f8 Add statefulset update strategy to elasticsearch data statefulset
This adds the helm-toolkit function for defining the update
strategy for the elasticsearch-data statefulset and sets the chart
default to RollingUpdate

Change-Id: Ia10ea7bf000474e597bdb36778118a96d85b93c1
2019-05-23 22:05:06 +00:00
Zuul
ead842d1d8 Merge "Elasticsearch: Heap configuration and ingest node updates" 2019-05-09 21:10:26 +00:00
Steve Wilkerson
88f21acf34 Curator: Update image, add separate configmaps for service
This updates the Curator image to use version 5.6.0, which adds
additional actions for use, such as the ability to shrink indices.

This also adds a separate configmap and config secret for Curator,
as this allows us to use separate configmap annotations on the
Elasticsearch component pods to prevent Curator config updates
from triggering recreation of Elasticsearch components. This helps
alleviate overhead associated with Elasticsearch service restarts.

Change-Id: I0aec7756b0dc09bc3981ede950dc88f821aeca4b
2019-05-09 10:22:25 -05:00
Steve Wilkerson
031ee3e6af Elasticsearch: Heap configuration and ingest node updates
This updates the Elasticsearch chart to allow for setting the
heap size per node type instead of for all nodes equally. This
also adds the required environment variable to configure whether
a node is an ingest node. This is set to false, as suggested for
elasticsearch versions <= 6.x

This also removes the ES_PLUGINS_INSTALL environment variable as
it is not used for anything in the current charts

Change-Id: I9096774db46dcbcd48b8a5448f0510984bf4108f
2019-05-06 14:55:45 -05:00
RAHUL KHIYANI
5be16a66d7 Elasticsearch: Fix security context
This PS fixes the use of the security context macros for the
elasticsearch chart.

Change-Id: I85a37aa4dec88222107323f17d10e5ff29f41648
2019-04-23 23:04:18 -05:00
Pete Birley
2abf62ff4d OSH-Infra: Add emptydirs for tmp
This PS adds emptydirs backing the /tmp directory in pods, which
is required in most cases for full operation when using a read only
filesystem backing the container.

Additionally some yaml indent issues are resolved.

Change-Id: I8b7f1614da059783254aa6efc09facf23fca3cad
Signed-off-by: Pete Birley <pete@port.direct>
2019-04-20 20:50:59 +00:00
Steve Wilkerson
84f30ec103 Add release-annotation to pod spec, add missing annotations
This adds the release-annotation to the pod spec for the charts in
openstack-helm-infra. This also adds missing configmap annotations
to charts in openstack-helm-infra

Change-Id: Ie23f0c16a7a21d3929e98928db2bbcef69ae6490
2019-03-21 09:10:48 -05:00
Luna Das
ae24ce9999 Add default-docker (enforce) AppArmor profile to Elasticsearch
Change-Id: I86930ee90170385008d5c674eab34d7c0e34e6e4
2019-01-01 08:12:56 -05:00
Chris Wedgwood
0c4e37391f 'NOP' cleanup for more consistent white-space use in charts
Where we have the style '{{ ...' we should use the style '... }}'.

Change-Id: Ic3e779e4681370d396f95d3804ca27db5b9d3642
2019-01-03 22:45:49 +00:00
Steve Wilkerson
a084769410 Elasticsearch S3 repo
This ps adds the ability to use the ceph radosgw s3 api for
snapshot repositories. It removes the ability to use a RWM pvc, as
the radosgw solution provides a more robust approach for storing
index snapshots

Change-Id: Ie56ac41ccdc61bfadcac52b400cceb35403e9fae
2018-09-19 15:53:21 -05:00
Pete Birley
bb3ff98d53 Add release uuid to pods and rc objects
This PS adds the ability to attach a release uuid to pods and rc
objects as desired. A follow up ps will add the ability to add arbitary
annotations to the same objects.

Change-Id: Iceedba457a03387f6fc44eb763a00fd57f9d84a5
Signed-off-by: Pete Birley <pete@port.direct>
2018-09-13 05:35:35 +00:00
Steve Wilkerson
9a311475ba Charts: Use secrets for configs in chart
This updates the osh-infra charts to use a secret for their
configuration files instead of a configmap, allowing for the
storage of sensitive information

Change-Id: Ia32587162288df0b297c45fd43b55cef381cb064
2018-08-24 15:56:53 -05:00
Seungkyu Ahn
a430533e6a Quoting node_select_value in Ingress Controller
In most cases, the ingress controller's nodeSelector key and value
are "node-role.kubernetes.io/ingress" and "true".
Using quote to treat the nodeSelector value as a string.

Change-Id: Ie1745629b90795e4d888d85f35565e6d6350e09b
2018-08-01 02:39:05 +00:00
Steve Wilkerson
cb7bf2c0b3 Add missing readiness probes to openstack-helm-infra charts
This adds missing readiness probes to the following charts in
openstack-helm-infra: elasticsearch, fluent-logging, kibana,
nagios, prometheus-kube-state-metrics, prometheus-node-exporter,
and prometheus-openstack-exporter

Change-Id: I6a2635b08667c31eadb1b05ba848c658935a17e5
2018-06-26 12:25:36 +00:00
Pete Birley
b6a51fb57f Use current kubernetes API version
This PS moves to use the current API version for kubernetes rcs'
that were previously using `apps/v1beta1`.

Story: 2002205
Task: 21735

Change-Id: Icb4e7aa2392da6867427a58926be2da6f424bd56
Signed-off-by: Pete Birley <pete@port.direct>
2018-06-12 17:35:13 -05:00
Steve Wilkerson
de9c46bcfa Charts: Tidy up openstack-helm-infra charts
This moves the charts in openstack-helm-infra closer towards a
standard structure. It addresses multiple deviations, including:
missing resources for init containers, incorrect indents for
disabled resources in some charts, incorrect indents for volumes
and volumemounts added via values, missing resources for some
helm test templates, missing helm-toolkit image functions, and
moving the resource template declarations to be under the image
template declarations

Change-Id: I4834a5d476ef7fc69c5583caacc0229050f20a76
2018-05-21 12:58:22 -07:00
Steve Wilkerson
aaffc4caf0 OSH-Infra: Update labels for chart components
This ps adds more granular node selectors for the charts in osh
infra to match what is currently done in osh

Change-Id: I8957a95053b9fb3ea329fd37ff049cd223a7695d
2018-04-13 08:44:33 -05:00
Pete Birley
b9336ca613 Helm-Toolkit: Kubernetes Entrypoint, simplify image dependencies
This PS simplify the logic for dyanmicly merging the image management
depenencies into pod deps when active.

Change-Id: I0cf6c93173bc5fbce697ac15be8697d3b1326d0a
2018-04-13 08:42:37 -05:00
Steve Wilkerson
8e4da9da55 Revert Elasticsearch/Kibana image change
This reverts the changes made to Elasticsearch, Kibana and fluent
logging charts in https://review.openstack.org/#/c/550229/7.

Specifically, this moves the images back to previous used versions
and makes the required changes to the fluent-logging elasticsearch
template job to include the correct mapping directives for the
elasticsearch template.

This change was made to give more time for evaluating a more
robust solution for switching to the official upstream images that
will not cause intermittent gate failures as seen since 550229 was
merged

Change-Id: I9f70b3412a8edc5cb1d80937b158aa2fe7b1ec82
2018-03-12 10:27:35 -05:00
Steve Wilkerson
3f44f4586a Elasticsearch: use endpoints section and lookups to set port
This PS moves elasticsearch to use the endpoints section and
lookups to set the port it serves on.

Change-Id: I4a73893124b6d988cd1f885cfc3dd62abeb4ae8c
2018-03-08 20:00:54 +00:00
Steve Wilkerson
d681396412 Address errors with Elasticsearch and Kibana
This moves Elasticsearch and Kibana to use the latest version
(6.2.2), as the images we were using are no longer supported with
the 6.x release.  There was a change in the doc reference in the
log entries that prevented the previous ES version from indexing
those entries, resulting in a busted gate.  Moving Kibana to 6.2.2
was required to match major/minor versions with Elasticsearch

The Elasticsearch version change also required changing config file
locations, changing the entrypoint used for launching the service,
changing the running user for the elasticsearch service, and
updated the ES tests as some of the API responses changed between
versions

This also required updating the elasticsearch template job as the
mapping definition entries changed between versions

Change-Id: Ia4cd9a66851754a1bb8f225c7e24513c43568e93
2018-03-08 10:27:06 -06:00
Pete Birley
3c101a6324 dependencies: move dynamic common deps under a 'dynamic.common' key
This PS moves existing dynamic common dependencies under a
'dynamic.common' key to simplify the yaml tree.

Change-Id: I4332bcfdf11197488e7bd5d8cf4c25565ea1c7b6
2018-02-24 17:42:10 -05:00
Pete Birley
e0c688d7ee dependencies: move static dependencies under a 'static' key
This PS moves static dependencies unser a 'static' key to allow
expansion to cover dynamic dependencies.

Change-Id: Ia0e853564955e0fbbe5a9e91a8b8924c703b1b02
2018-02-24 17:39:55 -05:00
portdirect
515494ca98 RBAC: Include release name in cluster roles to prevent collision
This PS includes the release name in the cluster role to prevent
colision if the chart is deployed multiple times in the same
cluster.

Change-Id: I7166e5ee25b3d4c89879393c5f84c869585a2681
2018-02-19 13:13:56 -06:00
Sean Eagan
641c79c902 Add deep merge utility to helm-toolkit
Adds "helm-toolkit.utils.merge" which is a replacement for the
upstream sprig "merge" function which didn't quite do what we
wanted, specifically it didn't merge slices, it just overrode
one with the other.  This PS also updates existing callsites
of the sprig merge with "helm-toolkit.utils.merge".

Change-Id: I456349558d4cf941d1bcb07fc76d0688b0a10782
2018-02-13 10:08:50 -06:00
Steve Wilkerson
220310bd2f Add node selectors to elasticsearch chart
This adds node selectors to all templates in the elasticsearch
chart, as they were previously missed

Change-Id: I34ea5751663b2e993c5f73a78a1f91133919752c
2018-01-31 15:19:28 -06:00
portdirect
97d60dcccd Elasticsearch Snapshot: Fix permisions for PVC
This PS fixes the permisions for the PVC backing the
Elasticsearch Snapshot

Change-Id: I7b9897a7e0f34096ce1f2a04aceab7796d3a89c5
2018-01-10 13:40:11 -05:00
Zuul
f41cec95a3 Merge "Fix elasticsearch repository configuration entry" 2018-01-06 23:30:18 +00:00
Steve Wilkerson
99ee859b66 Fix elasticsearch repository configuration entry
This dynamically adds the elasticsearch path.repo configuration
entry if it's not defined.  This solves issues arising when the
storage settings are disabled in favor of emptydirs for simpler
ES deployments.  If elasticsearch attempts to configure the repo
path with an invalid entry (inaccesible external or shared fs
path), the service will crash.

Change-Id: I089b77104107dfb1f8e6ea2d8a560384718e63f9
2018-01-05 10:54:00 -06:00
portdirect
a8fe16cd42 ElasticSearch: tidy rbac roles and bindings to live with appropriate rc
This PS brings ElasticSearch inline with other charts by placing the
RBAC roles and bindings in the same template as the pod rc they are
assocated with.


Change-Id: I6d541a18d6750d42d31326f77a9aacb06195ddac
2018-01-05 06:52:49 +00:00
Steve Wilkerson
9b32ba17f4 Rename elasticsearch configmaps and provide config via toYaml
This brings the elasticsearch configmaps, volume and mount names
inline with other charts by naming them after the service.

This also moves the configuration for elasticsearch to the values
file to bring it inline with other charts that do the same

Change-Id: I61f7c740d830a9a0567f8b72a0f815a09407b90c
2017-12-28 19:48:58 +00:00
Steve Wilkerson
45ba95a2de Elasticsearch: Add curator snapshot action and PVC for fs repo
This provides an example action in the curator config for taking
snapshots of the elasticsearch indexes. As the snapshot action
requires a repository registered with Elasticsearch, this also
adds a PVC for a filesystem repository backed with NFS and a job
for registering the repository with Elasticsearch.

Change-Id: I26b788c58f52844e997bde5002459bddc1bb685e
2017-12-28 13:42:47 -06:00
Tin Lam
628fd3007d RBAC: Consolidate serviceaccounts and restrict rbac
Currently, services have two serviceaccounts: one specified in the
chart that cannot read anything, and one injected via helm-toolkit
that can read everything. This patch set refactors the logic to:

- cleanup the roles and their binding automatically when the helm
  chart is deleted;
- remove the need to separately mount a serviceaccount  with secret;
- better handling of namespaces resource restriction.

Co-Authored-By: portdirect <pete@port.direct>

Change-Id: I47d41e0cad9b5b002f59fc9652bad2cc025538dc
2017-12-19 20:22:57 -05:00
Steve Wilkerson
bea44e53bf Add Elasticsearch liveness/readiness probes
The elasticsearch tests fail because the pods don't have
readiness or liveliness probes in the templates. This adds those
definitions

Change-Id: I4fd25aec5ae02d89ae1b933d8b083a3e9cafc55a
2017-12-11 18:47:46 -06:00
Steve Wilkerson
4b94e47c94 Add Elasticsearch to OSH Infra
This moves the Elasticsearch chart to OSH infra, along with rbac
rules for running Elasticsearch. It includes a cronjob for running
ES Curator for cleaning up old indices

Change-Id: I69fcbe8b77de8b594eba5340a6e4340f389ba5bf
2017-12-01 17:42:23 -06:00