80 Commits

Author SHA1 Message Date
Steve Wilkerson
3f44f4586a Elasticsearch: use endpoints section and lookups to set port
This PS moves elasticsearch to use the endpoints section and
lookups to set the port it serves on.

Change-Id: I4a73893124b6d988cd1f885cfc3dd62abeb4ae8c
2018-03-08 20:00:54 +00:00
Steve Wilkerson
d681396412 Address errors with Elasticsearch and Kibana
This moves Elasticsearch and Kibana to use the latest version
(6.2.2), as the images we were using are no longer supported with
the 6.x release.  There was a change in the doc reference in the
log entries that prevented the previous ES version from indexing
those entries, resulting in a busted gate.  Moving Kibana to 6.2.2
was required to match major/minor versions with Elasticsearch

The Elasticsearch version change also required changing config file
locations, changing the entrypoint used for launching the service,
changing the running user for the elasticsearch service, and
updated the ES tests as some of the API responses changed between
versions

This also required updating the elasticsearch template job as the
mapping definition entries changed between versions

Change-Id: Ia4cd9a66851754a1bb8f225c7e24513c43568e93
2018-03-08 10:27:06 -06:00
Hemanth Nakkina
5392ac810b Remove Api version requirement for CronJob Curator
Curator job in Elasticsearch helm chart has a condition on
api version batch/v2alpha1. Cronjob resource is deprecated
in batch/v2alpha1 from k8s 1.8 and batch/v1beta1 is enabled
by default.

Remove the condition on API version as it is no more required.

Closes-Bug: #1753524
Change-Id: Ia296b3742e655fae508e5d4402e7f3881db31688
2018-03-06 08:51:08 +05:30
Zuul
d6e9fa2c5a Merge "Add clusterrolebinding and clusterrole for elasticsearch-master" 2018-03-01 10:21:11 +00:00
Pete Birley
3c101a6324 dependencies: move dynamic common deps under a 'dynamic.common' key
This PS moves existing dynamic common dependencies under a
'dynamic.common' key to simplify the yaml tree.

Change-Id: I4332bcfdf11197488e7bd5d8cf4c25565ea1c7b6
2018-02-24 17:42:10 -05:00
Pete Birley
e0c688d7ee dependencies: move static dependencies under a 'static' key
This PS moves static dependencies unser a 'static' key to allow
expansion to cover dynamic dependencies.

Change-Id: Ia0e853564955e0fbbe5a9e91a8b8924c703b1b02
2018-02-24 17:39:55 -05:00
Steve Wilkerson
94d6e2899e Add clusterrolebinding and clusterrole for elasticsearch-master
The template for elasticsearch-master was missing the clusterrole
and clusterrolebinding. This adds them to bring it in line with
the other templates

Change-Id: I34bc7e889018411b3791c1b7f24d150e1f6a24e5
2018-02-23 10:51:34 -06:00
portdirect
515494ca98 RBAC: Include release name in cluster roles to prevent collision
This PS includes the release name in the cluster role to prevent
colision if the chart is deployed multiple times in the same
cluster.

Change-Id: I7166e5ee25b3d4c89879393c5f84c869585a2681
2018-02-19 13:13:56 -06:00
Zuul
f808c525f8 Merge "Add deep merge utility to helm-toolkit" 2018-02-14 01:29:27 +00:00
Sean Eagan
641c79c902 Add deep merge utility to helm-toolkit
Adds "helm-toolkit.utils.merge" which is a replacement for the
upstream sprig "merge" function which didn't quite do what we
wanted, specifically it didn't merge slices, it just overrode
one with the other.  This PS also updates existing callsites
of the sprig merge with "helm-toolkit.utils.merge".

Change-Id: I456349558d4cf941d1bcb07fc76d0688b0a10782
2018-02-13 10:08:50 -06:00
Zuul
4297f786b0 Merge "Add elasticsearch exporter" 2018-02-07 23:30:04 +00:00
Steve Wilkerson
094e0103a9 Add elasticsearch exporter
This adds a chart for an elasticsearch exporter to expose metrics
for prometheus. This also moves the exporter to be included as
part of the chart it's meant to target as opposed to its own
separate chart

Change-Id: I491f4d1efba633827d8a6255218daeb9d427f922
2018-02-07 10:59:27 -06:00
Zuul
d5c6d42ecc Merge "Disable Elasticsearch NFS snapshot repository by default" 2018-02-04 16:54:25 +00:00
Steve Wilkerson
220310bd2f Add node selectors to elasticsearch chart
This adds node selectors to all templates in the elasticsearch
chart, as they were previously missed

Change-Id: I34ea5751663b2e993c5f73a78a1f91133919752c
2018-01-31 15:19:28 -06:00
Steve Wilkerson
012f5ec8d0 Disable Elasticsearch NFS snapshot repository by default
This disables the Elasticsearch snapshot repository backed by
NFS by default as the curator job for snapshots is disabled by
default, and should make no assumption that NFS is deployed by
default

Change-Id: Idc74cfb80fcb4c4741d82c6d0ce63fd90a8c919f
2018-01-29 16:25:27 +00:00
Steve Wilkerson
0bfc5bd12f Remove kibana template in elasticsearch templates
The _kibana.sh.tpl template ended up in the elasticsearch chart
templates folder. This removes it

Change-Id: I636d5b716c266d7ccd266a5f9c051a6eec56e3e1
2018-01-29 13:04:56 +00:00
Steve Wilkerson
3ec7f5f0ff Gate fix: httpd image, elasticsearch, openstack-exporter
There was a change in the upstream reference httpd image for
apache that changed how modules were built for apache.
This change adds the required fix to accomodate the change.
See isssue here https://github.com/docker-library/httpd/pull/87

The Elasticsearch image tag was updated to accomodate the kernel
versions used in the gate as part of the kernel update playbook
See https://github.com/elastic/elasticsearch/issues/28349#issuecomment-360233779

The openstack-exporter binary was changed to reflect changes made
to the openstack-exporter image

Change-Id: I1deb9e7cde794421dd33fade566c2a9fdb5007e6
2018-01-28 15:07:24 -06:00
Steve Wilkerson
d197c4f9a2 Run elasticsearch behind apache
Run elasticsearch behind apache as a reverse proxy to supply basic
auth for elasticsearch, as xpack requires a suscription to support
security for elasticsearch

Change-Id: I72d06ed9cd2179ead86ddc67db33c68a1e40c437
2018-01-16 08:14:47 -06:00
portdirect
97d60dcccd Elasticsearch Snapshot: Fix permisions for PVC
This PS fixes the permisions for the PVC backing the
Elasticsearch Snapshot

Change-Id: I7b9897a7e0f34096ce1f2a04aceab7796d3a89c5
2018-01-10 13:40:11 -05:00
Zuul
f41cec95a3 Merge "Fix elasticsearch repository configuration entry" 2018-01-06 23:30:18 +00:00
Steve Wilkerson
99ee859b66 Fix elasticsearch repository configuration entry
This dynamically adds the elasticsearch path.repo configuration
entry if it's not defined.  This solves issues arising when the
storage settings are disabled in favor of emptydirs for simpler
ES deployments.  If elasticsearch attempts to configure the repo
path with an invalid entry (inaccesible external or shared fs
path), the service will crash.

Change-Id: I089b77104107dfb1f8e6ea2d8a560384718e63f9
2018-01-05 10:54:00 -06:00
portdirect
a8fe16cd42 ElasticSearch: tidy rbac roles and bindings to live with appropriate rc
This PS brings ElasticSearch inline with other charts by placing the
RBAC roles and bindings in the same template as the pod rc they are
assocated with.


Change-Id: I6d541a18d6750d42d31326f77a9aacb06195ddac
2018-01-05 06:52:49 +00:00
Steve Wilkerson
9b32ba17f4 Rename elasticsearch configmaps and provide config via toYaml
This brings the elasticsearch configmaps, volume and mount names
inline with other charts by naming them after the service.

This also moves the configuration for elasticsearch to the values
file to bring it inline with other charts that do the same

Change-Id: I61f7c740d830a9a0567f8b72a0f815a09407b90c
2017-12-28 19:48:58 +00:00
Steve Wilkerson
45ba95a2de Elasticsearch: Add curator snapshot action and PVC for fs repo
This provides an example action in the curator config for taking
snapshots of the elasticsearch indexes. As the snapshot action
requires a repository registered with Elasticsearch, this also
adds a PVC for a filesystem repository backed with NFS and a job
for registering the repository with Elasticsearch.

Change-Id: I26b788c58f52844e997bde5002459bddc1bb685e
2017-12-28 13:42:47 -06:00
Zuul
558ed8cd27 Merge "RBAC: Consolidate serviceaccounts and restrict rbac" 2017-12-20 04:38:15 +00:00
Tin Lam
628fd3007d RBAC: Consolidate serviceaccounts and restrict rbac
Currently, services have two serviceaccounts: one specified in the
chart that cannot read anything, and one injected via helm-toolkit
that can read everything. This patch set refactors the logic to:

- cleanup the roles and their binding automatically when the helm
  chart is deleted;
- remove the need to separately mount a serviceaccount  with secret;
- better handling of namespaces resource restriction.

Co-Authored-By: portdirect <pete@port.direct>

Change-Id: I47d41e0cad9b5b002f59fc9652bad2cc025538dc
2017-12-19 20:22:57 -05:00
Steve Wilkerson
1753d19a68 Fix elasticsearch curator file entry in configmap
Fixes an issue preventing the elastic curator configuration being
populated via elasticsearch's values.yaml

Change-Id: I74901c1aa99abc56a06ea95ca9ea9d818761d79b
2017-12-18 08:13:12 -06:00
portdirect
6e5fe71d9c Elasticsearch: Move default storage access-mode to ReadWriteOnce
This PS moves the default storage access-mode to ReadWriteOnce, as
the PVC is created inline with the statefulset. So ReadWriteMany will
have no effect, as a volume is created per pod.

Change-Id: I2a6a28832c0b1beedeb3e280572b3717628f7b88
2017-12-17 21:34:10 +00:00
Steve Wilkerson
bea44e53bf Add Elasticsearch liveness/readiness probes
The elasticsearch tests fail because the pods don't have
readiness or liveliness probes in the templates. This adds those
definitions

Change-Id: I4fd25aec5ae02d89ae1b933d8b083a3e9cafc55a
2017-12-11 18:47:46 -06:00
Steve Wilkerson
4b94e47c94 Add Elasticsearch to OSH Infra
This moves the Elasticsearch chart to OSH infra, along with rbac
rules for running Elasticsearch. It includes a cronjob for running
ES Curator for cleaning up old indices

Change-Id: I69fcbe8b77de8b594eba5340a6e4340f389ba5bf
2017-12-01 17:42:23 -06:00