2142 Commits

Author SHA1 Message Date
Kabanov, Dmitrii
cd6f3442e9 [Ceph-OSD] Update exit code for flock
The PS updates the exit code for flock. Now we are using default value (1)
if timeout happened.

Change-Id: I2d1cd051c61695a12aa11af1ecb356f91b9e8279
2019-12-30 09:41:35 -08:00
Zuul
e95a8b8b09 Merge "Enable runtime apparmor for ceph-mon,ceph-mds & ceph-mgr." 2019-12-18 20:24:23 +00:00
Steve Wilkerson
803f0e8435 Disable kubeadm-aio jobs while issues addressed
This disables the keystone-auth single node job and all multinode
periodic and experimental jobs while standing issues with the
kubeadm-aio image deployment are sorted out

Change-Id: I3ce0afba155e923b6dd50f83fa6b529908b9a79b
Signed-off-by: Steve Wilkerson <sw5822@att.com>
2019-12-18 13:56:21 -06:00
Zuul
98e0d86f8d Merge "[Elasticsearch] update ceph Nautilus images" 2019-12-18 18:27:14 +00:00
diwakar thyagaraj
841feb7e82 Enable runtime apparmor for ceph-mon,ceph-mds & ceph-mgr.
Also changed ceph apparmor gate job.
Change-Id: I92b9a467b2a77d607dd431f031ec566cc18a86a4
2019-12-18 14:52:12 +00:00
Chinasubbareddy Mallavarapu
0c18f272df [Elasticsearch] update ceph Nautilus images
This is to update ceph images  to Nautilus based images since
ceph cluster is now upgraded to Nautilus.

Change-Id: Ib57f29a4dba89de762a9824ba398ad49b0bd397b
2019-12-18 01:45:10 +00:00
Chinasubbareddy Mallavarapu
027c8497a2 [ceph-client] remove rbd pool init step
This it to remove "rbd"  pool intilization since its not
required as appliction enable in next step does the same.

Change-Id: I07ecdbe417f0156005ebf7cef8fd9e40bace3920
2019-12-18 01:44:55 +00:00
Zuul
13b2384134 Merge "[ceph-client] update pool validation logic" 2019-12-18 01:43:59 +00:00
Chinasubbareddy Mallavarapu
61419dd305 [ceph-client] force to set ceph-mgr module configs
This is to force setting the config values for all moduels since nautilus
version will not let us set them  before mgr starts.

Change-Id: I0e370b525b628fce040b33ab2e403b8b71e948cb
2019-12-17 21:46:19 +00:00
Chinasubbareddy Mallavarapu
02f63af995 [ceph-client] update pool validation logic
Starting in Nautilus, setting pgp_num step is no longer necessary
as long as pgp_num and pg_num currently match, pgp_num will automatically
track any pg_num changes.
More importantly, the adjustment of pgp_num to migrate data and (eventually)
converge to pg_num is done gradually to limit the data migration load on the system.

Change-Id: I491b6eac35b486698c0eef256ca91dac217f8929
2019-12-17 21:46:04 +00:00
Stephen Taylor
016b56e586 Ceph Nautilus compatibility
This change updates the Ceph charts to use Ceph Nautilus images
built on Ubuntu Bionic instead of Xenial. The mirror that hosts
Ceph packages only provides Nautilus packages for Bionic at
present, so this is necessary for Nautilus deployment.

There are also several configuration and scripting changes
included to provide compatibility with Ceph Nautilus. Most of
these simply allow existing logic to execute for Nautilus
deployments, but some logical changes are required to support
Nautilus as well.

NOTE: The cephfs test has been disabled because it was failing
the gate. This test has passed in multiple dev environments, and
since cephfs isn't used by any openstack-helm-infra components we
don't want this to block getting this change merged. The gate
issue will be investigated and addressed in a subsequent patch
set.

Change-Id: Id2d9d7b35d4dc66e93a0aacc9ea514e85ae13467
2019-12-17 18:47:24 +00:00
Steve Wilkerson
edd6ffd712 Reduce osh-infra-logging job scope
This updates the osh-infra-logging single node job to omit the
fluentbit deployment step, as having multiple logging daemonsets
deployed to the single node jobs is causing IO issues. Also, it
was noted that the fluentd-deployment step was missing the
overrides to move the fluentd-deployment release from utilizing a
daemonset to a deployment. This resulted in 3 logging daemons
being deployed to a single host

Change-Id: I4a0c5550e6ea6a331aab0082a975f161e65704bf
Signed-off-by: Steve Wilkerson <sw5822@att.com>
2019-12-17 12:43:12 -06:00
Steven Fitzpatrick
472097d7eb Kafka - Implement SASL Authentication
This change implements SASL authentication in the Kafka chart.
Kafka and Exporter credentials are defined in the endpoints section,
while other credentials for producers and consumers can be defined
in the jaas section.

Additionally, a few server settings are provided to enable SASL auth,
and a jvm_options key is introduced. Any options specified here will
be set when starting Kafka, including the location of the jaas file
in this case.

Change-Id: I43469c5bb5734b62cf69be924fe9cf7078e82a9c
2019-12-16 16:55:55 -06:00
Zuul
3494fe8489 Merge "Add support to set vhost-iommu-support as global option" 2019-12-16 22:39:41 +00:00
Zuul
6bb7039368 Merge "Fix apparmor annotation of libvirt." 2019-12-16 19:22:22 +00:00
Zuul
69b9c7dd0b Merge "Fix incompatible curator version" 2019-12-16 19:14:19 +00:00
Zuul
79da084a07 Merge "[ceph-client][ceph-osd] Adjust required_percent_of_osds parameter" 2019-12-16 17:39:30 +00:00
Kaspars Skels
4fb0b49169 Fix incompatible curator version
Elasticsearch version 7.1.0 incompatible with this version of Curator (5.6.0)

Change-Id: If9323a6d742ddf2915ca9ec167eb8585c694cf1e
2019-12-13 15:20:57 -06:00
Itxaka
cace08c016
Fix openstack release in gates
Change I7def8df68371deda0b75a685363c8a73b818dd45 removed one
line by mistake passing the openstack release var down to the zuul
jobs, so all jobs are currently running under ocata.

This patch restores the missing line, thus fixing the opensuse jobs
and making sure the other jobs run under the correct release.

Change-Id: Ia7a488928e521de1afb821f141d77d2b0268ff0a
2019-12-12 10:51:19 +01:00
Roy Tang
57c2c0620c Fix apparmor annotation of libvirt.
Change-Id: I9b37bea8fe3476e4586c261156f15ca63f4658be
2019-12-11 10:44:53 -06:00
Steve Wilkerson
03580ec90a Elasticsearch: Make node selectors more granular
This updates the Elasticsearch chart to make the values keys used
for defining node selectors for the various elasticsearch
components more granular

Change-Id: Ic1ac343b1d6ee48fc7cb456afe4cd9588c4aa13b
Signed-off-by: Steve Wilkerson <sw5822@att.com>
2019-12-10 15:55:38 -06:00
Zuul
3af64d1111 Merge "Fluentd: Add support for arbitrary secret env variables" 2019-12-09 22:37:11 +00:00
Roy Tang
9a8516867a Add support to set vhost-iommu-support as global option
Also update other optional parm

Change-Id: I9a36acd6a331255d01722ed5961b08e1fbca80d1
2019-12-09 22:19:21 +00:00
Zuul
cecd63ef62 Merge "Grafana: Remove default dashboards from chart" 2019-12-09 21:34:53 +00:00
Zuul
160370eec9 Merge "Fix feature gate envvar overriding" 2019-12-09 21:25:08 +00:00
Steve Wilkerson
005ece16d4 Fluentd: Add support for arbitrary secret env variables
This adds a helm-toolkit util for consuming arbitrary secret env
variables via pod env variables. It also updates the Fluentd chart
to add a release secret that is used to house the secret env
variables defined in the chart's values.yaml. This can be used as
an example to expand to other charts where this functionality is
desired

Change-Id: I9ef606840af92e54b2204e637c58442085e2c748
Signed-off-by: Steve Wilkerson <sw5822@att.com>
2019-12-09 14:15:40 -06:00
Steve Wilkerson
a9652653fb Update Armada manifests for osh-infra
This updates the Armada manifests to remove the explicit enabling
of helm tests (as the default armada behavior is now to test by
default) and updates the ceph-osd chart document to set the native
helm wait behavior to false (required for the update-uuid job to
complete successfully)

Change-Id: Ia84f20ce0f38be5f07dedce70b3fbe424a037ba2
Signed-off-by: Steve Wilkerson <sw5822@att.com>
2019-12-09 13:50:12 +00:00
Steve Wilkerson
3a6df3b544 Grafana: Remove default dashboards from chart
This removes the default dashboards from the Grafana chart and
instead places them in the values_overrides directory, similar to
what was done for the Prometheus rules. As Grafana dashboards
will likely be heavily dependent upon end-user needs, the old
default dashboard configs should only be used as a reference
instead of opinionated defaults that are difficult to override.
The previous defaults made using specialized labels for dashboard
variables difficult, as they were making dangerous assumptions
about deployed namespaces and host fqdns. By removing the defaults
entirely, end users can define their own dashboards to meet their
specialized needs

Change-Id: I7def8df68371deda0b75a685363c8a73b818dd45
Signed-off-by: Steve Wilkerson <sw5822@att.com>
2019-12-09 13:39:13 +00:00
Zuul
385803eef2 Merge "Fix OVS-DPDK readiness probe for OVS < v2.10.0" 2019-12-08 20:54:19 +00:00
Zuul
9b77752723 Merge "Initialize PCI module if used before DPDK for hotplug to work" 2019-12-08 10:28:52 +00:00
Tin Lam
ac18e6acf9 Fix feature gate envvar overriding
Currently using envsubst to perform substitution of value overrides in
the feature gate caused conflicts as gotpl gets templated into those
overrides. This adds in '%%%REPLACE_${var}%%%' and uses sed to perform
the substitution instead to address the issue.

Change-Id: I9d3d630b53a2f3d828866229a5072bb04440ae15
Signed-off-by: Tin Lam <tin@irrational.io>
2019-12-07 12:22:16 -06:00
Pete Birley
eb5ae14dcd HTK: Omit port from uri for http/https when 80/443 used
This PS updates htk to omit the port used in the url when this
corresponds to the standard ports for the http and https protocols.

Change-Id: I46e2237dde99460fd096bd6fe58fe154b220041f
Signed-off-by: Pete Birley <pete@port.direct>
2019-12-06 17:48:38 -06:00
Zuul
d216fbf731 Merge "Elasticsearch: Remove unnecessary rbac definitions" 2019-12-06 18:16:06 +00:00
Zuul
bb7c2787c3 Merge "Elasticsearch/Kibana: Update version to 7.1.0" 2019-12-06 18:16:05 +00:00
bw6938
53a1343400 [ceph-client][ceph-osd] Adjust required_percent_of_osds parameter
and incorporate in ceph-osd testing.

This parameter is used by helm test to check if a set % of OSDs
out of the total are considered in & up. Adjusting to 75% and
adding to helm-test.sh for ceph-osd along with more robust
ceph osd validation function

Change-Id: Ib1f37b901f8656f0c6f2ed6a3ec27f0357e82278
2019-12-06 16:16:00 +00:00
Hemachandra Reddy
066a7e0ab4 Initialize PCI module if used before DPDK for hotplug to work
Change-Id: I9f552035eb70fd2a828ff7fa4852ba66055b885f
2019-12-06 15:56:14 +00:00
Tin Lam
daefed7218 Add feature gate capability to OSH-Infra
This patch set adds the feature gate capability to OpenStack-Helm-Infra
repository without depending on the main OpenStack-Helm repository.

Change-Id: I70b8fac4fd2365f8eedcf50519f125eb34534f2f
Signed-off-by: Tin Lam <tlam@omegaprime.dev>
Signed-off-by: Tin Lam <tin@irrational.io>
2019-12-03 16:55:00 -06:00
Zuul
8bd11d1ad2 Merge "[ceph-client] Validate failure domain support for replica count per pool" 2019-12-03 22:23:08 +00:00
Zuul
f9479c31c9 Merge "Create Chart to Deploy Apache Kafka" 2019-12-03 22:02:39 +00:00
Zuul
9632d8719f Merge "Nagios: Add support for arbitrary object definitions via overrides" 2019-12-03 21:09:55 +00:00
Steven Fitzpatrick
e8f3d84ccc Create Chart to Deploy Apache Kafka
This proposes adding a kafka chart to osh-infra that aligns
with the design patterns laid out by the other charts in osh-infra
and osh.

danielqsj's kafka-exporter image is leveraged to deploy a prometheus
exporter for kafka alongside the main application if enabled in
values.yaml

Change-Id: I5997b0994fc3aef9bd1b222c373cc3a013112566
Co-Authored-By: Meghan Heisler <mh783g@att.com>
2019-12-03 11:37:54 -06:00
Georg Kunz
cd27f2714c Fix OVS-DPDK readiness probe for OVS < v2.10.0
A recently introduced readiness probe for OVS with DPDK makes use of an
OVSDB table entry 'dpdk_initialized' which does not exist in OVS
versions preceeding v2.10.0. This patch changes the readiness probe to
exit successfully if this table entry does not exit.

Change-Id: I1776ac4bf736220267a49042f1b7092f3cf5ed16
2019-12-03 16:48:13 +00:00
Steve Wilkerson
fd7067649a Elasticsearch: Remove unnecessary rbac definitions
This removes the cluster role definition from the Elasticsearch
component templates, as these are not needed for the service to
function correctly.

Change-Id: I671272affbed8984a47121187024e4b831937123
Signed-off-by: Steve Wilkerson <sw5822@att.com>
2019-12-03 09:06:13 -06:00
Steve Wilkerson
6c4404ee4d Nagios: Disable Nagios page tours by default
This disables the Nagios page tours option. This option is enabled
by default, which results in a youtube video being overlaid on
each Nagios page.

Change-Id: Ifd80a8d122dcbe145315b37753a72e1309e1d210
Signed-off-by: Steve Wilkerson <sw5822@att.com>
2019-12-03 14:48:41 +00:00
Steve Wilkerson
2d3c9575ff Elasticsearch/Kibana: Update version to 7.1.0
This updates the Elasticsearch and Kibana charts to deploy
version 7.1.0. This move required significant changes to both
charts, including: changing elasticsearch masters to a statefulset
to utilize reliable dns names for the discovery process, config
updates to reflect deprecated/updated/removed values, use the
kibana saved objects api for managing index patterns and setting
the default index, and updating the elasticsearch entrypoint
scripts to reflect the use of elastic-keystore for storing s3
credentials instead of defining them in the configuration file

Change-Id: I270d905f266fc15492e47d8376714ba80603e66d
Signed-off-by: Steve Wilkerson <sw5822@att.com>
2019-12-03 07:43:29 -06:00
Dustin Specker
ae8a6c5d50 refactor(deploy-k8s): remove explicit wait on etcd pod
Using `--network-plugin=cni` for `minikube start` will have minikube
wait for Kubernetes components to spin up and not require the Node to be
in ready status.

Change-Id: I08bf40ac4790955c107e8fee6a004b930c333d16
2019-12-02 19:21:19 +00:00
bw6938
699ea1acba [ceph-client] Validate failure domain support for replica count per pool
Ensure each pool is configured with enough failure domains to
satisfy the pool's replica size requirements. If any pool does
not have enough failure domains to satisfy the pool's replica size,
then fail the ceph deployment.

Change-Id: I9dd1cafd05e81f145d1eb8c916591203946bc8f1
2019-12-02 15:22:54 +00:00
Steve Wilkerson
6f7790e451 Nagios: Add support for arbitrary object definitions via overrides
This adds support for arbitrary object definitions via the conf
key in the Nagios chart. This allows for customizing the
definitions required by different deployment targets instead of
assuming all nagios deployments are monitoring and targeting the
same hosts and executing the same service checks and commands.

This also adds reference overrides to the chart for elasticsearch,
postgresql, and openstack nagios objects that are deployed in the
single and multinode jobs here

Change-Id: I6475ca980447591b5b691220eb841a2ab958e854
Signed-off-by: Steve Wilkerson <sw5822@att.com>
2019-12-02 13:46:20 +00:00
Oleh Hryhorov
9492a8cde0 Fixing typo in exporter-deployment.yaml PUBLISH_PORT
The patch fixes typo in PUBLISH_PORT and adds quotes for
PUBLISH_PORT because of the fact that it is string values
otherwise it leads to the error below:

error updating the release: rpc error: code = Unknown desc = release
rabbitmq failed: Deployment in version "v1" cannot be handled as
a Deployment: v1.Deployment.Spec: v1.DeploymentSpec.Template: v1.PodTemplateSpec.Spec:
v1.PodSpec.Containers: []v1.Container: v1.Container.Env: []v1.EnvVar: v1.EnvVar.Value:
ReadString: expects " or n, but found 9, error found in #10 byte of ...|,"value":9095},{"nam|...,
bigger context ...|value":"no_sort"},{"name":"PUBLISH_PORT","value":9095},{"name":"LOG_LEVEL","value":"info"},{"name":"|...

Change-Id: I027c91ee48df8eb5b4b2bf3fd28036b8eca47238
2019-11-28 17:26:27 +02:00
Drew Walters
992e82fc1d tools: Sort resolv.conf minikube K8s script
The way that the minikube K8s script orders a host's resolv.conf file
leaves service endpoints inaccessible from the host itself even though
they are accessible within the cluster, leaving the OpenStack client
unusable from the minikube node. This change resolves the service access
issues by reordering the DNS entries in the host's resolv.conf file.

Change-Id: I58bf6d541e59f3049a0e350291e07241f6a6b544
Signed-off-by: Drew Walters <andrew.walters@att.com>
2019-11-25 21:13:29 +00:00