842 Commits

Author SHA1 Message Date
Pete Birley
f6e84fe15f MariaDB: Galera cluster refactor
This PS updates the MariaDB chart to better support clustering,
using a configmap to track cluster state.

Change-Id: Ifd9c3d63353a9b587384b6f13c0863ecc4fbd956
Signed-off-by: Pete Birley <pete@port.direct>
2018-10-25 06:21:01 +00:00
Pete Birley
8bc03bf88c MariaDB: Move to use mariabackup instead of xtrabackup-v2
This PS moves to use mariabackup instead of xtrabackup-v2, for info
see:
 * https://mariadb.com/kb/en/library/upgrading-from-mariadb-102-to-mariadb-103/#mariadb-backup-and-percona-xtrabackup
 * https://mariadb.com/kb/en/library/mariabackup-overview/#about-mariabackup

Additionally the readyness script is updated to match the order of
validation tests described in the mariadb/galera documentation.

Change-Id: I031c63d6305f1514ffdd53d77d621bc7edc0e68c
Signed-off-by: Pete Birley <pete@port.direct>
2018-10-25 05:43:59 +00:00
Tin Lam
653b84a2e1 Fix k8s-auth job
This patch set addresses a cross-repo conflict with the enablement of
network policy in gate script override.

Change-Id: I284d6b04940424a87e5b239ccc9d30ae01075f38
Signed-off-by: Tin Lam <tin@irrational.io>
2018-10-24 20:49:17 -05:00
Pete Birley
1144ccbbb2 Ceph: Update MGR check to allow use on hosts with fqdns defined
This PS updates the mgr check to allow use on hosts with fqdns
defined.

Change-Id: If1cb740e8093fbcafce846234c96db931409b436
Signed-off-by: Pete Birley <pete@port.direct>
2018-10-24 00:57:12 +00:00
Zuul
eabf53253f Merge "ceph-mgr: make prometheus module port configurable" 2018-10-23 23:19:09 +00:00
Chinasubbareddy M
e23e372120 ceph-mgr: make prometheus module port configurable
this is to give example for prometheus module port configurable

Change-Id: I66844bb8ee59a58f7bfd3e3002a183779810e881
2018-10-23 15:40:43 -05:00
Zuul
5c446bb2d3 Merge "Use supplied HELM variable for dep up in Makefile" 2018-10-23 20:36:22 +00:00
Zuul
f49461acc4 Merge "cronjob-checkPGs failure fix" 2018-10-23 20:21:46 +00:00
Zuul
860a897aee Merge "[gate] allow pip caching" 2018-10-23 18:30:20 +00:00
Zuul
4c4e947e17 Merge "Ceph: A script to check object replication across the hosts" 2018-10-23 18:25:43 +00:00
Zuul
1e3693f1a3 Merge "[gate] Put nfs-provisioner in it's own namespace (docker-registry)" 2018-10-23 18:22:01 +00:00
Zuul
bad8427b21 Merge "[gate] Put nfs-provisioner in it's own namespace" 2018-10-23 18:22:00 +00:00
Zuul
11ec46bdce Merge "Prometheus kubelet.rules change" 2018-10-23 17:57:26 +00:00
Bryan Strassner
dacb01c82a Use supplied HELM variable for dep up in Makefile
Updates the helm dep up command to use the $(HELM) variable instead of
the locally istalled helm for the host machine. This bring this line of
code in alignment with the other uses of helm in the same Makefile.

Change-Id: I91bfdceedd3bac0ac49daf5b9410c05e0e840168
2018-10-23 11:26:16 -05:00
Zuul
a0d58decff Merge "[Calico] Allow resource configuration using chart (overrides)" 2018-10-22 22:49:08 +00:00
Zuul
19e7e0fb61 Merge "Use the correct socket file for the Ceph mon check." 2018-10-22 20:03:47 +00:00
Chris Wedgwood
02f400e442 [Calico] Allow resource configuration using chart (overrides)
Allow Calico resources such as NetworkPolicy, GlobalNetworkPolicy,
WorkloadEndpoint, etc to be specified using values.

To avoid the complexities of list management with helm we use a
dictionary that contains a relative priority and set of objects
(called rules).

For example:

network:
  policy:

    someName:
      priority: 0
      rules:
       - apiVersion: projectcalico.org/v3
... some useful resource object ...
       - apiVersion: projectcalico.org/v3
... some other useful resource object ...

    someOtherName:
      priority: 1
      rules:
       - apiVersion: projectcalico.org/v3
... rules that come later ...

    lastSetOfRules:
      priority: 9
      rules:
       - apiVersion: projectcalico.org/v3
... rules that come last ... maybe hostendpoints ...

By having named groups of rules each with it's own priority you can
update, delete and amend individual sets of rules without provided you
set the appropriate "priority" value.

Change-Id: Id441350bcc8b95a91ef4d1b89d1bc3c417f50b13
2018-10-22 18:49:18 +00:00
Jean-Philippe Evrard
e7f21a6bd0 Remove dependency to OSH repo
This removes yet another time the dependency towards OSH repo.
With each repository independant, we can later introduce abstract
jobs that will be re-usable but with a clean dependency map: only
bring jobs from one single location, openstack-helm-infra.

Change-Id: I72844a944cfea5380de25dbd7cf7231c8d39f4ec
2018-10-22 10:50:02 +02:00
Matthew Heler
154fcd894f Use the correct socket file for the Ceph mon check.
Change-Id: If8c40c3c0501b78db88d3a7f33bf3838c0e60199
Closes-Bug: 1796313
2018-10-22 04:56:13 +00:00
Chris Wedgwood
d4ac063163 [gate] allow pip caching
The pip cache is useful for repeat operations and doesn't seem to have
any real downsides.

Change-Id: Iadb21a118f8d725911a9baa6a9264b8644012af9
2018-10-22 00:11:25 +00:00
Chris Wedgwood
c08c78f1d1 [gate] Put nfs-provisioner in it's own namespace (docker-registry)
Use the 'docker-nfs' namespace to back the docker registry.  This
means we can delete the registry namespace without causing IO lockups.

Change-Id: I1706dd96653598dcfbb81904fde8c0bf92294b06
2018-10-21 23:42:20 +00:00
Chris Wedgwood
8f5aaa3fd0 [gate] Put nfs-provisioner in it's own namespace
Having storage (backend) components in their own namespace means we
can delete the namespaces containing the openstack without causing
system hangs which occur when storage is remove whilst in use.

Change-Id: Ie489709b08929f25cf0e626a8541620a06506b8b
2018-10-21 23:37:56 +00:00
Chinasubbareddy M
26991ad182 Ceph: A script to check object replication across the hosts
this script will  create  an object and see if the object is
getting replicated across diffrent hosts  or not.

Change-Id: Ic5056c1a07dc5d5b6a5d6fc24e3d9a75fa46458f
2018-10-21 15:38:26 +00:00
Huang, Scott (sh2725)
b99d39dd95 [467551] Mount Nagios Logfile
Mount Nagios logfile to host to enable log streaming to elasticsearch

Change-Id: I297f61067c0ff3e870e14b124a5c6fdd49e12b01
2018-10-21 15:37:40 +00:00
Zuul
23730808d4 Merge "Gates: Update fluent-logging/elasticsearch configurations" 2018-10-21 09:17:19 +00:00
Zuul
91f38f22b7 Merge "Organize aio gates by function" 2018-10-21 09:16:23 +00:00
Matthew Heler
ec8bb513eb Add RBD-NBD support to openstack-helm
By default use rbd-nbd (librbd) instead of krbd.

Applying this change on existing nodes will
require reboots.

Change-Id: I81829fb8666541e856ab402128a5192984b6fe05
2018-10-19 16:20:10 -05:00
Steve Wilkerson
4c29bafcbc Gates: Update fluent-logging/elasticsearch configurations
This updates the fluentd buffer output configurations to account
for the restraints of the jobs deploying fluentd. This also
renames the fluentd configuration key from td_agent to fluentd to
reflect the fact we're no longer deploying td-agent

This also updates the Elasticsearch default replicas and overrides
the replica counts in each Elasticsearch deployment to account for
resource constraints

Change-Id: I55dee410eced99c3e1645f7452e4306ad646e601
2018-10-19 17:30:08 +00:00
Steve Wilkerson
538d51e991 Organize aio gates by function
This organizes the single node gates for osh-infra by function.
This organization aims to improve the single node gates in the
following ways:

1. Reduce number of services deployed in single node jobs
2. Only deploy Ceph for logging job, as Elasticsearch requires
   RGW for snapshot repositories.
3. Use NFS for storage for monitoring job, as Ceph is not a
   requirement for any of the services here.
4. Remove duplicate services deployed to multiple single node jobs
5. Remove storage from openstack-support job, as the only service
   requiring storage is rabbitmq. Rabbitmq is deployed with
   storage enabled in the openstack-helm checks/gates.

This also removes the documentation for the single node deployments,
as those deployments do not make sense with this change. This should
be revisited as a follow-on once we have a clear path forward for
the larger gate refactoring work

Change-Id: I46951f76904fa2ab245a202d55f76019b7503362
2018-10-19 12:28:18 -05:00
Chris Wedgwood
d9457c8860 Remove dependency to OSH repository of new jobs
Without this patch, there is a dependency between the two
repositories OSH and OSH-infra, which was recently introduced, and
which will cause a circular dependency problem when trying to remove
the duplicated jobs that will appear in OSH.

Change-Id: Ief4461a66f7139ae0650e4a240a3e65800821f78
Required-By: https://review.openstack.org/610481/
Co-Authored-By: Jean-Philippe Evrard <jean-philippe@evrard.me>
2018-10-18 21:06:21 +00:00
Zuul
27ea2a53a6 Merge "Fix grep logic around weighting OSDs during ceph-client chart." 2018-10-18 09:04:29 +00:00
Zuul
cd4b8e9b87 Merge "Ceph: Remove fluentbit sidecars, mount hostpath for logs" 2018-10-17 21:41:38 +00:00
Matthew Heler
0de1d23895 Fix grep logic around weighting OSDs during ceph-client chart.
Change-Id: I7831ac07a53b9aaf3000e9f64bf8c17344723a8f
2018-10-17 15:58:24 -05:00
Steve Wilkerson
92717bdc72 Ceph: Remove fluentbit sidecars, mount hostpath for logs
This removes the fluentbit sidecars from the ceph-mon and ceph-osd
charts. Instead, we mount /var/log/ceph as a hostpath, and use the
fluentbit daemonset to target the mounted log files instead

This also updates the fluentd configuration to better handle the
correct configuration type for flush_interval (time vs int), as
well as updates the fluentd elasticsearch output values to help
address the gate failures resulting from the Elasticsearch bulk
endpoints failing

Change-Id: If3f2ff6371f267ed72379de25ff463079ba4cddc
2018-10-17 11:05:03 -05:00
Chinasubbareddy M
793b3631b5 Ceph-mgr: make liveness to check through admin scoket
This is to update the mgr liveness script to use admin socket
instead of resolving ceph  mon fqdn

Change-Id: Id95f78afef44103a834312d0667d49947ee803a4
Co-Authored-By: Jean-Charles Lopez <jl970p@att.com>
2018-10-17 14:40:42 +00:00
Zuul
b3b4e6858b Merge "Add LDAP support for k8s-keystone-auth in gate" 2018-10-17 08:39:14 +00:00
Zuul
1b7240c64c Merge "Secure pool during deployment" 2018-10-17 07:37:38 +00:00
Samuel Pilla
6fe001361a Add LDAP support for k8s-keystone-auth in gate
This patch set changes the keystone in the k8s-keystone-auth to
be backed by LDAP. It also updates the test to use the LDAP users
instead of created users in the database.

Co-Authored-By: Samuel Pilla <sp516w@att.com>
Change-Id: Ia34dac51b36a300068ad5fd936c48b0f30821a52
Signed-off-by: Tin Lam <tin@irrational.io>
2018-10-17 06:19:20 +00:00
Jean-Charles Lopez
55f1d2db57 Secure pool during deployment
Change-Id: Ifbeb956ab2c015deaed501ee4bff22dfc1e0404f
2018-10-17 04:53:53 +00:00
Pete Birley
be7b01d798 Helm-Toolkit: Document and fix the anti-affinity function
This PS document use of and fixes the anti-affinity function to
properly support hard anti affinity.

Change-Id: I2ec643d7720036b34fc249a2e230b3bed3aac41f
Signed-off-by: Pete Birley <pete@port.direct>
2018-10-17 04:50:02 +00:00
Zuul
7d3bda1307 Merge "Ceph-RGW: Use hostname not podname for pod specific config" 2018-10-17 04:24:49 +00:00
Zuul
51bab02b24 Merge "Rename mandatory access control annotation func" 2018-10-17 04:24:40 +00:00
Zuul
21f46d294b Merge "[Open vSwitch] Remove auto_bridge_add support" 2018-10-17 04:23:52 +00:00
Zuul
23fba51fbb Merge "[MariaDB] Bump to version 10.2.18 to avoid shutdown hangs" 2018-10-17 04:23:51 +00:00
Zuul
570355b1d9 Merge "Initialize OSDs with a crush weight of 0 to prevent automatic rebalancing." 2018-10-17 02:45:45 +00:00
Pete Birley
a01e2db6ab Ceph-RGW: Use hostname not podname for pod specific config
This PS moves to use the hostname, not the pod name for the
instances specific config sections.

Change-Id: If2bc60c9f4f12038e8aa70fbd33a009cdf652b75
Signed-off-by: Pete Birley <pete@port.direct>
2018-10-17 01:38:34 +00:00
Cliff Parsons
c5b10d155f Rename mandatory access control annotation func
This patch set renames the existing apparmor annotation
function to a more generic MAC (Mandatory Access Control)
name to be flexible enough to handle other MAC annotations
in the future.

Change-Id: I98a34484cebc2b420ad8f2664e4aaa84cfb9dca1
2018-10-17 01:35:49 +00:00
Matthew Heler
5efac315f7 Initialize OSDs with a crush weight of 0 to prevent automatic rebalancing.
Weight the OSDs based on reported disk size when ceph-client chart runs.

Change-Id: I9f4080a9843f1a63564cf71154841b351382bfe2
2018-10-16 21:33:49 +00:00
Steve Wilkerson
f3d8bda9d6 Grafana: Support multiple Ceph clusters with dashboards
This updates the Grafana Ceph dashboards to use templating to
determine which ceph-mgr to use for displaying ceph related
metrics.  This required setting the appropriate labels on the
ceph-mgr service to be able to distinguish between releases

Change-Id: Id2eceacadc5b6366d7bc6668bc16ccf5ba878e4a
2018-10-16 21:32:13 +00:00
Chris Wedgwood
8dad346f3f [MariaDB] Bump to version 10.2.18 to avoid shutdown hangs
We see sporadic shutdown hangs that look to be the issue described at
https://jira.mariadb.org/browse/MDEV-15554

Upgrade minor version to address this.

Change-Id: Idf8403b44e871b5a32173bd153a8367519b239ec
2018-10-16 21:30:22 +00:00