828 Commits

Author SHA1 Message Date
Zuul
1e3693f1a3 Merge "[gate] Put nfs-provisioner in it's own namespace (docker-registry)" 2018-10-23 18:22:01 +00:00
Zuul
bad8427b21 Merge "[gate] Put nfs-provisioner in it's own namespace" 2018-10-23 18:22:00 +00:00
Zuul
11ec46bdce Merge "Prometheus kubelet.rules change" 2018-10-23 17:57:26 +00:00
Zuul
a0d58decff Merge "[Calico] Allow resource configuration using chart (overrides)" 2018-10-22 22:49:08 +00:00
Zuul
19e7e0fb61 Merge "Use the correct socket file for the Ceph mon check." 2018-10-22 20:03:47 +00:00
Chris Wedgwood
02f400e442 [Calico] Allow resource configuration using chart (overrides)
Allow Calico resources such as NetworkPolicy, GlobalNetworkPolicy,
WorkloadEndpoint, etc to be specified using values.

To avoid the complexities of list management with helm we use a
dictionary that contains a relative priority and set of objects
(called rules).

For example:

network:
  policy:

    someName:
      priority: 0
      rules:
       - apiVersion: projectcalico.org/v3
... some useful resource object ...
       - apiVersion: projectcalico.org/v3
... some other useful resource object ...

    someOtherName:
      priority: 1
      rules:
       - apiVersion: projectcalico.org/v3
... rules that come later ...

    lastSetOfRules:
      priority: 9
      rules:
       - apiVersion: projectcalico.org/v3
... rules that come last ... maybe hostendpoints ...

By having named groups of rules each with it's own priority you can
update, delete and amend individual sets of rules without provided you
set the appropriate "priority" value.

Change-Id: Id441350bcc8b95a91ef4d1b89d1bc3c417f50b13
2018-10-22 18:49:18 +00:00
Jean-Philippe Evrard
e7f21a6bd0 Remove dependency to OSH repo
This removes yet another time the dependency towards OSH repo.
With each repository independant, we can later introduce abstract
jobs that will be re-usable but with a clean dependency map: only
bring jobs from one single location, openstack-helm-infra.

Change-Id: I72844a944cfea5380de25dbd7cf7231c8d39f4ec
2018-10-22 10:50:02 +02:00
Matthew Heler
154fcd894f Use the correct socket file for the Ceph mon check.
Change-Id: If8c40c3c0501b78db88d3a7f33bf3838c0e60199
Closes-Bug: 1796313
2018-10-22 04:56:13 +00:00
Chris Wedgwood
c08c78f1d1 [gate] Put nfs-provisioner in it's own namespace (docker-registry)
Use the 'docker-nfs' namespace to back the docker registry.  This
means we can delete the registry namespace without causing IO lockups.

Change-Id: I1706dd96653598dcfbb81904fde8c0bf92294b06
2018-10-21 23:42:20 +00:00
Chris Wedgwood
8f5aaa3fd0 [gate] Put nfs-provisioner in it's own namespace
Having storage (backend) components in their own namespace means we
can delete the namespaces containing the openstack without causing
system hangs which occur when storage is remove whilst in use.

Change-Id: Ie489709b08929f25cf0e626a8541620a06506b8b
2018-10-21 23:37:56 +00:00
Huang, Scott (sh2725)
b99d39dd95 [467551] Mount Nagios Logfile
Mount Nagios logfile to host to enable log streaming to elasticsearch

Change-Id: I297f61067c0ff3e870e14b124a5c6fdd49e12b01
2018-10-21 15:37:40 +00:00
Zuul
23730808d4 Merge "Gates: Update fluent-logging/elasticsearch configurations" 2018-10-21 09:17:19 +00:00
Zuul
91f38f22b7 Merge "Organize aio gates by function" 2018-10-21 09:16:23 +00:00
Matthew Heler
ec8bb513eb Add RBD-NBD support to openstack-helm
By default use rbd-nbd (librbd) instead of krbd.

Applying this change on existing nodes will
require reboots.

Change-Id: I81829fb8666541e856ab402128a5192984b6fe05
2018-10-19 16:20:10 -05:00
Steve Wilkerson
4c29bafcbc Gates: Update fluent-logging/elasticsearch configurations
This updates the fluentd buffer output configurations to account
for the restraints of the jobs deploying fluentd. This also
renames the fluentd configuration key from td_agent to fluentd to
reflect the fact we're no longer deploying td-agent

This also updates the Elasticsearch default replicas and overrides
the replica counts in each Elasticsearch deployment to account for
resource constraints

Change-Id: I55dee410eced99c3e1645f7452e4306ad646e601
2018-10-19 17:30:08 +00:00
Steve Wilkerson
538d51e991 Organize aio gates by function
This organizes the single node gates for osh-infra by function.
This organization aims to improve the single node gates in the
following ways:

1. Reduce number of services deployed in single node jobs
2. Only deploy Ceph for logging job, as Elasticsearch requires
   RGW for snapshot repositories.
3. Use NFS for storage for monitoring job, as Ceph is not a
   requirement for any of the services here.
4. Remove duplicate services deployed to multiple single node jobs
5. Remove storage from openstack-support job, as the only service
   requiring storage is rabbitmq. Rabbitmq is deployed with
   storage enabled in the openstack-helm checks/gates.

This also removes the documentation for the single node deployments,
as those deployments do not make sense with this change. This should
be revisited as a follow-on once we have a clear path forward for
the larger gate refactoring work

Change-Id: I46951f76904fa2ab245a202d55f76019b7503362
2018-10-19 12:28:18 -05:00
Chris Wedgwood
d9457c8860 Remove dependency to OSH repository of new jobs
Without this patch, there is a dependency between the two
repositories OSH and OSH-infra, which was recently introduced, and
which will cause a circular dependency problem when trying to remove
the duplicated jobs that will appear in OSH.

Change-Id: Ief4461a66f7139ae0650e4a240a3e65800821f78
Required-By: https://review.openstack.org/610481/
Co-Authored-By: Jean-Philippe Evrard <jean-philippe@evrard.me>
2018-10-18 21:06:21 +00:00
Zuul
27ea2a53a6 Merge "Fix grep logic around weighting OSDs during ceph-client chart." 2018-10-18 09:04:29 +00:00
Zuul
cd4b8e9b87 Merge "Ceph: Remove fluentbit sidecars, mount hostpath for logs" 2018-10-17 21:41:38 +00:00
Matthew Heler
0de1d23895 Fix grep logic around weighting OSDs during ceph-client chart.
Change-Id: I7831ac07a53b9aaf3000e9f64bf8c17344723a8f
2018-10-17 15:58:24 -05:00
Steve Wilkerson
92717bdc72 Ceph: Remove fluentbit sidecars, mount hostpath for logs
This removes the fluentbit sidecars from the ceph-mon and ceph-osd
charts. Instead, we mount /var/log/ceph as a hostpath, and use the
fluentbit daemonset to target the mounted log files instead

This also updates the fluentd configuration to better handle the
correct configuration type for flush_interval (time vs int), as
well as updates the fluentd elasticsearch output values to help
address the gate failures resulting from the Elasticsearch bulk
endpoints failing

Change-Id: If3f2ff6371f267ed72379de25ff463079ba4cddc
2018-10-17 11:05:03 -05:00
Chinasubbareddy M
793b3631b5 Ceph-mgr: make liveness to check through admin scoket
This is to update the mgr liveness script to use admin socket
instead of resolving ceph  mon fqdn

Change-Id: Id95f78afef44103a834312d0667d49947ee803a4
Co-Authored-By: Jean-Charles Lopez <jl970p@att.com>
2018-10-17 14:40:42 +00:00
Zuul
b3b4e6858b Merge "Add LDAP support for k8s-keystone-auth in gate" 2018-10-17 08:39:14 +00:00
Zuul
1b7240c64c Merge "Secure pool during deployment" 2018-10-17 07:37:38 +00:00
Samuel Pilla
6fe001361a Add LDAP support for k8s-keystone-auth in gate
This patch set changes the keystone in the k8s-keystone-auth to
be backed by LDAP. It also updates the test to use the LDAP users
instead of created users in the database.

Co-Authored-By: Samuel Pilla <sp516w@att.com>
Change-Id: Ia34dac51b36a300068ad5fd936c48b0f30821a52
Signed-off-by: Tin Lam <tin@irrational.io>
2018-10-17 06:19:20 +00:00
Jean-Charles Lopez
55f1d2db57 Secure pool during deployment
Change-Id: Ifbeb956ab2c015deaed501ee4bff22dfc1e0404f
2018-10-17 04:53:53 +00:00
Pete Birley
be7b01d798 Helm-Toolkit: Document and fix the anti-affinity function
This PS document use of and fixes the anti-affinity function to
properly support hard anti affinity.

Change-Id: I2ec643d7720036b34fc249a2e230b3bed3aac41f
Signed-off-by: Pete Birley <pete@port.direct>
2018-10-17 04:50:02 +00:00
Zuul
7d3bda1307 Merge "Ceph-RGW: Use hostname not podname for pod specific config" 2018-10-17 04:24:49 +00:00
Zuul
51bab02b24 Merge "Rename mandatory access control annotation func" 2018-10-17 04:24:40 +00:00
Zuul
21f46d294b Merge "[Open vSwitch] Remove auto_bridge_add support" 2018-10-17 04:23:52 +00:00
Zuul
23fba51fbb Merge "[MariaDB] Bump to version 10.2.18 to avoid shutdown hangs" 2018-10-17 04:23:51 +00:00
Zuul
570355b1d9 Merge "Initialize OSDs with a crush weight of 0 to prevent automatic rebalancing." 2018-10-17 02:45:45 +00:00
Pete Birley
a01e2db6ab Ceph-RGW: Use hostname not podname for pod specific config
This PS moves to use the hostname, not the pod name for the
instances specific config sections.

Change-Id: If2bc60c9f4f12038e8aa70fbd33a009cdf652b75
Signed-off-by: Pete Birley <pete@port.direct>
2018-10-17 01:38:34 +00:00
Cliff Parsons
c5b10d155f Rename mandatory access control annotation func
This patch set renames the existing apparmor annotation
function to a more generic MAC (Mandatory Access Control)
name to be flexible enough to handle other MAC annotations
in the future.

Change-Id: I98a34484cebc2b420ad8f2664e4aaa84cfb9dca1
2018-10-17 01:35:49 +00:00
Matthew Heler
5efac315f7 Initialize OSDs with a crush weight of 0 to prevent automatic rebalancing.
Weight the OSDs based on reported disk size when ceph-client chart runs.

Change-Id: I9f4080a9843f1a63564cf71154841b351382bfe2
2018-10-16 21:33:49 +00:00
Steve Wilkerson
f3d8bda9d6 Grafana: Support multiple Ceph clusters with dashboards
This updates the Grafana Ceph dashboards to use templating to
determine which ceph-mgr to use for displaying ceph related
metrics.  This required setting the appropriate labels on the
ceph-mgr service to be able to distinguish between releases

Change-Id: Id2eceacadc5b6366d7bc6668bc16ccf5ba878e4a
2018-10-16 21:32:13 +00:00
Chris Wedgwood
8dad346f3f [MariaDB] Bump to version 10.2.18 to avoid shutdown hangs
We see sporadic shutdown hangs that look to be the issue described at
https://jira.mariadb.org/browse/MDEV-15554

Upgrade minor version to address this.

Change-Id: Idf8403b44e871b5a32173bd153a8367519b239ec
2018-10-16 21:30:22 +00:00
Pete Birley
a4111037b0 Gate: Fix kubeadm-aio image
This PS resores the kubeadm-aio image to a functioning state, by
updating the requests package.

Change-Id: I706a8ca5661a8e773386c8d82c049e2a9a04e94e
Signed-off-by: Pete Birley <pete@port.direct>
2018-10-16 16:09:49 -05:00
Zuul
6e092c908c Merge "Externalize some repo URL vars to allow runtime modification" 2018-10-16 00:04:06 +00:00
Zuul
580522c42a Merge "Ceph-client: make pool creation depedent on ceph-mgr service" 2018-10-15 22:05:40 +00:00
Zuul
1f9c8d7f42 Merge "Nagios: Update image with Elasticsearch plugin headers" 2018-10-15 17:58:17 +00:00
Zuul
b3e777c596 Merge "Add network policy toolkit function" 2018-10-15 17:45:35 +00:00
Roman Gorshunov
da31cacafd Externalize some repo URL vars to allow runtime modification
This is to be able to use local mirror of certain packages.

Change-Id: Ia06c6df0628ce5a44ed072c875eaa65d1343c65d
2018-10-15 17:10:10 +00:00
Chinasubbareddy M
616aecd80a Ceph-client: make pool creation depedent on ceph-mgr service
This is to add dependency for pool creation untill ceph-mgr fully up.

Change-Id: Id3111810a855bedff62970091b225358c269cecd
2018-10-15 10:00:27 -05:00
Steve Wilkerson
19248c11e9 Nagios: Update image with Elasticsearch plugin headers
This updates the Nagios image to include an update to the
Elasticsearch plugin that adds the appropriate headers to the
request sent to Elasticsearch. As Elasticsearch >=6.0 no longer
tries to determine the request data type, we need to explicitly
tell Elasticsearch the request body is JSON. Since we use
Elasticsearch 5.6.4 as default, this change will make the
deprecation warnings for the 6.0 breaking change go away.

Change-Id: I0dbd8859ca8d0bd0893832b4edd92742e575598b
2018-10-15 14:20:22 +00:00
Tin Lam
92e68d33ea Add network policy toolkit function
This patch set implements the helm toolkit function to generate a
kubernetes network policy manifest based on overrideable values.
This also adds a chart that shuts down all the ingress and egress
traffics in the namespace. This can be used to ensure the
whitelisted network policy works as intended.

Additionally, implementation is done for some infrastructure charts.

Change-Id: I78e87ef3276e948ae4dd2eb462b4b8012251c8c8
Co-Authored-By: Mike Pham <tp6510@att.com>
Signed-off-by: Tin Lam <tin@irrational.io>
2018-10-15 13:50:50 +00:00
Jean-Philippe Evrard
0dcceacf7d Remove dependency to OSH repository for test jobs
Without this patch, there is a dependency between the two
repositories OSH and OSH-infra, which will cause a circular
dependency problem when trying to remove the duplicated jobs
that will appear in OSH.

Change-Id: Ibeee0a853d0c1358519b0391c879137d8a214be2
2018-10-15 13:34:08 +02:00
Zuul
be7dbf6c28 Merge "[MariaDB] Update/remove deprecated configuration" 2018-10-13 21:40:48 +00:00
Zuul
75ea67e591 Merge "Fluent-logging: Update helm tests for checking index entries" 2018-10-13 03:11:39 +00:00
Zuul
c39b29e351 Merge "Fluentd: Update logging interval values" 2018-10-13 03:02:04 +00:00