214 Commits

Author SHA1 Message Date
Matthew Heler
3e7ba37290 Ensure latest Ceph packages during deployment
Change-Id: Ia5bc0802577e2b72a1de078085f5fe7e60f63604
2018-10-31 02:16:50 -05:00
Tin Lam
5730631ba6 Clean-up script
This patch set cleans up the script to be consistent with other OSH
installation scripts.

Change-Id: I212cd0cf0e818f1fc924b9b690d18f5d107b850b
Signed-off-by: Tin Lam <tin@irrational.io>
2018-10-30 16:22:45 +00:00
Chris Wedgwood
b10ebbb63a [gate] Use Kubernetes 1.10.9
Change-Id: I5bb951f455fa6d7d344a264336a2a9b985fd85f4
2018-10-29 15:10:35 +00:00
Jean-Philippe Evrard
52f41c0af0 Define OSH_PATH by default
OSH_PATH is not defined by default outside OpenStack's CI.

This is a problem if a user wants to run scripts manually on its
machine for local testing.

This fixes it by having, by default, the OSH_PATH defined
in the scripts using OSH relatively to current folder.

For user experience, the script returns to the same path after
running.

Change-Id: I915e7d3c945f2002a2008b2b033a2b7725320b17
2018-10-26 01:15:08 +00:00
Pete Birley
f6e84fe15f MariaDB: Galera cluster refactor
This PS updates the MariaDB chart to better support clustering,
using a configmap to track cluster state.

Change-Id: Ifd9c3d63353a9b587384b6f13c0863ecc4fbd956
Signed-off-by: Pete Birley <pete@port.direct>
2018-10-25 06:21:01 +00:00
Tin Lam
653b84a2e1 Fix k8s-auth job
This patch set addresses a cross-repo conflict with the enablement of
network policy in gate script override.

Change-Id: I284d6b04940424a87e5b239ccc9d30ae01075f38
Signed-off-by: Tin Lam <tin@irrational.io>
2018-10-24 20:49:17 -05:00
Zuul
860a897aee Merge "[gate] allow pip caching" 2018-10-23 18:30:20 +00:00
Chris Wedgwood
d4ac063163 [gate] allow pip caching
The pip cache is useful for repeat operations and doesn't seem to have
any real downsides.

Change-Id: Iadb21a118f8d725911a9baa6a9264b8644012af9
2018-10-22 00:11:25 +00:00
Chris Wedgwood
c08c78f1d1 [gate] Put nfs-provisioner in it's own namespace (docker-registry)
Use the 'docker-nfs' namespace to back the docker registry.  This
means we can delete the registry namespace without causing IO lockups.

Change-Id: I1706dd96653598dcfbb81904fde8c0bf92294b06
2018-10-21 23:42:20 +00:00
Chris Wedgwood
8f5aaa3fd0 [gate] Put nfs-provisioner in it's own namespace
Having storage (backend) components in their own namespace means we
can delete the namespaces containing the openstack without causing
system hangs which occur when storage is remove whilst in use.

Change-Id: Ie489709b08929f25cf0e626a8541620a06506b8b
2018-10-21 23:37:56 +00:00
Zuul
23730808d4 Merge "Gates: Update fluent-logging/elasticsearch configurations" 2018-10-21 09:17:19 +00:00
Zuul
91f38f22b7 Merge "Organize aio gates by function" 2018-10-21 09:16:23 +00:00
Matthew Heler
ec8bb513eb Add RBD-NBD support to openstack-helm
By default use rbd-nbd (librbd) instead of krbd.

Applying this change on existing nodes will
require reboots.

Change-Id: I81829fb8666541e856ab402128a5192984b6fe05
2018-10-19 16:20:10 -05:00
Steve Wilkerson
4c29bafcbc Gates: Update fluent-logging/elasticsearch configurations
This updates the fluentd buffer output configurations to account
for the restraints of the jobs deploying fluentd. This also
renames the fluentd configuration key from td_agent to fluentd to
reflect the fact we're no longer deploying td-agent

This also updates the Elasticsearch default replicas and overrides
the replica counts in each Elasticsearch deployment to account for
resource constraints

Change-Id: I55dee410eced99c3e1645f7452e4306ad646e601
2018-10-19 17:30:08 +00:00
Steve Wilkerson
538d51e991 Organize aio gates by function
This organizes the single node gates for osh-infra by function.
This organization aims to improve the single node gates in the
following ways:

1. Reduce number of services deployed in single node jobs
2. Only deploy Ceph for logging job, as Elasticsearch requires
   RGW for snapshot repositories.
3. Use NFS for storage for monitoring job, as Ceph is not a
   requirement for any of the services here.
4. Remove duplicate services deployed to multiple single node jobs
5. Remove storage from openstack-support job, as the only service
   requiring storage is rabbitmq. Rabbitmq is deployed with
   storage enabled in the openstack-helm checks/gates.

This also removes the documentation for the single node deployments,
as those deployments do not make sense with this change. This should
be revisited as a follow-on once we have a clear path forward for
the larger gate refactoring work

Change-Id: I46951f76904fa2ab245a202d55f76019b7503362
2018-10-19 12:28:18 -05:00
Samuel Pilla
6fe001361a Add LDAP support for k8s-keystone-auth in gate
This patch set changes the keystone in the k8s-keystone-auth to
be backed by LDAP. It also updates the test to use the LDAP users
instead of created users in the database.

Co-Authored-By: Samuel Pilla <sp516w@att.com>
Change-Id: Ia34dac51b36a300068ad5fd936c48b0f30821a52
Signed-off-by: Tin Lam <tin@irrational.io>
2018-10-17 06:19:20 +00:00
Pete Birley
a4111037b0 Gate: Fix kubeadm-aio image
This PS resores the kubeadm-aio image to a functioning state, by
updating the requests package.

Change-Id: I706a8ca5661a8e773386c8d82c049e2a9a04e94e
Signed-off-by: Pete Birley <pete@port.direct>
2018-10-16 16:09:49 -05:00
Zuul
6e092c908c Merge "Externalize some repo URL vars to allow runtime modification" 2018-10-16 00:04:06 +00:00
Roman Gorshunov
da31cacafd Externalize some repo URL vars to allow runtime modification
This is to be able to use local mirror of certain packages.

Change-Id: Ia06c6df0628ce5a44ed072c875eaa65d1343c65d
2018-10-15 17:10:10 +00:00
Tin Lam
92e68d33ea Add network policy toolkit function
This patch set implements the helm toolkit function to generate a
kubernetes network policy manifest based on overrideable values.
This also adds a chart that shuts down all the ingress and egress
traffics in the namespace. This can be used to ensure the
whitelisted network policy works as intended.

Additionally, implementation is done for some infrastructure charts.

Change-Id: I78e87ef3276e948ae4dd2eb462b4b8012251c8c8
Co-Authored-By: Mike Pham <tp6510@att.com>
Signed-off-by: Tin Lam <tin@irrational.io>
2018-10-15 13:50:50 +00:00
Pete Birley
8bb71f6659 Gate: Cleanup scripts for k8s keystone auth gate
This PS cleans up the scripts for the k8s k8s keystone auth gate.

Change-Id: I248439f9b8ffa372dfaba5acba0c8c587231d901
Signed-off-by: Pete Birley <pete@port.direct>
2018-10-12 13:43:41 -05:00
Pete Birley
7f4a37440f VBMC: Move and update vbmc image to osh-infra
This PS updates and moves the vmbc image to osh infra.

Change-Id: I9f8d21df8974d1484d9f087ee296fede2a87e545
Signed-off-by: Pete Birley <pete@port.direct>
2018-10-08 09:38:24 -05:00
Zuul
7c839c82b2 Merge "Gate: Move to K8s 1.10.8" 2018-10-06 06:30:08 +00:00
Zuul
77393ab143 Merge "Libvirt: Fix image" 2018-10-06 00:58:41 +00:00
Pete Birley
f8880d27ad Libvirt: Fix image
This PS fixes the libvirt image, buy removing the ubuntu-cloud
archive repo and pinning to a good version.

Change-Id: I5097d8893b92d020f7a5a1cb5925dec0b01d4da2
Signed-off-by: Pete Birley <pete@port.direct>
2018-10-05 15:41:12 -05:00
Gupta, Sangeet (sg774j)
a34a7d8e50 Adding Falco
This commit adds falco daemonset of the node for behavioral activity
monitor designed to detect anomalous activity.

Change-Id: I783a2acc03592471c81a8a54e1dc0df140b34a42
2018-10-05 16:18:51 +00:00
Pete Birley
19376ee9e6 Gate: Move to K8s 1.10.8
This PS moves to use k8s 1.10.8, which includes a couple of fixes
for PVC mounts.

* https://github.com/kubernetes/kubernetes/pull/66863

Change-Id: Ica30950a8200f5755897b51fd2b4d24c69a10e61
Signed-off-by: Pete Birley <pete@port.direct>
2018-10-05 07:44:42 -05:00
Pete Birley
feeeed4d5d Gate: Remove unused helm chart deployment role and playbook
This PS removes the unused helm chart deployment role and playbook.

Change-Id: I01c58a628589ec35af2557c8cc93ea47fe084089
Signed-off-by: Pete Birley <pete@port.direct>
2018-10-05 01:43:33 +00:00
Pete Birley
25985f7b43 Libvirt: escape kube cgroups and pid reaper
This PS moves to run the Libvirt process as a transient unit
on the host, free fom k8s controlled cgroups. In addition it
also uses the cloud archive provided libvirt/qemu packages.

Change-Id: Idfe9ae6f072acd86f877df0c3dfe3db4c20902d6
Signed-off-by: Pete Birley <pete@port.direct>
2018-10-03 19:11:00 +00:00
Steve Wilkerson
fa09705867 Fluentbit: Add kernel, kubelet, and dockerd logs
This adds inputs for kernel logs on the host, as well as dockerd
and kubelet logs via the systemd plugin. This also adds a filter
for adding the hostname to the kernel log events, for renaming the
fields for systemd logs as kibana can not visualize fields that
begin with an underscore, and adds elasticsearch indexes for both
kernel and systemd logs

Change-Id: I026470dd45a971047f1e5bd1cd49bd0889589d12
2018-10-01 11:56:58 +00:00
sai battina
e155c92f14 Helm: Update helm to 2.11.0
This helps to fix a bug when adding stable repos

Change-Id: I3eb28a037f7eb22016a29bc36e4a791a5bfda852
2018-09-25 18:14:49 +00:00
Steve Wilkerson
a084769410 Elasticsearch S3 repo
This ps adds the ability to use the ceph radosgw s3 api for
snapshot repositories. It removes the ability to use a RWM pvc, as
the radosgw solution provides a more robust approach for storing
index snapshots

Change-Id: Ie56ac41ccdc61bfadcac52b400cceb35403e9fae
2018-09-19 15:53:21 -05:00
Zuul
333fdc931b Merge "Gate: Trim dev-deploy gates" 2018-09-19 15:52:48 +00:00
Pete Birley
101f58ae4b Gate/Dev: Fix perms for upstream-resolv.conf
This PS fixes the permissions for the upstream resolv.conf used by the
dns redirector.

Change-Id: Ieef113a6e7b72767318516c63cf48dcac202cf4d
Signed-off-by: Pete Birley <pete@port.direct>
2018-09-18 11:55:20 -05:00
Steve Wilkerson
bc6e22f392 Gate: Trim dev-deploy gates
This attempts to trim down the dev-deploy gates until further
gate refactoring is complete. This disables the elasticsearch and
fluentd exporters and removes the openstack exporter from the
single node deployment gates to ease the load on nodepool vms

Change-Id: If211511e8f52fe39d293966abbd7e62b45b65970
2018-09-17 13:56:51 +00:00
Zuul
d208d55a46 Merge "Gate: only restart network manager if required" 2018-09-17 13:49:01 +00:00
Zuul
0f2dace4e8 Merge "Gate/Dev: Allow custom upstream dns servers to be preserved" 2018-09-15 16:59:22 +00:00
Pete Birley
620d374730 Gate: only restart network manager if required
We only need to restart network manager if disabling dns management.

Change-Id: Idfdf68678a68c2808527de4226ff91e9ea5f8d67
Signed-off-by: Pete Birley <pete@port.direct>
2018-09-15 14:39:17 +00:00
Pete Birley
79d11e4044 Gate/Dev: Allow custom upstream dns servers to be preserved
This PS updates the dns redirect pod deployment to support a persistant
set of customised upstream nameservers to be used.

Change-Id: Ib163f8ed9ceadca69b56cd5f146ffd194d98cdc3
Signed-off-by: Pete Birley <pete@port.direct>
2018-09-15 13:13:29 +00:00
Zuul
2fce7e8212 Merge "Correct the task name for tiller installed" 2018-09-14 21:03:16 +00:00
Zuul
accaf8aabf Merge "Update the env usage of docker_container module" 2018-09-13 13:53:54 +00:00
Zuul
e6966ffdaf Merge "Gate: Update gate permit running on ubuntu 18.04" 2018-09-11 19:19:27 +00:00
Pete Birley
e67f7bafd5 Gate: Update gate permit running on ubuntu 18.04
This PS updates the gate to permit running on the current LTS ubuntu
release.

Change-Id: I7e32a4ab0dc79e4b5f7a16f8a8cb5e9ee182ee08
Signed-off-by: Pete Birley <pete@port.direct>
2018-09-11 15:20:48 +00:00
Steve Wilkerson
6b944f557b Libvirt: Move chart to openstack-helm-infra
This moves the libvirt chart to openstack-helm-infra as part of
the effort to move charts to their appropriate repositories

Change-Id: I02ce197f8d100da74c086d84e2f9d2b902a69e97
Story: 2002204
Task: 21723
2018-09-10 09:45:55 -06:00
Steve Wilkerson
3dcbfae101 Openvswitch: Move chart to openstack-helm-infra
This moves the openvswitch chart to openstack-helm-infra as part of
the effort to move charts to their appropriate repositories

Change-Id: I6e00231b8de54c01bc9bb31e0433753a9f281542
Story: 2002204
Task: 21730
2018-09-07 12:35:40 +00:00
Zuul
40e0ddba33 Merge "Gate: Add process exporter to gate deployments and docs" 2018-09-05 20:52:24 +00:00
Steve Wilkerson
93630ac6e3 MariaDB: Move chart to openstack-helm-infra
This moves the mariadb chart to openstack-helm-infra as part of
the effort to move charts to the appropriate repositories

Change-Id: Ife56e28de46c536108cebb4f4cdf6bad2a415289
Story: 2002204
Task: 21725
2018-09-04 18:57:53 -05:00
Steve Wilkerson
0bfb2979ec Gate: Add gate for openstack support infrastructure
As part of the effort to move the supporting infrastructure
services to openstack-helm-infra, this adds a gate that will be
used for those services specifically

Change-Id: Id7c5649330eb41a0017a740ade9465fd66abb32f
2018-09-04 16:42:21 -05:00
Steve Wilkerson
0aae608aa0 Gate: Add process exporter to gate deployments and docs
This adds the process exporter to both the developer and multinode
gates, along with adding the relevant deployment steps to the docs

Change-Id: I85d5c398fbbb62145c9bb4e3a885e9a774725e5a
2018-09-04 15:54:25 -05:00
Zuul
b6f7ff7db5 Merge "Add Ceph to osh-infra gates" 2018-08-30 04:24:53 +00:00