3299 Commits

Author SHA1 Message Date
Steven Fitzpatrick
d3c6069be3 Elasticsearch: Make templates job more robust
This change primarily changes the type of the api_objects yaml structure
to a map, which allows for additional objects to be added by values
overrides (Arrays/Lists are not mutable like this)

Also, in the previous change, some scripts in HTK were modified, while
other were copied over to the Elasticsearch chart. To simplify the chart's
structure, this change also moves the create_s3_bucket script to Elasticsearch,
and reverts the changes in HTK.

Those HTK scripts are no longer referenced by osh charts, and could be candidates
for removal if that chart needed to be pruned

Change-Id: I7d8d7ef28223948437450dcb64bd03f2975ad54d
2021-04-12 18:40:11 +00:00
Gage Hugo
25c897fb89 Move shaker chart from osh-addons
This change moves the shaker chart from the osh-addons repo
to this one.

Change-Id: Ica2c7668a7ab047f8ed2361234b5810eedc9c1e2
2021-04-08 04:06:00 +00:00
Zuul
c6786de152 Merge "Enable TLS for Ceph RGW" 2021-04-06 20:53:03 +00:00
Ritchie, Frank (fr801x)
e954253a1a Enable TLS for Ceph RGW
This PS is to optionally enable tls for ceph-rgw.

Change-Id: I4797ef41612143f8065ac8fec20ddeae2c0218a3
2021-04-06 18:44:59 +00:00
Steven Fitzpatrick
6de864110e Elasticsearch S3 Update
This change updates how the Elasticsearch chart handles
S3 configuration and snapshot repository registration.

This allows for
  - Multiple snapshot destinations to be configued
  - Repositories to use a specific placement target
  - Management of multiple account credentials

Change-Id: I12de918adc5964a4ded46f6f6cd3fa94c7235112
2021-04-06 15:12:34 +00:00
Chris Wedgwood
20cf2db961 [htk] Jobs; put labels only in the template spec
This is an update to address a behavior change introduced with
0ae8f4d21ac2a091f1612e50f4786da5065d4398.

Job labels if empty/unspecified are taken from the template.  If (any)
labels are specified on the job we do not get this behavior.

Specifically if we *apply*:

    apiVersion: batch/v1
    kind: Job
    metadata:
      # no "labels:" here
      name: placement-db-init
      namespace: openstack
    spec:
      template:
        metadata:
          labels:
            application: placement
            component: db-init
            release_group: placement
        spec:
          containers:
          # do stuffs

then *query* we see:

    apiVersion: batch/v1
    kind: Job
    metadata:
      # k8s did this for us!
      labels:
        application: placement
        component: db-init
        job-name: placement-db-init
        release_group: placement
      name: placement-db-init
      namespace: openstack
    spec:
      template:
        metadata:
          labels:
            application: placement
            component: db-init
            release_group: placement
        spec:
          containers:
          # do stuffs

The aforementioned change causes objects we apply and query to look
like:

    apiVersion: batch/v1
    kind: Job
    metadata:
      # k8s did this for us!
      labels:
        application: placement
        # nothing else!
      name: placement-db-init
      namespace: openstack
    spec:
      template:
        metadata:
          labels:
            application: placement
            component: db-init
            release_group: placement
        spec:
          containers:
          # do stuffs

Current users rely on this behavior and deployment systems use job
labels for synchronization, those labels being only specified in the
template and propagating to the job.

This change preserves functionality added recently and restores the
previous behavior.

The explicit "application" label is no longer needed as the
helm-toolkit.snippets.kubernetes_metadata_labels macro provides it.

Change-Id: I1582d008217b8848103579b826fae065c538aaf0
2021-04-02 16:54:03 -05:00
Zuul
7351586a7d Merge "Allow Ceph RBD pool job to leave failed pods" 2021-03-30 08:04:06 +00:00
Zuul
3ef3ad1432 Merge "HTK: Override the expiry of Ingress TLS certificate" 2021-03-29 21:23:14 +00:00
Zuul
f1384caca6 Merge "[Update] NPD systemd-monitor lookback duration" 2021-03-29 19:54:10 +00:00
Zuul
4ed24de14b Merge "[ceph-osd] Update directory-based OSD deployment for image changes" 2021-03-29 19:48:43 +00:00
Parsons, Cliff (cp769u)
f20eff164f Allow Ceph RBD pool job to leave failed pods
This patchset will add the capability to configure the
Ceph RBD pool job to leave failed pods behind for debugging
purposes, if it is desired. Default is to not leave them
behind, which is the current behavior.

Change-Id: Ife63b73f89996d59b75ec617129818068b060d1c
2021-03-29 19:38:55 +00:00
Chinasubbareddy Mallavarapu
734b344bf6 [ceph-provisioners] Update ceph_mon config as per new ceph clients
As new ceph clients expecting the ceph_mon config as shown below , this
ps will update the configmap.

mon_host = [v1:172.29.1.139:6789/0,v2:172.29.1.139:3300/0],
[v1:172.29.1.140:6789/0,v2:172.29.1.140:3300/0],
[v1:172.29.1.145:6789/0,v2:172.29.1.145:3300/0]

Change-Id: I6b96bf5bd4fb29bf1e004fc2ce8514979da706ed
2021-03-29 15:02:08 +00:00
Stephen Taylor
131ea21512 [ceph-osd] Update directory-based OSD deployment for image changes
Directory-based OSDs are failing to deploy because 'python' has
been replaced with 'python3' in the image. This change updates the
python commands to use python3 instead.

There is also a dependency on forego, which has been removed from
the image. This change also modifies the deployment so that it
doesn't depend on forego.

Ownership of the OSD keyring file has also been changed so that it
is owned by the 'ceph' user, and the ceph-osd process now uses
--setuser and --setgroup to run as the same user.

Change-Id: If825df283bca0b9f54406084ac4b8f958a69eab7
2021-03-29 14:40:28 +00:00
Zuul
1f52a1c24c Merge "Set strict permission on mariadb data dir" 2021-03-26 22:20:32 +00:00
Zuul
0d8331d7ec Merge "fix(script): removes replacement overrides" 2021-03-26 18:00:33 +00:00
Radhika Pai
e9fce11161 [Update] NPD systemd-monitor lookback duration
This ps adds the lookback duration of 5m to the systemd-monitor to avoid
looking back indefinitely in journal log and causing the alert to stick around.

Change-Id: Ia32f043c0c7484d0bb92cfc4b68b506eae8e9d72
2021-03-26 15:24:10 +00:00
Gupta, Sangeet (sg774j)
f4ce1c8681 HTK: Override the expiry of Ingress TLS certificate
v1.2.0 of cert-manager noew supports overriding the default value
of ingress certificate expiry via annotations. This PS add the
required annotation.

Change-Id: Ic81e47f24d4e488eb4fc09688c36a6cea324e9e2
2021-03-25 22:18:57 +00:00
Huang, Sophie (sh879n)
6eec615b39 Set strict permission on mariadb data dir
For security reasons, strict access permission is given to
the mariadb data directory /var/lib/mysql

Change-Id: I9e55a7e564d66874a35a54a72817fa1237a162e9
2021-03-24 20:20:03 +00:00
Zuul
b3888df131 Merge "Elasticsearch Disable Curator in Gate & Chart Defaults" 2021-03-24 02:08:39 +00:00
Parsons, Cliff (cp769u)
167b9eb1a8 Fix ceph-client helm test
This patch resolves a helm test problem where the test was failing
if it found a PG state of "activating". It could also potentially
find a number of other states, like premerge or unknown, that
could also fail the test. Note that if these transient PG states are
found for more than 3 minutes, the helm test fails.

Change-Id: I071bcfedf7e4079e085c2f72d2fbab3adc0b027c
2021-03-22 22:06:27 +00:00
Steven Fitzpatrick
4fb159f7a3 Elasticsearch Disable Curator in Gate & Chart Defaults
Since chart v0.1.3 SLM policies have been supported, but we still
run curator in the gate, and its manifest toggles still default to
true

Change-Id: I5d8a29ae78fa4f93cb71bdf6c7d1ab3254c31325
2021-03-22 21:16:59 +00:00
Tin Lam
b72f750e87 fix(script): removes replacement overrides
This removes the functionality to perform envsubst in the feature
gate script to prevent users with specific env set running into
unexpected error. This feature will be re-visited in the future to
be made more robust.

Signed-off-by: Tin Lam <tin@irrational.io>
Change-Id: I6dcfd4dad138573294a9222e4e7af80c9bff4ac0
2021-03-19 01:14:09 -05:00
Zuul
43226de6e3 Merge "Enable TLS between Prometheus and Grafana" 2021-03-18 15:28:34 +00:00
Zuul
f78cbde672 Merge "Enable TLS for Prometheus" 2021-03-18 07:00:03 +00:00
Lo, Chi (cl566n)
86112314ed Enable TLS between Prometheus and Grafana
This patchset enables TLS path between Prometheus and Grafana.
Grafana pull data from Prometheus. As such, Prometheus is the
server and Grafana is the client for TLS handshake.

Change-Id: I50cb6f59472155415cff16a81ebaebd192064d65
2021-03-18 02:12:16 +00:00
Lo, Chi (cl566n)
1892fca645 Enable TLS for Prometheus
This patchset enabled TLS path for Prometheus when it acts as
a server.  Note that TLS is not directly terminated at Prometheus.
TLS is terminated at apache proxy which in turn route request
to Prometheus.

Change-Id: I0db366b6237a34da2e9a31345d96ae8f63815fa2
2021-03-17 17:06:07 -07:00
Zuul
8c2bcb1429 Merge "Disable mariadb mysql history client logging" 2021-03-17 19:15:32 +00:00
Smith, David (ds3330)
96b751465a Upgrade Prometheus to v2.25 change/Remove deprecated flags
The flag storage.tsdb.retention is deprecated and generates warnings
on startup storage.tsdb.retention.time is the new flag.

storage.tsdb.wal-compression is now set as the default in v2.20
and above and is no longer needed

Change-Id: I66f861a354a3cdde69a712ca5fd8a1d1a1eca60a
2021-03-16 18:19:49 +00:00
Zuul
58d9a62e73 Merge "Pin a few Java configuration values to 8-13" 2021-03-16 05:50:45 +00:00
Ritchie, Frank (fr801x)
05cad716e5 Add support for rgw placement targets
This PS adds support for rgw placement targets:

https://docs.ceph.com/en/latest/radosgw/placement/#placement-targets

Change-Id: I6fc643994dcf2c15a04f07b8703968a76c009c18
2021-03-12 22:16:41 +00:00
Huang, Sophie (sh879n)
87429ebb86 Disable mariadb mysql history client logging
Environment variable MYSQL_HISTFILE is added to mariadb container
to disable storing client mysql history to ~/.mysql_history file.

Change-Id: Ie95bc1f830fbf34d30c73de07513299115d8e8c5
2021-03-12 20:50:15 +00:00
Stephen Taylor
69a7916b92 [ceph-client] Disable autoscaling before pools are created
When autoscaling is disabled after pools are created, there is an
opportunity for some autoscaling to take place before autoscaling
is disabled. This change checks to see if autoscaling needs to be
disabled before creating pools, then checks to see if it needs to
be enabled after creating pools. This ensures that autoscaling
won't happen when autoscaler is disabled and autoscaling won't
start prematurely as pools are being created when it is enabled.

Change-Id: I8803b799b51735ecd3a4878d62be45ec50bbbe19
2021-03-12 15:03:51 +00:00
Kiran Kumar Surapathi (ks342f)
4b42f3f57f Fix Helm tests for the Ceph provisioners
We are adding the node selectors to helm tests for Ceph provisioners

Change-Id: I0fc9a78dcd27a92486dc724ce9294da96826eac9
2021-03-11 17:34:12 +00:00
Zuul
ff81e97301 Merge "Stop using fsGroup inside container securityContext" 2021-03-11 03:11:42 +00:00
Zuul
8a3151a7c6 Merge "Replace brace expansion with more standardized Posix approach" 2021-03-10 23:57:29 +00:00
Mohammed Naser
737f5610e3 Pin a few Java configuration values to 8-13
The newer versions of ElasticSearch use Java 15 which has dropped
some of those options, we can keep backwards compatibility by
pinning to certain versions[1].

[1]: https://discuss.elastic.co/t/elasticsearch-wont-start-after-7-9-1-to-7-9-2-upgrade/249878/2

Change-Id: Iaa29bc202d9eb9c5eda3040b38596f0524a0c453
2021-03-10 17:23:36 -05:00
Mohammed Naser
3fee13c5cd Stop using fsGroup inside container securityContext
fsGroup is not supported inside the container securityContext,
only inside the pod.  This drops a configuration that is not
valid and makes things deployable.

Change-Id: I956a1de107768c3fadc704722db83eb661cd25d2
2021-03-10 16:51:16 -05:00
bw6938
bb3ce70a10 [ceph-client] enhance logic to enable and disable the autoscaler
The autoscaler was introduced in the Nautilus release. This
change only sets the pg_num value for a pool if the autoscaler
is disabled or the Ceph release is earlier than Nautilus.

When pools are created with the autoscaler enabled, a pg_num_min
value specifies the minimum value of pg_num that the autoscaler
will target. That default was recently changed from 8 to 32
which severely limits the number of pools in a small cluster per
https://github.com/rook/rook/issues/5091. This change overrides
the default pg_num_min value of 32 with a value of 8 (matching
the default pg_num value of 8) using the optional --pg-num-min
<value> argument at pool creation and pg_num_min value for
existing pools.

Change-Id: Ie08fb367ec8b1803fcc6e8cd22dc8da43c90e5c4
2021-03-09 22:11:47 +00:00
Neely, Travis (tn720x)
2300e76904 Replace brace expansion with more standardized Posix approach
There is also an extra . which causes the expansion to fail.

Change-Id: Id0c02e2f293a72048e33078548a588d5cf1b62ce
2021-03-09 16:01:02 -06:00
Kabanov, Dmitrii
cda359ef1f [Ceph] Add Ceph CSI plugin
The PS adds Ceph CSI plugin (RBD only)

Change-Id: I3ddc69e49d12ff178263f38ac10aff90bb82b902
2021-03-09 16:54:45 +00:00
Zuul
3e4a46d450 Merge "Update rbac api version for elastic-metricbeat" 2021-03-05 19:29:21 +00:00
Zuul
345f63383e Merge "Rename mariadb backup identities" 2021-03-05 19:29:07 +00:00
Zuul
fc418a30dd Merge "Update rbac api version for kubernetes-node-problem-detector" 2021-03-05 19:29:01 +00:00
okozachenko
5db88a5fb4 Rename mariadb backup identities
Challenge:
Now remote_ks_admin and remote_rgw_user are using for user labels
of backup target openstack cloud.
When the backup user doesn't exist and we can enable job_ks_user
manifest.
But job_ks_user uses .Vaules.secrets.identity.admin and mariadb,
while secret-rgw and cron-job-backup-mariadb use .Values.secrets.
identity.remote_ks_admin and remote_rgw_user.
It requires to use same values for admin and remote_ks_admin,
and for mariadb and remote_rgw_user.
Seems it isbreaking values consistency.

Suggestion:
Now providing 2 kinds of backup - pvc and swift.
"remote_" means the swift backup.
In fact, mariadb chart has no case to access to keystone except
swift backup. So we can remove remote_xx_* prefix and there is
no confusion.

Change-Id: Ib82120611659bd36bae35f2e90054642fb8ee31f
2021-03-03 20:46:51 +02:00
Zuul
618c064d25 Merge "Add metadata in job templates" 2021-03-03 17:07:23 +00:00
Zuul
c107f44f86 Merge "[mariadb-ingress] Uplift Mariadb-ingress to 0.42.0" 2021-03-03 16:32:54 +00:00
Zuul
a6867a7953 Merge "Remove kafka residue" 2021-03-02 21:49:25 +00:00
Gupta, Sangeet (sg774j)
d64c5fea64 TLS gates from infra monitoring
Change-Id: If41dae31763abac8c9c65b1650556438a9b6b0df
2021-03-02 16:10:24 +00:00
jinyuanliu
d3d16964da Remove kafka residue
About kafka chart,It's been removed,Remove kafka residue now.

Change-Id: Ia9b4b9ea1070e74172f10505709e68063054810a
2021-03-02 16:44:55 +08:00
Zuul
0b308f7752 Merge "Enforce release notes updates" 2021-03-02 06:32:56 +00:00