openstack-helm-infra/mariadb
Markin, Sergiy (sm515x) 931ba39e87 [MariaDB] Add liveness probe to restart a pod that got stuck in a transfer wsrep_local_state_comment
Readiness probe that we currently have does not help with restarting a
pod that got stuck in a transfer state reported by
wsrep_local_state_comment.

root@mariadb-server-2:/# mysql_status_query wsrep_ready
OFF
root@mariadb-server-2:/# mysql_status_query wsrep_connected
ON
root@mariadb-server-2:/# mysql_status_query wsrep_cluster_status
non-Primary
root@mariadb-server-2:/# mysql_status_query wsrep_local_state_comment
Transfer

So the idea is to add a liveness probe that will take care of this.

Change-Id: I2ccecc75349667fe19c6f7f9dccc2dbbd17d0a5e
2022-06-21 20:32:52 +00:00
..
files Migrate CronJob resources to batch/v1 and PodDisruptionBudget resources to policy/v1 2022-05-10 15:12:53 -04:00
templates [MariaDB] Add liveness probe to restart a pod that got stuck in a transfer wsrep_local_state_comment 2022-06-21 20:32:52 +00:00
values_overrides Remove panko residue 2021-05-18 11:01:40 +08:00
.helmignore MariaDB: Move chart to openstack-helm-infra 2018-09-04 18:57:53 -05:00
Chart.yaml [MariaDB] Add liveness probe to restart a pod that got stuck in a transfer wsrep_local_state_comment 2022-06-21 20:32:52 +00:00
README.rst MariaDB: Move chart to openstack-helm-infra 2018-09-04 18:57:53 -05:00
requirements.yaml Update htk requirements 2021-10-06 01:02:28 +00:00
values.yaml [MariaDB] Add liveness probe to restart a pod that got stuck in a transfer wsrep_local_state_comment 2022-06-21 20:32:52 +00:00

openstack-helm/mariadb

By default, this chart creates a 3-member mariadb galera cluster.

This chart leverages StatefulSets, with persistent storage.

It creates a job that acts as a temporary standalone galera cluster. This host is bootstrapped with authentication and then the WSREP bindings are exposed publicly. The cluster members being StatefulSets are provisioned one at a time. The first host must be marked as Ready before the next host will be provisioned. This is determined by the readinessProbes which actually validate that MySQL is up and responsive.

The configuration leverages xtrabackup-v2 for synchronization. This may later be augmented to leverage rsync which has some benefits.

Once the seed job completes, which completes only when galera reports that it is Synced and all cluster members are reporting in thus matching the cluster count according to the job to the replica count in the helm values configuration, the job is terminated. When the job is no longer active, future StatefulSets provisioned will leverage the existing cluster members as gcomm endpoints. It is only when the job is running that the cluster members leverage the seed job as their gcomm endpoint. This ensures you can restart members and scale the cluster.

The StatefulSets all leverage PVCs to provide stateful storage to /var/lib/mysql.

You must ensure that your control nodes that should receive mariadb instances are labeled with openstack-control-plane=enabled, or whatever you have configured in values.yaml for the label configuration:

kubectl label nodes openstack-control-plane=enabled --all