HTK - added verify_databases_backup_in_directory function that is
going to be defined inside mariadb/postgresql/etcd charts.
Mariadb chart - added verify_databases_backup_archives function
implementation.
Added mariadb-verify container to mariadb-backup cronjob to run
verification process.
Added remove backup verification pocess - comparition of local and remote file md5 hashes.
PostgreSQL chart - added empty implementation of verify_databases_backup_archives() function. This is a subject for future realization.
Change-Id: I361cdb92c66b0b27539997d697adfd1e93c9a29d
In an environment with helmv3, it was noticed that the mariadb
helmrelease is failing to render properly due to unsupported map key
type (int).
This change quickly fix this problem by quoting the value, forcing it to
be rendered as a string.
Signed-off-by: Thales Elero Cervi <thaleselero.cervi@windriver.com>
Change-Id: I2f2be87d0f79ca439e731d07354bcd5f149790d5
Based on spec in openstack-helm repo,
support-OCI-image-registry-with-authentication-turned-on.rst
Each Helm chart can configure an OCI image registry and
credentials to use. A Kubernetes secret is then created with these
info. Service Accounts then specify an imagePullSecret specifying
the Secret with creds for the registry. Then any pod using one
of these ServiceAccounts may pull images from an authenticated
container registry.
Change-Id: Iebda4c7a861aa13db921328776b20c14ba346269
Readiness probe that we currently have does not help with restarting a
pod that got stuck in a transfer state reported by
wsrep_local_state_comment.
root@mariadb-server-2:/# mysql_status_query wsrep_ready
OFF
root@mariadb-server-2:/# mysql_status_query wsrep_connected
ON
root@mariadb-server-2:/# mysql_status_query wsrep_cluster_status
non-Primary
root@mariadb-server-2:/# mysql_status_query wsrep_local_state_comment
Transfer
So the idea is to add a liveness probe that will take care of this.
Change-Id: I2ccecc75349667fe19c6f7f9dccc2dbbd17d0a5e
This patch adds database sys to the list of databases
to be ignored by backup/restore scripts in mariadb chart.
Change-Id: Ida7965bc583ada2c7ca4800c8ff5d6761fb3913a
This patchset is adding update priviledge to ingress cluster role in order to let it to update mariadb state configmap. The problem appeared after upgrading nginx controller up to v1.1.3 in https://review.opendev.org/c/openstack/openstack-helm-infra/+/840691
Change-Id: I962ac336bf6b3588db88b04e2259de1aa20b1221
This change updates the default image values in the mariadb chart
up to using Wallaby for the ones that use openstack images.
Change-Id: Id28da22932362c0400766a564b382ddbcada8c61
CHG: Updated naming variable to change based on global values
subchart_release_name for the following:
* mariadb
* rabbitmq
* memcached
This is a required change for the chart to be included
as a subchart. if subchart_release_name is not present the
yaml will render the same as prior to this change, leaving
existing deployments unaffected.
Change-Id: Ib7a449f3b21d5169b8003cf4464f3ed95e942c14
This adds taint toleration support for openstack jobs
Signed-off-by: Lucas Cavalcante <lucasmedeiros.cavalcante@windriver.com>
Change-Id: Iab78370182b15b48df964eb2dfdc957a9868c708
The mariadb chart currently fails to deploy due to
differences in handling comparison between helm v2
and v3. This change updates the comparison to work
in both versions.
Change-Id: I9143a16f3011c0c0ae5420e6ec41ad7745a28cab
The set -x has produced 6 identical log strings every time the
log_backup_error_exit function is called. Prometheus is using
the occurrence and number of some logs over a period of time to
evaluate database backup failure or not. Only one log should be
generated when a particular database backup scenario failed.
Upon discussion with database backup and restore SME, it is
recommended to remove the set -x once and for all.
Change-Id: I846b5c16908f04ac40ee8f4d87d3b7df86036512
* Add capability to retry uploading backup to remote server configured
number of times and delay the retires randomly between configured
minimum/maximum seconds.
* Enhanced error checking, logging and retrying logic.
Change-Id: Ida3649420bdd6d39ac6ba7412c8c7078a75e0a10
If thread launch_cluster_Monitor() and launch_leader_election() operates on the configmap at the same time, Will cause a error 'Exception in thread "Thread-1"'.
This error will cause the thread to get stuck. Configmap will not be updated and the error "data too old" will be reported.
Just passing kubernetes_API exceptions is not enough, all are more appropriate.
Change-Id: I6baa9ece474f9c937fe9bce2231ef500562e0406
This change updates the helm-toolkit path in each chart as part
of the move to helm v3. This is due to a lack of helm serve.
Change-Id: I011e282616bf0b5a5c72c1db185c70d8c721695e
If labels are not specified on a Job, kubernetes defaults them
to include the labels of their underlying Pod template. Helm 3
injects metadata into all resources [0] including a
`app.kubernetes.io/managed-by: Helm` label. Thus when kubernetes
sees a Job's labels they are no longer empty and thus do not get
defaulted to the underlying Pod template's labels. This is a
problem since Job labels are depended on by
- Armada pre-upgrade delete hooks
- Armada wait logic configurations
- kubernetes-entrypoint dependencies
Thus for each Job template this adds labels matching the
underlying Pod template to retain the same labels that were
present with Helm 2.
[0]: https://github.com/helm/helm/pull/7649
Change-Id: I3b6b25fcc6a1af4d56f3e2b335615074e2f04b6d
exporter-jpb-create-user was failing due to the field immutability
which was resulting in the manual delete of the job for every helm
upgrade to be successful. Reason being job being upgraded before the
other manifest that are required been updated. It can be avoided by
using helm-hook post-install and post-upgrade which will force the
job manifest to be applied only after all other manifest are applied.
Hook annotation is provided "5" so that the if other jobs are annotated,
exporter job will be last to created.
helm3_hook value is used for the condition which will enable the disable
of the hook.
Change-Id: I2039abb5bad07a19fd09fc5e245485c3c772beca
This will ease mirroring capabilities for the docker official images.
Signed-off-by: Thiago Brito <thiago.brito@windriver.com>
Change-Id: I0f9177b0b83e4fad599ae0c3f3820202bf1d450d
Since k8s v1.11+, the annotation `service.alpha.kubernetes.io/tolerate-unready-endpoints` is deprecated. we should use Service.spec.publishNotReadyAddresses instead.
Change-Id: Ic4f82b8e78770ff29637937c4bcb9af71b53f8d3
The current start logic when existing cluster state is reboot can
lead to a split brain condition under certain circumstances. This
patchset adds some additional step to ensure cluster is set to
live state once leader node is ready to start, instead of relying
on slave nodes to handle. Also add some simple retry when there
is collision detected while trying to write to configmap.
The existing hair-trigger that will put the cluster state from
"live" into "reboot" can use some fine tuning, but updating it
properly should require additional investigation and testing,
hence should be done as a separate activity outside the scope
of this patchset.
Change-Id: Ieb2861d6fbc435e24e20d13c7b358c751890b4c4
This change updates the default images for mariadb, both the version
to 10.5.9 and the ubuntu release to focal.
Change-Id: Iff99ebe78554197db4d459bef0dda01b6b2710b7
There seems to be a race condition involving the grastate.dat file.
Upon creation of a new mariad-server pod the file would exist however,
it is not populated for a short period of time. It seems to take
around 15-20 seconds for this file to be populated. However there is
a separate thread which is attempting to read the file and tends to
end in an IndexError exception killing the thread which maintains the
grastate.dat file until the pod is restarted. This patchset adds a
loop to check for up to 60 seconds for the file to be populated
before attempting to continue, thus giving the file time to be
populated.
Change-Id: I2f2a801aa4528a7af61797419422572be1c82e75
For security reasons, strict access permission is given to
the mariadb data directory /var/lib/mysql
Change-Id: I9e55a7e564d66874a35a54a72817fa1237a162e9
Environment variable MYSQL_HISTFILE is added to mariadb container
to disable storing client mysql history to ~/.mysql_history file.
Change-Id: Ie95bc1f830fbf34d30c73de07513299115d8e8c5
Challenge:
Now remote_ks_admin and remote_rgw_user are using for user labels
of backup target openstack cloud.
When the backup user doesn't exist and we can enable job_ks_user
manifest.
But job_ks_user uses .Vaules.secrets.identity.admin and mariadb,
while secret-rgw and cron-job-backup-mariadb use .Values.secrets.
identity.remote_ks_admin and remote_rgw_user.
It requires to use same values for admin and remote_ks_admin,
and for mariadb and remote_rgw_user.
Seems it isbreaking values consistency.
Suggestion:
Now providing 2 kinds of backup - pvc and swift.
"remote_" means the swift backup.
In fact, mariadb chart has no case to access to keystone except
swift backup. So we can remove remote_xx_* prefix and there is
no confusion.
Change-Id: Ib82120611659bd36bae35f2e90054642fb8ee31f
- Uplifts the image to nginx 0.42.0 to address CVEs
- Updates nginx.tmpl accordingly for nginx 0.42.0
- Adds CLusterRole and labels needed for nginx 0.42.0
- Updates release notes for mariadb
Change-Id: Ie4e2a66873bc130c547ff8f30d8e1b2ee9a62186
ClusterIssuer does not belong to a single namespace (unlike Issuer)
and can be referenced by Certificate resources from multiple different
namespaces. When internal TLS is added to multiple namespaces, same
ClusterIssuer can be used instead of one Issuer per namespace.
Change-Id: I1576f486f30d693c4bc6b15e25c238d8004b4568
Enabling ability to automate testing and auto promotion.
Unpinning ovs, mariadb and node-problem-detector images.
Change-Id: I6256452d575d23f84f4fd5c728437b0e4e9423f3
Signed-off-by: Andrii Ostapenko <andrii.ostapenko@att.com>
When multiple users are granted access to a database, the
MariaDB backup script failed to retrieve the grants for that
database, which caused the backup job to fail. This patchset
updates the script.
Change-Id: I9076b2e7363ae0ec216d4e822f385fa949df8f54
This commit ensures the below mariadb settings with reference to [0]:
- 'local_infile' Is Disabled
- 'have_symlink' Is Disabled
- 'secure_file_priv' Is Not Empty
- 'sql_mode' Contains 'STRICT_ALL_TABLES'
[0] https://dev.mysql.com/doc/mysql-security-excerpt/8.0/en/general-security-issues.html
Change-Id: I701b9bc2bdfb91d67aef91e88f953a09ac72d8be
Since we introduced chart version check in gates, requirements are not
satisfied with strict check of 0.1.0
Change-Id: I15950b735b4f8566bc0018fe4f4ea9ba729235fc
Signed-off-by: Andrii Ostapenko <andrii.ostapenko@att.com>
Added chart lint in zuul CI to enhance the stability for charts.
Fixed some lint errors in the current charts.
Change-Id: I9df4024c7ccf8b3510e665fc07ba0f38871fcbdb
Fallback to old dump file naming for read operation to support archives
with legacy naming.
Change-Id: I0c9c7b2c1feaac9aca817041dae617b4d1056b84
Signed-off-by: Andrii Ostapenko <andrii.ostapenko@att.com>
This patch fixes following issues:
1. The existing envvar DATA_SOURCE_NAME overrides the setting specified
in the mysql_user.cnf file, ignore setting placed there;
2. Version 0.10 of the exporter does not support TLS, moving this to
minimally 0.11; and
3. Changed the host to the internal long name rather than the short
name.
Change-Id: I7259d23391ed31c423d74a8d9dc002e597adfb95
Signed-off-by: Tin Lam <tin@irrational.io>
This PS adds the capability to Mariadb and Postgresql to backup a
single database (as an optional parameter to the backup script).
Change-Id: I9bc1eb0173063638b2cf58465c063f602ed20bc1
