openstack/openstack-helm-infra
Code Issues Proposed changes

[ceph] Update queries to filter pgs correctly

Browse Source 
The PS updates queries in wait_for_pgs function in ceph-client and
ceph-osd charts. It allows more accurately check the status of PGs.
The output of the "ceph pg ls" command may contain many PG statuses,
like "active+clean", "active+undersized+degraded", "active+recovering",
"peering" and etc. But along with these statuses there may be such as
"stale+active+clean". To avoid the wrong interpretation of the status
of the PSs the filter was changed from "startswith(active+)" to
"contains(active)".
Also PS adds a delay after restart of the pods to post-apply job.
It allows to reduce the number of useless queries to kubernetes.

Change-Id: I0eff2ce036ad543bf2554bd586c2a2d3e91c052b
This commit is contained in:
Kabanov, Dmitrii 2020-08-13 22:45:01 -07:00
parent adf9fc7fc4
commit 4557f6fbe8
2 changed files with 5 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
ceph-client/templates/bin/pool/_init.sh.tpl
View File

@ -33,7 +33,7 @@ function wait_for_pgs () {
echo "#### Start: Checking pgs ####" echo "#### Start: Checking pgs ####"
pgs_ready=0 pgs_ready=0
query='map({state: .state}) | group_by(.state) | map({state: .[0].state, count: length}) | .[] | select(.state | startswith("active+") | not)' query='map({state: .state}) | group_by(.state) | map({state: .[0].state, count: length}) | .[] | select(.state | contains("active") | not)'
if [[ $(ceph tell mon.* version | egrep -q "nautilus"; echo $?) -eq 0 ]]; then if [[ $(ceph tell mon.* version | egrep -q "nautilus"; echo $?) -eq 0 ]]; then
query=".pg_stats | ${query}" query=".pg_stats | ${query}"

5
ceph-osd/templates/bin/_post-apply.sh.tpl
View File

@ -83,7 +83,7 @@ function wait_for_pgs () {
echo "#### Start: Checking pgs ####" echo "#### Start: Checking pgs ####"
pgs_ready=0 pgs_ready=0
query='map({state: .state}) | group_by(.state) | map({state: .[0].state, count: length}) | .[] | select(.state | startswith("active+") | not)' query='map({state: .state}) | group_by(.state) | map({state: .[0].state, count: length}) | .[] | select(.state | contains("active") | not)'
if [[ $(ceph tell mon.* version | egrep -q "nautilus"; echo $?) -eq 0 ]]; then if [[ $(ceph tell mon.* version | egrep -q "nautilus"; echo $?) -eq 0 ]]; then
query=".pg_stats | ${query}" query=".pg_stats | ${query}"
@ -136,6 +136,9 @@ function restart_by_rack() {
fi fi
done done
echo "waiting for the pods under rack $rack from restart" echo "waiting for the pods under rack $rack from restart"
# The pods will not be ready in first 60 seconds. Thus we can reduce
# amount of queries to kubernetes.
sleep 60
wait_for_pods $CEPH_NAMESPACE wait_for_pods $CEPH_NAMESPACE
echo "waiting for inactive pgs after osds restarted from rack $rack" echo "waiting for inactive pgs after osds restarted from rack $rack"
wait_for_pgs wait_for_pgs