Add conditions to when RBD devices are unmounted

ceph-preshutdown.sh is called as a post operation when docker is
stopped/restarted. Based on current service dependencies, when docker is
restarted this will also trigger a restart of containerd.

Puppet manifests will restart containerd and docker for various
operations both on system boot and during runtime operations when their
configuration has changed.

This update adds conditions to ensure that the RBD devices are only
unmounted when the system is shutting down. This avoids the RBD backed
persistent volumes from being forcibly removed from running pods and
being remounted read-only during these restart scenarios.

Change-Id: I7adfddf135debcc8bcaa1f93866e1a276b554c88
Closes-Bug: #1901449
Signed-off-by: Robert Church <robert.church@windriver.com>

ceph/ceph/files/ceph-preshutdown.sh

@@ -10,6 +10,27 @@ script=$(basename $0)
 # Set nullglob so wildcards will return empty string if no match
 shopt -s nullglob
 
+state=$(timeout 10 systemctl is-system-running)
+case $? in
+    124)
+        # If systemctl hangs, proceed with unmounting RBD devices to prevent
+        # shutdown hang. This maintains any existing edge-case behavior
+        logger -t ${script} "systemctl timed out. System state unknown."
+        ;;
+
+    [01])
+        # 0 - running; 1 - initializing, starting, degraded, maintenance, stopping
+        logger -t ${script} "System is $state"
+        if [ "$state" != "stopping" ]; then
+            logger -t ${script} "System is not shutting down. Leaving RBD devices mounted"
+            exit 0
+        fi
+        ;;
+esac
+
+logger -t ${script} "Unmounting RBD devices"
+
+# Unmount the RBD devices as the system is shutting down.
 for dev in /dev/rbd[0-9]*; do
     for mnt in $(mount | awk -v dev=$dev '($1 == dev) {print $3}'); do
         logger -t ${script} "Unmounting $mnt"
@@ -27,4 +48,3 @@ lsmod | grep -q '^rbd\>' && /usr/sbin/modprobe -r rbd
 lsmod | grep -q '^libceph\>' && /usr/sbin/modprobe -r libceph
 
 exit 0
-