An intermittent KeyError exception was found in an execution of
activation script 06-security-kubeapi-service-parameters-apply.py,
when accessing the transition time for kubeapi-server pods.
This transition times are initialized for each host relying on a
call to k8s api ('kubectl get nodes'). If the k8s api is
unavailable, this call can fail, resulting in the values not being
initialized correctly.
Fixed this by adding robustness to the script, now throwing an
exception if the call to get the nodes doesn't succeed, and also
retrying upon exceptions for 3 times, waiting 5 seconds.
Test plan:
PASS: Perform upgrade activation (DX SystemController).
PASS: Forced the method '__register_last_transition_time' to
throw an exception in the first call, to verify the
exception handling and retry logic.
PASS: SX subcloud upgrade.
Closes-bug: 2093786
Change-Id: Id187af98053c086eb730b6d6c992c41d429139fc
Signed-off-by: Marcelo de Castro Loebens <Marcelo.DeCastroLoebens@windriver.com>
6feccede02