Add backend support for orchestrated prestaging for one or a group of
subclouds serially or in parallel.
Similar to other types of orchestration; prestage strategy can be
created, applied, aborted and deleted. When a prestage-strategy is in
progress, no other strategy can be created.
Orchestrated subcloud prestaging will also consist of 4 states:
- precheck – prestage conditions verification
- preparing packages – prestage packages generation
- prestaging packages – upgrade packages prestaging
- prestaging images – upgrade container images prestaging
The prestaging progress of each subcloud in the batch can be monitored
via dcmanager strategy-step list.
Unlike other types of orchestration:
- sysadmin-password is required as a mandatory input
- interaction with vim is not required
Test Plan:
PASS:
- API-level lifecycle commands for dcmanger prestage-strategy
- Test via REST API (see curl samples below)
- Test via CLI commands
- Create a prestage strategy
- For a single subcloud
- For a subcloud group
- basic validation during creation for both cases
- unmanaged/offline/invalid deploy_status
- Fail during create if subcloud offline
- Apply the strategy
- Ensure prestage operations proceed in order for each subcloud, as
per the unorchestrated case
- Test various failure scenarios:
- non AIO-SX subcloud
- unmanaged/offline/invalid deploy_status
- failure of ansible script
- Apply with various values for max parallel subclouds
- ranging from 1 to 50
- Apply strategy after unmanaging subcloud(s) in the strategy-list
- Show the strategy (in various states)
- Delete strategy
- Abort the strategy while in progress
- Result: in-progress operations run to completion, remaining
subclouds are aborted
- Regression testing of the original, unorchestrated 'dcmanager
subcloud prestage' command
- Run prestage operations with no uploaded image list file
- Issue an individual 'dcmanager subcloud prestage' command
while prestage orchestration is in progress. The command is rejected.
- Upgrade scenario with included image list file
- ensure all ansible scripts are correctly run (success case)
- ensure ansible failures are handled correctly
- Test failure of prestage prepare. The first failure should cause the
next subclouds in the strategy step to fail. We don't want to keep
retrying prestage prepare for all subclouds in the strategy list.
- Handle case where prestage prepare ansible fails. This
should only be attempted once per strategy step, i.e. we don't
want to keep retrying the failed script per subcloud.
- The issue is that the rescue deletes the
'.prestage_preparation_completed' file along with the packages
output directory. So although it is synchronized, after getting
the lock, each subcloud retries the same script
- Perform host-swact after successful upgrade
- ensure prestage preparation is replicated across to
the newly active controller
Sample Curl Commands used in testing:
APIURL=$(system oam-show | awk '/oam_floating_ip/ { print "http://["$4"]:8119/v1.0"; }')
# Token - is valid for 1 hour
TOKEN=$(openstack token issue | awk '/ id / { print $4; }')
Create prestage strategy:
curl -g -X POST $APIURL/sw-update-strategy \
-H "X-Auth-Token: ${TOKEN}" \
-H 'Content-Type: application/json' \
-d '{"type": "prestage", \
"sysadmin_password": "TGk2OW51eCo=", \
"force": "true", \
"max-parallel-subclouds": "50", \
"group": "Default"}'
Delete prestage strategy:
curl -g -X DELETE $APIURL/sw-update-strategy?type=prestage \
-H "X-Auth-Token: ${TOKEN}" \
-H 'Content-Type: application/json'
Get prestage strategy:
curl -g -X GET $APIURL/sw-update-strategy?type=prestage \
-H "X-Auth-Token: ${TOKEN}" \
-H 'Content-Type: application/json'
Apply:
curl -g -X POST $APIURL/sw-update-strategy/actions?type=prestage \
-H "X-Auth-Token: ${TOKEN}" \
-H 'Content-Type: application/json'
-d '{"action": "apply"}'
Malformed data:
curl -g -X POST $APIURL/sw-update-strategy \
-H "X-Auth-Token: ${TOKEN}" \
-H 'Content-Type: application/json' \
-d 'abc'
Missing mandatory parameter:
curl -g -X POST $APIURL/sw-update-strategy \
-H "X-Auth-Token: ${TOKEN}" \
-H 'Content-Type: application/json' \
-d '{"type": "prestage"}'
Sysadmin password not encoded:
curl -g -X POST $APIURL/sw-update-strategy \
-H "X-Auth-Token: ${TOKEN}" \
-H 'Content-Type: application/json' \
-d '{"type": "prestage", "sysadmin_password": "abc"}'
Invalid force option value:
curl -g -X POST $APIURL/sw-update-strategy \
-H "X-Auth-Token: ${TOKEN}" \
-H 'Content-Type: application/json' \
-d '{"type": "prestage", \
"sysadmin_password": "TGk2OW51eCo=", \
"force": "abc"}'
Depends-On: https://review.opendev.org/c/starlingx/ansible-playbooks/+/831208
Story: 2009799
Task: 44592
Change-Id: Ie28970e8fb1b862b5659ff264967e91ce8be2ab0
Signed-off-by: Kyle MacLeod <kyle.macleod@windriver.com>
55a86904a3