openstack/openstack-helm-infra
Code Issues Proposed changes

Added detailed FiXME for ks-service script bug and code changes in [0]

Browse Source 
On somewhat rare occasions the openstack service list call fails with
a connection aborted OSError 104 ECONNRESET. During an upgrade this failure
causes the script to think that the service it is checking for does not
exist and therefore it recreates the script. In turn this causes further
issues when other services try to use this duplicate service.
This is a temporary change in order to alliviate the issue while the root
cause is investigated.

[0] https://review.opendev.org/c/openstack/openstack-helm-infra/+/772416

Change-Id: Id0971a95eb54eca9486a9811f7ec6f603a007cbb
This commit is contained in:
Neely, Travis (tn720x) 2021-01-29 10:30:52 -06:00
parent ff3b0aa9e0
commit 69c525d791
2 changed files with 16 additions and 4 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
helm-toolkit/Chart.yaml
View File

@ -15,7 +15,7 @@ apiVersion: v1
appVersion: v1.0.0
description: OpenStack-Helm Helm-Toolkit
name: helm-toolkit
version: 0.2.3
version: 0.2.4
home: https://docs.openstack.org/openstack-helm
icon: https://www.openstack.org/themes/openstack/images/project-mascots/OpenStack-Helm/OpenStack_Project_OpenStackHelm_vertical.png
sources:

18
helm-toolkit/templates/scripts/_ks-service.sh.tpl
View File

@ -37,9 +37,21 @@ OS_SERVICE_DESC="${OS_REGION_NAME}: ${OS_SERVICE_NAME} (${OS_SERVICE_TYPE}) serv
# Get Service ID if it exists
unset OS_SERVICE_ID
# If OS_SERVICE_ID is blank (due to the service not being ready yet)
# then wait a few seconds to give it additional time to be ready
# and try again
# FIXME - There seems to be an issue once in a while where the
# openstack service list fails and encounters an error message such as:
# Unable to establish connection to
# https://keystone-api.openstack.svc.cluster.local:5000/v3/auth/tokens:
# ('Connection aborted.', OSError("(104, 'ECONNRESET')",))
# During an upgrade scenario, this would cause the OS_SERVICE_ID to be blank
# and it would attempt to create a new service when it was not needed.
# This duplciate service would sometimes be used by other services such as
# Horizon and would give an 'Invalid Service Catalog' error.
# This loop allows for a 'retry' of the openstack service list in an
# attempt to get the service list as expected if it does ecounter an error.
# This loop and recheck can be reverted once the underlying issue is addressed.
# If OS_SERVICE_ID is blank then wait a few seconds to give it
# additional time and try again
for i in {1...3}
do
OS_SERVICE_ID=$( openstack service list -f csv --quote none | \