openstack/charm-glance
Code Issues Proposed changes

Add support for cinder storage backend

Browse Source 
Create new glance_api.conf template from Ussuri release to
use default_backend and enabled_backends configuration
parameters instead of deprecated stores, default_store
parameters.
Add new config option cinder-volume-types to specify the
volume types in cinder that can be used to store glance
images.
Add logic to update cinder in glance-api configurations
if cinder-volume-service relation is joined.

Also add two flags, cinder_http_retries and
cinder_state_transition_timeout

Closes-Bug: #1905042
Change-Id: Ife649defc9b765b433d7973ab31778f9cb1efdd9
This commit is contained in:
Hemanth Nakkina 2021-10-21 15:11:12 +05:30
parent 3c32cab855
commit c2f877a7d4
13 changed files with 524 additions and 6 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

28
README.md
View File

@ -28,7 +28,7 @@ pool type' and 'Ceph backed storage' for more information.
## Deployment
This section includes four different deployment scenarios (with their
This section includes five different deployment scenarios (with their
respective backends). Each scenario requires these applications to be present:
keystone, nova-cloud-controller, nova-compute, and a cloud database.
@ -49,7 +49,7 @@ Ceph cluster via the ceph-mon charm:
juju deploy --to lxd:1 glance
juju add-relation glance:ceph ceph-mon:client
Proceed with a group of commands common to all three scenarios:
Proceed with a group of commands common to all four scenarios:
juju add-relation glance:identity-service keystone:identity-service
juju add-relation glance:image-service nova-cloud-controller:image-service
@ -116,6 +116,25 @@ S3 server information can be passed via charm config options:
Enabling S3 backend overrides `expose-image-locations` as false not to
expose S3 credentials through Glance API.
### Cinder LVM-backed storage
Glance can also use Cinder LVM-backed storage as its storage backend, the
configuration option cinder-volume-types can be used to specify the volume
types in Cinder. The steps below assume a pre-existing Cinder LVM-backed
deployment (see the [cinder][cinder-charm] and [cinder-lvm][cinder-lvm-charm]
charms).
Here, Glance is deployed to machine '1' and related to Cinder:
juju deploy --to 1 glance
juju add-relation cinder:image-service glance:image-service
juju add-relation cinder:cinder-volume-service glance:cinder-volume-service
Proceed with the common group of commands from the Ceph scenario.
> **Note**: To support Cinder LVM-backed storage, Glance unit should be deployed
on a baremetal machine or a virtual machine.
### Local storage
Glance can simply use the storage available on the application unit's machine
@ -133,6 +152,7 @@ Multiple storage backend support allows for one backend of each type:
* Ceph
* Swift (includes Ceph RADOS Gateway)
* S3
* Cinder
* local
With multiple backends configured, the cloud operator can specify, at image
@ -142,7 +162,7 @@ using the `--store` option to the `glance` CLI client:
glance image-create --store <backend-name> ...
The default order of precedence is given by the following backend names:
'ceph', 'swift', 's3', and then 'local'.
'ceph', 'swift', 's3', 'cinder', and then 'local'.
> **Important**: The backend name of 'swift' denotes both object storage
solutions: Swift and Ceph RADOS Gateway. Only one of these solutions can
@ -359,3 +379,5 @@ Please report bugs on [Launchpad][lp-bugs-charm-glance].
[ceph-bluestore-compression]: https://docs.ceph.com/en/latest/rados/configuration/bluestore-config-ref/#inline-compression
[upstream-glance]: https://docs.openstack.org/glance/latest/
[juju-docs-actions]: https://jaas.ai/docs/actions
[cinder-charm]: https://jaas.ai/cinder
[cinder-lvm-charm]: https://jaas.ai/cinder-lvm

18
config.yaml
View File

@ -213,6 +213,24 @@ options:
Device class from CRUSH map to use for placement groups for
erasure profile - valid values: ssd, hdd or nvme (or leave
unset to not use a device class).
cinder-volume-types:
type: string
default:
description: |
Comma separated list of cinder volume_types that can be configured
as cinder storage backends. The first one in the list will be
configured as the default backend.
cinder-http-retries:
type: int
default: 3
description: |
Number of cinderclient retries on failed http calls.
cinder-state-transition-timeout:
type: int
default: 30
description: |
Time period of time in seconds to wait for a cinder volume transition
to complete.
worker-multiplier:
type: float
default:

53
hooks/glance_contexts.py
View File

@ -31,7 +31,8 @@ from charmhelpers.core.hookenv import (
from charmhelpers.contrib.openstack.context import (
OSContextGenerator,
ApacheSSLContext as SSLContext,
BindHostContext
BindHostContext,
VolumeAPIContext,
)
from charmhelpers.contrib.hahelpers.cluster import (
@ -275,6 +276,13 @@ class MultiBackendContext(OSContextGenerator):
}
return ctx
def _get_cinder_config(self):
cinder_ctx = CinderStoreContext()()
if not cinder_ctx or cinder_ctx.get("cinder_store", False) is False:
return
return cinder_ctx
def __call__(self):
ctxt = {
"enabled_backend_configs": {},
@ -310,11 +318,54 @@ class MultiBackendContext(OSContextGenerator):
if not ctxt["default_store_backend"]:
ctxt["default_store_backend"] = "s3"
cinder_ctx = self._get_cinder_config()
if cinder_ctx:
cinder_volume_types = config('cinder-volume-types')
volume_types_str = cinder_volume_types or 'cinder'
volume_types = volume_types_str.split(',')
default_backend = volume_types[0]
for volume_type in volume_types:
backends.append(volume_type+':cinder')
# Add backend cinder_volume_type if cinder-volume-types configured
# In case cinder-volume-types not configured in charm, glance-api
# backend cinder_volume_type should be left blank so that glance
# creates volume in cinder without specifying any volume type.
if cinder_volume_types:
for volume_type in volume_types:
ctxt['enabled_backend_configs'][volume_type] = {
'cinder_volume_type': volume_type,
'cinder_http_retries': config('cinder-http-retries'),
'cinder_state_transition_timeout': config(
'cinder-state-transition-timeout'),
}
else:
# default cinder volume type cinder
ctxt['enabled_backend_configs']['cinder'] = {
'cinder_http_retries': config('cinder-http-retries'),
'cinder_state_transition_timeout': config(
'cinder-state-transition-timeout'),
}
# Add internal endpoints if use-internal-endpoints set to true
if config('use-internal-endpoints'):
vol_api_ctxt = VolumeAPIContext('glance-common')()
volume_catalog_info = vol_api_ctxt['volume_catalog_info']
for volume_type in volume_types:
if 'volume_type' not in ctxt['enabled_backend_configs']:
ctxt['enabled_backend_configs'][volume_type] = {}
ctxt['enabled_backend_configs'][volume_type].update(
{'cinder_catalog_info': volume_catalog_info})
if not ctxt["default_store_backend"]:
ctxt["default_store_backend"] = default_backend
if local_fs and not ctxt["default_store_backend"]:
ctxt["default_store_backend"] = "local"
if len(backends) > 0:
ctxt["enabled_backends"] = ", ".join(backends)
return ctxt

14
hooks/glance_relations.py
View File

@ -78,9 +78,11 @@ from charmhelpers.core.hookenv import (
)
from charmhelpers.core.host import (
is_container,
# restart_on_change,
service_reload,
service_restart,
service_start,
service_stop,
)
from charmhelpers.fetch import (
@ -188,6 +190,12 @@ def install_hook():
'glance',
restart_handler=lambda: service_restart('glance-api'))
# Make sure iscsid has a unique InitiatorName by starting iscsid
# and invoking /lib/open-iscsi/startup-checks.sh indirectly as
# ExecStartPre script of it
if not is_container():
service_start('iscsid')
@hooks.hook('shared-db-relation-joined')
def db_joined():
@ -494,6 +502,12 @@ def upgrade_charm():
'glance',
restart_handler=lambda: service_restart('glance-api'))
# Make sure iscsid has a unique InitiatorName by starting iscsid
# and invoking /lib/open-iscsi/startup-checks.sh indirectly as
# ExecStartPre script of it
if not is_container():
service_start('iscsid')
@hooks.hook('ha-relation-joined')
def ha_relation_joined(relation_id=None):

6
hooks/glance_utils.py
View File

@ -44,6 +44,7 @@ from charmhelpers.core.hookenv import (
from charmhelpers.core.host import (
CompareHostReleases,
is_container,
lsb_release,
mkdir,
service_stop,
@ -519,6 +520,11 @@ def check_optional_config_and_relations(configs):
except ValueError as e:
return ('blocked', 'Invalid external S3 config: {}'.format(str(e)))
if relation_ids('cinder-volume-service') and is_container():
return (
'blocked',
'Glance with cinder storage backend is not supported on container')
# return 'unknown' as the lowest priority to not clobber an existing
# status.
return "unknown", ""

86
templates/ussuri/glance-api.conf Normal file
View File

@ -0,0 +1,86 @@
[DEFAULT]
verbose = {{ verbose }}
use_syslog = {{ use_syslog }}
debug = {{ debug }}
workers = {{ workers }}
bind_host = {{ bind_host }}
{% if ext -%}
bind_port = {{ ext }}
{% elif bind_port -%}
bind_port = {{ bind_port }}
{% else -%}
bind_port = 9292
{% endif -%}
{% if transport_url %}
transport_url = {{ transport_url }}
{% endif %}
log_file = /var/log/glance/api.log
backlog = 4096
{% if expose_image_locations -%}
show_multiple_locations = {{ expose_image_locations }}
show_image_direct_url = {{ expose_image_locations }}
{% endif -%}
{% if api_config_flags -%}
{% for key, value in api_config_flags.items() -%}
{{ key }} = {{ value }}
{% endfor -%}
{% endif -%}
delayed_delete = False
scrub_time = 43200
scrubber_datadir = /var/lib/glance/scrubber
image_cache_dir = /var/lib/glance/image-cache/
db_enforce_mysql_charset = False
{% if image_size_cap -%}
image_size_cap = {{ image_size_cap }}
{% endif -%}
{% if enabled_backends %}
enabled_backends = {{ enabled_backends }}
{% endif %}
[glance_store]
{% if default_store_backend %}
default_backend = {{ default_store_backend }}
{% endif %}
[image_format]
disk_formats = {{ disk_formats }}
{% if container_formats -%}
container_formats = {{ container_formats }}
{% endif -%}
{% include "section-keystone-authtoken-v3only" %}
{% if auth_host -%}
[paste_deploy]
flavor = keystone
{% endif %}
[barbican]
auth_endpoint = {{ service_protocol }}://{{ service_host }}:{{ service_port }}/v3
{% include "parts/section-database" %}
{% include "section-oslo-messaging-rabbit" %}
{% include "section-oslo-notifications" %}
{% include "section-oslo-middleware" %}
{% include "parts/section-storage" %}
{% for name, cfg in enabled_backend_configs.items() %}
[{{name}}]
{% for key, val in cfg.items() -%}
{{ key }} = {{ val }}
{% endfor -%}
{% endfor%}
{% include "parts/section-image-import" %}

47
tests/bundles/focal-xena.yaml
View File

@ -25,6 +25,7 @@ machines:
'13':
'14':
'15':
'16':
applications:
@ -34,6 +35,9 @@ applications:
glance-mysql-router:
charm: ch:mysql-router
channel: latest/edge
cinder-mysql-router:
charm: ch:mysql-router
channel: latest/edge
mysql-innodb-cluster:
charm: ch:mysql-innodb-cluster
@ -115,6 +119,31 @@ applications:
to:
- '14'
cinder:
expose: True
charm: ch:cinder
num_units: 1
storage:
block-devices: '10G'
options:
openstack-origin: *openstack-origin
glance-api-version: 2
block-device: None
to:
- '16'
channel: latest/edge
cinder-lvm:
charm: ch:/cinder-lvm
options:
block-device: '/tmp/vol1|4G'
alias: zaza-lvm
overwrite: "true"
ephemeral-unmount: /mnt
allocation-type: auto
config-flags: target_helper=lioadm
channel: latest/edge
relations:
- - 'keystone:shared-db'
- 'keystone-mysql-router:shared-db'
@ -144,3 +173,21 @@ relations:
- 'keystone:identity-service'
- - 'ceph-radosgw:object-store'
- 'glance:object-store'
- - 'cinder:shared-db'
- 'cinder-mysql-router:shared-db'
- - 'cinder-mysql-router:db-router'
- 'mysql-innodb-cluster:db-router'
- - 'cinder:identity-service'
- 'keystone:identity-service'
- - 'cinder:amqp'
- 'rabbitmq-server:amqp'
- - 'cinder:image-service'
- 'glance:image-service'
- - 'cinder:cinder-volume-service'
- 'glance:cinder-volume-service'
- - 'cinder-lvm:storage-backend'
- 'cinder:storage-backend'

47
tests/bundles/focal-yoga.yaml
View File

@ -25,6 +25,7 @@ machines:
'13':
'14':
'15':
'16':
applications:
@ -34,6 +35,9 @@ applications:
glance-mysql-router:
charm: ch:mysql-router
channel: latest/edge
cinder-mysql-router:
charm: ch:mysql-router
channel: latest/edge
mysql-innodb-cluster:
charm: ch:mysql-innodb-cluster
@ -115,6 +119,31 @@ applications:
to:
- '14'
cinder:
expose: True
charm: ch:cinder
num_units: 1
storage:
block-devices: '10G'
options:
openstack-origin: *openstack-origin
glance-api-version: 2
block-device: None
to:
- '16'
channel: latest/edge
cinder-lvm:
charm: ch:cinder-lvm
options:
block-device: '/tmp/vol1|4G'
alias: zaza-lvm
overwrite: "true"
ephemeral-unmount: /mnt
allocation-type: auto
config-flags: target_helper=lioadm
channel: latest/edge
relations:
- - 'keystone:shared-db'
- 'keystone-mysql-router:shared-db'
@ -144,3 +173,21 @@ relations:
- 'keystone:identity-service'
- - 'ceph-radosgw:object-store'
- 'glance:object-store'
- - 'cinder:shared-db'
- 'cinder-mysql-router:shared-db'
- - 'cinder-mysql-router:db-router'
- 'mysql-innodb-cluster:db-router'
- - 'cinder:identity-service'
- 'keystone:identity-service'
- - 'cinder:amqp'
- 'rabbitmq-server:amqp'
- - 'cinder:image-service'
- 'glance:image-service'
- - 'cinder:cinder-volume-service'
- 'glance:cinder-volume-service'
- - 'cinder-lvm:storage-backend'
- 'cinder:storage-backend'

47
tests/bundles/impish-xena.yaml
View File

@ -25,6 +25,7 @@ machines:
'13':
'14':
'15':
'16':
applications:
@ -34,6 +35,9 @@ applications:
glance-mysql-router:
charm: ch:mysql-router
channel: latest/edge
cinder-mysql-router:
charm: ch:mysql-router
channel: latest/edge
mysql-innodb-cluster:
charm: ch:mysql-innodb-cluster
@ -115,6 +119,31 @@ applications:
to:
- '14'
cinder:
expose: True
charm: ch:cinder
num_units: 1
storage:
block-devices: '10G'
options:
openstack-origin: *openstack-origin
glance-api-version: 2
block-device: None
to:
- '16'
channel: latest/edge
cinder-lvm:
charm: ch:cinder-lvm
options:
block-device: '/tmp/vol1|4G'
alias: zaza-lvm
overwrite: "true"
ephemeral-unmount: /mnt
allocation-type: auto
config-flags: target_helper=lioadm
channel: latest/edge
relations:
- - 'keystone:shared-db'
- 'keystone-mysql-router:shared-db'
@ -144,3 +173,21 @@ relations:
- 'keystone:identity-service'
- - 'ceph-radosgw:object-store'
- 'glance:object-store'
- - 'cinder:shared-db'
- 'cinder-mysql-router:shared-db'
- - 'cinder-mysql-router:db-router'
- 'mysql-innodb-cluster:db-router'
- - 'cinder:identity-service'
- 'keystone:identity-service'
- - 'cinder:amqp'
- 'rabbitmq-server:amqp'
- - 'cinder:image-service'
- 'glance:image-service'
- - 'cinder:cinder-volume-service'
- 'glance:cinder-volume-service'
- - 'cinder-lvm:storage-backend'
- 'cinder:storage-backend'

47
tests/bundles/jammy-yoga.yaml
View File

@ -25,6 +25,7 @@ machines:
'13':
'14':
'15':
'16':
applications:
@ -34,6 +35,9 @@ applications:
glance-mysql-router:
charm: ch:mysql-router
channel: latest/edge
cinder-mysql-router:
charm: ch:mysql-router
channel: latest/edge
mysql-innodb-cluster:
charm: ch:mysql-innodb-cluster
@ -115,6 +119,31 @@ applications:
to:
- '14'
cinder:
expose: True
charm: ch:cinder
num_units: 1
storage:
block-devices: '10G'
options:
openstack-origin: *openstack-origin
glance-api-version: 2
block-device: None
to:
- '16'
channel: latest/edge
cinder-lvm:
charm: ch:cinder-lvm
options:
block-device: '/tmp/vol1|4G'
alias: zaza-lvm
overwrite: "true"
ephemeral-unmount: /mnt
allocation-type: auto
config-flags: target_helper=lioadm
channel: latest/edge
relations:
- - 'keystone:shared-db'
- 'keystone-mysql-router:shared-db'
@ -144,3 +173,21 @@ relations:
- 'keystone:identity-service'
- - 'ceph-radosgw:object-store'
- 'glance:object-store'
- - 'cinder:shared-db'
- 'cinder-mysql-router:shared-db'
- - 'cinder-mysql-router:db-router'
- 'mysql-innodb-cluster:db-router'
- - 'cinder:identity-service'
- 'keystone:identity-service'
- - 'cinder:amqp'
- 'rabbitmq-server:amqp'
- - 'cinder:image-service'
- 'glance:image-service'
- - 'cinder:cinder-volume-service'
- 'glance:cinder-volume-service'
- - 'cinder-lvm:storage-backend'
- 'cinder:storage-backend'

1
tests/tests.yaml
View File

@ -28,6 +28,7 @@ tests:
- zaza.openstack.charm_tests.glance.tests.GlanceTest
- zaza.openstack.charm_tests.glance.tests.GlanceCephRGWBackendTest
- zaza.openstack.charm_tests.glance.tests.GlanceExternalS3Test
- zaza.openstack.charm_tests.glance.tests.GlanceCinderBackendTest
- zaza.openstack.charm_tests.policyd.tests.GlanceTests
- zaza.openstack.charm_tests.ceph.tests.CheckPoolTypes
- zaza.openstack.charm_tests.ceph.tests.BlueStoreCompressionCharmOperation

135
unit_tests/test_glance_contexts.py
View File

@ -294,8 +294,15 @@ class TestGlanceContexts(CharmTestCase):
})
def test_multi_backend_with_swift(self):
# return relation_ids only for swift but not for cinder
def _relation_ids(*args, **kwargs):
if args[0] == 'object-store':
return ["object-store:0"]
return []
self.maxDiff = None
self.relation_ids.return_value = ["object-store:0"]
self.relation_ids.side_effect = _relation_ids
self.is_relation_made.return_value = False
data_dir = '/some/data/dir'
conf_dict = {
@ -321,6 +328,78 @@ class TestGlanceContexts(CharmTestCase):
'default_store_backend': 'swift',
})
def test_multi_backend_with_cinder(self):
# return relation_ids only for cinder but not for swift
def _relation_ids(*args, **kwargs):
if args[0] == 'cinder-volume-service':
return ["cinder-volume-service:0"]
return []
self.maxDiff = None
self.relation_ids.side_effect = _relation_ids
self.is_relation_made.return_value = False
data_dir = '/some/data/dir'
conf_dict = {
'filesystem-store-datadir': data_dir,
'cinder-http-retries': 3,
'cinder-state-transition-timeout': 30,
}
self.config.side_effect = lambda x: conf_dict.get(x)
self.assertEqual(
contexts.MultiBackendContext()(),
{
'enabled_backend_configs': {
'local': {
'filesystem_store_datadir': data_dir,
'store_description': 'Local filesystem store',
},
'cinder': {
'cinder_http_retries': 3,
'cinder_state_transition_timeout': 30,
}
},
'enabled_backends': 'local:file, cinder:cinder',
'default_store_backend': 'cinder',
})
def test_multi_backend_with_cinder_volume_types_defined(self):
# return relation_ids only for cinder but not for swift
def _relation_ids(*args, **kwargs):
if args[0] == 'cinder-volume-service':
return ["cinder-volume-service:0"]
return []
self.maxDiff = None
self.relation_ids.side_effect = _relation_ids
self.is_relation_made.return_value = False
data_dir = '/some/data/dir'
conf_dict = {
'filesystem-store-datadir': data_dir,
'cinder-volume-types': 'volume-type-test',
'cinder-http-retries': 3,
'cinder-state-transition-timeout': 30,
}
self.config.side_effect = lambda x: conf_dict.get(x)
self.assertEqual(
contexts.MultiBackendContext()(),
{
'enabled_backend_configs': {
'local': {
'filesystem_store_datadir': data_dir,
'store_description': 'Local filesystem store',
},
'volume-type-test': {
'cinder_volume_type': 'volume-type-test',
'cinder_http_retries': 3,
'cinder_state_transition_timeout': 30,
}
},
'enabled_backends': 'local:file, volume-type-test:cinder',
'default_store_backend': 'volume-type-test',
})
def test_multi_backend_with_ceph_no_swift(self):
self.maxDiff = None
self.relation_ids.return_value = []
@ -354,8 +433,15 @@ class TestGlanceContexts(CharmTestCase):
})
def test_multi_backend_with_ceph_and_swift(self):
# return relation_ids only for swift but not for cinder
def _relation_ids(*args, **kwargs):
if args[0] == 'object-store':
return ["object-store:0"]
return []
self.maxDiff = None
self.relation_ids.return_value = ["object-store:0"]
self.relation_ids.side_effect = _relation_ids
self.is_relation_made.return_value = True
service = 'glance'
self.service_name.return_value = service
@ -391,6 +477,51 @@ class TestGlanceContexts(CharmTestCase):
'default_store_backend': 'ceph',
})
def test_multi_backend_with_ceph_and_cinder(self):
# return relation_ids only for cinder but not for swift
def _relation_ids(*args, **kwargs):
if args[0] == 'cinder-volume-service':
return ["cinder-volume-service:0"]
return []
self.maxDiff = None
self.relation_ids.side_effect = _relation_ids
self.is_relation_made.return_value = True
service = 'glance'
self.service_name.return_value = service
data_dir = '/some/data/dir'
conf_dict = {
'filesystem-store-datadir': data_dir,
'rbd-pool-name': 'mypool',
'cinder-http-retries': 3,
'cinder-state-transition-timeout': 30,
}
self.config.side_effect = lambda x: conf_dict.get(x)
self.assertEqual(
contexts.MultiBackendContext()(),
{
'enabled_backend_configs': {
'local': {
'filesystem_store_datadir': data_dir,
'store_description': 'Local filesystem store',
},
'ceph': {
"rbd_store_chunk_size": 8,
"rbd_store_pool": 'mypool',
"rbd_store_user": service,
"rados_connect_timeout": 0,
"rbd_store_ceph_conf": "/etc/ceph/ceph.conf",
},
'cinder': {
'cinder_http_retries': 3,
'cinder_state_transition_timeout': 30,
}
},
'enabled_backends': 'local:file, ceph:rbd, cinder:cinder',
'default_store_backend': 'ceph',
})
def test_multi_backend_with_external_s3(self):
self.maxDiff = None
self.os_release.return_value = 'ussuri'

1
unit_tests/test_glance_relations.py
View File

@ -72,6 +72,7 @@ TO_PATCH = [
'service_reload',
'service_stop',
'service_restart',
'service_start',
# charmhelpers.contrib.openstack.utils
'configure_installation_source',
'is_db_initialised',