openstack/ironic
Code Issues Proposed changes

Remove endpoint_type from configuration

Browse Source 
python-swiftclient stopped supporting the temp url structure used when radosgw
was set as the endpoint_type in ocata, meaning only Newton and older versions
of python-swiftclient will work.  Newton is deprecated, so remove the option.

This breaks the deprecation cycle, but since it has been not working for so
long it needs to just be dropped.

Change-Id: Ibdc93b049b7e1ae34cac9e1f599786439c46a685
This commit is contained in:
Matthew Thode 2018-05-10 11:02:07 -05:00 committed by Vladyslav Drok
parent 9e63f0e97f
commit e9e4d8870c
14 changed files with 92 additions and 251 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

56
doc/source/admin/radosgw.rst
View File

@ -8,7 +8,7 @@ Overview
========
Ceph project is a powerful distributed storage system. It contains object store
and provides a RADOS Gateway Swift API which is compatible with OpenStack Swift
API. These two APIs use different formats for their temporary URLs.
API.
Ironic added support for RADOS Gateway temporary URL in the Mitaka release.
@ -17,56 +17,56 @@ Configure Ironic and Glance with RADOS Gateway
#. Install Ceph storage with RADOS Gateway. See `Ceph documentation <http://docs.ceph.com/docs>`_.
#. Create RADOS Gateway credentials for Glance by executing the following
commands on the RADOS Gateway admin host::
#. Configure RADOS Gateway to use keystone for authentication. See
`Integrating with OpenStack Keystone <http://docs.ceph.com/docs/master/radosgw/keystone/>`_
sudo radosgw-admin user create --uid="GLANCE_USERNAME" --display-name="User for Glance"
#. Register RADOS Gateway endpoint in the keystone catalog, with the same
format swift uses, as the ``object-store`` service. URL example:
sudo radosgw-admin subuser create --uid=GLANCE_USERNAME --subuser=GLANCE_USERNAME:swift --access=full
``http://rados.example.com:8080/swift/v1/AUTH_$(project_id)s``.
sudo radosgw-admin key create --subuser=GLANCE_USERNAME:swift --key-type=swift --secret=STORE_KEY
In the ceph configuration, make sure radosgw is configured with the
following value::
sudo radosgw-admin user modify --uid=GLANCE_USERNAME --temp-url-key=TEMP_URL_KEY
Replace GLANCE_USERNAME with a user name for Glance access, and replace
STORE_KEY and TEMP_URL_KEY with suitable keys.
Note: Do not use "--gen-secret" CLI parameter because it will cause the
"radosgw-admin" utility to generate keys with slash symbols which do not
work with Glance.
rgw swift account in url = True
#. Configure Glance API service for RADOS Swift API as backend. Edit the
configuration file for the Glance API service (is typically located at
``/etc/glance/glance-api.conf``). Replace RADOS_IP and PORT with the IP/port
of the RADOS Gateway API service::
``/etc/glance/glance-api.conf``)::
[glance_store]
stores = file, http, swift
default_store = swift
swift_store_auth_version = 1
swift_store_auth_address = http://RADOS_IP:PORT/auth/1.0
swift_store_user = GLANCE_USERNAME:swift
swift_store_key = STORE_KEY
default_swift_reference=ref1
swift_store_config_file=/etc/glance/glance-swift-creds.conf
swift_store_container = glance
swift_store_create_container_on_put = True
In the file referenced in ``swift_store_config_file`` option, add the
following::
[ref1]
user = <service project>:<service user name>
key = <service user password>
user_domain_id = default
project_domain_id = default
auth_version = 3
auth_address = http://keystone.example.com/identity
Values for user and key options correspond to keystone credentials for
RADOS Gateway service user.
Note: RADOS Gateway uses FastCGI protocol for interacting with HTTP server.
Read your HTTP server documentation if you want to enable HTTPS support.
#. Restart Glance API service and upload all needed images.
#. Change Ironic configuration file on the conductor host(s) as follows::
#. If you're using custom container name in RADOS, change Ironic configuration
file on the conductor host(s) as follows::
[glance]
swift_container = glance
swift_api_version = v1
swift_endpoint_url = http://RADOS_IP:PORT
swift_temp_url_key = TEMP_URL_KEY
[deploy]
object_store_endpoint_type = radosgw
#. Restart Ironic conductor service(s).

1
doc/source/admin/report.txt
View File

@ -382,7 +382,6 @@ glance:
swift_temp_url_duration = 3600
swift_temp_url_expected_download_start_delay = 0
swift_temp_url_key = ***
temp_url_endpoint_type = swift
timeout = None
ilo:

5
doc/source/install/configdrive.rst
View File

@ -61,8 +61,8 @@ Configuration drive storage in an object store
Under normal circumstances, the configuration drive can be stored in the
Bare Metal service when the size is less than 64KB. Optionally, if the size
is larger than 64KB there is support to store it in swift or radosgw backed
object store. Both swift and radosgw use swift-style APIs.
is larger than 64KB there is support to store it in a swift endpoint. Both
swift and radosgw use swift-style APIs.
The following option in ``/etc/ironic/ironic.conf`` enables swift as an object
store backend to store config drive. This uses the Identity service to
@ -83,7 +83,6 @@ instead. ::
...
configdrive_use_object_store = True
object_store_endpoint_type = radosgw
[swift]
...

1
doc/source/install/configure-glance-swift.rst
View File

@ -67,7 +67,6 @@ and Object Storage service as described below.
[glance]
temp_url_endpoint_type = swift
swift_endpoint_url = http://openstack/swift
swift_account = AUTH_bc39f1d9dcf9486899088007789ae643
swift_container = glance

53
ironic/common/glance_service/v2/image_service.py
View File

@ -149,40 +149,26 @@ class GlanceImageService(base_image_service.BaseImageService,
endpoint_url = re.sub('/v1/AUTH_[^/]+/?$', '', endpoint_url)
key = CONF.glance.swift_temp_url_key
if CONF.deploy.object_store_endpoint_type == 'radosgw':
chunks = urlparse.urlsplit(CONF.glance.swift_endpoint_url)
if not chunks.path:
endpoint_url = urlparse.urljoin(
endpoint_url, 'swift')
elif chunks.path != '/swift':
raise exc.InvalidParameterValue(
_('Swift endpoint URL should only contain scheme, '
'hostname, optional port and optional /swift path '
'without trailing slash; provided value is: %s')
% endpoint_url)
account = CONF.glance.swift_account
if not account:
swift_session = swift.get_swift_session()
auth_ref = swift_session.auth.get_auth_ref(swift_session)
account = 'AUTH_%s' % auth_ref.project_id
template = '/{api_version}/{container}/{object_id}'
else:
account = CONF.glance.swift_account
if not account:
swift_session = swift.get_swift_session()
auth_ref = swift_session.auth.get_auth_ref(swift_session)
account = 'AUTH_%s' % auth_ref.project_id
if not key:
swift_api = swift.SwiftAPI()
key_header = 'x-account-meta-temp-url-key'
key = swift_api.connection.head_account().get(key_header)
if not key:
swift_api = swift.SwiftAPI()
key_header = 'x-account-meta-temp-url-key'
key = swift_api.connection.head_account().get(key_header)
if not key:
raise exc.MissingParameterValue(_(
'Swift temporary URLs require a shared secret to be '
'created. You must provide "swift_temp_url_key" as a '
'config option or pre-generate the key on the project '
'used to access Swift.'))
if not key:
raise exc.MissingParameterValue(_(
'Swift temporary URLs require a shared secret to be '
'created. You must provide "swift_temp_url_key" as a '
'config option or pre-generate the key on the project '
'used to access Swift.'))
url_fragments['account'] = account
template = '/{api_version}/{account}/{container}/{object_id}'
url_fragments['account'] = account
template = '/{api_version}/{account}/{container}/{object_id}'
url_path = template.format(**url_fragments)
@ -197,11 +183,6 @@ class GlanceImageService(base_image_service.BaseImageService,
def _validate_temp_url_config(self):
"""Validate the required settings for a temporary URL."""
if (not CONF.glance.swift_temp_url_key
and CONF.deploy.object_store_endpoint_type != 'swift'):
raise exc.MissingParameterValue(_(
'Swift temporary URLs require a shared secret to be created. '
'You must provide "swift_temp_url_key" as a config option.'))
if (CONF.glance.swift_temp_url_duration
< CONF.glance.swift_temp_url_expected_download_start_delay):
raise exc.InvalidParameterValue(_(

62
ironic/common/swift.py
View File

@ -44,45 +44,39 @@ class SwiftAPI(object):
"""Underlying Swift connection object."""
def __init__(self):
"""Initialize the connection with swift or radosgw
"""Initialize the connection with swift
:raises: ConfigInvalid if required keystone authorization credentials
with swift are missing.
"""
params = {'retries': CONF.swift.swift_max_retries}
if CONF.deploy.object_store_endpoint_type == 'radosgw':
params.update({'authurl': CONF.swift.auth_url,
'user': CONF.swift.username,
'key': CONF.swift.password})
else:
# NOTE(pas-ha) swiftclient still (as of 3.3.0) does not use
# (adapter-based) SessionClient, and uses the passed in session
# only to resolve endpoint and get a token,
# but not to make further requests to Swift itself (LP 1736135).
# Thus we need to deconstruct back all the adapter- and
# session-related args as loaded by keystoneauth from config
# to pass them to the client explicitly.
# TODO(pas-ha) re-write this when swiftclient is brought on par
# with other OS clients re auth plugins, sessions and adapters
# support.
# TODO(pas-ha) pass the context here and use token from context
# with service auth
params['session'] = session = get_swift_session()
adapter = keystone.get_adapter('swift', session=session)
params['os_options'] = {
'object_storage_url': adapter.get_endpoint()}
# deconstruct back session-related options
params['timeout'] = session.timeout
if session.verify is False:
params['insecure'] = True
elif isinstance(session.verify, six.string_types):
params['cacert'] = session.verify
if session.cert:
# NOTE(pas-ha) although setting cert as path to single file
# with both client cert and key is supported by Session,
# keystoneauth loading always sets the session.cert
# as tuple of cert and key.
params['cert'], params['cert_key'] = session.cert
# NOTE(pas-ha) swiftclient still (as of 3.3.0) does not use
# (adapter-based) SessionClient, and uses the passed in session
# only to resolve endpoint and get a token,
# but not to make further requests to Swift itself (LP 1736135).
# Thus we need to deconstruct back all the adapter- and
# session-related args as loaded by keystoneauth from config
# to pass them to the client explicitly.
# TODO(pas-ha) re-write this when swiftclient is brought on par
# with other OS clients re auth plugins, sessions and adapters
# support.
# TODO(pas-ha) pass the context here and use token from context
# with service auth
params['session'] = session = get_swift_session()
adapter = keystone.get_adapter('swift', session=session)
params['os_options'] = {'object_storage_url': adapter.get_endpoint()}
# deconstruct back session-related options
params['timeout'] = session.timeout
if session.verify is False:
params['insecure'] = True
elif isinstance(session.verify, six.string_types):
params['cacert'] = session.verify
if session.cert:
# NOTE(pas-ha) although setting cert as path to single file
# with both client cert and key is supported by Session,
# keystoneauth loading always sets the session.cert
# as tuple of cert and key.
params['cert'], params['cert_key'] = session.cert
self.connection = swift_client.Connection(**params)

14
ironic/conductor/base_manager.py
View File

@ -87,8 +87,6 @@ class BaseConductorManager(object):
:raises: DriverLoadError if an enabled driver cannot be loaded.
:raises: DriverNameConflict if a classic driver and a dynamic driver
are both enabled and have the same name.
:raises: ConfigInvalid if required config options for connection with
radosgw are missing while storing config drive.
"""
if self._started:
raise RuntimeError(_('Attempt to start an already running '
@ -151,18 +149,6 @@ class BaseConductorManager(object):
self._collect_periodic_tasks(admin_context)
# Check for required config options if object_store_endpoint_type is
# radosgw
if (CONF.deploy.configdrive_use_object_store
and CONF.deploy.object_store_endpoint_type == "radosgw"):
if (None in (CONF.swift.auth_url, CONF.swift.username,
CONF.swift.password)):
msg = _("Parameters missing to make a connection with "
"radosgw. Ensure that [swift]/auth_url, "
"[swift]/username, and [swift]/password are all "
"configured.")
raise exception.ConfigInvalid(msg)
# clear all target_power_state with locks by this conductor
self.dbapi.clear_node_target_power_state(self.host)
# clear all locks held by this conductor before registering

4
ironic/conductor/manager.py
View File

@ -3191,14 +3191,14 @@ def _get_configdrive_obj_name(node):
def _store_configdrive(node, configdrive):
"""Handle the storage of the config drive.
If configured, the config drive data are uploaded to swift or radosgw.
If configured, the config drive data are uploaded to a swift endpoint.
The Node's instance_info is updated to include either the temporary
Swift URL from the upload, or if no upload, the actual config drive data.
:param node: an Ironic node object.
:param configdrive: A gzipped and base64 encoded configdrive.
:raises: SwiftOperationError if an error occur when uploading the
config drive to swift or radosgw.
config drive to the swift endpoint.
:raises: ConfigInvalid if required keystone authorization credentials
with swift are missing.

10
ironic/conf/deploy.py
View File

@ -79,15 +79,7 @@ opts = [
deprecated_name='configdrive_use_swift',
help=_('Whether to upload the config drive to object store. '
'Set this option to True to store config drive '
'in swift or radosgw.')),
cfg.StrOpt('object_store_endpoint_type',
default='swift',
deprecated_group='glance',
deprecated_name='temp_url_endpoint_type',
choices=[('swift', _('use Object Storage service')),
('radosgw', _('use RADOS object store'))],
help=_('Type of object store endpoint type to be '
'used as a backend')),
'in a swift endpoint.')),
]

8
ironic/conf/glance.py
View File

@ -68,7 +68,7 @@ opts = [
'swift_endpoint_url',
help=_('The "endpoint" (scheme, hostname, optional port) for '
'the Swift URL of the form '
'"endpoint_url/api_version/[account/]container/object_id". '
'"endpoint_url/api_version/account/container/object_id". '
'Do not include trailing "/". '
'For example, use "https://swift.example.com". If using RADOS '
'Gateway, endpoint may also contain /swift path; if it does '
@ -79,7 +79,7 @@ opts = [
default='v1',
help=_('The Swift API version to create a temporary URL for. '
'Defaults to "v1". Swift temporary URL format: '
'"endpoint_url/api_version/[account/]container/object_id"')),
'"endpoint_url/api_version/account/container/object_id"')),
cfg.StrOpt(
'swift_account',
help=_('The account that Glance uses to communicate with '
@ -89,7 +89,7 @@ opts = [
'If not set, the default value is calculated based on the ID '
'of the project used to access Swift (as set in the [swift] '
'section). Swift temporary URL format: '
'"endpoint_url/api_version/[account/]container/object_id"')),
'"endpoint_url/api_version/account/container/object_id"')),
cfg.StrOpt(
'swift_container',
default='glance',
@ -97,7 +97,7 @@ opts = [
'images in. Defaults to "glance", which is the default '
'in glance-api.conf. '
'Swift temporary URL format: '
'"endpoint_url/api_version/[account/]container/object_id"')),
'"endpoint_url/api_version/account/container/object_id"')),
cfg.IntOpt('swift_store_multiple_containers_seed',
default=0,
help=_('This should match a config by the same name in the '

74
ironic/tests/unit/common/test_glance_service.py
View File

@ -23,7 +23,6 @@ from keystoneauth1 import loading as kaloading
import mock
from oslo_config import cfg
from oslo_utils import uuidutils
from six.moves.urllib import parse as urlparse
import testtools
from ironic.common import context
@ -686,65 +685,6 @@ class TestGlanceSwiftTempURL(base.TestCase):
self.service.swift_temp_url, image_info)
self.assertFalse(tempurl_mock.called)
@mock.patch('swiftclient.utils.generate_temp_url', autospec=True)
def test_swift_temp_url_radosgw(self, tempurl_mock):
self.config(object_store_endpoint_type='radosgw', group='deploy')
path = ('/v1'
'/glance'
'/757274c4-2856-4bd2-bb20-9a4a231e187b')
tempurl_mock.return_value = (
path + '?temp_url_sig=hmacsig&temp_url_expires=1400001200')
self.service._validate_temp_url_config = mock.Mock()
temp_url = self.service.swift_temp_url(image_info=self.fake_image)
self.assertEqual(
(urlparse.urljoin(CONF.glance.swift_endpoint_url, 'swift')
+ tempurl_mock.return_value),
temp_url)
tempurl_mock.assert_called_with(
path=path,
seconds=CONF.glance.swift_temp_url_duration,
key=CONF.glance.swift_temp_url_key,
method='GET')
@mock.patch('swiftclient.utils.generate_temp_url', autospec=True)
def test_swift_temp_url_radosgw_endpoint_with_swift(self, tempurl_mock):
self.config(swift_endpoint_url='https://swift.radosgw.com/swift',
group='glance')
self.config(object_store_endpoint_type='radosgw', group='deploy')
path = ('/v1'
'/glance'
'/757274c4-2856-4bd2-bb20-9a4a231e187b')
tempurl_mock.return_value = (
path + '?temp_url_sig=hmacsig&temp_url_expires=1400001200')
self.service._validate_temp_url_config = mock.Mock()
temp_url = self.service.swift_temp_url(image_info=self.fake_image)
self.assertEqual(
CONF.glance.swift_endpoint_url + tempurl_mock.return_value,
temp_url)
tempurl_mock.assert_called_with(
path=path,
seconds=CONF.glance.swift_temp_url_duration,
key=CONF.glance.swift_temp_url_key,
method='GET')
@mock.patch('swiftclient.utils.generate_temp_url', autospec=True)
def test_swift_temp_url_radosgw_endpoint_invalid(self, tempurl_mock):
self.config(swift_endpoint_url='https://swift.radosgw.com/eggs/',
group='glance')
self.config(object_store_endpoint_type='radosgw', group='deploy')
self.service._validate_temp_url_config = mock.Mock()
self.assertRaises(exception.InvalidParameterValue,
self.service.swift_temp_url,
self.fake_image)
self.assertFalse(tempurl_mock.called)
@mock.patch('swiftclient.utils.generate_temp_url', autospec=True)
def test_swift_temp_url_multiple_containers(self, tempurl_mock):
@ -778,20 +718,8 @@ class TestGlanceSwiftTempURL(base.TestCase):
def test__validate_temp_url_config(self):
self.service._validate_temp_url_config()
def test__validate_temp_url_key_no_exception(self):
def test__validate_temp_url_no_key_no_exception(self):
self.config(swift_temp_url_key=None, group='glance')
self.config(object_store_endpoint_type='swift', group='deploy')
self.service._validate_temp_url_config()
def test__validate_temp_url_key_exception(self):
self.config(swift_temp_url_key=None, group='glance')
self.config(object_store_endpoint_type='radosgw', group='deploy')
self.assertRaises(exception.MissingParameterValue,
self.service._validate_temp_url_config)
def test__validate_temp_url_no_account_exception_radosgw(self):
self.config(swift_account=None, group='glance')
self.config(object_store_endpoint_type='radosgw', group='deploy')
self.service._validate_temp_url_config()
def test__validate_temp_url_endpoint_less_than_download_delay(self):

24
ironic/tests/unit/common/test_swift.py
View File

@ -57,30 +57,6 @@ class SwiftTestCase(base.TestCase):
os_options={'object_storage_url': 'http://example.com/objects'}
)
def test___init___radosgw(self, connection_mock, swift_session_mock):
"""Check if client is properly initialized with radosgw"""
auth_url = 'http://1.2.3.4'
username = 'foo'
password = 'foo_password'
CONF.set_override('object_store_endpoint_type', 'radosgw',
group='deploy')
opts = [cfg.StrOpt('auth_url'), cfg.StrOpt('username'),
cfg.StrOpt('password')]
CONF.register_opts(opts, group='swift')
CONF.set_override('auth_url', auth_url, group='swift')
CONF.set_override('username', username, group='swift')
CONF.set_override('password', password, group='swift')
swift.SwiftAPI()
params = {'authurl': auth_url,
'user': username,
'key': password,
'retries': 2}
connection_mock.assert_called_once_with(**params)
self.assertFalse(swift_session_mock.called)
@mock.patch.object(__builtin__, 'open', autospec=True)
def test_create_object(self, open_mock, connection_mock, keystone_mock):
swiftapi = swift.SwiftAPI()

22
ironic/tests/unit/conductor/test_base_manager.py
View File

@ -262,28 +262,6 @@ class StartStopTestCase(mgr_utils.ServiceSetUpMixin, db_base.DbTestCase):
self.service.del_host()
self.assertTrue(wait_mock.called)
def test_start_fails_on_missing_config_for_configdrive(self):
"""Check to fail conductor on missing config options"""
missing_parameters_error = ("Parameters missing to make a "
"connection with radosgw")
CONF.set_override('configdrive_use_object_store', True,
group='deploy')
CONF.set_override('object_store_endpoint_type', 'radosgw',
group='deploy')
params = {'auth_url': 'http://1.2.3.4',
'username': 'foo', 'password': 'foo_pass'}
CONF.register_opts((cfg.StrOpt(x) for x in params),
group='swift')
for key, value in params.items():
test_params = params.copy()
test_params[key] = None
for test_key, test_value in test_params.items():
CONF.set_override(key, test_value, group='swift')
with self.assertRaisesRegex(exception.ConfigInvalid,
missing_parameters_error):
self._start_service()
def test_conductor_shutdown_flag(self):
self._start_service()
self.assertFalse(self.service._shutdown)

9
releasenotes/notes/remove-radosgw-config-b664f3023dc8403c.yaml Normal file
View File

@ -0,0 +1,9 @@
---
upgrade:
- |
The ``swift/endpoint_type`` configuration option is now removed.
python-swiftclient 3.2.0 (Ocata) and above removed support for the native
URL type used by radosgw. Since using a ``swift/endpoint_type`` value of
``radosgw`` would fail anyway, it is removed. Deployers must now configure
ceph with ``rgw swift account in url = True``. This must be set before
upgrading to this release.