Rethinking filesystem access

In Rocky multiple backend support is added as experimental feature. In
order to take advantage of this feature it is decided to deprecate
work_dir and node_staging_uri configuration options
and reserve two filesystem stores 'os_glance_tasks_store' and
'os_glance_staging_store', which can be used to get rid of initializing
store via internal functions.

These internal stores are considered "reserved stores" by Glance.
For the time being, these are hard-coded as filesystem stores.  The
store prefix 'os_glance_' is reserved for internal Glance use and
the glance-api service will refuse to start if a store with this
prefix is included in the enabled_backends config option in
glance-api.conf.

NOTE: Because there are no sensible default values for the location
of the datadir for each of these stores, the operator must define
'os_glance_tasks_store' and 'os_glance_staging_store' in
glance-api.conf configuration file as shown below.

[os_glance_tasks_store]
filesystem_store_datadir = /var/lib/glance/tasks_work_dir/

[os_glance_staging_store]
filesystem_store_datadir = /var/lib/glance/staging/

Each filesystem store must have a unique datadir.

Depends-On: https://review.openstack.org/#/c/639765/
Implements: blueprint rethinking-filesystem-access
Change-Id: I86ec513c5fc653dbb97b79d953d8430f014e684f

doc/source/admin/multistores.rst

@@ -16,11 +16,8 @@
 Multi Store Support
 ===================
 
-.. note:: The Multi Store feature is introduced as EXPERIMENTAL in Rocky and
+.. note:: The Multi Store feature was introduced as EXPERIMENTAL in Rocky
-          its use in production systems is currently **not supported**.
+          and is now fully supported in the Train release.
-          However we encourage people to use this feature for testing
-          purposes and report the issues so that we can make it stable and
-          fully supported in Stein release.
 
 Scope of this document
 ----------------------
@@ -53,10 +50,10 @@ operators to enable multiple stores support.
     multiple stores operator can specify multiple key:value separated by
     comma.
 
-    Due to the special read only nature and characteristics of the
+    .. warning::
-    http store type, we do not encourage nor support configuring
+       The store identifier prefix ``os_glance_`` is reserved.  If you
-    multiple instances of the http type store even though it's
+       define a store identifier with this prefix, the glance service will
-    possible.
+       refuse to start.
 
     The http store type is always treated by Glance as a read-only
     store.  This is indicated in the response to the ``/v2/stores/info``
@@ -112,15 +109,98 @@ operators to enable multiple stores support.
 
   .. note ::
        ``store_description`` is a new config option added to each store where
-       operator can add meaningful description about that store. This description
+       operator can add meaningful description about that store. This
-       is displayed in the GET /v2/info/stores response.
+       description is displayed in the GET /v2/info/stores response.
 
-* For new image import workflow glance will reserve a ``os_staging`` file
+Store Configuration Issues
-  store identifier for staging the images data during staging operation. This
+~~~~~~~~~~~~~~~~~~~~~~~~~~
-  should be added by default in ``glance-api.conf`` as shown below:
+
+Please keep the following points in mind.
+
+* Due to the special read only nature and characteristics of the
+  http store type, configuring multiple instances of the http type
+  store **is not supported**.  (This constraint is not currently
+  enforced in the code.)
+
+* Each instance of the filesystem store **must** have a different value
+  for the ``filesystem_store_datadir``.  (This constraint is not currently
+  enforced in the code.)
+
+
+Reserved Stores
+---------------
+
+With the Train release, Glance is beginning a transition from its former
+reliance upon local directories for temporary data storage to the ability
+to use backend stores accessed via the glance_store library.
+
+In the Train release, the use of backend stores for this purpose is optional
+**unless you are using the multi store support feature**.  Since you are
+reading this document, this situation most likely applies to you.
+
+.. note::
+   Currently, only the filesystem store type is supported as a Glance
+   reserved store.
+
+The reserved stores are not intended to be exposed to end users.  Thus
+they will not appear in the response to the store discovery call, GET
+/v2/info/stores, or as values in the ``OpenStack-image-store-ids``
+response header of the image-create call.
+
+You do not get to select the name of a reserved store; these are defined
+by Glance and begin with the prefix ``os_glance_``.  In the Train release,
+you do not get to select the store type: all reserved stores must be of
+type filesystem.
+
+Currently, there are two reserved stores:
+
+``os_glance_tasks_store``
+    This store is used for the tasks engine.  It replaces the use of the
+    DEPRECATED configuration option ``[task]/work_dir``.
+
+``os_glance_staging_store``
+    This store is used for the staging area for the interoperable image
+    import process.  It replaces  the use of the DEPRECATED configuration
+    option ``[DEFAULT]/node_staging_uri``.
+
+Configuration
+~~~~~~~~~~~~~
+
+As mentioned above, you do not get to select the name or the type of
+a reserved store (though we anticipate that you will be able configure
+the store type in a future release).
+
+The reserved stores *must* be of type filesystem.  Hence, you must
+provide configuration for them in your ``glance-api.conf`` file.  You
+do this by introducing a section in ``glance-api.conf`` for each reserved
+store as follows:
 
 .. code-block:: ini
 
-        [os_staging]
+    [os_glance_tasks_store]
-        filesystem_store_datadir = /opt/stack/data/glance/os_staging/
+    filesystem_store_datadir = /var/lib/glance/tasks_work_dir
-        store_description = "Filesystem store for staging purpose"
+
+    [os_glance_staging_store]
+    filesystem_store_datadir = /var/lib/glance/staging
+
+Since these are both filesystem stores (remember, you do not get a choice)
+the only option you must configure for each is the
+``filesystem_store_datadir``.  Please keep the following points in mind:
+
+* The path for ``filesystem_store_datadir`` used for the reserved
+  stores must be **different** from the path you are using for
+  any filesystem store you have listed in ``enabled_backends``.
+  Using the same data directory for multiple filesystem stores is
+  **unsupported** and may lead to data loss.
+
+* The identifiers for reserved stores, that is, ``os_glance_tasks_store``
+  and ``os_glance_staging_store``, must **not** be included in the
+  ``enabled_backends`` list.
+
+* The reserved stores will **not** appear in the store discovery response
+  or as values in the ``OpenStack-image-store-ids`` response header of
+  the image-create call.
+
+* The reserved stores will **not** be accepted as the value of the
+  ``X-Image-Meta-Store`` header on the image-data-upload call or
+  the image-import call.

glance/api/v2/discovery.py

@@ -46,6 +46,9 @@ class InfoController(object):
 
         backends = []
         for backend in enabled_backends:
+            if backend.startswith("os_glance_"):
+                continue
+
             stores = {}
             stores['id'] = backend
             description = getattr(CONF, backend).store_description

glance/api/v2/image_data.py

@@ -17,6 +17,7 @@ import os
 from cursive import exception as cursive_exception
 import glance_store
 from glance_store import backend
+from glance_store import location
 from oslo_config import cfg
 from oslo_log import log as logging
 from oslo_utils import encodeutils
@@ -75,10 +76,18 @@ class ImageDataController(object):
         :param image: The image will be restored
         :param staging_store: The store used for staging
         """
-        # NOTE(abhishek): staging_store not being used in this function
+        if CONF.enabled_backends:
-        # because of bug #1803498
+            file_path = "%s/%s" % (getattr(
-        # TODO(abhishek): refactor to use the staging_store when the
+                CONF, 'os_glance_staging_store').filesystem_store_datadir,
-        # "Rethinking Filesystem Access" spec is implemented in Train
+                image.image_id)
+            try:
+                loc = location.get_location_from_uri_and_backend(
+                    file_path, 'os_glance_staging_store')
+                staging_store.delete(loc)
+            except (glance_store.exceptions.NotFound,
+                    glance_store.exceptions.UnknownScheme):
+                pass
+        else:
             file_path = str(CONF.node_staging_uri + '/' + image.image_id)[7:]
             if os.path.exists(file_path):
                 try:
@@ -320,6 +329,13 @@ class ImageDataController(object):
                 raise exception.BadStoreUri(message=msg)
             return staging_store
 
+        # NOTE(abhishekk): Use reserved 'os_glance_staging_store' for staging
+        # the data, the else part will be removed once multiple backend feature
+        # is declared as stable.
+        if CONF.enabled_backends:
+            staging_store = glance_store.get_store_from_store_identifier(
+                'os_glance_staging_store')
+        else:
             staging_store = _build_staging_store()
 
         try:

glance/api/v2/images.py

@@ -20,6 +20,7 @@ import re
 from castellan.common import exception as castellan_exception
 from castellan import key_manager
 import glance_store
+from glance_store import location
 from oslo_config import cfg
 from oslo_log import log as logging
 from oslo_serialization import jsonutils as json
@@ -368,8 +369,22 @@ class ImagesController(object):
             image = image_repo.get(image_id)
 
             if image.status == 'uploading':
+                if CONF.enabled_backends:
+                    file_path = "%s/%s" % (getattr(
+                        CONF, 'os_glance_staging_store'
+                    ).filesystem_store_datadir, image_id)
+                    try:
+                        fn_call = glance_store.get_store_from_store_identifier
+                        staging_store = fn_call('os_glance_staging_store')
+                        loc = location.get_location_from_uri_and_backend(
+                            file_path, 'os_glance_staging_store')
+                        staging_store.delete(loc)
+                    except (glance_store.exceptions.NotFound,
+                            glance_store.exceptions.UnknownScheme):
+                        pass
+                else:
                     file_path = str(
-                    CONF.node_staging_uri + '/' + image.image_id)[7:]
+                        CONF.node_staging_uri + '/' + image_id)[7:]
                     if os.path.exists(file_path):
                         try:
                             LOG.debug(

glance/async_/flows/_internal_plugins/web_download.py

@@ -12,7 +12,7 @@
 #    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
 #    License for the specific language governing permissions and limitations
 #    under the License.
-
+import glance_store as store_api
 from glance_store import backend
 from oslo_config import cfg
 from oslo_log import log as logging
@@ -44,6 +44,13 @@ class _WebDownload(task.Task):
         super(_WebDownload, self).__init__(
             name='%s-WebDownload-%s' % (task_type, task_id))
 
+        # NOTE(abhishekk): Use reserved 'os_glance_staging_store' for
+        # staging the data, the else part will be removed once old way
+        # of configuring store is deprecated.
+        if CONF.enabled_backends:
+            self.store = store_api.get_store_from_store_identifier(
+                'os_glance_staging_store')
+        else:
             if CONF.node_staging_uri is None:
                 msg = (_("%(task_id)s of %(task_type)s not configured "
                          "properly. Missing node_staging_uri: %(work_dir)s") %
@@ -111,7 +118,6 @@ class _WebDownload(task.Task):
                            "task_id": self.task_id})
 
         path = self.store.add(self.image_id, data, 0)[0]
-
         return path
 
     def revert(self, result, **kwargs):

glance/async_/flows/api_image_import.py

@@ -14,6 +14,7 @@
 #    under the License.
 import os
 
+import glance_store as store_api
 from glance_store import backend
 from oslo_config import cfg
 from oslo_log import log as logging
@@ -86,8 +87,12 @@ class _DeleteFromFS(task.Task):
 
         :param file_path: path to the file being deleted
         """
-        # TODO(abhishekk): After removal of backend module from glance_store
+        if CONF.enabled_backends:
-        # need to change this to use multi_backend module.
+            store_api.delete(file_path, 'os_glance_staging_store')
+        else:
+            # TODO(abhishekk): After removal of backend module from
+            # glance_store need to change this to use multi_backend
+            # module.
             file_path = file_path[7:]
             if os.path.exists(file_path):
                 try:
@@ -102,7 +107,8 @@ class _DeleteFromFS(task.Task):
             else:
                 LOG.warning(_("After upload to backend, deletion of staged "
                               "image data has failed because "
-                          "it cannot be found at %(fn)s"), {'fn': file_path})
+                              "it cannot be found at %(fn)s"), {
+                    'fn': file_path})
 
 
 class _VerifyStaging(task.Task):
@@ -132,6 +138,7 @@ class _VerifyStaging(task.Task):
                     'task_type': self.task_type})
             raise exception.BadTaskConfiguration(msg)
 
+        if not CONF.enabled_backends:
             # NOTE(jokke): We really don't need the store for anything but
             # verifying that we actually can build the store will allow us to
             # fail the flow early with clear message why that happens.
@@ -332,10 +339,14 @@ def get_flow(**kwargs):
     backend = kwargs.get('backend')
 
     separator = ''
-    if not CONF.node_staging_uri.endswith('/'):
+    if not CONF.enabled_backends and not CONF.node_staging_uri.endswith('/'):
         separator = '/'
 
     if not uri and import_method == 'glance-direct':
+        if CONF.enabled_backends:
+            separator, staging_dir = _get_dir_separator()
+            uri = separator.join((staging_dir, str(image_id)))
+        else:
             uri = separator.join((CONF.node_staging_uri, str(image_id)))
 
     flow = lf.Flow(task_type, retry=retry.AlwaysRevert())
@@ -343,6 +354,10 @@ def get_flow(**kwargs):
     if import_method == 'web-download':
         downloadToStaging = internal_plugins.get_import_plugin(**kwargs)
         flow.add(downloadToStaging)
+        if CONF.enabled_backends:
+            separator, staging_dir = _get_dir_separator()
+            file_uri = separator.join((staging_dir, str(image_id)))
+        else:
             file_uri = separator.join((CONF.node_staging_uri, str(image_id)))
     else:
         file_uri = uri
@@ -381,3 +396,12 @@ def get_flow(**kwargs):
     image_repo.save(image, from_state=from_state)
 
     return flow
+
+
+def _get_dir_separator():
+    separator = ''
+    staging_dir = "file://%s" % getattr(
+        CONF, 'os_glance_staging_store').filesystem_store_datadir
+    if not staging_dir.endswith('/'):
+        separator = '/'
+    return separator, staging_dir

glance/async_/flows/base_import.py

@@ -95,6 +95,13 @@ class _ImportToFS(task.Task):
         super(_ImportToFS, self).__init__(
             name='%s-ImportToFS-%s' % (task_type, task_id))
 
+        # NOTE(abhishekk): Use reserved 'os_glance_tasks_store' for tasks,
+        # the else part will be removed once old way of configuring store
+        # is deprecated.
+        if CONF.enabled_backends:
+            self.store = store_api.get_store_from_store_identifier(
+                'os_glance_tasks_store')
+        else:
             if CONF.task.work_dir is None:
                 msg = (_("%(task_id)s of %(task_type)s not configured "
                          "properly. Missing work dir: %(work_dir)s") %
@@ -185,6 +192,9 @@ class _ImportToFS(task.Task):
             return
 
         if os.path.exists(result.split("file://")[-1]):
+            if CONF.enabled_backends:
+                store_api.delete(result, 'os_glance_tasks_store')
+            else:
                 store_api.delete_from_backend(result)
 
 
@@ -201,16 +211,20 @@ class _DeleteFromFS(task.Task):
 
         :param file_path: path to the file being deleted
         """
+        if CONF.enabled_backends:
+            store_api.delete(file_path, 'os_glance_tasks_store')
+        else:
             store_api.delete_from_backend(file_path)
 
 
 class _ImportToStore(task.Task):
 
-    def __init__(self, task_id, task_type, image_repo, uri):
+    def __init__(self, task_id, task_type, image_repo, uri, backend):
         self.task_id = task_id
         self.task_type = task_type
         self.image_repo = image_repo
         self.uri = uri
+        self.backend = backend
         super(_ImportToStore, self).__init__(
             name='%s-ImportToStore-%s' % (task_type, task_id))
 
@@ -311,7 +325,8 @@ class _ImportToStore(task.Task):
         # image_import.set_image_data(image, image_path, None)
         try:
             image_import.set_image_data(image,
-                                        file_path or self.uri, self.task_id)
+                                        file_path or self.uri, self.task_id,
+                                        backend=self.backend)
         except IOError as e:
             msg = (_('Uploading the image failed due to: %(exc)s') %
                    {'exc': encodeutils.exception_to_unicode(e)})
@@ -423,11 +438,13 @@ def get_flow(**kwargs):
     image_repo = kwargs.get('image_repo')
     image_factory = kwargs.get('image_factory')
     uri = kwargs.get('uri')
+    backend = kwargs.get('backend')
 
     flow = lf.Flow(task_type, retry=retry.AlwaysRevert()).add(
         _CreateImage(task_id, task_type, task_repo, image_repo, image_factory))
 
-    import_to_store = _ImportToStore(task_id, task_type, image_repo, uri)
+    import_to_store = _ImportToStore(task_id, task_type, image_repo, uri,
+                                     backend)
 
     try:
         # NOTE(flaper87): ImportToLocal and DeleteFromLocal shouldn't be here.

glance/async_/flows/convert.py

@@ -114,7 +114,13 @@ class _Convert(task.Task):
         # shields us from being vulnerable to an attack vector described here
         # https://bugs.launchpad.net/glance/+bug/1449062
 
-        dest_path = os.path.join(CONF.task.work_dir, "%s.converted" % image_id)
+        data_dir = CONF.task.work_dir
+        # NOTE(abhishekk): Use reserved 'os_glance_tasks_store' for tasks.
+        if CONF.enabled_backends:
+            data_dir = getattr(
+                CONF, 'os_glance_tasks_store').filesystem_store_datadir
+
+        dest_path = os.path.join(data_dir, "%s.converted" % image_id)
         stdout, stderr = putils.trycmd('qemu-img', 'convert',
                                        '-f', src_format,
                                        '-O', conversion_format,

glance/async_/flows/ovf_process.py

@@ -59,7 +59,13 @@ class _OVF_Process(task.Task):
             name='%s-OVF_Process-%s' % (task_type, task_id))
 
     def _get_extracted_file_path(self, image_id):
-        return os.path.join(CONF.task.work_dir,
+        file_path = CONF.task.work_dir
+        # NOTE(abhishekk): Use reserved 'os_glance_tasks_store' for tasks.
+        if CONF.enabled_backends:
+            file_path = getattr(
+                CONF, 'os_glance_tasks_store').filesystem_store_datadir
+
+        return os.path.join(file_path,
                             "%s.extracted" % image_id)
 
     def _get_ova_iter_objects(self, uri):

glance/async_/taskflow_executor.py

@@ -119,7 +119,8 @@ class TaskExecutor(glance.async_.TaskExecutor):
                 'context': self.context,
                 'task_repo': self.task_repo,
                 'image_repo': self.image_repo,
-                'image_factory': self.image_factory
+                'image_factory': self.image_factory,
+                'backend': task_input.get('backend')
             }
 
             if task.type == "import":
@@ -129,7 +130,6 @@ class TaskExecutor(glance.async_.TaskExecutor):
             if task.type == 'api_image_import':
                 kwds['image_id'] = task_input['image_id']
                 kwds['import_req'] = task_input['import_req']
-                kwds['backend'] = task_input['backend']
             return driver.DriverManager('glance.flows', task.type,
                                         invoke_on_load=True,
                                         invoke_kwds=kwds).driver

glance/common/scripts/api_image_import/main.py

@@ -116,13 +116,13 @@ def import_image(image_repo, image_factory, task_input, task_id, uri):
                         location)
 
 
-def set_image_data(image, uri, task_id):
+def set_image_data(image, uri, task_id, backend=None):
     data_iter = None
     try:
         LOG.info("Task %(task_id)s: Got image data uri %(data_uri)s to be "
                  "imported", {"data_uri": uri, "task_id": task_id})
         data_iter = script_utils.get_image_data_iter(uri)
-        image.set_data(data_iter)
+        image.set_data(data_iter, backend=backend)
     except Exception as e:
         with excutils.save_and_reraise_exception():
             LOG.warn("Task %(task_id)s failed with exception %(error)s" %

glance/common/store_utils.py

@@ -32,6 +32,13 @@ CONF.import_opt('use_user_token', 'glance.registry.client')
 RESTRICTED_URI_SCHEMAS = frozenset(['file', 'filesystem', 'swift+config'])
 
 
+def check_reserved_stores(enabled_stores):
+    for store in enabled_stores:
+        if store.startswith("os_glance_"):
+            return True
+    return False
+
+
 def safe_delete_from_backend(context, image_id, location):
     """
     Given a location, delete an image from the store and
@@ -159,7 +166,8 @@ def _get_store_id_from_uri(uri):
         return
     for store in location_map[scheme]:
         store_instance = location_map[scheme][store]['store']
-        if uri.startswith(store_instance.url_prefix):
+        url_prefix = store_instance.url_prefix
+        if url_prefix and uri.startswith(url_prefix):
             url_matched = True
             break
 

glance/common/wsgi.py

@@ -54,6 +54,7 @@ from webob import multidict
 
 from glance.common import config
 from glance.common import exception
+from glance.common import store_utils
 from glance.common import utils
 from glance import i18n
 from glance.i18n import _, _LE, _LI, _LW
@@ -370,6 +371,12 @@ except ImportError:
     LOG.debug('Detected not running under uwsgi')
     uwsgi = None
 
+# Reserved file stores for staging and tasks operations
+RESERVED_STORES = {
+    'os_glance_staging_store': 'file',
+    'os_glance_tasks_store': 'file'
+}
+
 
 def register_cli_opts():
     CONF.register_cli_opts(cli_opts)
@@ -492,8 +499,8 @@ def initialize_glance_store():
 
 def initialize_multi_store():
     """Initialize glance multi store backends."""
-    glance_store.register_store_opts(CONF)
+    glance_store.register_store_opts(CONF, reserved_stores=RESERVED_STORES)
-    glance_store.create_multi_stores(CONF)
+    glance_store.create_multi_stores(CONF, reserved_stores=RESERVED_STORES)
     glance_store.verify_store()
 
 
@@ -619,6 +626,11 @@ class BaseServer(object):
         self.configure_socket(old_conf, has_changed)
         if self.initialize_glance_store:
             if CONF.enabled_backends:
+                if store_utils.check_reserved_stores(CONF.enabled_backends):
+                    msg = _("'os_glance_' prefix should not be used in "
+                            "enabled_backends config option. It is reserved "
+                            "for internal use only.")
+                    raise RuntimeError(msg)
                 initialize_multi_store()
             else:
                 initialize_glance_store()

glance/common/wsgi_app.py

@@ -18,6 +18,7 @@ from oslo_log import log as logging
 import osprofiler.initializer
 
 from glance.common import config
+from glance.common import store_utils
 from glance import notifier
 
 CONF = cfg.CONF
@@ -29,6 +30,12 @@ CONFIG_FILES = ['glance-api-paste.ini',
                 'glance-image-import.conf',
                 'glance-api.conf']
 
+# Reserved file stores for staging and tasks operations
+RESERVED_STORES = {
+    'os_glance_staging_store': 'file',
+    'os_glance_tasks_store': 'file'
+}
+
 
 def _get_config_files(env=None):
     if env is None:
@@ -63,8 +70,13 @@ def init_app():
     logging.setup(CONF, "glance")
 
     if CONF.enabled_backends:
-        glance_store.register_store_opts(CONF)
+        if store_utils.check_reserved_stores(CONF.enabled_backends):
-        glance_store.create_multi_stores(CONF)
+            msg = _("'os_glance_' prefix should not be used in "
+                    "enabled_backends config option. It is reserved "
+                    "for internal use only.")
+            raise RuntimeError(msg)
+        glance_store.register_store_opts(CONF, reserved_stores=RESERVED_STORES)
+        glance_store.create_multi_stores(CONF, reserved_stores=RESERVED_STORES)
         glance_store.verify_store()
     else:
         glance_store.register_opts(CONF)

glance/tests/functional/__init__.py

@@ -565,6 +565,7 @@ class ApiServerForMultipleBackend(Server):
         self.metadata_encryption_key = "012345678901234567890123456789ab"
         self.image_dir_backend_1 = os.path.join(self.test_dir, "images_1")
         self.image_dir_backend_2 = os.path.join(self.test_dir, "images_2")
+        self.staging_dir = os.path.join(self.test_dir, "staging")
         self.pid_file = pid_file or os.path.join(self.test_dir,
                                                  "multiple_backend_api.pid")
         self.log_file = os.path.join(self.test_dir, "multiple_backend_api.log")
@@ -651,6 +652,8 @@ filesystem_store_datadir=%(image_dir_backend_1)s
 filesystem_store_datadir=%(image_dir_backend_2)s
 [import_filtering_opts]
 allowed_ports = []
+[os_glance_staging_store]
+filesystem_store_datadir=%(staging_dir)s
 """
         self.paste_conf_base = """[pipeline:glance-api]
 pipeline =

glance/tests/functional/v2/test_images.py

@@ -4506,9 +4506,11 @@ class TestImagesMultipleBackend(functional.MultipleBackendFunctionalTest):
         self.assertEqual(http.OK, response.status_code)
         discovery_calls = jsonutils.loads(
             response.text)['stores']
+        # os_glance_staging_store should not be available in discovery response
         for stores in discovery_calls:
             self.assertIn('id', stores)
             self.assertIn(stores['id'], available_stores)
+            self.assertFalse(stores["id"].startswith("os_glance_"))
 
         # Create an image
         path = self._url('/v2/images')
@@ -4667,9 +4669,11 @@ class TestImagesMultipleBackend(functional.MultipleBackendFunctionalTest):
         self.assertEqual(http.OK, response.status_code)
         discovery_calls = jsonutils.loads(
             response.text)['stores']
+        # os_glance_staging_store should not be available in discovery response
         for stores in discovery_calls:
             self.assertIn('id', stores)
             self.assertIn(stores['id'], available_stores)
+            self.assertFalse(stores["id"].startswith("os_glance_"))
 
         # Create an image
         path = self._url('/v2/images')
@@ -4829,9 +4833,11 @@ class TestImagesMultipleBackend(functional.MultipleBackendFunctionalTest):
         self.assertEqual(http.OK, response.status_code)
         discovery_calls = jsonutils.loads(
             response.text)['stores']
+        # os_glance_staging_store should not be available in discovery response
         for stores in discovery_calls:
             self.assertIn('id', stores)
             self.assertIn(stores['id'], available_stores)
+            self.assertFalse(stores["id"].startswith("os_glance_"))
 
         # Create an image
         path = self._url('/v2/images')
@@ -4990,9 +4996,11 @@ class TestImagesMultipleBackend(functional.MultipleBackendFunctionalTest):
         self.assertEqual(http.OK, response.status_code)
         discovery_calls = jsonutils.loads(
             response.text)['stores']
+        # os_glance_staging_store should not be available in discovery response
         for stores in discovery_calls:
             self.assertIn('id', stores)
             self.assertIn(stores['id'], available_stores)
+            self.assertFalse(stores["id"].startswith("os_glance_"))
 
         # Create an image
         path = self._url('/v2/images')
@@ -5142,9 +5150,11 @@ class TestImagesMultipleBackend(functional.MultipleBackendFunctionalTest):
         self.assertEqual(http.OK, response.status_code)
         discovery_calls = jsonutils.loads(
             response.text)['stores']
+        # os_glance_staging_store should not be available in discovery response
         for stores in discovery_calls:
             self.assertIn('id', stores)
             self.assertIn(stores['id'], available_stores)
+            self.assertFalse(stores["id"].startswith("os_glance_"))
 
         # Create an image (with two deployer-defined properties)
         path = self._url('/v2/images')
@@ -5309,9 +5319,11 @@ class TestImagesMultipleBackend(functional.MultipleBackendFunctionalTest):
         self.assertEqual(http.OK, response.status_code)
         discovery_calls = jsonutils.loads(
             response.text)['stores']
+        # os_glance_staging_store should not be available in discovery response
         for stores in discovery_calls:
             self.assertIn('id', stores)
             self.assertIn(stores['id'], available_stores)
+            self.assertFalse(stores["id"].startswith("os_glance_"))
 
         # Create an image (with two deployer-defined properties)
         path = self._url('/v2/images')

glance/tests/unit/common/test_wsgi.py

@@ -568,6 +568,16 @@ class ServerTest(test_utils.BaseTestCase):
         actual = wsgi.Server(threads=1).create_pool()
         self.assertIsInstance(actual, eventlet.greenpool.GreenPool)
 
+    @mock.patch.object(prefetcher, 'Prefetcher')
+    @mock.patch.object(wsgi.Server, 'configure_socket')
+    def test_reserved_stores_not_allowed(self, mock_configure_socket,
+                                         mock_prefetcher):
+        """Ensure the reserved stores are not allowed"""
+        enabled_backends = {'os_glance_file_store': 'file'}
+        self.config(enabled_backends=enabled_backends)
+        server = wsgi.Server(threads=1, initialize_glance_store=True)
+        self.assertRaises(RuntimeError, server.configure)
+
     @mock.patch.object(prefetcher, 'Prefetcher')
     @mock.patch.object(wsgi.Server, 'configure_socket')
     def test_http_keepalive(self, mock_configure_socket, mock_prefetcher):

glance/tests/unit/v2/test_discovery_stores.py

@@ -59,3 +59,16 @@ class TestInfoControllers(base.MultiStoreClearingUnitTest):
                 self.assertTrue(stores['read-only'])
             else:
                 self.assertIsNone(stores.get('read-only'))
+
+    def test_get_stores_reserved_stores_excluded(self):
+        enabled_backends = {
+            'fast': 'file',
+            'cheap': 'file'
+        }
+        self.config(enabled_backends=enabled_backends)
+        req = unit_test_utils.get_fake_request()
+        output = self.controller.get_stores(req)
+        self.assertIn('stores', output)
+        self.assertEqual(2, len(output['stores']))
+        for stores in output["stores"]:
+            self.assertFalse(stores["id"].startswith("os_glance_"))

releasenotes/notes/rethinking-filesystem-access-120bc46064b3d40a.yaml

@@ -0,0 +1,57 @@
+---
+features:
+  - |
+    With the introduction of the Glance multiple stores feature, introduced
+    on an experimental basis in Rocky and now established as a full feature
+    in the Train release, it is now possible for Glance to use backends
+    accessed via the glance_store library for the temporary storage of
+    data that previously required access to the local filesystem.  Please
+    note the following:
+
+    * In this release, the use of stores (instead of local directories) is
+      optional, but it will become mandatory for the 'U' release.
+
+    * In this release, the stores used *must* be the filesystem store type.
+      Our goal is that in a future release, operators will be able to
+      configure a store of their choice for these functions.  In Train,
+      however, each of these *must* be a filesystem store.
+
+    Please see the Upgrades section of this document and the "Multi Store
+    Support" chapter of the Glance Administration Guide for more information.
+
+upgrade:
+  - |
+    The configuration options ``work_dir`` and ``node_staging_uri`` are
+    deprecated and will be removed early in the 'U' development cycle.
+
+    These local directories are used by Glance for the temporary storage
+    of data during the interoperable image import process and by the
+    tasks engine.  This release introduces the ability to instead use a
+    backend filesystem store accessed via the glance_store library for this
+    temporary storage.  Please note the following:
+
+    * If you wish to use the backend store feature now, please see the
+      "Reserved Stores" section of the "Multi Store Support" chapter of
+      the Glance Administration Guide for configuration information.
+
+    * If you use the Glance multiple stores feature, introduced on an
+      experimental basis in Rocky and now fully supported in the Train
+      release, then you *must* use backing stores instead of ``work_dir``
+      and ``node_staging_uri`` for Glance's temporary storage **beginning
+      right now with the current release**.  See the "Reserved Stores"
+      section of the "Multi Store Support" chapter of the Glance
+      Administration Guide for more information.
+
+  - |
+    The store name prefix ``os_glance_*`` is reserved by Glance for internal
+    stores.  Glance will refuse to start if a store with this prefix is
+    included in the ``enabled_backends`` option.
+
+    The internal store identifiers introduced in this release are
+    ``os_glance_tasks_store`` and ``os_glance_staging_store``.
+
+issues:
+  - |
+    When using the multiple stores feature, each filesystem store **must**
+    be configured with a different value for the ``filesystem_store_datadir``
+    option.  This is not currently enforced in the code.