Replication =========== For backend devices that offer replication features, Cinder provides a common mechanism for exposing that functionality on a per volume basis while still trying to allow flexibility for the varying implementation and requirements of all the different backend devices. There are 2 sides to Cinder's replication feature, the core mechanism and the driver specific functionality, and in this document we'll only be covering the driver side of things aimed at helping vendors implement this functionality in their drivers in a way consistent with all other drivers. Although we'll be focusing on the driver implementation there will also be some mentions on deployment configurations to provide a clear picture to developers and help them avoid implementing custom solutions to solve things that were meant to be done via the cloud configuration. Overview -------- As a general rule replication is enabled and configured via the cinder.conf file under the driver's section, and volume replication is requested through the use of volume types. *NOTE*: Current replication implementation is v2.1 and it's meant to solve a very specific use case, the "smoking hole" scenario. It's critical that you read the Use Cases section of the spec here: https://specs.openstack.org/openstack/cinder-specs/specs/mitaka/cheesecake.html From a user's perspective volumes will be created using specific volume types, even if it is the default volume type, and they will either be replicated or not, which will be reflected on the ``replication_status`` field of the volume. So in order to know if a snapshot is replicated we'll have to check its volume. After the loss of the primary storage site all operations on the resources will fail and VMs will no longer have access to the data. It is then when the Cloud Administrator will issue the ``failover-host`` command to make the cinder-volume service perform the failover. After the failover is completed, the Cinder volume service will start using the failed-over secondary storage site for all operations and the user will once again be able to perform actions on all resources that were replicated, while all other resources will be in error status since they are no longer available. Storage Device configuration ---------------------------- Most storage devices will require configuration changes to enable the replication functionality, and this configuration process is vendor and storage device specific so it is not contemplated by the Cinder core replication functionality. It is up to the vendors whether they want to handle this device configuration in the Cinder driver or as a manual process, but the most common approach is to avoid including this configuration logic into Cinder and having the Cloud Administrators do a manual process following a specific guide to enable replication on the storage device before configuring the cinder volume service. Service configuration --------------------- The way to enable and configure replication is common to all drivers and it is done via the ``replication_device`` configuration option that goes in the driver's specific section in the ``cinder.conf`` configuration file. ``replication_device`` is a multi dictionary option, that should be specified for each replication target device the admin wants to configure. While it is true that all drivers use the same ``replication_device`` configuration option this doesn't mean that they will all have the same data, as there is only one standardized and **REQUIRED** key in the configuration entry, all others are vendor specific: - backend_id: Values of ``backend_id`` keys are used to uniquely identify within the driver each of the secondary sites, although they can be reused on different driver sections. These unique identifiers will be used by the failover mechanism as well as in the driver initialization process, and the only requirement is that is must never have the value "default". An example driver configuration for a device with multiple replication targets is show below:: ..... [driver-biz] volume_driver=xxxx volume_backend_name=biz [driver-baz] volume_driver=xxxx volume_backend_name=baz [driver-foo] volume_driver=xxxx volume_backend_name=foo replication_device = backend_id:vendor-id-1,unique_key:val.... replication_device = backend_id:vendor-id-2,unique_key:val.... In this example the result of calling ``self.configuration.safe_get('replication_device)`` within the driver is the following list:: [{backend_id: vendor-id-1, unique_key: val1}, {backend_id: vendor-id-2, unique_key: val2}] It is expected that if a driver is configured with multiple replication targets, that replicated volumes are actually replicated on **all targets**. Besides specific replication device keys defined in the ``replication_device``, a driver may also have additional normal configuration options in the driver section related with the replication to allow Cloud Administrators to configure things like timeouts. Capabilities reporting ---------------------- There are 2 new replication stats/capability keys that drivers supporting replication v2.1 should be reporting: ``replication_enabled`` and ``replication_targets``:: stats["replication_enabled"] = True|False stats["replication_targets"] = [...] If a driver is behaving correctly we can expect the ``replication_targets`` field to be populated whenever ``replication_enabled`` is set to ``True``, and it is expected to either be set to ``[]`` or be missing altogether when ``replication_enabled`` is set to ``False``. The purpose of the ``replication_enabled`` field is to be used by the scheduler in volume types for creation and migrations. As for the ``replication_targets`` field it is only provided for informational purposes so it can be retrieved through the ``get_capabilities`` using the admin REST API, but it will not be used for validation at the API layer. That way Cloud Administrators will be able to know available secondary sites where they can failover. Volume Types / Extra Specs --------------------------- The way to control the creation of volumes on a cloud with backends that have replication enabled is, like with many other features, through the use of volume types. We won't go into the details of volume type creation, but suffice to say that you will most likely want to use volume types to discriminate between replicated and non replicated volumes and be explicit about it so that non replicated volumes won't end up in a replicated backend. Since the driver is reporting the ``replication_enabled`` key, we just need to require it for replication volume types adding ``replication_enabled=' True``` and also specifying it for all non replicated volume types ``replication_enabled=' False'``. It's up to the driver to parse the volume type info on create and set things up as requested. While the scoping key can be anything, it's strongly recommended that all backends utilize the same key (replication) for consistency and to make things easier for the Cloud Administrator. Additional replication parameters can be supplied to the driver using vendor specific properties through the volume type's extra-specs so they can be used by the driver at volume creation time, or retype. It is up to the driver to parse the volume type info on create and retype to set things up as requested. A good pattern to get a custom parameter from a given volume instance is this:: extra_specs = getattr(volume.volume_type, 'extra_specs', {}) custom_param = extra_specs.get('custom_param', 'default_value') It may seem convoluted, but we must be careful when retrieving the ``extra_specs`` from the ``volume_type`` field as it could be ``None``. Vendors should try to avoid obfuscating their custom properties and expose them using the ``_init_vendor_properties`` method so they can be checked by the Cloud Administrator using the ``get_capabilities`` REST API. *NOTE*: For storage devices doing per backend/pool replication the use of volume types is also recommended. Volume creation --------------- Drivers are expected to honor the replication parameters set in the volume type during creation, retyping, or migration. When implementing the replication feature there are some driver methods that will most likely need modifications -if they are implemented in the driver (since some are optional)- to make sure that the backend is replicating volumes that need to be replicated and not replicating those that don't need to be: - ``create_volume`` - ``create_volume_from_snapshot`` - ``create_cloned_volume`` - ``retype`` - ``clone_image`` - ``migrate_volume`` In these methods the driver will have to check the volume type to see if the volumes need to be replicated, we could use the same pattern described in the `Volume Types / Extra Specs`_ section:: def _is_replicated(self, volume): specs = getattr(volume.volume_type, 'extra_specs', {}) return specs.get('replication_enabled') == ' True' But it is **not** the recommended mechanism, and the ``is_replicated`` method available in volumes and volume types versioned objects instances should be used instead. Drivers are expected to keep the ``replication_status`` field up to date and in sync with reality, usually as specified in the volume type. To do so in above mentioned methods' implementation they should use the update model mechanism provided for each one of those methods. One must be careful since the update mechanism may be different from one method to another. What this means is that most of these methods should be returning a ``replication_status`` key with the value set to ``enabled`` in the model update dictionary if the volume type is enabling replication. There is no need to return the key with the value of ``disabled`` if it is not enabled since that is the default value. In the case of the ``create_volume``, and ``retype`` method there is no need to return the ``replication_status`` in the model update since it has already been set by the scheduler on creation using the extra spec from the volume type. And on ``migrate_volume`` there is no need either since there is no change to the ``replication_status``. *NOTE*: For storage devices doing per backend/pool replication it is not necessary to check the volume type for the ``replication_enabled`` key since all created volumes will be replicated, but they are expected to return the ``replication_status`` in all those methods, including the ``create_volume`` method since the driver may receive a volume creation request without the replication enabled extra spec and therefore the driver will not have set the right ``replication_status`` and the driver needs to correct this. Besides the ``replication_status`` field that drivers need to update there are other fields in the database related to the replication mechanism that the drivers can use: - ``replication_extended_status`` - ``replication_driver_data`` These fields are string type fields with a maximum size of 255 characters and they are available for drivers to use internally as they see fit for their normal replication operation. So they can be assigned in the model update and later on used by the driver, for example during the failover. To avoid using magic strings drivers must use values defined by the ``ReplicationStatus`` class in ``cinder/objects/fields.py`` file and these are: - ``ERROR``: When setting the replication failed on creation, retype, or migrate. This should be accompanied by the volume status ``error``. - ``ENABLED``: When the volume is being replicated. - ``DISABLED``: When the volume is not being replicated. - ``FAILED_OVER``: After a volume has been successfully failed over. - ``FAILOVER_ERROR``: When there was an error during the failover of this volume. - ``NOT_CAPABLE``: When we failed-over but the volume was not replicated. The first 3 statuses revolve around the volume creation and the last 3 around the failover mechanism. The only status that should not be used for the volume's ``replication_status`` is the ``FAILING_OVER`` status. Whenever we are referring to values of the ``replication_status`` in this document we will be referring to the ``ReplicationStatus`` attributes and not a literal string, so ``ERROR`` means ``cinder.objects.field.ReplicationStatus.ERROR`` and not the string "ERROR". Failover -------- This is the mechanism used to instruct the cinder volume service to fail over to a secondary/target device. Keep in mind the use case is that the primary backend has died a horrible death and is no longer valid, so any volumes that were on the primary and were not being replicated will no longer be available. The method definition required from the driver to implement the failback mechanism is as follows:: def failover_host(self, context, volumes, secondary_id=None): There are several things that are expected of this method: - Promotion of a secondary storage device to primary - Generating the model updates - Changing internally to access the secondary storage device for all future requests. If no secondary storage device is provided to the driver via the ``backend_id`` argument (it is equal to ``None``), then it is up to the driver to choose which storage device to failover to. In this regard it is important that the driver takes into consideration that it could be failing over from a secondary (there was a prior failover request), so it should discard current target from the selection. If the ``secondary_id`` is not a valid one the driver is expected to raise ``InvalidReplicationTarget``, for any other non recoverable errors during a failover the driver should raise ``UnableToFailOver`` or any child of ``VolumeDriverException`` class and revert to a state where the previous backend is in use. The failover method in the driver will receive a list of replicated volumes that need to be failed over. Replicated volumes passed to the driver may have diverse ``replication_status`` values, but they will always be one of: ``ENABLED``, ``FAILED_OVER``, or ``FAILOVER_ERROR``. The driver must return a 2-tuple with the new storage device target id as the first element and a list of dictionaries with the model updates required for the volumes so that the driver can perform future actions on those volumes now that they need to be accessed on a different location. It's not a requirement for the driver to return model updates for all the volumes, or for any for that matter as it can return ``None`` or an empty list if there's no update necessary. But if elements are returned in the model update list then it is a requirement that each of the dictionaries contains 2 key-value pairs, ``volume_id`` and ``updates`` like this:: [{ 'volume_id': volumes[0].id, 'updates': { 'provider_id': new_provider_id1, ... }, 'volume_id': volumes[1].id, 'updates': { 'provider_id': new_provider_id2, 'replication_status': fields.ReplicationStatus.FAILOVER_ERROR, ... }, }] In these updates there is no need to set the ``replication_status`` to ``FAILED_OVER`` if the failover was successful, as this will be performed by the manager by default, but it won't create additional DB queries if it is returned. It is however necessary to set it to ``FAILOVER_ERROR`` for those volumes that had errors during the failover. Drivers don't have to worry about snapshots or non replicated volumes, since the manager will take care of those in the following manner: - All non replicated volumes will have their current ``status`` field saved in the ``previous_status`` field, the ``status`` field changed to ``error``, and their ``replication_status`` set to ``NOT_CAPABLE``. - All snapshots from non replicated volumes will have their statuses changed to ``error``. - All replicated volumes that failed on the failover will get their ``status`` changed to ``error``, their current ``status`` preserved in ``previous_status``, and their ``replication_status`` set to ``FAILOVER_ERROR`` . - All snapshots from volumes that had errors during the failover will have their statuses set to ``error``. Any model update request from the driver that changes the ``status`` field will trigger a change in the ``previous_status`` field to preserve the current status. Once the failover is completed the driver should be pointing to the secondary and should be able to create and destroy volumes and snapshots as usual, and it is left to the Cloud Administrator's discretion whether resource modifying operations are allowed or not. Failback -------- Drivers are not required to support failback, but they are required to raise a ``InvalidReplicationTarget`` exception if the failback is requested but not supported. The way to request the failback is quite simple, the driver will receive the argument ``secondary_id`` with the value of ``default``. That is why it was forbidden to use the ``default`` on the target configuration in the cinder configuration file. Expected driver behavior is the same as the one explained in the `Failover`_ section: - Promotion of the original primary to primary - Generating the model updates - Changing internally to access the original primary storage device for all future requests. If the failback of any of the volumes fail the driver must return ``replication_status`` set to ``ERROR`` in the volume updates for those volumes. If they succeed it is not necessary to change the ``replication_status`` since the default behavior will be to set them to ``ENABLED``, but it won't create additional DB queries if it is set. The manager will update resources in a slightly different way than in the failover case: - All non replicated volumes will not have any model modifications. - All snapshots from non replicated volumes will not have any model modifications. - All replicated volumes that failed on the failback will get their ``status`` changed to ``error``, have their current ``status`` preserved in the ``previous_status`` field, and their ``replication_status`` set to ``FAILOVER_ERROR``. - All snapshots from volumes that had errors during the failover will have their statuses set to ``error``. We can avoid using the "default" magic string by using the ``FAILBACK_SENTINEL`` class attribute from the ``VolumeManager`` class. Initialization -------------- It stands to reason that a failed over Cinder volume service may be restarted, so there needs to be a way for a driver to know on start which storage device should be used to access the resources. So, to let drivers know which storage device they should use the manager passes drivers the ``active_backend_id`` argument to their ``__init__`` method during the initialization phase of the driver. Default value is ``None`` when the default (primary) storage device should be used. Drivers should store this value if they will need it, as the base driver is not storing it, for example to determine the current storage device when a failover is requested and we are already in a failover state, as mentioned above. Freeze / Thaw ------------- In many cases, after a failover has been completed we'll want to allow changes to the data in the volumes as well as some operations like attach and detach while other operations that modify the number of existing resources, like delete or create, are not allowed. And that is where the freezing mechanism comes in; freezing a backend puts the control plane of the specific Cinder volume service into a read only state, or at least most of it, while allowing the data plane to proceed as usual. While this will mostly be handled by the Cinder core code, drivers are informed when the freezing mechanism is enabled or disabled via these 2 calls:: freeze_backend(self, context) thaw_backend(self, context) In most cases the driver may not need to do anything, and then it doesn't need to define any of these methods as long as its a child class of the ``BaseVD`` class that already implements them as noops. Raising a `VolumeDriverException` exception in any of these methods will result in a 500 status code response being returned to the caller and the manager will not log the exception, so it's up to the driver to log the error if it is appropriate. If the driver wants to give a more meaningful error response, then it can raise other exceptions that have different status codes. When creating the `freeze_backend` and `thaw_backend` driver methods we must remember that this is a Cloud Administrator operation, so we can return errors that reveal internals of the cloud, for example the type of storage device, and we must use the appropriate internationalization translation methods when raising exceptions; for `VolumeDriverException` no translation is necessary since the manager doesn't log it or return to the user in any way, but any other exception should use the ``_()`` translation method since it will be returned to the REST API caller. For example, if a storage device doesn't support the thaw operation when failed over, then it should raise an `Invalid` exception:: def thaw_backend(self, context): if self.failed_over: msg = _('Thaw is not supported by driver XYZ.') raise exception.Invalid(msg)