[NetApp] Add FlexGroup volume support

The NetApp driver has been working  with FlexVol ONTAP volumes.
The driver does not support scaling FlexVol volumes higher than
100 TiB, which was a theoretical limit for the large namespace that
these containers were meant to handle. ONTAP's Flexgroup volumes
eliminate such limitations. So, added the support for provisioning
share as FlexGroup in the NetApp driver.

The FlexGroup provision is enabled by new option
`netapp_enable_flexgroup`, which will make the driver report a single
pool represeting all aggregates. The selection on which aggregates the
FlexGroup share will reside is up to ONTAP. If the administrator desires
to control that selection through Manila scheduler, it must inform the set
of aggregates that formss FlexGroup pool in the new  option
`netapp_flexgroup_pool`.

Each NetApp pool will report now the capability: `netapp_flexgroup`
informing which type the pool is.

The following operations are allowed with FlexGroup shares (DHSS
True/False and NFS/CIFS):

- Create/Delete share;
- Shrink/Extend share;
- Create/Delete snapshot;
- Revert to snapshot;
- Manage/Unmanage snapshots;
- Create from snapshot;
- Replication[1]
- Manage/Unmanage shares;

The backend with one FlexGroup pool configured will drop the consistent
snapshot support for all pools.

The driver FlexGroup support requires ONTAP version 9.8 or greater.

[1] FlexGroup is limited to one single replica for ONTAP version
lower than 9.9.1.

DocImpact

Depends-On: If525e97a5d456d6ddebb4bf9bc8ff6190c95a555
Depends-On: I646f782c3e2be5ac799254f08a248a22cb9e0358
Implements: bp netapp-flexgroup-support
Change-Id: I4f68a9bb33be85f9a22e0be4ccf673647e713459
Signed-off-by: Felipe Rodrigues <felipefuty01@gmail.com>

doc/source/admin/netapp_cluster_mode_driver.rst

@@ -20,6 +20,12 @@ NetApp Clustered Data ONTAP
 The Shared File Systems service can be configured to use
 NetApp Clustered Data ONTAP (cDOT) version 8.2 and later.
 
+The driver can work with two types of pools: FlexGroup and FlexVol. By default,
+it only works with FlexVol, if desired, the FlexGroup pool can be enabled
+together or standalone.
+
+FlexGroup pool requires ONTAP version 9.8 or later.
+
 Supported Operations
 --------------------
 
@@ -57,6 +63,19 @@ The following operations are supported on Clustered Data ONTAP:
 - Create a replicated snapshot (DHSS=False)
 - Delete a replicated snapshot (DHSS=False)
 - Update a replicated snapshot (DHSS=False)
+- Migrate share
+- Migrate share server
+
+.. note::
+
+    The operations are not fully supported configuring FlexGroup pool:
+
+    - Consistency group operations are only supported configuring the driver
+      without any FlexGroup pool.
+    - For FlexGroup share, create more than one replica is only allowed with
+      ONTAP 9.9.1 and newer.
+    - Migration of FlexGroup shares is not allowed.
+    - Migration of share servers containing FlexGroup share is not allowed.
 
 .. note::
 

manila/exception.py

@@ -797,6 +797,11 @@ class NetAppException(ManilaException):
     message = _("Exception due to NetApp failure.")
 
 
+class NetAppBusyAggregateForFlexGroupException(ManilaException):
+    message = _("Exception due to an aggregate being busy while trying to "
+                "provision the FlexGroup.")
+
+
 class VserverNotFound(NetAppException):
     message = _("Vserver %(vserver)s not found.")
 

manila/share/driver.py

@@ -2608,7 +2608,7 @@ class ShareDriver(object):
         """
         raise NotImplementedError()
 
-    def get_filter_function(self):
+    def get_filter_function(self, pool=None):
         """Get filter_function string.
 
         Returns either the string from the driver instance or global section
@@ -2616,14 +2616,16 @@ class ShareDriver(object):
         find the default filter_function. When None is returned the scheduler
         will always pass the driver instance.
 
+        :param pool: pool name to get the filter or None
         :return: a filter_function string or None
         """
         ret_function = self.configuration.filter_function
         if not ret_function:
             ret_function = CONF.filter_function
         if not ret_function:
+            kwargs = {'pool': pool} if pool else {}
             # pylint: disable=assignment-from-none
-            ret_function = self.get_default_filter_function()
+            ret_function = self.get_default_filter_function(**kwargs)
             # pylint: enable=assignment-from-none
         return ret_function
 
@@ -2646,12 +2648,13 @@ class ShareDriver(object):
             # pylint: enable=assignment-from-none
         return ret_function
 
-    def get_default_filter_function(self):
+    def get_default_filter_function(self, pool=None):
         """Get the default filter_function string.
 
         Each driver could overwrite the method to return a well-known
         default string if it is available.
 
+        :param pool: pool name to get the filter or None
         :return: None
         """
         return None

manila/share/drivers/netapp/dataontap/client/client_cmode.py

@@ -74,6 +74,8 @@ class NetAppCmodeClient(client_base.NetAppBaseClient):
         ontapi_1_120 = ontapi_version >= (1, 120)
         ontapi_1_140 = ontapi_version >= (1, 140)
         ontapi_1_150 = ontapi_version >= (1, 150)
+        ontapi_1_180 = ontapi_version >= (1, 180)
+        ontapi_1_191 = ontapi_version >= (1, 191)
         ontap_9_10 = self.get_system_version()['version-tuple'] >= (9, 10, 0)
 
         self.features.add_feature('SNAPMIRROR_V2', supported=ontapi_1_20)
@@ -96,6 +98,8 @@ class NetAppCmodeClient(client_base.NetAppBaseClient):
                                   supported=ontapi_1_150)
         self.features.add_feature('LDAP_LDAP_SERVERS',
                                   supported=ontapi_1_120)
+        self.features.add_feature('FLEXGROUP', supported=ontapi_1_180)
+        self.features.add_feature('FLEXGROUP_FAN_OUT', supported=ontapi_1_191)
         self.features.add_feature('SVM_MIGRATE', supported=ontap_9_10)
 
     def _invoke_vserver_api(self, na_element, vserver):
@@ -123,7 +127,8 @@ class NetAppCmodeClient(client_base.NetAppBaseClient):
         self.connection.set_vserver(vserver)
 
     def send_iter_request(self, api_name, api_args=None,
-                          max_page_length=DEFAULT_MAX_PAGE_LENGTH):
+                          max_page_length=DEFAULT_MAX_PAGE_LENGTH,
+                          enable_tunneling=True):
         """Invoke an iterator-style getter API."""
 
         if not api_args:
@@ -132,7 +137,8 @@ class NetAppCmodeClient(client_base.NetAppBaseClient):
         api_args['max-records'] = max_page_length
 
         # Get first page
-        result = self.send_request(api_name, api_args)
+        result = self.send_request(api_name, api_args,
+                                   enable_tunneling=enable_tunneling)
 
         # Most commonly, we can just return here if there is no more data
         next_tag = result.get_child_content('next-tag')
@@ -150,7 +156,8 @@ class NetAppCmodeClient(client_base.NetAppBaseClient):
         while next_tag is not None:
             next_api_args = copy.deepcopy(api_args)
             next_api_args['tag'] = next_tag
-            next_result = self.send_request(api_name, next_api_args)
+            next_result = self.send_request(api_name, next_api_args,
+                                            enable_tunneling=enable_tunneling)
 
             next_attributes_list = next_result.get_child_by_name(
                 'attributes-list') or netapp_api.NaElement('none')
@@ -2044,6 +2051,63 @@ class NetAppCmodeClient(client_base.NetAppBaseClient):
             'containing-aggr-name': aggregate_name,
             'size': six.text_type(size_gb) + 'g',
             'volume': volume_name,
+        }
+        api_args.update(self._get_create_volume_api_args(
+            volume_name, thin_provisioned, snapshot_policy, language,
+            snapshot_reserve, volume_type, qos_policy_group, encrypt,
+            adaptive_qos_policy_group))
+
+        self.send_request('volume-create', api_args)
+
+        self.update_volume_efficiency_attributes(volume_name,
+                                                 dedup_enabled,
+                                                 compression_enabled)
+        if max_files is not None:
+            self.set_volume_max_files(volume_name, max_files)
+
+    @na_utils.trace
+    def create_volume_async(self, aggregate_list, volume_name, size_gb,
+                            thin_provisioned=False, snapshot_policy=None,
+                            language=None, snapshot_reserve=None,
+                            volume_type='rw', qos_policy_group=None,
+                            encrypt=False, adaptive_qos_policy_group=None,
+                            auto_provisioned=False, **options):
+        """Creates a volume asynchronously."""
+
+        if adaptive_qos_policy_group and not self.features.ADAPTIVE_QOS:
+            msg = 'Adaptive QoS not supported on this backend ONTAP version.'
+            raise exception.NetAppException(msg)
+
+        api_args = {
+            'size': size_gb * units.Gi,
+            'volume-name': volume_name,
+        }
+        if auto_provisioned:
+            api_args['auto-provision-as'] = 'flexgroup'
+        else:
+            api_args['aggr-list'] = [{'aggr-name': aggr}
+                                     for aggr in aggregate_list]
+        api_args.update(self._get_create_volume_api_args(
+            volume_name, thin_provisioned, snapshot_policy, language,
+            snapshot_reserve, volume_type, qos_policy_group, encrypt,
+            adaptive_qos_policy_group))
+
+        result = self.send_request('volume-create-async', api_args)
+        job_info = {
+            'status': result.get_child_content('result-status'),
+            'jobid': result.get_child_content('result-jobid'),
+            'error-code': result.get_child_content('result-error-code'),
+            'error-message': result.get_child_content('result-error-message')
+        }
+
+        return job_info
+
+    def _get_create_volume_api_args(self, volume_name, thin_provisioned,
+                                    snapshot_policy, language,
+                                    snapshot_reserve, volume_type,
+                                    qos_policy_group, encrypt,
+                                    adaptive_qos_policy_group):
+        api_args = {
             'volume-type': volume_type,
         }
         if volume_type != 'dp':
@@ -2055,8 +2119,7 @@ class NetAppCmodeClient(client_base.NetAppBaseClient):
         if language is not None:
             api_args['language-code'] = language
         if snapshot_reserve is not None:
-            api_args['percentage-snapshot-reserve'] = six.text_type(
+            api_args['percentage-snapshot-reserve'] = str(snapshot_reserve)
-                snapshot_reserve)
         if qos_policy_group is not None:
             api_args['qos-policy-group-name'] = qos_policy_group
         if adaptive_qos_policy_group is not None:
@@ -2070,13 +2133,7 @@ class NetAppCmodeClient(client_base.NetAppBaseClient):
             else:
                 api_args['encrypt'] = 'true'
 
-        self.send_request('volume-create', api_args)
+        return api_args
-
-        self.update_volume_efficiency_attributes(volume_name,
-                                                 dedup_enabled,
-                                                 compression_enabled)
-        if max_files is not None:
-            self.set_volume_max_files(volume_name, max_files)
 
     @na_utils.trace
     def enable_dedup(self, volume_name):
@@ -2108,6 +2165,36 @@ class NetAppCmodeClient(client_base.NetAppBaseClient):
         }
         self.send_request('sis-set-config', api_args)
 
+    @na_utils.trace
+    def enable_dedupe_async(self, volume_name):
+        """Enable deduplication on FlexVol/FlexGroup volume asynchronously."""
+        api_args = {'volume-name': volume_name}
+        self.connection.send_request('sis-enable-async', api_args)
+
+    @na_utils.trace
+    def disable_dedupe_async(self, volume_name):
+        """Disable deduplication on FlexVol/FlexGroup volume asynchronously."""
+        api_args = {'volume-name': volume_name}
+        self.connection.send_request('sis-disable-async', api_args)
+
+    @na_utils.trace
+    def enable_compression_async(self, volume_name):
+        """Enable compression on FlexVol/FlexGroup volume asynchronously."""
+        api_args = {
+            'volume-name': volume_name,
+            'enable-compression': 'true'
+        }
+        self.connection.send_request('sis-set-config-async', api_args)
+
+    @na_utils.trace
+    def disable_compression_async(self, volume_name):
+        """Disable compression on FlexVol/FlexGroup volume asynchronously."""
+        api_args = {
+            'volume-name': volume_name,
+            'enable-compression': 'false'
+        }
+        self.connection.send_request('sis-set-config-async', api_args)
+
     @na_utils.trace
     def get_volume_efficiency_status(self, volume_name):
         """Get dedupe & compression status for a volume."""
@@ -2312,7 +2399,24 @@ class NetAppCmodeClient(client_base.NetAppBaseClient):
                       qos_policy_group=None, hide_snapdir=None,
                       autosize_attributes=None,
                       adaptive_qos_policy_group=None, **options):
-        """Update backend volume for a share as necessary."""
+        """Update backend volume for a share as necessary.
+
+        :param aggregate_name: either a list or a string. List for aggregate
+            names where the FlexGroup resides, while a string for the aggregate
+            name where FlexVol volume is.
+        :param volume_name: name of the modified volume.
+        :param thin_provisioned: volume is thin.
+        :param snapshot_policy: policy of volume snapshot.
+        :param language: language of the volume.
+        :param dedup_enabled: is the deduplication enabled for the volume.
+        :param compression_enabled: is the compression enabled for the volume.
+        :param max_files: number of maximum files in the volume.
+        :param qos_policy_group: name of the QoS policy.
+        :param hide_snapdir: hide snapshot directory.
+        :param autosize_attributes: autosize for the volume.
+        :param adaptive_qos_policy_group: name of the adaptive QoS policy.
+        """
+
         if adaptive_qos_policy_group and not self.features.ADAPTIVE_QOS:
             msg = 'Adaptive QoS not supported on this backend ONTAP version.'
             raise exception.NetAppException(msg)
@@ -2321,7 +2425,6 @@ class NetAppCmodeClient(client_base.NetAppBaseClient):
             'query': {
                 'volume-attributes': {
                     'volume-id-attributes': {
-                        'containing-aggregate-name': aggregate_name,
                         'name': volume_name,
                     },
                 },
@@ -2341,6 +2444,16 @@ class NetAppCmodeClient(client_base.NetAppBaseClient):
                 },
             },
         }
+        if isinstance(aggregate_name, str):
+            is_flexgroup = False
+            api_args['query']['volume-attributes']['volume-id-attributes'][
+                'containing-aggregate-name'] = aggregate_name
+        elif isinstance(aggregate_name, list):
+            is_flexgroup = True
+            aggr_list = [{'aggr-name': aggr_name} for aggr_name in
+                         aggregate_name]
+            api_args['query']['volume-attributes']['volume-id-attributes'][
+                'aggr-list'] = aggr_list
         if language:
             api_args['attributes']['volume-attributes'][
                 'volume-language-attributes']['language'] = language
@@ -2373,11 +2486,13 @@ class NetAppCmodeClient(client_base.NetAppBaseClient):
         # Efficiency options must be handled separately
         self.update_volume_efficiency_attributes(volume_name,
                                                  dedup_enabled,
-                                                 compression_enabled)
+                                                 compression_enabled,
+                                                 is_flexgroup=is_flexgroup)
 
     @na_utils.trace
     def update_volume_efficiency_attributes(self, volume_name, dedup_enabled,
-                                            compression_enabled):
+                                            compression_enabled,
+                                            is_flexgroup=False):
         """Update dedupe & compression attributes to match desired values."""
         efficiency_status = self.get_volume_efficiency_status(volume_name)
 
@@ -2386,15 +2501,27 @@ class NetAppCmodeClient(client_base.NetAppBaseClient):
 
         # enable/disable dedup if needed
         if dedup_enabled and not efficiency_status['dedupe']:
-            self.enable_dedup(volume_name)
+            if is_flexgroup:
+                self.enable_dedupe_async(volume_name)
+            else:
+                self.enable_dedup(volume_name)
         elif not dedup_enabled and efficiency_status['dedupe']:
-            self.disable_dedup(volume_name)
+            if is_flexgroup:
+                self.disable_dedupe_async(volume_name)
+            else:
+                self.disable_dedup(volume_name)
 
         # enable/disable compression if needed
         if compression_enabled and not efficiency_status['compression']:
-            self.enable_compression(volume_name)
+            if is_flexgroup:
+                self.enable_compression_async(volume_name)
+            else:
+                self.enable_compression(volume_name)
         elif not compression_enabled and efficiency_status['compression']:
-            self.disable_compression(volume_name)
+            if is_flexgroup:
+                self.disable_compression_async(volume_name)
+            else:
+                self.disable_compression(volume_name)
 
     @na_utils.trace
     def volume_exists(self, volume_name):
@@ -2470,6 +2597,9 @@ class NetAppCmodeClient(client_base.NetAppBaseClient):
             'desired-attributes': {
                 'volume-attributes': {
                     'volume-id-attributes': {
+                        'aggr-list': {
+                            'aggr-name': None,
+                        },
                         'containing-aggregate-name': None,
                         'name': None,
                     },
@@ -2487,6 +2617,11 @@ class NetAppCmodeClient(client_base.NetAppBaseClient):
 
         aggregate = volume_id_attributes.get_child_content(
             'containing-aggregate-name')
+        if not aggregate:
+            aggr_list_attr = volume_id_attributes.get_child_by_name(
+                'aggr-list') or netapp_api.NaElement('none')
+            aggregate = [aggr_elem.get_content()
+                         for aggr_elem in aggr_list_attr.get_children()]
 
         if not aggregate:
             msg = _('Could not find aggregate for volume %s.')
@@ -2515,9 +2650,8 @@ class NetAppCmodeClient(client_base.NetAppBaseClient):
         return self._has_records(result)
 
     @na_utils.trace
-    def volume_has_junctioned_volumes(self, volume_name):
+    def volume_has_junctioned_volumes(self, junction_path):
         """Checks if volume has volumes mounted beneath its junction path."""
-        junction_path = self.get_volume_junction_path(volume_name)
         if not junction_path:
             return False
 
@@ -2575,12 +2709,16 @@ class NetAppCmodeClient(client_base.NetAppBaseClient):
             'desired-attributes': {
                 'volume-attributes': {
                     'volume-id-attributes': {
+                        'aggr-list': {
+                            'aggr-name': None,
+                        },
                         'containing-aggregate-name': None,
                         'junction-path': None,
                         'name': None,
                         'owning-vserver-name': None,
                         'type': None,
                         'style': None,
+                        'style-extended': None,
                     },
                     'volume-qos-attributes': {
                         'policy-group-name': None,
@@ -2613,9 +2751,19 @@ class NetAppCmodeClient(client_base.NetAppBaseClient):
         volume_space_attributes = volume_attributes.get_child_by_name(
             'volume-space-attributes') or netapp_api.NaElement('none')
 
+        aggregate = volume_id_attributes.get_child_content(
+            'containing-aggregate-name')
+        aggregate_list = []
+        if not aggregate:
+            aggregate = ''
+            aggr_list_attr = volume_id_attributes.get_child_by_name(
+                'aggr-list') or netapp_api.NaElement('none')
+            aggregate_list = [aggr_elem.get_content()
+                              for aggr_elem in aggr_list_attr.get_children()]
+
         volume = {
-            'aggregate': volume_id_attributes.get_child_content(
+            'aggregate': aggregate,
-                'containing-aggregate-name'),
+            'aggr-list': aggregate_list,
             'junction-path': volume_id_attributes.get_child_content(
                 'junction-path'),
             'name': volume_id_attributes.get_child_content('name'),
@@ -2625,7 +2773,9 @@ class NetAppCmodeClient(client_base.NetAppBaseClient):
             'style': volume_id_attributes.get_child_content('style'),
             'size': volume_space_attributes.get_child_content('size'),
             'qos-policy-group-name': volume_qos_attributes.get_child_content(
-                'policy-group-name')
+                'policy-group-name'),
+            'style-extended': volume_id_attributes.get_child_content(
+                'style-extended')
         }
         return volume
 
@@ -2640,21 +2790,17 @@ class NetAppCmodeClient(client_base.NetAppBaseClient):
                 'volume-attributes': {
                     'volume-id-attributes': {
                         'junction-path': junction_path,
+                        'style-extended': '%s|%s' % (
+                            na_utils.FLEXGROUP_STYLE_EXTENDED,
+                            na_utils.FLEXVOL_STYLE_EXTENDED),
                     },
                 },
             },
             'desired-attributes': {
                 'volume-attributes': {
                     'volume-id-attributes': {
-                        'containing-aggregate-name': None,
-                        'junction-path': None,
                         'name': None,
-                        'type': None,
-                        'style': None,
                     },
-                    'volume-space-attributes': {
-                        'size': None,
-                    }
                 },
             },
         }
@@ -2668,30 +2814,26 @@ class NetAppCmodeClient(client_base.NetAppBaseClient):
             'volume-attributes') or netapp_api.NaElement('none')
         volume_id_attributes = volume_attributes.get_child_by_name(
             'volume-id-attributes') or netapp_api.NaElement('none')
-        volume_space_attributes = volume_attributes.get_child_by_name(
-            'volume-space-attributes') or netapp_api.NaElement('none')
 
         volume = {
-            'aggregate': volume_id_attributes.get_child_content(
-                'containing-aggregate-name'),
-            'junction-path': volume_id_attributes.get_child_content(
-                'junction-path'),
             'name': volume_id_attributes.get_child_content('name'),
-            'type': volume_id_attributes.get_child_content('type'),
-            'style': volume_id_attributes.get_child_content('style'),
-            'size': volume_space_attributes.get_child_content('size'),
         }
         return volume
 
     @na_utils.trace
     def get_volume_to_manage(self, aggregate_name, volume_name):
-        """Get flexvol to be managed by Manila."""
+        """Get flexvol to be managed by Manila.
+
+        :param aggregate_name: either a list or a string. List for aggregate
+            names where the FlexGroup resides, while a string for the aggregate
+            name where FlexVol volume is.
+        :param volume_name: name of the managed volume.
+        """
 
         api_args = {
             'query': {
                 'volume-attributes': {
                     'volume-id-attributes': {
-                        'containing-aggregate-name': aggregate_name,
                         'name': volume_name,
                     },
                 },
@@ -2699,6 +2841,9 @@ class NetAppCmodeClient(client_base.NetAppBaseClient):
             'desired-attributes': {
                 'volume-attributes': {
                     'volume-id-attributes': {
+                        'aggr-list': {
+                            'aggr-name': None,
+                        },
                         'containing-aggregate-name': None,
                         'junction-path': None,
                         'name': None,
@@ -2715,6 +2860,15 @@ class NetAppCmodeClient(client_base.NetAppBaseClient):
                 },
             },
         }
+        if isinstance(aggregate_name, str):
+            api_args['query']['volume-attributes']['volume-id-attributes'][
+                'containing-aggregate-name'] = aggregate_name
+        elif isinstance(aggregate_name, list):
+            aggr_list = [{'aggr-name': aggr_name} for aggr_name in
+                         aggregate_name]
+            api_args['query']['volume-attributes']['volume-id-attributes'][
+                'aggr-list'] = aggr_list
+
         result = self.send_iter_request('volume-get-iter', api_args)
         if not self._has_records(result):
             return None
@@ -2730,9 +2884,19 @@ class NetAppCmodeClient(client_base.NetAppBaseClient):
         volume_space_attributes = volume_attributes.get_child_by_name(
             'volume-space-attributes') or netapp_api.NaElement('none')
 
+        aggregate = volume_id_attributes.get_child_content(
+            'containing-aggregate-name')
+        aggregate_list = []
+        if not aggregate:
+            aggregate = ''
+            aggr_list_attr = volume_id_attributes.get_child_by_name(
+                'aggr-list') or netapp_api.NaElement('none')
+            aggregate_list = [aggr_elem.get_content()
+                              for aggr_elem in aggr_list_attr.get_children()]
+
         volume = {
-            'aggregate': volume_id_attributes.get_child_content(
+            'aggregate': aggregate,
-                'containing-aggregate-name'),
+            'aggr-list': aggregate_list,
             'junction-path': volume_id_attributes.get_child_content(
                 'junction-path'),
             'name': volume_id_attributes.get_child_content('name'),
@@ -3991,8 +4155,8 @@ class NetAppCmodeClient(client_base.NetAppBaseClient):
     @na_utils.trace
     def create_snapmirror_vol(self, source_vserver, source_volume,
                               destination_vserver, destination_volume,
-                              schedule=None, policy=None,
+                              relationship_type, schedule=None,
-                              relationship_type='data_protection'):
+                              policy=na_utils.MIRROR_ALL_SNAP_POLICY):
         """Creates a SnapMirror relationship between volumes."""
         self._create_snapmirror(source_vserver, destination_vserver,
                                 source_volume=source_volume,
@@ -4003,7 +4167,7 @@ class NetAppCmodeClient(client_base.NetAppBaseClient):
     @na_utils.trace
     def create_snapmirror_svm(self, source_vserver, destination_vserver,
                               schedule=None, policy=None,
-                              relationship_type='data_protection',
+                              relationship_type=na_utils.DATA_PROTECTION_TYPE,
                               identity_preserve=True,
                               max_transfer_rate=None):
         """Creates a SnapMirror relationship between vServers."""
@@ -4017,7 +4181,7 @@ class NetAppCmodeClient(client_base.NetAppBaseClient):
     def _create_snapmirror(self, source_vserver, destination_vserver,
                            source_volume=None, destination_volume=None,
                            schedule=None, policy=None,
-                           relationship_type='data_protection',
+                           relationship_type=na_utils.DATA_PROTECTION_TYPE,
                            identity_preserve=None, max_transfer_rate=None):
         """Creates a SnapMirror relationship (cDOT 8.2 or later only)."""
         self._ensure_snapmirror_v2()
@@ -4167,7 +4331,7 @@ class NetAppCmodeClient(client_base.NetAppBaseClient):
         self._ensure_snapmirror_v2()
         api_args = {
             'query': {
-                'snapmirror-destination-info': dest_info
+                'snapmirror-destination-info': dest_info,
             },
             'relationship-info-only': (
                 'true' if relationship_info_only else 'false'),
@@ -5479,6 +5643,126 @@ class NetAppCmodeClient(client_base.NetAppBaseClient):
         """Checks if the cluster supports SVM Migrate."""
         return self.features.SVM_MIGRATE
 
+    def get_volume_state(self, name):
+        """Returns volume state for a given name"""
+
+        api_args = {
+            'query': {
+                'volume-attributes': {
+                    'volume-id-attributes': {
+                        'name': name,
+                    },
+                },
+            },
+            'desired-attributes': {
+                'volume-attributes': {
+                    'volume-state-attributes': {
+                        'state': None
+                    }
+                }
+            },
+        }
+
+        result = self.send_iter_request('volume-get-iter', api_args)
+
+        volume_state = ''
+        if self._has_records(result):
+            attributes_list = result.get_child_by_name(
+                'attributes-list') or netapp_api.NaElement('none')
+            volume_attributes = attributes_list.get_child_by_name(
+                'volume-attributes') or netapp_api.NaElement('none')
+            volume_state_attributes = volume_attributes.get_child_by_name(
+                'volume-state-attributes') or netapp_api.NaElement('none')
+            volume_state = volume_state_attributes.get_child_content('state')
+
+        return volume_state
+
+    @na_utils.trace
+    def is_flexgroup_volume(self, volume_name):
+        """Determines if the ONTAP volume is FlexGroup."""
+
+        if not self.is_flexgroup_supported():
+            return False
+
+        api_args = {
+            'query': {
+                'volume-attributes': {
+                    'volume-id-attributes': {
+                        'name': volume_name,
+                    },
+                },
+            },
+            'desired-attributes': {
+                'volume-attributes': {
+                    'volume-id-attributes': {
+                        'style-extended': None,
+                    },
+                },
+            },
+        }
+        result = self.send_request('volume-get-iter', api_args)
+
+        attributes_list = result.get_child_by_name(
+            'attributes-list') or netapp_api.NaElement('none')
+        volume_attributes_list = attributes_list.get_children()
+
+        if not self._has_records(result):
+            raise exception.StorageResourceNotFound(name=volume_name)
+        elif len(volume_attributes_list) > 1:
+            msg = _('More than one volume with volume name %(vol)s found.')
+            msg_args = {'vol': volume_name}
+            raise exception.NetAppException(msg % msg_args)
+
+        volume_attributes = volume_attributes_list[0]
+
+        volume_id_attributes = volume_attributes.get_child_by_name(
+            'volume-id-attributes') or netapp_api.NaElement('none')
+
+        return na_utils.is_style_extended_flexgroup(
+            volume_id_attributes.get_child_content('style-extended'))
+
+    @na_utils.trace
+    def is_flexgroup_supported(self):
+        return self.features.FLEXGROUP
+
+    @na_utils.trace
+    def is_flexgroup_fan_out_supported(self):
+        return self.features.FLEXGROUP_FAN_OUT
+
+    @na_utils.trace
+    def get_job_state(self, job_id):
+        """Returns job state for a given job id."""
+
+        api_args = {
+            'query': {
+                'job-info': {
+                    'job-id': job_id,
+                },
+            },
+            'desired-attributes': {
+                'job-info': {
+                    'job-state': None,
+                },
+            },
+        }
+
+        result = self.send_iter_request('job-get-iter', api_args,
+                                        enable_tunneling=False)
+
+        attributes_list = result.get_child_by_name(
+            'attributes-list') or netapp_api.NaElement('none')
+        job_info_list = attributes_list.get_children()
+        if not self._has_records(result):
+            msg = _('Could not find job with ID %(id)s.')
+            msg_args = {'id': job_id}
+            raise exception.NetAppException(msg % msg_args)
+        elif len(job_info_list) > 1:
+            msg = _('Could not find unique job for ID %(id)s.')
+            msg_args = {'id': job_id}
+            raise exception.NetAppException(msg % msg_args)
+
+        return job_info_list[0].get_child_content('job-state')
+
     # ------------------------ REST CALLS ONLY ------------------------
 
     @na_utils.trace

manila/share/drivers/netapp/dataontap/cluster_mode/data_motion.py

@@ -167,7 +167,8 @@ class DataMotionSession(object):
                                 'last-transfer-end-timestamp'])
         return snapmirrors
 
-    def create_snapmirror(self, source_share_obj, dest_share_obj, mount=False):
+    def create_snapmirror(self, source_share_obj, dest_share_obj,
+                          relationship_type, mount=False):
         """Sets up a SnapMirror relationship between two volumes.
 
         1. Create SnapMirror relationship.
@@ -188,6 +189,7 @@ class DataMotionSession(object):
                                           src_volume_name,
                                           dest_vserver,
                                           dest_volume_name,
+                                          relationship_type,
                                           schedule='hourly')
 
         # 2. Initialize async transfer of the initial data
@@ -204,7 +206,7 @@ class DataMotionSession(object):
                 timeout=replica_config.netapp_mount_replica_timeout)
 
     def delete_snapmirror(self, source_share_obj, dest_share_obj,
-                          release=True):
+                          release=True, relationship_info_only=False):
         """Ensures all information about a SnapMirror relationship is removed.
 
         1. Abort snapmirror
@@ -251,21 +253,16 @@ class DataMotionSession(object):
                                                     vserver_name=src_vserver)
             except Exception:
                 src_client = None
+
             # 3. Cleanup SnapMirror relationship on source
-            try:
+            if src_client:
-                if src_client:
+                src_config = get_backend_configuration(src_backend)
-                    src_client.release_snapmirror_vol(src_vserver,
+                release_timeout = (
-                                                      src_volume_name,
+                    src_config.netapp_snapmirror_release_timeout)
-                                                      dest_vserver,
+                self.wait_for_snapmirror_release_vol(
-                                                      dest_volume_name)
+                    src_vserver, dest_vserver, src_volume_name,
-            except netapp_api.NaApiError as e:
+                    dest_volume_name, relationship_info_only, src_client,
-                with excutils.save_and_reraise_exception() as exc_context:
+                    timeout=release_timeout)
-                    if (e.code == netapp_api.EOBJECTNOTFOUND or
-                            e.code == netapp_api.ESOURCE_IS_DIFFERENT or
-                            "(entry doesn't exist)" in e.message):
-                        # Handle the case where the snapmirror is already
-                        # cleaned up
-                        exc_context.reraise = False
 
     def update_snapmirror(self, source_share_obj, dest_share_obj):
         """Schedule a snapmirror update to happen on the backend."""
@@ -419,7 +416,8 @@ class DataMotionSession(object):
 
     def change_snapmirror_source(self, replica,
                                  orig_source_replica,
-                                 new_source_replica, replica_list):
+                                 new_source_replica, replica_list,
+                                 is_flexgroup=False):
         """Creates SnapMirror relationship from the new source to destination.
 
         1. Delete all snapmirrors involving the replica, but maintain
@@ -443,11 +441,16 @@ class DataMotionSession(object):
             if other_replica['id'] == replica['id']:
                 continue
 
-            # We need to delete ALL snapmirror relationships
+            # deletes all snapmirror relationships involving this replica to
-            # involving this replica but do not remove snapmirror metadata
+            # ensure new relation can be set. For efficient snapmirror, it
-            # so that the new snapmirror relationship is efficient.
+            # does not remove the snapshots, only releasing the relationship
-            self.delete_snapmirror(other_replica, replica, release=False)
+            # info if FlexGroup volume.
-            self.delete_snapmirror(replica, other_replica, release=False)
+            self.delete_snapmirror(other_replica, replica,
+                                   release=is_flexgroup,
+                                   relationship_info_only=is_flexgroup)
+            self.delete_snapmirror(replica, other_replica,
+                                   release=is_flexgroup,
+                                   relationship_info_only=is_flexgroup)
 
         # 2. vserver operations when driver handles share servers
         replica_config = get_backend_configuration(replica_backend)
@@ -471,11 +474,14 @@ class DataMotionSession(object):
 
         # 3. create
         # TODO(ameade): Update the schedule if needed.
+        relationship_type = na_utils.get_relationship_type(is_flexgroup)
         replica_client.create_snapmirror_vol(new_src_vserver,
                                              new_src_volume_name,
                                              replica_vserver,
                                              replica_volume_name,
+                                             relationship_type,
                                              schedule='hourly')
+
         # 4. resync
         replica_client.resync_snapmirror_vol(new_src_vserver,
                                              new_src_volume_name,
@@ -793,3 +799,42 @@ class DataMotionSession(object):
                     "because a snapmirror initialize operation is still in "
                     "progress. Retries exhausted. Not retrying.") % msg_args
             raise exception.NetAppException(message=msg)
+
+    def wait_for_snapmirror_release_vol(self, src_vserver, dest_vserver,
+                                        src_volume_name, dest_volume_name,
+                                        relationship_info_only, src_client,
+                                        timeout=300):
+        interval = 10
+        retries = (timeout / interval or 1)
+
+        @utils.retry(exception.NetAppException, interval=interval,
+                     retries=retries, backoff_rate=1)
+        def release_snapmirror():
+            snapmirrors = src_client.get_snapmirror_destinations(
+                source_vserver=src_vserver, dest_vserver=dest_vserver,
+                source_volume=src_volume_name, dest_volume=dest_volume_name)
+            if not snapmirrors:
+                LOG.debug("No snapmirrors to be released in source volume.")
+            else:
+                try:
+                    src_client.release_snapmirror_vol(
+                        src_vserver, src_volume_name, dest_vserver,
+                        dest_volume_name,
+                        relationship_info_only=relationship_info_only)
+                except netapp_api.NaApiError as e:
+                    if (e.code == netapp_api.EOBJECTNOTFOUND or
+                            e.code == netapp_api.ESOURCE_IS_DIFFERENT or
+                            "(entry doesn't exist)" in e.message):
+                        LOG.debug('Snapmirror relationship does not exist '
+                                  'anymore.')
+
+                msg = _('Snapmirror release sent to source volume. Waiting '
+                        'until it has been released.')
+                raise exception.NetAppException(vserver=msg)
+
+        try:
+            release_snapmirror()
+        except exception.NetAppException:
+            msg = _("Unable to release the snapmirror from source volume %s. "
+                    "Retries exhausted. Aborting") % src_volume_name
+            raise exception.NetAppException(message=msg)

manila/share/drivers/netapp/dataontap/cluster_mode/drv_multi_svm.py

@@ -113,13 +113,13 @@ class NetAppCmodeMultiSvmShareDriver(driver.ShareDriver):
 
     def _update_share_stats(self, data=None):
         data = self.library.get_share_stats(
-            filter_function=self.get_filter_function(),
+            get_filter_function=self.get_filter_function,
             goodness_function=self.get_goodness_function())
         super(NetAppCmodeMultiSvmShareDriver, self)._update_share_stats(
             data=data)
 
-    def get_default_filter_function(self):
+    def get_default_filter_function(self, pool=None):
-        return self.library.get_default_filter_function()
+        return self.library.get_default_filter_function(pool=pool)
 
     def get_default_goodness_function(self):
         return self.library.get_default_goodness_function()

manila/share/drivers/netapp/dataontap/cluster_mode/drv_single_svm.py

@@ -105,13 +105,13 @@ class NetAppCmodeSingleSvmShareDriver(driver.ShareDriver):
 
     def _update_share_stats(self, data=None):
         data = self.library.get_share_stats(
-            filter_function=self.get_filter_function(),
+            get_filter_function=self.get_filter_function,
             goodness_function=self.get_goodness_function())
         super(NetAppCmodeSingleSvmShareDriver, self)._update_share_stats(
             data=data)
 
-    def get_default_filter_function(self):
+    def get_default_filter_function(self, pool=None):
-        return self.library.get_default_filter_function()
+        return self.library.get_default_filter_function(pool=pool)
 
     def get_default_goodness_function(self):
         return self.library.get_default_goodness_function()

manila/share/drivers/netapp/dataontap/cluster_mode/lib_multi_svm.py

@@ -65,8 +65,13 @@ class NetAppCmodeMultiSVMFileStorageLibrary(
                     'configuration when the driver is managing share servers.')
             raise exception.InvalidInput(reason=msg)
 
+        # Ensure FlexGroup support
+        aggr_list = self._client.list_non_root_aggregates()
+        self._initialize_flexgroup_pools(set(aggr_list))
+
         # Ensure one or more aggregates are available.
-        if not self._find_matching_aggregates():
+        if (self.is_flexvol_pool_configured() and
+                not self._find_matching_aggregates(aggregate_names=aggr_list)):
             msg = _('No aggregates are available for provisioning shares. '
                     'Ensure that the configuration option '
                     'netapp_aggregate_name_search_pattern is set correctly.')
@@ -110,6 +115,7 @@ class NetAppCmodeMultiSVMFileStorageLibrary(
             'pools': {
                 'vserver': None,
                 'aggregates': self._find_matching_aggregates(),
+                'flexgroup_aggregates': self._flexgroup_pools,
             },
         }
 
@@ -124,9 +130,15 @@ class NetAppCmodeMultiSVMFileStorageLibrary(
             _handle_housekeeping_tasks())
 
     @na_utils.trace
-    def _find_matching_aggregates(self):
+    def _find_matching_aggregates(self, aggregate_names=None):
         """Find all aggregates match pattern."""
-        aggregate_names = self._client.list_non_root_aggregates()
+
+        if not self.is_flexvol_pool_configured():
+            return []
+
+        if not aggregate_names:
+            aggregate_names = self._client.list_non_root_aggregates()
+
         pattern = self.configuration.netapp_aggregate_name_search_pattern
         return [aggr_name for aggr_name in aggregate_names
                 if re.match(pattern, aggr_name)]
@@ -242,23 +254,26 @@ class NetAppCmodeMultiSVMFileStorageLibrary(
         ipspace_name = self._client.get_ipspace_name_for_vlan_port(
             node_name, port, vlan) or self._create_ipspace(network_info)
 
+        aggregate_names = self._find_matching_aggregates()
         if is_dp_destination:
             # Get Data ONTAP aggregate name as pool name.
             LOG.debug('Creating a new Vserver (%s) for data protection.',
                       vserver_name)
             self._client.create_vserver_dp_destination(
                 vserver_name,
-                self._find_matching_aggregates(),
+                aggregate_names,
                 ipspace_name)
             # Set up port and broadcast domain for the current ipspace
             self._create_port_and_broadcast_domain(ipspace_name, network_info)
         else:
             LOG.debug('Vserver %s does not exist, creating.', vserver_name)
+            aggr_set = set(aggregate_names).union(
+                self._get_flexgroup_aggr_set())
             self._client.create_vserver(
                 vserver_name,
                 self.configuration.netapp_root_volume_aggregate,
                 self.configuration.netapp_root_volume,
-                self._find_matching_aggregates(),
+                aggr_set,
                 ipspace_name)
 
             vserver_client = self._get_api_client(vserver=vserver_name)
@@ -680,8 +695,9 @@ class NetAppCmodeMultiSVMFileStorageLibrary(
                                    share_server=None, parent_share=None):
         # NOTE(dviroel): If both parent and child shares are in the same host,
         # they belong to the same cluster, and we can skip all the processing
-        # below.
+        # below. Group snapshot is always to the same host too, so we can skip.
-        if parent_share['host'] != share['host']:
+        is_group_snapshot = share.get('source_share_group_snapshot_member_id')
+        if not is_group_snapshot and parent_share['host'] != share['host']:
             # 1. Retrieve source and destination vservers from source and
             # destination shares
             dm_session = data_motion.DataMotionSession()
@@ -922,11 +938,24 @@ class NetAppCmodeMultiSVMFileStorageLibrary(
         method = SERVER_MIGRATE_SVM_DR
         if (not src_client.is_svm_dr_supported()
                 or not dest_client.is_svm_dr_supported()):
-            msg = _("Cannot perform server migration because at least one of "
+            msg = _("Cannot perform server migration because at leat one of "
                     "the backends doesn't support SVM DR.")
             LOG.error(msg)
             return method, False
 
+        # Check that server does not have any FlexGroup volume.
+        if src_client.is_flexgroup_supported():
+            dm_session = data_motion.DataMotionSession()
+            for req_spec in shares_request_spec.get('shares_req_spec', []):
+                share_instance = req_spec.get('share_instance_properties', {})
+                host = share_instance.get('host')
+                if self.is_flexgroup_destination_host(host, dm_session):
+                    msg = _("Cannot perform server migration since a "
+                            "FlexGroup was encountered in share server to be "
+                            "migrated.")
+                    LOG.error(msg)
+                    return method, False
+
         # Check capacity.
         server_total_size = (shares_request_spec.get('shares_size', 0) +
                              shares_request_spec.get('snapshots_size', 0))
@@ -1175,6 +1204,7 @@ class NetAppCmodeMultiSVMFileStorageLibrary(
         # requests.
         dst_client = data_motion.get_client_for_backend(dest_backend_name,
                                                         vserver_name=None)
+
         migration_method, compatibility = self._check_for_migration_support(
             src_client, dst_client, source_share_server, shares_request_spec,
             src_cluster_name, pools)

manila/share/drivers/netapp/dataontap/cluster_mode/lib_single_svm.py

@@ -63,8 +63,14 @@ class NetAppCmodeSingleSVMFileStorageLibrary(
                         'match supplied credentials.')
                 raise exception.InvalidInput(reason=msg)
 
+        # Ensure FlexGroup support
+        vserver_client = self._get_api_client(vserver=self._vserver)
+        aggr_list = vserver_client.list_vserver_aggregates()
+        self._initialize_flexgroup_pools(set(aggr_list))
+
         # Ensure one or more aggregates are available to the vserver.
-        if not self._find_matching_aggregates():
+        if (self.is_flexvol_pool_configured() and
+                not self._find_matching_aggregates(aggregate_names=aggr_list)):
             msg = _('No aggregates are available to Vserver %s for '
                     'provisioning shares. Ensure that one or more aggregates '
                     'are assigned to the Vserver and that the configuration '
@@ -105,6 +111,7 @@ class NetAppCmodeSingleSVMFileStorageLibrary(
             'pools': {
                 'vserver': self._vserver,
                 'aggregates': self._find_matching_aggregates(),
+                'flexgroup_aggregates': self._flexgroup_pools,
             },
         }
 
@@ -123,10 +130,15 @@ class NetAppCmodeSingleSVMFileStorageLibrary(
             _handle_housekeeping_tasks())
 
     @na_utils.trace
-    def _find_matching_aggregates(self):
+    def _find_matching_aggregates(self, aggregate_names=None):
-        """Find all aggregates match pattern."""
+        """Find all aggregates match pattern if FlexVol pool is configured."""
-        vserver_client = self._get_api_client(vserver=self._vserver)
+
-        aggregate_names = vserver_client.list_vserver_aggregates()
+        if not self.is_flexvol_pool_configured():
+            return []
+
+        if not aggregate_names:
+            vserver_client = self._get_api_client(vserver=self._vserver)
+            aggregate_names = vserver_client.list_vserver_aggregates()
 
         root_aggregate_names = []
         if self._have_cluster_creds:

manila/share/drivers/netapp/dataontap/cluster_mode/performance.py

@@ -131,9 +131,15 @@ class PerformanceLibrary(object):
 
         aggr_names = set()
         for pool_name, pool_info in aggregate_pools.items():
+            if pool_info.get('netapp_flexgroup', False):
+                continue
             aggr_names.add(pool_info.get('netapp_aggregate'))
+
         for pool_name, pool_info in flexvol_pools.items():
+            if pool_info.get('netapp_flexgroup', False):
+                continue
             aggr_names.add(pool_info.get('netapp_aggregate'))
+
         return list(aggr_names)
 
     def _get_nodes_for_aggregates(self, aggr_names):

manila/share/drivers/netapp/dataontap/protocols/base.py

@@ -71,7 +71,8 @@ class NetAppBaseHelper(object):
     @abc.abstractmethod
     def create_share(self, share, share_name,
                      clear_current_export_policy=True,
-                     ensure_share_already_exists=False, replica=False):
+                     ensure_share_already_exists=False, replica=False,
+                     is_flexgroup=False):
         """Creates NAS share."""
 
     @abc.abstractmethod

manila/share/drivers/netapp/dataontap/protocols/cifs_cmode.py

@@ -30,7 +30,8 @@ class NetAppCmodeCIFSHelper(base.NetAppBaseHelper):
     @na_utils.trace
     def create_share(self, share, share_name,
                      clear_current_export_policy=True,
-                     ensure_share_already_exists=False, replica=False):
+                     ensure_share_already_exists=False, replica=False,
+                     is_flexgroup=False):
         """Creates CIFS share if does not exist on Data ONTAP Vserver.
 
         The new CIFS share has Everyone access, so it removes all access after
@@ -41,6 +42,7 @@ class NetAppCmodeCIFSHelper(base.NetAppBaseHelper):
         :param clear_current_export_policy: ignored, NFS only.
         :param ensure_share_already_exists: ensures that CIFS share exists.
         :param replica: it is a replica volume (DP type).
+        :param is_flexgroup: whether the share is a FlexGroup or not.
         """
 
         cifs_exist = self._client.cifs_share_exists(share_name)

manila/share/drivers/netapp/dataontap/protocols/nfs_cmode.py

@@ -42,7 +42,8 @@ class NetAppCmodeNFSHelper(base.NetAppBaseHelper):
     @na_utils.trace
     def create_share(self, share, share_name,
                      clear_current_export_policy=True,
-                     ensure_share_already_exists=False, replica=False):
+                     ensure_share_already_exists=False, replica=False,
+                     is_flexgroup=False):
         """Ensures the share export policy is set correctly.
 
         The export policy must have the same name as the share. If it matches,
@@ -57,12 +58,18 @@ class NetAppCmodeNFSHelper(base.NetAppBaseHelper):
         the check.
         :param ensure_share_already_exists: ignored, CIFS only.
         :param replica: it is a replica volume (DP type).
+        :param is_flexgroup: whether the share is a FlexGroup or not.
         """
 
         if clear_current_export_policy:
             self._client.clear_nfs_export_policy_for_volume(share_name)
         self._ensure_export_policy(share, share_name)
-        export_path = self._client.get_volume_junction_path(share_name)
+
+        if is_flexgroup:
+            volume_info = self._client.get_volume(share_name)
+            export_path = volume_info['junction-path']
+        else:
+            export_path = self._client.get_volume_junction_path(share_name)
 
         # Return a callback that may be used for generating export paths
         # for this share.

manila/share/drivers/netapp/options.py

@@ -21,6 +21,7 @@ place to ensure re usability and better management of configuration options.
 """
 
 from oslo_config import cfg
+from oslo_config import types
 
 netapp_proxy_opts = [
     cfg.StrOpt('netapp_storage_family',
@@ -134,7 +135,48 @@ netapp_provisioning_opts = [
                default=60,
                help='The maximum time in seconds that the cached aggregates '
                     'status will be considered valid. Trying to read the '
-                    'expired cache leads to refreshing it.'), ]
+                    'expired cache leads to refreshing it.'),
+    cfg.BoolOpt('netapp_enable_flexgroup',
+                default=False,
+                help='Specify if the FlexGroup pool is enabled. When it is '
+                     'enabled, the driver will report a single pool '
+                     'representing all aggregates (ONTAP chooses on which the '
+                     'share will be allocated). If you want to Manila control '
+                     'the aggregate selection, you can configure its custom '
+                     'FlexGroup pools through netapp_flexgroup_pools option. '
+                     'The FlexGroup placement is done either by ONTAP or '
+                     'Manila, not both.'),
+    cfg.MultiOpt('netapp_flexgroup_pools',
+                 item_type=types.Dict(value_type=types.String()),
+                 default={},
+                 help="Multi opt of dict to represent the FlexGroup pools. "
+                      "A FlexGroup pool is configured with its name and its "
+                      "list of aggregates. Specify this option as many times "
+                      "as you have FlexGroup pools. Each entry takes the "
+                      "dict config form: "
+                      "netapp_flexgroup_pools = "
+                      "<pool_name>: <aggr_name1> <aggr_name2> .."),
+    cfg.BoolOpt('netapp_flexgroup_pool_only',
+                default=False,
+                help='Specify if the FlexVol pools must not be reported when '
+                     'the netapp_enable_flexgroup is enabled.'),
+    cfg.IntOpt('netapp_flexgroup_volume_online_timeout',
+               min=60,
+               default=360,  # Default to six minutes
+               help='Sets time in seconds to wait for a FlexGroup volume '
+                    'create to complete and go online.'),
+    cfg.IntOpt('netapp_flexgroup_aggregate_not_busy_timeout',
+               min=60,
+               default=360,  # Default to six minutes
+               help='Provisioning FlexGroup share requires that all of its '
+                    'aggregates to not be busy deploying another volume. So, '
+                    'sets time in seconds to retry to create the FlexGroup '
+                    'share.'),
+    cfg.IntOpt('netapp_delete_busy_flexgroup_snapshot_timeout',
+               min=60,
+               default=360,  # Default to six minutes
+               help='Sets time in seconds to wait for a FlexGroup snapshot '
+                    'to not be busy with clones after splitting them.'), ]
 
 netapp_cluster_opts = [
     cfg.StrOpt('netapp_vserver',

manila/share/drivers/netapp/utils.py

@@ -43,6 +43,15 @@ MIGRATION_STATE_READY_FOR_SOURCE_CLEANUP = 'ready_for_source_cleanup'
 MIGRATION_STATE_MIGRATE_COMPLETE = 'migrate_complete'
 MIGRATION_STATE_MIGRATE_PAUSED = 'migrate_paused'
 
+EXTENDED_DATA_PROTECTION_TYPE = 'extended_data_protection'
+MIRROR_ALL_SNAP_POLICY = 'MirrorAllSnapshots'
+DATA_PROTECTION_TYPE = 'data_protection'
+
+FLEXGROUP_STYLE_EXTENDED = 'flexgroup'
+FLEXVOL_STYLE_EXTENDED = 'flexvol'
+
+FLEXGROUP_DEFAULT_POOL_NAME = 'flexgroup_auto'
+
 
 def validate_driver_instantiation(**kwargs):
     """Checks if a driver is instantiated other than by the unified driver.
@@ -123,6 +132,76 @@ def convert_string_to_list(string, separator=','):
     return [elem.strip() for elem in string.split(separator)]
 
 
+def get_relationship_type(is_flexgroup):
+    """Returns the snapmirror relationship type."""
+    return (EXTENDED_DATA_PROTECTION_TYPE if is_flexgroup
+            else DATA_PROTECTION_TYPE)
+
+
+def is_style_extended_flexgroup(style_extended):
+    """Returns whether the style is extended type or not."""
+    return style_extended == FLEXGROUP_STYLE_EXTENDED
+
+
+def parse_flexgroup_pool_config(config, cluster_aggr_set={}, check=False):
+    """Returns the dict with the FlexGroup pools and if it is auto provisioned.
+
+    :param config: the configuration flexgroup list of dict.
+    :param cluster_aggr_set: the set of aggregates in the cluster.
+    :param check: should check the config is correct.
+    """
+
+    flexgroup_pools_map = {}
+    aggr_list_used = []
+    for pool_dic in config:
+        for pool_name, aggr_str in pool_dic.items():
+            aggr_name_list = aggr_str.split()
+
+            if not check:
+                aggr_name_list.sort()
+                flexgroup_pools_map[pool_name] = aggr_name_list
+                continue
+
+            if pool_name in cluster_aggr_set:
+                msg = _('The %s FlexGroup pool name is not valid, because '
+                        'it is a cluster aggregate name. Ensure that the '
+                        'configuration option netapp_flexgroup_pools is '
+                        'set correctly.')
+                raise exception.NetAppException(msg % pool_name)
+
+            aggr_name_set = set(aggr_name_list)
+            if len(aggr_name_set) != len(aggr_name_list):
+                msg = _('There is a repeated aggregate name in the '
+                        'FlexGroup pool %s definition. Ensure that the '
+                        'configuration option netapp_flexgroup_pools is '
+                        'set correctly.')
+                raise exception.NetAppException(msg % pool_name)
+
+            not_found_aggr = aggr_name_set - cluster_aggr_set
+            if not_found_aggr:
+                not_found_list = [str(s) for s in not_found_aggr]
+                not_found_str = ", ".join(not_found_list)
+                msg = _('There is an aggregate name in the FlexGroup pool '
+                        '%(pool)s that is not in the cluster: %(aggr)s. '
+                        'Ensure that the configuration option '
+                        'netapp_flexgroup_pools is set correctly.')
+                msg_args = {'pool': pool_name, 'aggr': not_found_str}
+                raise exception.NetAppException(msg % msg_args)
+
+            aggr_name_list.sort()
+            aggr_name_list_str = "".join(aggr_name_list)
+            if aggr_name_list_str in aggr_list_used:
+                msg = _('The FlexGroup pool %s is duplicated. Ensure that '
+                        'the configuration option netapp_flexgroup_pools '
+                        'is set correctly.')
+                raise exception.NetAppException(msg % pool_name)
+
+            aggr_list_used.append(aggr_name_list_str)
+            flexgroup_pools_map[pool_name] = aggr_name_list
+
+    return flexgroup_pools_map
+
+
 class OpenStackInfo(object):
     """OS/distribution, release, and version.
 

manila/tests/share/drivers/netapp/dataontap/client/fakes.py

@@ -56,6 +56,8 @@ SHARE_AGGREGATE_DISK_TYPES = ['SATA', 'SSD']
 SHARE_NAME = 'fake_share'
 SHARE_SIZE = '1000000000'
 SHARE_NAME_2 = 'fake_share_2'
+FLEXGROUP_STYLE_EXTENDED = 'flexgroup'
+FLEXVOL_STYLE_EXTENDED = 'flexvol'
 SNAPSHOT_NAME = 'fake_snapshot'
 CG_SNAPSHOT_ID = 'fake_cg_id'
 PARENT_SHARE_NAME = 'fake_parent_share'
@@ -120,6 +122,8 @@ FPOLICY_EXT_TO_INCLUDE_LIST = ['avi']
 FPOLICY_EXT_TO_EXCLUDE = 'jpg,mp3'
 FPOLICY_EXT_TO_EXCLUDE_LIST = ['jpg', 'mp3']
 
+JOB_ID = 123
+JOB_STATE = 'success'
 
 NETWORK_INTERFACES = [{
     'interface_name': 'fake_interface',
@@ -2057,6 +2061,26 @@ GET_AGGREGATE_FOR_VOLUME_RESPONSE = etree.XML("""
     'share': SHARE_NAME
 })
 
+GET_AGGREGATE_FOR_FLEXGROUP_VOL_RESPONSE = etree.XML("""
+  <results status="passed">
+    <attributes-list>
+      <volume-attributes>
+        <volume-id-attributes>
+          <aggr-list>
+            <aggr-name>%(aggr)s</aggr-name>
+          </aggr-list>
+          <name>%(share)s</name>
+          <owning-vserver-name>os_aa666789-5576-4835-87b7-868069856459</owning-vserver-name>
+        </volume-id-attributes>
+      </volume-attributes>
+    </attributes-list>
+    <num-records>1</num-records>
+  </results>
+""" % {
+    'aggr': SHARE_AGGREGATE_NAME,
+    'share': SHARE_NAME
+})
+
 VOLUME_AUTOSIZE_GET_RESPONSE = etree.XML("""
   <results status="passed">
     <grow-threshold-percent>%(grow_percent)s</grow-threshold-percent>
@@ -2228,6 +2252,7 @@ VOLUME_GET_ITER_VOLUME_TO_MANAGE_RESPONSE = etree.XML("""
           <owning-vserver-name>%(vserver)s</owning-vserver-name>
           <style>flex</style>
           <type>rw</type>
+          <style-extended>%(style-extended)s</style-extended>
         </volume-id-attributes>
         <volume-space-attributes>
           <size>%(size)s</size>
@@ -2245,6 +2270,41 @@ VOLUME_GET_ITER_VOLUME_TO_MANAGE_RESPONSE = etree.XML("""
     'volume': SHARE_NAME,
     'size': SHARE_SIZE,
     'qos-policy-group-name': QOS_POLICY_GROUP_NAME,
+    'style-extended': FLEXVOL_STYLE_EXTENDED,
+})
+
+VOLUME_GET_ITER_FLEXGROUP_VOLUME_TO_MANAGE_RESPONSE = etree.XML("""
+  <results status="passed">
+    <attributes-list>
+      <volume-attributes>
+        <volume-id-attributes>
+          <aggr-list>
+            <aggr-name>%(aggr)s</aggr-name>
+          </aggr-list>
+          <junction-path>/%(volume)s</junction-path>
+          <name>%(volume)s</name>
+          <owning-vserver-name>%(vserver)s</owning-vserver-name>
+          <style>flex</style>
+          <type>rw</type>
+          <style-extended>%(style-extended)s</style-extended>
+        </volume-id-attributes>
+        <volume-space-attributes>
+          <size>%(size)s</size>
+        </volume-space-attributes>
+        <volume-qos-attributes>
+          <policy-group-name>%(qos-policy-group-name)s</policy-group-name>
+        </volume-qos-attributes>
+      </volume-attributes>
+    </attributes-list>
+    <num-records>1</num-records>
+  </results>
+""" % {
+    'aggr': SHARE_AGGREGATE_NAME,
+    'vserver': VSERVER_NAME,
+    'volume': SHARE_NAME,
+    'size': SHARE_SIZE,
+    'qos-policy-group-name': QOS_POLICY_GROUP_NAME,
+    'style-extended': FLEXGROUP_STYLE_EXTENDED,
 })
 
 VOLUME_GET_ITER_NO_QOS_RESPONSE = etree.XML("""
@@ -2258,6 +2318,7 @@ VOLUME_GET_ITER_NO_QOS_RESPONSE = etree.XML("""
           <owning-vserver-name>%(vserver)s</owning-vserver-name>
           <style>flex</style>
           <type>rw</type>
+          <style-extended>%(style-extended)s</style-extended>
         </volume-id-attributes>
         <volume-space-attributes>
           <size>%(size)s</size>
@@ -2271,6 +2332,7 @@ VOLUME_GET_ITER_NO_QOS_RESPONSE = etree.XML("""
     'vserver': VSERVER_NAME,
     'volume': SHARE_NAME,
     'size': SHARE_SIZE,
+    'style-extended': FLEXVOL_STYLE_EXTENDED,
 })
 
 CLONE_CHILD_1 = 'fake_child_1'
@@ -3097,3 +3159,81 @@ FAKE_MIGRATION_JOB_SUCCESS = {
     "state": "migrate_complete",
     "uuid": "4ea7a442-86d1-11e0-ae1c-123478563412"
 }
+
+
+VOLUME_GET_ITER_STATE_RESPONSE = etree.XML("""
+    <results status="passed">
+        <num-records>1</num-records>
+        <attributes-list>
+            <volume-attributes>
+                <volume-state-attributes>
+                    <state>online</state>
+                </volume-state-attributes>
+            </volume-attributes>
+        </attributes-list>
+    </results>
+""")
+
+ASYNC_OPERATION_RESPONSE = etree.XML("""
+  <results status="passed">
+    <result-status>in_progress</result-status>
+    <result-jobid>123</result-jobid>
+  </results>
+""")
+
+VOLUME_GET_ITER_STYLE_FLEXGROUP_RESPONSE = etree.XML("""
+    <results status="passed">
+        <num-records>1</num-records>
+        <attributes-list>
+            <volume-attributes>
+                <volume-id-attributes>
+                    <style-extended>%(style)s</style-extended>
+                </volume-id-attributes>
+            </volume-attributes>
+        </attributes-list>
+    </results>
+""" % {
+    'style': FLEXGROUP_STYLE_EXTENDED,
+})
+
+VOLUME_GET_ITER_STYLE_FLEXVOL_RESPONSE = etree.XML("""
+    <results status="passed">
+        <num-records>1</num-records>
+        <attributes-list>
+            <volume-attributes>
+                <volume-id-attributes>
+                    <style-extended>flexvol</style-extended>
+                </volume-id-attributes>
+            </volume-attributes>
+        </attributes-list>
+    </results>
+""")
+
+JOB_GET_STATE_RESPONSE = etree.XML("""
+    <results status="passed">
+        <num-records>1</num-records>
+        <attributes-list>
+            <job-info>
+                <job-state>%(state)s</job-state>
+            </job-info>
+        </attributes-list>
+    </results>
+""" % {
+    'state': JOB_STATE,
+})
+
+JOB_GET_STATE_NOT_UNIQUE_RESPONSE = etree.XML("""
+    <results status="passed">
+        <num-records>1</num-records>
+        <attributes-list>
+            <job-info>
+                <job-state>%(state)s</job-state>
+            </job-info>
+            <job-info>
+                <job-state>%(state)s</job-state>
+            </job-info>
+        </attributes-list>
+    </results>
+""" % {
+    'state': JOB_STATE,
+})

manila/tests/share/drivers/netapp/dataontap/client/test_client_cmode.py

@@ -27,6 +27,7 @@ from manila import exception
 from manila.share.drivers.netapp.dataontap.client import api as netapp_api
 from manila.share.drivers.netapp.dataontap.client import client_base
 from manila.share.drivers.netapp.dataontap.client import client_cmode
+from manila.share.drivers.netapp import utils as na_utils
 from manila import test
 from manila.tests.share.drivers.netapp.dataontap.client import fakes as fake
 
@@ -317,9 +318,9 @@ class NetAppClientCmodeTestCase(test.TestCase):
         args3['tag'] = 'next_tag_2'
 
         mock_send_request.assert_has_calls([
-            mock.call('storage-disk-get-iter', args1),
+            mock.call('storage-disk-get-iter', args1, enable_tunneling=True),
-            mock.call('storage-disk-get-iter', args2),
+            mock.call('storage-disk-get-iter', args2, enable_tunneling=True),
-            mock.call('storage-disk-get-iter', args3),
+            mock.call('storage-disk-get-iter', args3, enable_tunneling=True),
         ])
 
     def test_send_iter_request_single_page(self):
@@ -348,7 +349,7 @@ class NetAppClientCmodeTestCase(test.TestCase):
         args['max-records'] = 10
 
         mock_send_request.assert_has_calls([
-            mock.call('storage-disk-get-iter', args),
+            mock.call('storage-disk-get-iter', args, enable_tunneling=True),
         ])
 
     def test_send_iter_request_not_found(self):
@@ -366,7 +367,7 @@ class NetAppClientCmodeTestCase(test.TestCase):
         args = {'max-records': client_cmode.DEFAULT_MAX_PAGE_LENGTH}
 
         mock_send_request.assert_has_calls([
-            mock.call('storage-disk-get-iter', args),
+            mock.call('storage-disk-get-iter', args, enable_tunneling=True),
         ])
 
     @ddt.data(fake.INVALID_GET_ITER_RESPONSE_NO_ATTRIBUTES,
@@ -3086,129 +3087,155 @@ class NetAppClientCmodeTestCase(test.TestCase):
             mock.call('cifs-domain-preferred-dc-remove',
                       preferred_dc_add_args)])
 
-    def test_create_volume(self):
+    @ddt.data(True, False)
-
+    def test_create_volume(self, set_max_files):
-        self.mock_object(self.client, 'send_request')
-        self.mock_object(self.client, 'update_volume_efficiency_attributes')
-
-        self.client.create_volume(
-            fake.SHARE_AGGREGATE_NAME, fake.SHARE_NAME, 100)
-
-        volume_create_args = {
-            'containing-aggr-name': fake.SHARE_AGGREGATE_NAME,
-            'size': '100g',
-            'volume': fake.SHARE_NAME,
-            'volume-type': 'rw',
-            'junction-path': '/%s' % fake.SHARE_NAME,
-        }
-
-        self.client.send_request.assert_called_once_with('volume-create',
-                                                         volume_create_args)
-
-    @ddt.data({'qos_policy_group_name': None,
-               'adaptive_policy_group_name': None},
-              {'qos_policy_group_name': fake.QOS_POLICY_GROUP_NAME,
-               'adaptive_policy_group_name': None},
-              {'qos_policy_group_name': None,
-               'adaptive_policy_group_name':
-                   fake.ADAPTIVE_QOS_POLICY_GROUP_NAME},
-              )
-    @ddt.unpack
-    def test_create_volume_with_extra_specs(self, qos_policy_group_name,
-                                            adaptive_policy_group_name):
         self.client.features.add_feature('ADAPTIVE_QOS')
         self.mock_object(self.client, 'set_volume_max_files')
-        self.mock_object(self.client, 'enable_dedup')
-        self.mock_object(self.client, 'enable_compression')
         self.mock_object(self.client, 'send_request')
+        self.mock_object(self.client, 'update_volume_efficiency_attributes')
         self.mock_object(
-            self.client,
+            self.client, '_get_create_volume_api_args',
-            'get_volume_efficiency_status',
+            mock.Mock(return_value={}))
-            mock.Mock(return_value={'dedupe': False, 'compression': False}))
 
         self.client.create_volume(
             fake.SHARE_AGGREGATE_NAME, fake.SHARE_NAME, 100,
-            thin_provisioned=True, language='en-US',
+            max_files=fake.MAX_FILES if set_max_files else None)
-            snapshot_policy='default', dedup_enabled=True,
-            compression_enabled=True, max_files=5000, snapshot_reserve=15,
-            qos_policy_group=qos_policy_group_name,
-            adaptive_qos_policy_group=adaptive_policy_group_name)
 
         volume_create_args = {
             'containing-aggr-name': fake.SHARE_AGGREGATE_NAME,
             'size': '100g',
             'volume': fake.SHARE_NAME,
-            'junction-path': '/%s' % fake.SHARE_NAME,
-            'space-reserve': 'none',
-            'language-code': 'en-US',
-            'volume-type': 'rw',
-            'snapshot-policy': 'default',
-            'percentage-snapshot-reserve': '15',
         }
 
-        if qos_policy_group_name:
+        self.client._get_create_volume_api_args.assert_called_once_with(
-            volume_create_args.update(
+            fake.SHARE_NAME, False, None, None, None, 'rw', None, False, None)
-                {'qos-policy-group-name': qos_policy_group_name})
-        if adaptive_policy_group_name:
-            volume_create_args.update(
-                {'qos-adaptive-policy-group-name': adaptive_policy_group_name})
-
         self.client.send_request.assert_called_with('volume-create',
                                                     volume_create_args)
-        self.client.set_volume_max_files.assert_called_once_with(
+        (self.client.update_volume_efficiency_attributes.
-            fake.SHARE_NAME, fake.MAX_FILES)
+            assert_called_once_with(fake.SHARE_NAME, False, False))
-        self.client.enable_dedup.assert_called_once_with(fake.SHARE_NAME)
+        if set_max_files:
-        self.client.enable_compression.assert_called_once_with(fake.SHARE_NAME)
+            self.client.set_volume_max_files.assert_called_once_with(
+                fake.SHARE_NAME, fake.MAX_FILES)
+        else:
+            self.client.set_volume_max_files.assert_not_called()
 
-    def test_create_encrypted_volume(self):
+    def test_create_volume_adaptive_not_supported(self):
 
+        self.client.features.add_feature('ADAPTIVE_QOS', supported=False)
         self.mock_object(self.client, 'send_request')
-        self.mock_object(self.client, 'update_volume_efficiency_attributes')
-        self.client.features.add_feature('FLEXVOL_ENCRYPTION')
-
-        self.client.create_volume(
-            fake.SHARE_AGGREGATE_NAME, fake.SHARE_NAME, 100, encrypt=True)
-
-        volume_create_args = {
-            'containing-aggr-name': fake.SHARE_AGGREGATE_NAME,
-            'size': '100g',
-            'volume': fake.SHARE_NAME,
-            'volume-type': 'rw',
-            'junction-path': '/%s' % fake.SHARE_NAME,
-            'encrypt': 'true',
-        }
-
-        self.client.send_request.assert_called_once_with('volume-create',
-                                                         volume_create_args)
-
-    def test_create_non_encrypted_volume(self):
-
-        self.mock_object(self.client, 'send_request')
-        self.mock_object(self.client, 'update_volume_efficiency_attributes')
-        self.client.features.add_feature('FLEXVOL_ENCRYPTION')
-
-        self.client.create_volume(
-            fake.SHARE_AGGREGATE_NAME, fake.SHARE_NAME, 100, encrypt=False)
-
-        volume_create_args = {
-            'containing-aggr-name': fake.SHARE_AGGREGATE_NAME,
-            'size': '100g',
-            'volume': fake.SHARE_NAME,
-            'volume-type': 'rw',
-            'junction-path': '/%s' % fake.SHARE_NAME,
-        }
-
-        self.client.send_request.assert_called_once_with('volume-create',
-                                                         volume_create_args)
-
-    def test_create_encrypted_volume_not_supported(self):
-
         self.assertRaises(exception.NetAppException,
                           self.client.create_volume,
                           fake.SHARE_AGGREGATE_NAME,
                           fake.SHARE_NAME,
                           100,
-                          encrypt=True)
+                          adaptive_qos_policy_group='fake')
+        self.client.send_request.assert_not_called()
+
+    @ddt.data(True, False)
+    def test_create_volume_async(self, auto_provisioned):
+        api_response = netapp_api.NaElement(fake.ASYNC_OPERATION_RESPONSE)
+        self.mock_object(self.client, 'send_request',
+                         mock.Mock(return_value=api_response))
+        self.mock_object(
+            self.client, '_get_create_volume_api_args',
+            mock.Mock(return_value={}))
+
+        result = self.client.create_volume_async(
+            [fake.SHARE_AGGREGATE_NAME], fake.SHARE_NAME, 1,
+            auto_provisioned=auto_provisioned)
+
+        volume_create_args = {
+            'size': 1073741824,
+            'volume-name': fake.SHARE_NAME,
+        }
+        if auto_provisioned:
+            volume_create_args['auto-provision-as'] = 'flexgroup'
+        else:
+            volume_create_args['aggr-list'] = [
+                {'aggr-name': fake.SHARE_AGGREGATE_NAME}]
+
+        expected_result = {
+            'status': 'in_progress',
+            'jobid': '123',
+            'error-code': None,
+            'error-message': None,
+        }
+
+        self.client._get_create_volume_api_args.assert_called_once_with(
+            fake.SHARE_NAME, False, None, None, None, 'rw', None, False, None)
+        self.client.send_request.assert_called_with('volume-create-async',
+                                                    volume_create_args)
+        self.assertEqual(expected_result, result)
+
+    def test_create_volume_async_adaptive_not_supported(self):
+
+        self.client.features.add_feature('ADAPTIVE_QOS', supported=False)
+        self.mock_object(self.client, 'send_request')
+        self.assertRaises(exception.NetAppException,
+                          self.client.create_volume_async,
+                          [fake.SHARE_AGGREGATE_NAME],
+                          fake.SHARE_NAME,
+                          100,
+                          adaptive_qos_policy_group='fake')
+        self.client.send_request.assert_not_called()
+
+    def test_get_create_volume_api_args_with_extra_specs(self):
+
+        self.client.features.add_feature('FLEXVOL_ENCRYPTION')
+        volume_type = 'rw'
+        thin_provisioned = 'none'
+        snapshot_policy = 'default'
+        language = 'en-US'
+        reserve = 15
+        qos_name = 'fake_qos'
+        encrypt = True
+        qos_adaptive_name = 'fake_adaptive_qos'
+
+        result_api_args = self.client._get_create_volume_api_args(
+            fake.SHARE_NAME, thin_provisioned, snapshot_policy, language,
+            reserve, volume_type, qos_name, encrypt, qos_adaptive_name)
+
+        expected_api_args = {
+            'volume-type': volume_type,
+            'junction-path': '/fake_share',
+            'space-reserve': thin_provisioned,
+            'snapshot-policy': snapshot_policy,
+            'language-code': language,
+            'percentage-snapshot-reserve': str(reserve),
+            'qos-policy-group-name': qos_name,
+            'qos-adaptive-policy-group-name': qos_adaptive_name,
+            'encrypt': 'true',
+        }
+        self.assertEqual(expected_api_args, result_api_args)
+
+    def test_get_create_volume_api_args_no_extra_specs(self):
+
+        self.client.features.add_feature('FLEXVOL_ENCRYPTION')
+        volume_type = 'dp'
+        thin_provisioned = False
+        snapshot_policy = None
+        language = None
+        reserve = None
+        qos_name = None
+        encrypt = False
+        qos_adaptive_name = None
+
+        result_api_args = self.client._get_create_volume_api_args(
+            fake.SHARE_NAME, thin_provisioned, snapshot_policy, language,
+            reserve, volume_type, qos_name, encrypt, qos_adaptive_name)
+
+        expected_api_args = {
+            'volume-type': volume_type,
+        }
+        self.assertEqual(expected_api_args, result_api_args)
+
+    def test_get_create_volume_api_args_encrypted_not_supported(self):
+
+        encrypt = True
+        self.assertRaises(exception.NetAppException,
+                          self.client._get_create_volume_api_args,
+                          fake.SHARE_NAME, True, 'default', 'en-US',
+                          15, 'rw', 'fake_qos', encrypt, 'fake_qos_adaptive')
 
     def test_is_flexvol_encrypted_unsupported(self):
 
@@ -3349,6 +3376,53 @@ class NetAppClientCmodeTestCase(test.TestCase):
         self.client.send_request.assert_called_once_with('sis-set-config',
                                                          sis_set_config_args)
 
+    def test_enable_dedupe_async(self):
+        self.mock_object(self.client.connection, 'send_request')
+
+        self.client.enable_dedupe_async(fake.SHARE_NAME)
+
+        sis_enable_args = {'volume-name': fake.SHARE_NAME}
+
+        self.client.connection.send_request.assert_called_once_with(
+            'sis-enable-async', sis_enable_args)
+
+    def test_disable_dedupe_async(self):
+
+        self.mock_object(self.client.connection, 'send_request')
+
+        self.client.disable_dedupe_async(fake.SHARE_NAME)
+
+        sis_enable_args = {'volume-name': fake.SHARE_NAME}
+
+        self.client.connection.send_request.assert_called_once_with(
+            'sis-disable-async', sis_enable_args)
+
+    def test_enable_compression_async(self):
+        self.mock_object(self.client.connection, 'send_request')
+
+        self.client.enable_compression_async(fake.SHARE_NAME)
+
+        sis_set_config_args = {
+            'volume-name': fake.SHARE_NAME,
+            'enable-compression': 'true'
+        }
+
+        self.client.connection.send_request.assert_called_once_with(
+            'sis-set-config-async', sis_set_config_args)
+
+    def test_disable_compression_async(self):
+        self.mock_object(self.client.connection, 'send_request')
+
+        self.client.disable_compression_async(fake.SHARE_NAME)
+
+        sis_set_config_args = {
+            'volume-name': fake.SHARE_NAME,
+            'enable-compression': 'false'
+        }
+
+        self.client.connection.send_request.assert_called_once_with(
+            'sis-set-config-async', sis_set_config_args)
+
     def test_get_volume_efficiency_status(self):
 
         api_response = netapp_api.NaElement(fake.SIS_GET_ITER_RESPONSE)
@@ -3443,19 +3517,23 @@ class NetAppClientCmodeTestCase(test.TestCase):
             'vserver-rename', vserver_api_args
         )
 
-    def test_modify_volume_no_optional_args(self):
+    @ddt.data(True, False)
+    def test_modify_volume_no_optional_args(self, is_flexgroup):
 
         self.mock_object(self.client, 'send_request')
         mock_update_volume_efficiency_attributes = self.mock_object(
             self.client, 'update_volume_efficiency_attributes')
 
-        self.client.modify_volume(fake.SHARE_AGGREGATE_NAME, fake.SHARE_NAME)
+        aggr = fake.SHARE_AGGREGATE_NAME
+        if is_flexgroup:
+            aggr = list(fake.SHARE_AGGREGATE_NAMES)
+
+        self.client.modify_volume(aggr, fake.SHARE_NAME)
 
         volume_modify_iter_api_args = {
             'query': {
                 'volume-attributes': {
                     'volume-id-attributes': {
-                        'containing-aggregate-name': fake.SHARE_AGGREGATE_NAME,
                         'name': fake.SHARE_NAME,
                     },
                 },
@@ -3473,10 +3551,19 @@ class NetAppClientCmodeTestCase(test.TestCase):
             },
         }
 
+        if is_flexgroup:
+            volume_modify_iter_api_args['query']['volume-attributes'][
+                'volume-id-attributes']['aggr-list'] = [
+                {'aggr-name': aggr[0]}, {'aggr-name': aggr[1]}]
+        else:
+            volume_modify_iter_api_args['query']['volume-attributes'][
+                'volume-id-attributes'][
+                'containing-aggregate-name'] = aggr
+
         self.client.send_request.assert_called_once_with(
             'volume-modify-iter', volume_modify_iter_api_args)
         mock_update_volume_efficiency_attributes.assert_called_once_with(
-            fake.SHARE_NAME, False, False)
+            fake.SHARE_NAME, False, False, is_flexgroup=is_flexgroup)
 
     @ddt.data((fake.QOS_POLICY_GROUP_NAME, None),
               (None, fake.ADAPTIVE_QOS_POLICY_GROUP_NAME))
@@ -3550,21 +3637,30 @@ class NetAppClientCmodeTestCase(test.TestCase):
         self.client.send_request.assert_called_once_with(
             'volume-modify-iter', volume_modify_iter_api_args)
         mock_update_volume_efficiency_attributes.assert_called_once_with(
-            fake.SHARE_NAME, True, False)
+            fake.SHARE_NAME, True, False, is_flexgroup=False)
 
     @ddt.data(
-        {'existing': (True, True), 'desired': (True, True)},
+        {'existing': (True, True), 'desired': (True, True), 'fg': False},
-        {'existing': (True, True), 'desired': (False, False)},
+        {'existing': (True, True), 'desired': (False, False), 'fg': False},
-        {'existing': (True, True), 'desired': (True, False)},
+        {'existing': (True, True), 'desired': (True, False), 'fg': False},
-        {'existing': (True, False), 'desired': (True, False)},
+        {'existing': (True, False), 'desired': (True, False), 'fg': False},
-        {'existing': (True, False), 'desired': (False, False)},
+        {'existing': (True, False), 'desired': (False, False), 'fg': False},
-        {'existing': (True, False), 'desired': (True, True)},
+        {'existing': (True, False), 'desired': (True, True), 'fg': False},
-        {'existing': (False, False), 'desired': (False, False)},
+        {'existing': (False, False), 'desired': (False, False), 'fg': False},
-        {'existing': (False, False), 'desired': (True, False)},
+        {'existing': (False, False), 'desired': (True, False), 'fg': False},
-        {'existing': (False, False), 'desired': (True, True)},
+        {'existing': (False, False), 'desired': (True, True), 'fg': False},
+        {'existing': (True, True), 'desired': (True, True), 'fg': True},
+        {'existing': (True, True), 'desired': (False, False), 'fg': True},
+        {'existing': (True, True), 'desired': (True, False), 'fg': True},
+        {'existing': (True, False), 'desired': (True, False), 'fg': True},
+        {'existing': (True, False), 'desired': (False, False), 'fg': True},
+        {'existing': (True, False), 'desired': (True, True), 'fg': True},
+        {'existing': (False, False), 'desired': (False, False), 'fg': True},
+        {'existing': (False, False), 'desired': (True, False), 'fg': True},
+        {'existing': (False, False), 'desired': (True, True), 'fg': True},
     )
     @ddt.unpack
-    def test_update_volume_efficiency_attributes(self, existing, desired):
+    def test_update_volume_efficiency_attributes(self, existing, desired, fg):
 
         existing_dedupe = existing[0]
         existing_compression = existing[1]
@@ -3578,33 +3674,66 @@ class NetAppClientCmodeTestCase(test.TestCase):
                                     'compression': existing_compression}))
         mock_enable_compression = self.mock_object(self.client,
                                                    'enable_compression')
+        mock_enable_compression_async = self.mock_object(
+            self.client, 'enable_compression_async')
         mock_disable_compression = self.mock_object(self.client,
                                                     'disable_compression')
+        mock_disable_compression_async = self.mock_object(
+            self.client, 'disable_compression_async')
         mock_enable_dedup = self.mock_object(self.client, 'enable_dedup')
+        mock_enable_dedup_async = self.mock_object(self.client,
+                                                   'enable_dedupe_async')
         mock_disable_dedup = self.mock_object(self.client, 'disable_dedup')
+        mock_disable_dedup_async = self.mock_object(self.client,
+                                                    'disable_dedupe_async')
 
         self.client.update_volume_efficiency_attributes(
-            fake.SHARE_NAME, desired_dedupe, desired_compression)
+            fake.SHARE_NAME, desired_dedupe, desired_compression,
+            is_flexgroup=fg)
 
         if existing_dedupe == desired_dedupe:
-            self.assertFalse(mock_enable_dedup.called)
+            if fg:
-            self.assertFalse(mock_disable_dedup.called)
+                self.assertFalse(mock_enable_dedup_async.called)
+                self.assertFalse(mock_disable_dedup_async.called)
+            else:
+                self.assertFalse(mock_enable_dedup.called)
+                self.assertFalse(mock_disable_dedup.called)
         elif existing_dedupe and not desired_dedupe:
-            self.assertFalse(mock_enable_dedup.called)
+            if fg:
-            self.assertTrue(mock_disable_dedup.called)
+                self.assertFalse(mock_enable_dedup_async.called)
+                self.assertTrue(mock_disable_dedup_async.called)
+            else:
+                self.assertFalse(mock_enable_dedup.called)
+                self.assertTrue(mock_disable_dedup.called)
         elif not existing_dedupe and desired_dedupe:
-            self.assertTrue(mock_enable_dedup.called)
+            if fg:
-            self.assertFalse(mock_disable_dedup.called)
+                self.assertTrue(mock_enable_dedup_async.called)
+                self.assertFalse(mock_disable_dedup_async.called)
+            else:
+                self.assertTrue(mock_enable_dedup.called)
+                self.assertFalse(mock_disable_dedup.called)
 
         if existing_compression == desired_compression:
-            self.assertFalse(mock_enable_compression.called)
+            if fg:
-            self.assertFalse(mock_disable_compression.called)
+                self.assertFalse(mock_enable_compression_async.called)
+                self.assertFalse(mock_disable_compression_async.called)
+            else:
+                self.assertFalse(mock_enable_compression.called)
+                self.assertFalse(mock_disable_compression.called)
         elif existing_compression and not desired_compression:
-            self.assertFalse(mock_enable_compression.called)
+            if fg:
-            self.assertTrue(mock_disable_compression.called)
+                self.assertFalse(mock_enable_compression_async.called)
+                self.assertTrue(mock_disable_compression_async.called)
+            else:
+                self.assertFalse(mock_enable_compression.called)
+                self.assertTrue(mock_disable_compression.called)
         elif not existing_compression and desired_compression:
-            self.assertTrue(mock_enable_compression.called)
+            if fg:
-            self.assertFalse(mock_disable_compression.called)
+                self.assertTrue(mock_enable_compression_async.called)
+                self.assertFalse(mock_disable_compression_async.called)
+            else:
+                self.assertTrue(mock_enable_compression.called)
+                self.assertFalse(mock_disable_compression.called)
 
     def test_set_volume_size(self):
 
@@ -3865,10 +3994,12 @@ class NetAppClientCmodeTestCase(test.TestCase):
                           fake.SNAPSHOT_NAME,
                           fake.SHARE_NAME)
 
-    def test_get_aggregate_for_volume(self):
+    @ddt.data(True, False)
+    def test_get_aggregate_for_volume(self, is_flexgroup):
 
         api_response = netapp_api.NaElement(
-            fake.GET_AGGREGATE_FOR_VOLUME_RESPONSE)
+            fake.GET_AGGREGATE_FOR_FLEXGROUP_VOL_RESPONSE if is_flexgroup
+            else fake.GET_AGGREGATE_FOR_VOLUME_RESPONSE)
         self.mock_object(self.client,
                          'send_iter_request',
                          mock.Mock(return_value=api_response))
@@ -3886,6 +4017,9 @@ class NetAppClientCmodeTestCase(test.TestCase):
             'desired-attributes': {
                 'volume-attributes': {
                     'volume-id-attributes': {
+                        'aggr-list': {
+                            'aggr-name': None,
+                        },
                         'containing-aggregate-name': None,
                         'name': None
                     }
@@ -3895,7 +4029,10 @@ class NetAppClientCmodeTestCase(test.TestCase):
 
         self.client.send_iter_request.assert_has_calls([
             mock.call('volume-get-iter', volume_get_iter_args)])
-        self.assertEqual(fake.SHARE_AGGREGATE_NAME, result)
+        if is_flexgroup:
+            self.assertEqual([fake.SHARE_AGGREGATE_NAME], result)
+        else:
+            self.assertEqual(fake.SHARE_AGGREGATE_NAME, result)
 
     def test_get_aggregate_for_volume_not_found(self):
 
@@ -3954,11 +4091,7 @@ class NetAppClientCmodeTestCase(test.TestCase):
                          mock.Mock(return_value=api_response))
 
         fake_junction_path = '/%s' % fake.SHARE_NAME
-        self.mock_object(self.client,
+        result = self.client.volume_has_junctioned_volumes(fake_junction_path)
-                         'get_volume_junction_path',
-                         mock.Mock(return_value=fake_junction_path))
-
-        result = self.client.volume_has_junctioned_volumes(fake.SHARE_NAME)
 
         volume_get_iter_args = {
             'query': {
@@ -3982,11 +4115,7 @@ class NetAppClientCmodeTestCase(test.TestCase):
 
     def test_volume_has_junctioned_volumes_no_junction_path(self):
 
-        self.mock_object(self.client,
+        result = self.client.volume_has_junctioned_volumes(None)
-                         'get_volume_junction_path',
-                         mock.Mock(return_value=''))
-
-        result = self.client.volume_has_junctioned_volumes(fake.SHARE_NAME)
 
         self.assertFalse(result)
 
@@ -3998,18 +4127,17 @@ class NetAppClientCmodeTestCase(test.TestCase):
                          mock.Mock(return_value=api_response))
 
         fake_junction_path = '/%s' % fake.SHARE_NAME
-        self.mock_object(self.client,
+        result = self.client.volume_has_junctioned_volumes(fake_junction_path)
-                         'get_volume_junction_path',
-                         mock.Mock(return_value=fake_junction_path))
-
-        result = self.client.volume_has_junctioned_volumes(fake.SHARE_NAME)
 
         self.assertFalse(result)
 
-    def test_get_volume(self):
+    @ddt.data(True, False)
+    def test_get_volume(self, is_flexgroup):
 
         api_response = netapp_api.NaElement(
-            fake.VOLUME_GET_ITER_VOLUME_TO_MANAGE_RESPONSE)
+            fake.VOLUME_GET_ITER_FLEXGROUP_VOLUME_TO_MANAGE_RESPONSE
+            if is_flexgroup
+            else fake.VOLUME_GET_ITER_VOLUME_TO_MANAGE_RESPONSE)
         self.mock_object(self.client,
                          'send_request',
                          mock.Mock(return_value=api_response))
@@ -4027,12 +4155,16 @@ class NetAppClientCmodeTestCase(test.TestCase):
             'desired-attributes': {
                 'volume-attributes': {
                     'volume-id-attributes': {
+                        'aggr-list': {
+                            'aggr-name': None,
+                        },
                         'containing-aggregate-name': None,
                         'junction-path': None,
                         'name': None,
                         'owning-vserver-name': None,
                         'type': None,
                         'style': None,
+                        'style-extended': None,
                     },
                     'volume-space-attributes': {
                         'size': None,
@@ -4045,7 +4177,8 @@ class NetAppClientCmodeTestCase(test.TestCase):
         }
 
         expected = {
-            'aggregate': fake.SHARE_AGGREGATE_NAME,
+            'aggregate': '' if is_flexgroup else fake.SHARE_AGGREGATE_NAME,
+            'aggr-list': [fake.SHARE_AGGREGATE_NAME] if is_flexgroup else [],
             'junction-path': '/%s' % fake.SHARE_NAME,
             'name': fake.SHARE_NAME,
             'type': 'rw',
@@ -4053,6 +4186,9 @@ class NetAppClientCmodeTestCase(test.TestCase):
             'size': fake.SHARE_SIZE,
             'owning-vserver-name': fake.VSERVER_NAME,
             'qos-policy-group-name': fake.QOS_POLICY_GROUP_NAME,
+            'style-extended': (fake.FLEXGROUP_STYLE_EXTENDED
+                               if is_flexgroup
+                               else fake.FLEXVOL_STYLE_EXTENDED),
         }
         self.client.send_request.assert_has_calls([
             mock.call('volume-get-iter', volume_get_iter_args)])
@@ -4078,12 +4214,16 @@ class NetAppClientCmodeTestCase(test.TestCase):
             'desired-attributes': {
                 'volume-attributes': {
                     'volume-id-attributes': {
+                        'aggr-list': {
+                            'aggr-name': None,
+                        },
                         'containing-aggregate-name': None,
                         'junction-path': None,
                         'name': None,
                         'owning-vserver-name': None,
                         'type': None,
                         'style': None,
+                        'style-extended': None,
                     },
                     'volume-space-attributes': {
                         'size': None,
@@ -4097,6 +4237,7 @@ class NetAppClientCmodeTestCase(test.TestCase):
 
         expected = {
             'aggregate': fake.SHARE_AGGREGATE_NAME,
+            'aggr-list': [],
             'junction-path': '/%s' % fake.SHARE_NAME,
             'name': fake.SHARE_NAME,
             'type': 'rw',
@@ -4104,6 +4245,7 @@ class NetAppClientCmodeTestCase(test.TestCase):
             'size': fake.SHARE_SIZE,
             'owning-vserver-name': fake.VSERVER_NAME,
             'qos-policy-group-name': None,
+            'style-extended': fake.FLEXVOL_STYLE_EXTENDED,
         }
         self.client.send_request.assert_has_calls([
             mock.call('volume-get-iter', volume_get_iter_args)])
@@ -4148,31 +4290,20 @@ class NetAppClientCmodeTestCase(test.TestCase):
                 'volume-attributes': {
                     'volume-id-attributes': {
                         'junction-path': fake_junction_path,
+                        'style-extended': 'flexgroup|flexvol',
                     },
                 },
             },
             'desired-attributes': {
                 'volume-attributes': {
                     'volume-id-attributes': {
-                        'containing-aggregate-name': None,
-                        'junction-path': None,
                         'name': None,
-                        'type': None,
-                        'style': None,
                     },
-                    'volume-space-attributes': {
-                        'size': None,
-                    }
                 },
             },
         }
         expected = {
-            'aggregate': fake.SHARE_AGGREGATE_NAME,
-            'junction-path': fake_junction_path,
             'name': fake.SHARE_NAME,
-            'type': 'rw',
-            'style': 'flex',
-            'size': fake.SHARE_SIZE,
         }
         self.client.send_iter_request.assert_has_calls([
             mock.call('volume-get-iter', volume_get_iter_args)])
@@ -4196,22 +4327,26 @@ class NetAppClientCmodeTestCase(test.TestCase):
 
         self.assertIsNone(result)
 
-    def test_get_volume_to_manage(self):
+    @ddt.data(True, False)
+    def test_get_volume_to_manage(self, is_flexgroup):
 
         api_response = netapp_api.NaElement(
-            fake.VOLUME_GET_ITER_VOLUME_TO_MANAGE_RESPONSE)
+            fake.VOLUME_GET_ITER_FLEXGROUP_VOLUME_TO_MANAGE_RESPONSE
+            if is_flexgroup
+            else fake.VOLUME_GET_ITER_VOLUME_TO_MANAGE_RESPONSE)
         self.mock_object(self.client,
                          'send_iter_request',
                          mock.Mock(return_value=api_response))
 
-        result = self.client.get_volume_to_manage(fake.SHARE_AGGREGATE_NAME,
+        aggr = fake.SHARE_AGGREGATE_NAME
-                                                  fake.SHARE_NAME)
+        result = self.client.get_volume_to_manage(
+            [aggr] if is_flexgroup else aggr,
+            fake.SHARE_NAME)
 
         volume_get_iter_args = {
             'query': {
                 'volume-attributes': {
                     'volume-id-attributes': {
-                        'containing-aggregate-name': fake.SHARE_AGGREGATE_NAME,
                         'name': fake.SHARE_NAME,
                     },
                 },
@@ -4219,6 +4354,9 @@ class NetAppClientCmodeTestCase(test.TestCase):
             'desired-attributes': {
                 'volume-attributes': {
                     'volume-id-attributes': {
+                        'aggr-list': {
+                            'aggr-name': None,
+                        },
                         'containing-aggregate-name': None,
                         'junction-path': None,
                         'name': None,
@@ -4235,8 +4373,16 @@ class NetAppClientCmodeTestCase(test.TestCase):
                 },
             },
         }
+        if is_flexgroup:
+            volume_get_iter_args['query']['volume-attributes'][
+                'volume-id-attributes']['aggr-list'] = [{'aggr-name': aggr}]
+        else:
+            volume_get_iter_args['query']['volume-attributes'][
+                'volume-id-attributes']['containing-aggregate-name'] = aggr
+
         expected = {
-            'aggregate': fake.SHARE_AGGREGATE_NAME,
+            'aggregate': '' if is_flexgroup else aggr,
+            'aggr-list': [aggr] if is_flexgroup else [],
             'junction-path': '/%s' % fake.SHARE_NAME,
             'name': fake.SHARE_NAME,
             'type': 'rw',
@@ -6144,14 +6290,14 @@ class NetAppClientCmodeTestCase(test.TestCase):
         self.client.create_snapmirror_vol(
             fake.SM_SOURCE_VSERVER, fake.SM_SOURCE_VOLUME,
             fake.SM_DEST_VSERVER, fake.SM_DEST_VOLUME,
-            schedule=schedule, policy=policy)
+            na_utils.DATA_PROTECTION_TYPE, schedule=schedule, policy=policy)
 
         snapmirror_create_args = {
             'source-vserver': fake.SM_SOURCE_VSERVER,
             'source-volume': fake.SM_SOURCE_VOLUME,
             'destination-vserver': fake.SM_DEST_VSERVER,
             'destination-volume': fake.SM_DEST_VOLUME,
-            'relationship-type': 'data_protection',
+            'relationship-type': na_utils.DATA_PROTECTION_TYPE,
         }
         if schedule:
             snapmirror_create_args['schedule'] = schedule
@@ -6167,14 +6313,16 @@ class NetAppClientCmodeTestCase(test.TestCase):
 
         self.client.create_snapmirror_vol(
             fake.SM_SOURCE_VSERVER, fake.SM_SOURCE_VOLUME,
-            fake.SM_DEST_VSERVER, fake.SM_DEST_VOLUME)
+            fake.SM_DEST_VSERVER, fake.SM_DEST_VOLUME,
+            na_utils.DATA_PROTECTION_TYPE)
 
         snapmirror_create_args = {
             'source-vserver': fake.SM_SOURCE_VSERVER,
             'source-volume': fake.SM_SOURCE_VOLUME,
             'destination-vserver': fake.SM_DEST_VSERVER,
             'destination-volume': fake.SM_DEST_VOLUME,
-            'relationship-type': 'data_protection',
+            'relationship-type': na_utils.DATA_PROTECTION_TYPE,
+            'policy': na_utils.MIRROR_ALL_SNAP_POLICY,
         }
         self.client.send_request.assert_has_calls([
             mock.call('snapmirror-create', snapmirror_create_args)])
@@ -6187,7 +6335,8 @@ class NetAppClientCmodeTestCase(test.TestCase):
         self.assertRaises(netapp_api.NaApiError,
                           self.client.create_snapmirror_vol,
                           fake.SM_SOURCE_VSERVER, fake.SM_SOURCE_VOLUME,
-                          fake.SM_DEST_VSERVER, fake.SM_DEST_VOLUME)
+                          fake.SM_DEST_VSERVER, fake.SM_DEST_VOLUME,
+                          na_utils.DATA_PROTECTION_TYPE)
         self.assertTrue(self.client.send_request.called)
 
     def test_create_snapmirror_svm(self):
@@ -6200,7 +6349,7 @@ class NetAppClientCmodeTestCase(test.TestCase):
         snapmirror_create_args = {
             'source-vserver': fake.SM_SOURCE_VSERVER,
             'destination-vserver': fake.SM_DEST_VSERVER,
-            'relationship-type': 'data_protection',
+            'relationship-type': na_utils.DATA_PROTECTION_TYPE,
             'identity-preserve': 'true',
             'max-transfer-rate': 'fake_xfer_rate'
         }
@@ -6295,6 +6444,7 @@ class NetAppClientCmodeTestCase(test.TestCase):
         self.mock_object(self.client, 'send_request')
         self.mock_object(self.client, 'get_snapmirror_destinations',
                          mock.Mock(return_value=snapmirror_destinations_list))
+        self.mock_object(self.client, '_ensure_snapmirror_v2')
 
         self.client.release_snapmirror_vol(
             fake.SM_SOURCE_VSERVER, fake.SM_SOURCE_VOLUME,
@@ -6329,7 +6479,9 @@ class NetAppClientCmodeTestCase(test.TestCase):
                           fake.SM_DEST_VSERVER, fake.SM_DEST_VOLUME)
 
     def test_release_snapmirror_svm(self):
+
         self.mock_object(self.client, 'send_request')
+        self.mock_object(self.client, '_ensure_snapmirror_v2')
 
         self.client.release_snapmirror_svm(
             fake.SM_SOURCE_VSERVER, fake.SM_DEST_VSERVER)
@@ -8518,3 +8670,189 @@ class NetAppClientCmodeTestCase(test.TestCase):
             fake.FAKE_JOB_ID
         )
         self.client.get_job.assert_called_once_with(fake.FAKE_JOB_ID)
+
+    @ddt.data(True, False)
+    def test_get_volume_state(self, has_record):
+
+        api_response = netapp_api.NaElement(
+            fake.VOLUME_GET_ITER_STATE_RESPONSE)
+        mock_send_iter_request = self.mock_object(
+            self.client, 'send_iter_request',
+            mock.Mock(return_value=api_response))
+        mock_has_record = self.mock_object(self.client,
+                                           '_has_records',
+                                           mock.Mock(return_value=has_record))
+
+        state = self.client.get_volume_state(fake.SHARE_NAME)
+
+        volume_get_iter_args = {
+            'query': {
+                'volume-attributes': {
+                    'volume-id-attributes': {
+                        'name': fake.SHARE_NAME,
+                    },
+                },
+            },
+            'desired-attributes': {
+                'volume-attributes': {
+                    'volume-state-attributes': {
+                        'state': None
+                    }
+                }
+            },
+        }
+        mock_send_iter_request.assert_called_once_with(
+            'volume-get-iter', volume_get_iter_args)
+        mock_has_record.assert_called_once_with(api_response)
+        if has_record:
+            self.assertEqual('online', state)
+        else:
+            self.assertEqual('', state)
+
+    @ddt.data(True, False)
+    def test_is_flexgroup_volume(self, is_flexgroup):
+
+        self.client.features.add_feature('FLEXGROUP', supported=True)
+        api_response = netapp_api.NaElement(
+            fake.VOLUME_GET_ITER_STYLE_FLEXGROUP_RESPONSE
+            if is_flexgroup else fake.VOLUME_GET_ITER_STYLE_FLEXVOL_RESPONSE)
+        mock_send_iter_request = self.mock_object(
+            self.client, 'send_request',
+            mock.Mock(return_value=api_response))
+        mock_has_record = self.mock_object(self.client,
+                                           '_has_records',
+                                           mock.Mock(return_value=True))
+        mock_is_style_extended_flexgroup = self.mock_object(
+            na_utils, 'is_style_extended_flexgroup',
+            mock.Mock(return_value=is_flexgroup))
+
+        is_flexgroup_res = self.client.is_flexgroup_volume(fake.SHARE_NAME)
+
+        volume_get_iter_args = {
+            'query': {
+                'volume-attributes': {
+                    'volume-id-attributes': {
+                        'name': fake.SHARE_NAME,
+                    },
+                },
+            },
+            'desired-attributes': {
+                'volume-attributes': {
+                    'volume-id-attributes': {
+                        'style-extended': None,
+                    },
+                },
+            },
+        }
+        mock_send_iter_request.assert_called_once_with(
+            'volume-get-iter', volume_get_iter_args)
+        mock_has_record.assert_called_once_with(api_response)
+        mock_is_style_extended_flexgroup.assert_called_once_with(
+            fake.FLEXGROUP_STYLE_EXTENDED
+            if is_flexgroup else fake.FLEXVOL_STYLE_EXTENDED)
+        self.assertEqual(is_flexgroup, is_flexgroup_res)
+
+    def test_is_flexgroup_volume_not_found(self):
+
+        self.client.features.add_feature('FLEXGROUP', supported=True)
+        api_response = netapp_api.NaElement(fake.NO_RECORDS_RESPONSE)
+        self.mock_object(self.client,
+                         'send_request',
+                         mock.Mock(return_value=api_response))
+
+        self.assertRaises(exception.StorageResourceNotFound,
+                          self.client.is_flexgroup_volume,
+                          fake.SHARE_NAME)
+
+    def test_is_flexgroup_volume_not_unique(self):
+
+        self.client.features.add_feature('FLEXGROUP', supported=True)
+        api_response = netapp_api.NaElement(
+            fake.VOLUME_GET_ITER_NOT_UNIQUE_RESPONSE)
+        self.mock_object(self.client,
+                         'send_request',
+                         mock.Mock(return_value=api_response))
+
+        self.assertRaises(exception.NetAppException,
+                          self.client.is_flexgroup_volume,
+                          fake.SHARE_NAME)
+
+    def test_is_flexgroup_volume_unsupported(self):
+
+        self.client.features.add_feature('FLEXGROUP', supported=False)
+
+        result = self.client.is_flexgroup_volume(fake.SHARE_NAME)
+
+        self.assertFalse(result)
+
+    def test_is_flexgroup_supported(self):
+        self.client.features.add_feature('FLEXGROUP')
+
+        result = self.client.is_flexgroup_supported()
+
+        self.assertTrue(result)
+
+    def test_is_flexgroup_fan_out_supported(self):
+        self.client.features.add_feature('FLEXGROUP_FAN_OUT')
+
+        result = self.client.is_flexgroup_fan_out_supported()
+
+        self.assertTrue(result)
+
+    def test_get_job_state(self):
+
+        api_response = netapp_api.NaElement(fake.JOB_GET_STATE_RESPONSE)
+        mock_send_iter_request = self.mock_object(
+            self.client, 'send_iter_request',
+            mock.Mock(return_value=api_response))
+        mock_has_record = self.mock_object(self.client,
+                                           '_has_records',
+                                           mock.Mock(return_value=True))
+
+        job_state_res = self.client.get_job_state(fake.JOB_ID)
+
+        job_get_iter_args = {
+            'query': {
+                'job-info': {
+                    'job-id': fake.JOB_ID,
+                },
+            },
+            'desired-attributes': {
+                'job-info': {
+                    'job-state': None,
+                },
+            },
+        }
+        mock_send_iter_request.assert_called_once_with(
+            'job-get-iter', job_get_iter_args, enable_tunneling=False)
+        mock_has_record.assert_called_once_with(api_response)
+        self.assertEqual(fake.JOB_STATE, job_state_res)
+
+    def test_get_job_state_not_found(self):
+
+        api_response = netapp_api.NaElement(fake.NO_RECORDS_RESPONSE)
+        self.mock_object(self.client,
+                         '_has_records',
+                         mock.Mock(return_value=False))
+        self.mock_object(self.client,
+                         'send_iter_request',
+                         mock.Mock(return_value=api_response))
+
+        self.assertRaises(exception.NetAppException,
+                          self.client.get_job_state,
+                          fake.JOB_ID)
+
+    def test_get_job_state_not_unique(self):
+
+        api_response = netapp_api.NaElement(
+            fake.JOB_GET_STATE_NOT_UNIQUE_RESPONSE)
+        self.mock_object(self.client,
+                         '_has_records',
+                         mock.Mock(return_value=True))
+        self.mock_object(self.client,
+                         'send_iter_request',
+                         mock.Mock(return_value=api_response))
+
+        self.assertRaises(exception.NetAppException,
+                          self.client.get_job_state,
+                          fake.JOB_ID)

manila/tests/share/drivers/netapp/dataontap/cluster_mode/test_data_motion.py

@@ -26,6 +26,7 @@ from manila.share.drivers.netapp.dataontap.client import api as netapp_api
 from manila.share.drivers.netapp.dataontap.client import client_cmode
 from manila.share.drivers.netapp.dataontap.cluster_mode import data_motion
 from manila.share.drivers.netapp import options as na_opts
+from manila.share.drivers.netapp import utils as na_utils
 from manila.share import utils as share_utils
 from manila import test
 from manila.tests.share.drivers.netapp.dataontap import fakes as fake
@@ -228,11 +229,12 @@ class NetAppCDOTDataMotionSessionTestCase(test.TestCase):
                          mock.Mock(return_value=mock_backend_config))
 
         self.dm_session.create_snapmirror(self.fake_src_share,
-                                          self.fake_dest_share, mount=mount)
+                                          self.fake_dest_share,
+                                          'data_protection', mount=mount)
 
         mock_dest_client.create_snapmirror_vol.assert_called_once_with(
             mock.ANY, self.fake_src_vol_name, mock.ANY,
-            self.fake_dest_vol_name, schedule='hourly'
+            self.fake_dest_vol_name, 'data_protection', schedule='hourly'
         )
         mock_dest_client.initialize_snapmirror_vol.assert_called_once_with(
             mock.ANY, self.fake_src_vol_name, mock.ANY,
@@ -285,6 +287,12 @@ class NetAppCDOTDataMotionSessionTestCase(test.TestCase):
         self.mock_object(data_motion, 'get_client_for_backend',
                          mock.Mock(side_effect=[mock_dest_client,
                                                 mock_src_client]))
+        mock_backend_config = na_fakes.create_configuration()
+        mock_backend_config.netapp_snapmirror_release_timeout = 30
+        self.mock_object(data_motion, 'get_backend_configuration',
+                         mock.Mock(return_value=mock_backend_config))
+        mock_wait_for_snapmirror_release_vol = self.mock_object(
+            self.dm_session, 'wait_for_snapmirror_release_vol')
 
         self.dm_session.delete_snapmirror(self.fake_src_share,
                                           self.fake_dest_share)
@@ -297,9 +305,10 @@ class NetAppCDOTDataMotionSessionTestCase(test.TestCase):
             mock.ANY, self.fake_src_vol_name, mock.ANY,
             self.fake_dest_vol_name
         )
-        mock_src_client.release_snapmirror_vol.assert_called_once_with(
+        mock_wait_for_snapmirror_release_vol.assert_called_once_with(
-            mock.ANY, self.fake_src_vol_name, mock.ANY,
+            self.source_vserver, self.dest_vserver, self.fake_src_vol_name,
-            self.fake_dest_vol_name
+            self.fake_dest_vol_name, False, mock_src_client,
+            timeout=30
         )
 
     @ddt.data(True, False)
@@ -336,6 +345,12 @@ class NetAppCDOTDataMotionSessionTestCase(test.TestCase):
         self.mock_object(data_motion, 'get_client_for_backend',
                          mock.Mock(side_effect=[mock_dest_client,
                                                 mock_src_client]))
+        mock_backend_config = na_fakes.create_configuration()
+        mock_backend_config.netapp_snapmirror_release_timeout = 30
+        self.mock_object(data_motion, 'get_backend_configuration',
+                         mock.Mock(return_value=mock_backend_config))
+        mock_wait_for_snapmirror_release_vol = self.mock_object(
+            self.dm_session, 'wait_for_snapmirror_release_vol')
 
         self.dm_session.delete_snapmirror(self.fake_src_share,
                                           self.fake_dest_share)
@@ -348,9 +363,10 @@ class NetAppCDOTDataMotionSessionTestCase(test.TestCase):
             mock.ANY, self.fake_src_vol_name, mock.ANY,
             self.fake_dest_vol_name
         )
-        mock_src_client.release_snapmirror_vol.assert_called_once_with(
+        mock_wait_for_snapmirror_release_vol.assert_called_once_with(
-            mock.ANY, self.fake_src_vol_name, mock.ANY,
+            self.source_vserver, self.dest_vserver, self.fake_src_vol_name,
-            self.fake_dest_vol_name
+            self.fake_dest_vol_name, False, mock_src_client,
+            timeout=30
         )
 
     def test_delete_snapmirror_svm_does_not_exist(self):
@@ -387,6 +403,12 @@ class NetAppCDOTDataMotionSessionTestCase(test.TestCase):
         self.mock_object(data_motion, 'get_client_for_backend',
                          mock.Mock(side_effect=[mock_dest_client,
                                                 mock_src_client]))
+        mock_backend_config = na_fakes.create_configuration()
+        mock_backend_config.netapp_snapmirror_release_timeout = 30
+        self.mock_object(data_motion, 'get_backend_configuration',
+                         mock.Mock(return_value=mock_backend_config))
+        mock_wait_for_snapmirror_release_vol = self.mock_object(
+            self.dm_session, 'wait_for_snapmirror_release_vol')
 
         self.dm_session.delete_snapmirror(self.fake_src_share,
                                           self.fake_dest_share)
@@ -399,9 +421,10 @@ class NetAppCDOTDataMotionSessionTestCase(test.TestCase):
             mock.ANY, self.fake_src_vol_name, mock.ANY,
             self.fake_dest_vol_name
         )
-        mock_src_client.release_snapmirror_vol.assert_called_once_with(
+        mock_wait_for_snapmirror_release_vol.assert_called_once_with(
-            mock.ANY, self.fake_src_vol_name, mock.ANY,
+            self.source_vserver, self.dest_vserver, self.fake_src_vol_name,
-            self.fake_dest_vol_name
+            self.fake_dest_vol_name, False, mock_src_client,
+            timeout=30
         )
 
     def test_delete_snapmirror_svm_error_deleting(self):
@@ -429,38 +452,14 @@ class NetAppCDOTDataMotionSessionTestCase(test.TestCase):
             timeout=mock_backend_config.netapp_snapmirror_release_timeout
         )
 
-    def test_delete_snapmirror_error_releasing(self):
-        """Ensure delete succeeds when the snapmirror does not exist."""
-        mock_src_client = mock.Mock()
-        mock_dest_client = mock.Mock()
-        mock_src_client.release_snapmirror_vol.side_effect = (
-            netapp_api.NaApiError(code=netapp_api.EOBJECTNOTFOUND))
-        self.mock_object(data_motion, 'get_client_for_backend',
-                         mock.Mock(side_effect=[mock_dest_client,
-                                                mock_src_client]))
-
-        self.dm_session.delete_snapmirror(self.fake_src_share,
-                                          self.fake_dest_share)
-
-        mock_dest_client.abort_snapmirror_vol.assert_called_once_with(
-            mock.ANY, self.fake_src_vol_name, mock.ANY,
-            self.fake_dest_vol_name, clear_checkpoint=False
-        )
-        mock_dest_client.delete_snapmirror_vol.assert_called_once_with(
-            mock.ANY, self.fake_src_vol_name, mock.ANY,
-            self.fake_dest_vol_name
-        )
-        mock_src_client.release_snapmirror_vol.assert_called_once_with(
-            mock.ANY, self.fake_src_vol_name, mock.ANY,
-            self.fake_dest_vol_name
-        )
-
     def test_delete_snapmirror_without_release(self):
         mock_src_client = mock.Mock()
         mock_dest_client = mock.Mock()
         self.mock_object(data_motion, 'get_client_for_backend',
                          mock.Mock(side_effect=[mock_dest_client,
                                                 mock_src_client]))
+        mock_wait_for_snapmirror_release_vol = self.mock_object(
+            self.dm_session, 'wait_for_snapmirror_release_vol')
 
         self.dm_session.delete_snapmirror(self.fake_src_share,
                                           self.fake_dest_share,
@@ -474,14 +473,15 @@ class NetAppCDOTDataMotionSessionTestCase(test.TestCase):
             mock.ANY, self.fake_src_vol_name, mock.ANY,
             self.fake_dest_vol_name
         )
-        self.assertFalse(mock_src_client.release_snapmirror_vol.called)
+        self.assertFalse(mock_wait_for_snapmirror_release_vol.called)
 
     def test_delete_snapmirror_source_unreachable(self):
-        mock_src_client = mock.Mock()
         mock_dest_client = mock.Mock()
         self.mock_object(data_motion, 'get_client_for_backend',
                          mock.Mock(side_effect=[mock_dest_client,
                                                 Exception]))
+        mock_wait_for_snapmirror_release_vol = self.mock_object(
+            self.dm_session, 'wait_for_snapmirror_release_vol')
 
         self.dm_session.delete_snapmirror(self.fake_src_share,
                                           self.fake_dest_share)
@@ -494,8 +494,7 @@ class NetAppCDOTDataMotionSessionTestCase(test.TestCase):
             mock.ANY, self.fake_src_vol_name, mock.ANY,
             self.fake_dest_vol_name
         )
-
+        self.assertFalse(mock_wait_for_snapmirror_release_vol.called)
-        self.assertFalse(mock_src_client.release_snapmirror_vol.called)
 
     def test_break_snapmirror(self):
         self.mock_object(self.dm_session, 'quiesce_then_abort')
@@ -671,6 +670,8 @@ class NetAppCDOTDataMotionSessionTestCase(test.TestCase):
                                                 self.mock_src_client,
                                                 self.mock_dest_client,
                                                 mock_new_src_client]))
+        self.mock_object(na_utils, 'get_relationship_type',
+                         mock.Mock(return_value=na_utils.DATA_PROTECTION_TYPE))
 
         self.dm_session.change_snapmirror_source(
             self.fake_dest_share, self.fake_src_share, fake_new_src_share,
@@ -680,12 +681,14 @@ class NetAppCDOTDataMotionSessionTestCase(test.TestCase):
 
         self.assertEqual(4, self.dm_session.delete_snapmirror.call_count)
         self.dm_session.delete_snapmirror.assert_called_with(
-            mock.ANY, mock.ANY, release=False
+            mock.ANY, mock.ANY, release=False, relationship_info_only=False
         )
 
+        na_utils.get_relationship_type.assert_called_once_with(False)
         self.mock_dest_client.create_snapmirror_vol.assert_called_once_with(
             mock.ANY, fake_new_src_share_name, mock.ANY,
-            self.fake_dest_vol_name, schedule='hourly'
+            self.fake_dest_vol_name, na_utils.DATA_PROTECTION_TYPE,
+            schedule='hourly'
         )
 
         self.mock_dest_client.resync_snapmirror_vol.assert_called_once_with(
@@ -713,6 +716,8 @@ class NetAppCDOTDataMotionSessionTestCase(test.TestCase):
         peer_cluster_name = 'new_src_cluster_name'
         self.mock_object(self.mock_src_client, 'get_cluster_name',
                          mock.Mock(return_value=peer_cluster_name))
+        self.mock_object(na_utils, 'get_relationship_type',
+                         mock.Mock(return_value=na_utils.DATA_PROTECTION_TYPE))
 
         self.dm_session.change_snapmirror_source(
             self.fake_dest_share, self.fake_src_share, fake_new_src_share,
@@ -731,12 +736,14 @@ class NetAppCDOTDataMotionSessionTestCase(test.TestCase):
         self.mock_src_client.accept_vserver_peer.assert_called_once_with(
             fake_new_src_ss_name, self.dest_vserver
         )
+        na_utils.get_relationship_type.assert_called_once_with(False)
         self.dm_session.delete_snapmirror.assert_called_with(
-            mock.ANY, mock.ANY, release=False
+            mock.ANY, mock.ANY, release=False, relationship_info_only=False
         )
         self.mock_dest_client.create_snapmirror_vol.assert_called_once_with(
             mock.ANY, fake_new_src_share_name, mock.ANY,
-            self.fake_dest_vol_name, schedule='hourly'
+            self.fake_dest_vol_name, na_utils.DATA_PROTECTION_TYPE,
+            schedule='hourly'
         )
         self.mock_dest_client.resync_snapmirror_vol.assert_called_once_with(
             mock.ANY, fake_new_src_share_name, mock.ANY,
@@ -1054,13 +1061,8 @@ class NetAppCDOTDataMotionSessionTestCase(test.TestCase):
                       dest_vserver=fake.VSERVER2),
             mock.call(source_vserver=fake.VSERVER1,
                       dest_vserver=fake.VSERVER2)])
-        if release_snapmirror_ret.side_effect is None:
+        src_mock_client.release_snapmirror_svm.assert_called_once_with(
-            src_mock_client.release_snapmirror_svm.assert_called_once_with(
+            fake.VSERVER1, fake.VSERVER2)
-                fake.VSERVER1, fake.VSERVER2)
-        else:
-            src_mock_client.release_snapmirror_svm.assert_called_once_with(
-                fake.VSERVER1, fake.VSERVER2
-            )
 
     def test_wait_for_snapmirror_release_svm_timeout(self):
         src_mock_client = mock.Mock()
@@ -1124,3 +1126,56 @@ class NetAppCDOTDataMotionSessionTestCase(test.TestCase):
 
         mock_client.mount_volume.assert_called_once_with(fake.SHARE_NAME)
         mock_warning_log.assert_not_called()
+
+    @ddt.data(mock.Mock(),
+              mock.Mock(side_effect=netapp_api.NaApiError(
+                  code=netapp_api.EOBJECTNOTFOUND)))
+    def test_wait_for_snapmirror_release_vol(self, release_snapmirror_ret):
+        src_mock_client = mock.Mock()
+        get_snapmirrors_mock = self.mock_object(
+            src_mock_client, 'get_snapmirror_destinations',
+            mock.Mock(side_effect=[['fake_snapmirror'], []]))
+        self.mock_object(src_mock_client, 'release_snapmirror_vol',
+                         release_snapmirror_ret)
+
+        self.dm_session.wait_for_snapmirror_release_vol(fake.VSERVER1,
+                                                        fake.VSERVER2,
+                                                        fake.SHARE_NAME,
+                                                        fake.SHARE_NAME2,
+                                                        False,
+                                                        src_mock_client,
+                                                        timeout=20)
+        get_snapmirrors_mock.assert_has_calls([
+            mock.call(source_vserver=fake.VSERVER1,
+                      dest_vserver=fake.VSERVER2,
+                      source_volume=fake.SHARE_NAME,
+                      dest_volume=fake.SHARE_NAME2),
+            mock.call(source_vserver=fake.VSERVER1,
+                      dest_vserver=fake.VSERVER2,
+                      source_volume=fake.SHARE_NAME,
+                      dest_volume=fake.SHARE_NAME2)])
+        src_mock_client.release_snapmirror_vol.assert_called_once_with(
+            fake.VSERVER1, fake.SHARE_NAME, fake.VSERVER2, fake.SHARE_NAME2,
+            relationship_info_only=False)
+
+    def test_wait_for_snapmirror_release_vol_timeout(self):
+        src_mock_client = mock.Mock()
+        get_snapmirrors_mock = self.mock_object(
+            src_mock_client, 'get_snapmirror_destinations',
+            mock.Mock(side_effect=[['fake_snapmirror']]))
+        self.mock_object(src_mock_client, 'release_snapmirror_vol')
+
+        self.assertRaises(exception.NetAppException,
+                          self.dm_session.wait_for_snapmirror_release_vol,
+                          fake.VSERVER1, fake.VSERVER2, fake.SHARE_NAME,
+                          fake.SHARE_NAME2, False, src_mock_client,
+                          timeout=10)
+
+        get_snapmirrors_mock.assert_has_calls([
+            mock.call(source_vserver=fake.VSERVER1,
+                      dest_vserver=fake.VSERVER2,
+                      source_volume=fake.SHARE_NAME,
+                      dest_volume=fake.SHARE_NAME2)])
+        src_mock_client.release_snapmirror_vol.assert_called_once_with(
+            fake.VSERVER1, fake.SHARE_NAME, fake.VSERVER2, fake.SHARE_NAME2,
+            relationship_info_only=False)

manila/tests/share/drivers/netapp/dataontap/cluster_mode/test_lib_multi_svm.py

@@ -118,6 +118,14 @@ class NetAppFileStorageLibraryTestCase(test.TestCase):
 
     def test_check_for_setup_error_cluster_creds_no_vserver(self):
         self.library._have_cluster_creds = True
+        mock_list_non_root_aggregates = self.mock_object(
+            self.client, 'list_non_root_aggregates',
+            mock.Mock(return_value=fake.AGGREGATES))
+        mock_init_flexgroup = self.mock_object(self.library,
+                                               '_initialize_flexgroup_pools')
+        self.mock_object(self.library,
+                         'is_flexvol_pool_configured',
+                         mock.Mock(return_value=True))
         self.mock_object(self.library,
                          '_find_matching_aggregates',
                          mock.Mock(return_value=fake.AGGREGATES))
@@ -126,12 +134,23 @@ class NetAppFileStorageLibraryTestCase(test.TestCase):
 
         self.library.check_for_setup_error()
 
+        mock_list_non_root_aggregates.assert_called_once_with()
+        mock_init_flexgroup.assert_called_once_with(set(fake.AGGREGATES))
+        self.assertTrue(self.library.is_flexvol_pool_configured.called)
         self.assertTrue(self.library._find_matching_aggregates.called)
         mock_super.assert_called_once_with()
 
     def test_check_for_setup_error_cluster_creds_with_vserver(self):
         self.library._have_cluster_creds = True
         self.library.configuration.netapp_vserver = fake.VSERVER1
+        mock_list_non_root_aggregates = self.mock_object(
+            self.client, 'list_non_root_aggregates',
+            mock.Mock(return_value=fake.AGGREGATES))
+        mock_init_flexgroup = self.mock_object(self.library,
+                                               '_initialize_flexgroup_pools')
+        self.mock_object(self.library,
+                         'is_flexvol_pool_configured',
+                         mock.Mock(return_value=True))
         self.mock_object(self.library,
                          '_find_matching_aggregates',
                          mock.Mock(return_value=fake.AGGREGATES))
@@ -141,9 +160,33 @@ class NetAppFileStorageLibraryTestCase(test.TestCase):
         self.library.check_for_setup_error()
 
         mock_super.assert_called_once_with()
+        mock_list_non_root_aggregates.assert_called_once_with()
+        mock_init_flexgroup.assert_called_once_with(set(fake.AGGREGATES))
+        self.assertTrue(self.library.is_flexvol_pool_configured.called)
         self.assertTrue(self.library._find_matching_aggregates.called)
         self.assertTrue(lib_multi_svm.LOG.warning.called)
 
+    def test_check_for_setup_error_no_aggregates_no_flexvol_pool(self):
+        self.library._have_cluster_creds = True
+        mock_list_non_root_aggregates = self.mock_object(
+            self.client, 'list_non_root_aggregates',
+            mock.Mock(return_value=fake.AGGREGATES))
+        mock_init_flexgroup = self.mock_object(self.library,
+                                               '_initialize_flexgroup_pools')
+        self.mock_object(self.library,
+                         'is_flexvol_pool_configured',
+                         mock.Mock(return_value=False))
+        self.mock_object(self.library,
+                         '_find_matching_aggregates',
+                         mock.Mock(return_value=[]))
+
+        self.library.check_for_setup_error()
+
+        mock_list_non_root_aggregates.assert_called_once_with()
+        mock_init_flexgroup.assert_called_once_with(set(fake.AGGREGATES))
+        self.assertTrue(self.library.is_flexvol_pool_configured.called)
+        self.assertTrue(self.library._find_matching_aggregates.called)
+
     def test_check_for_setup_error_vserver_creds(self):
         self.library._have_cluster_creds = False
 
@@ -152,12 +195,24 @@ class NetAppFileStorageLibraryTestCase(test.TestCase):
 
     def test_check_for_setup_error_no_aggregates(self):
         self.library._have_cluster_creds = True
+        mock_list_non_root_aggregates = self.mock_object(
+            self.client, 'list_non_root_aggregates',
+            mock.Mock(return_value=fake.AGGREGATES))
+        mock_init_flexgroup = self.mock_object(self.library,
+                                               '_initialize_flexgroup_pools')
+        self.mock_object(self.library,
+                         'is_flexvol_pool_configured',
+                         mock.Mock(return_value=True))
         self.mock_object(self.library,
                          '_find_matching_aggregates',
                          mock.Mock(return_value=[]))
 
         self.assertRaises(exception.NetAppException,
                           self.library.check_for_setup_error)
+
+        mock_list_non_root_aggregates.assert_called_once_with()
+        mock_init_flexgroup.assert_called_once_with(set(fake.AGGREGATES))
+        self.assertTrue(self.library.is_flexvol_pool_configured.called)
         self.assertTrue(self.library._find_matching_aggregates.called)
 
     def test_get_vserver_no_share_server(self):
@@ -254,6 +309,7 @@ class NetAppFileStorageLibraryTestCase(test.TestCase):
         self.mock_object(self.library,
                          '_find_matching_aggregates',
                          mock.Mock(return_value=['aggr1', 'aggr2']))
+        self.library._flexgroup_pools = {'fg': ['aggr1', 'aggr2']}
 
         result = self.library._get_ems_pool_info()
 
@@ -261,6 +317,7 @@ class NetAppFileStorageLibraryTestCase(test.TestCase):
             'pools': {
                 'vserver': None,
                 'aggregates': ['aggr1', 'aggr2'],
+                'flexgroup_aggregates': {'fg': ['aggr1', 'aggr2']},
             },
         }
         self.assertEqual(expected, result)
@@ -358,6 +415,9 @@ class NetAppFileStorageLibraryTestCase(test.TestCase):
 
     def test_find_matching_aggregates(self):
 
+        mock_is_flexvol_pool_configured = self.mock_object(
+            self.library, 'is_flexvol_pool_configured',
+            mock.Mock(return_value=True))
         mock_list_non_root_aggregates = self.mock_object(
             self.client, 'list_non_root_aggregates',
             mock.Mock(return_value=fake.AGGREGATES))
@@ -367,7 +427,19 @@ class NetAppFileStorageLibraryTestCase(test.TestCase):
         result = self.library._find_matching_aggregates()
 
         self.assertListEqual([fake.AGGREGATES[0]], result)
+        mock_is_flexvol_pool_configured.assert_called_once_with()
         mock_list_non_root_aggregates.assert_called_once_with()
+        mock_list_non_root_aggregates.assert_called_once_with()
+
+    def test_find_matching_aggregates_no_flexvol_pool(self):
+
+        self.mock_object(self.library,
+                         'is_flexvol_pool_configured',
+                         mock.Mock(return_value=False))
+
+        result = self.library._find_matching_aggregates()
+
+        self.assertListEqual([], result)
 
     @ddt.data({'nfs_config_support': False},
               {'nfs_config_support': True,
@@ -523,6 +595,9 @@ class NetAppFileStorageLibraryTestCase(test.TestCase):
         self.mock_object(self.library,
                          '_find_matching_aggregates',
                          mock.Mock(return_value=fake.AGGREGATES))
+        self.mock_object(self.library,
+                         '_get_flexgroup_aggr_set',
+                         mock.Mock(return_value=fake.AGGREGATES))
         self.mock_object(self.library,
                          '_create_ipspace',
                          mock.Mock(return_value=fake.IPSPACE))
@@ -547,7 +622,7 @@ class NetAppFileStorageLibraryTestCase(test.TestCase):
                 fake.NETWORK_INFO)
         self.library._client.create_vserver.assert_called_once_with(
             vserver_name, fake.ROOT_VOLUME_AGGREGATE, fake.ROOT_VOLUME,
-            fake.AGGREGATES, fake.IPSPACE)
+            set(fake.AGGREGATES), fake.IPSPACE)
         self.library._get_api_client.assert_called_once_with(
             vserver=vserver_name)
         self.library._create_vserver_lifs.assert_called_once_with(
@@ -561,6 +636,7 @@ class NetAppFileStorageLibraryTestCase(test.TestCase):
         self.library._client.setup_security_services.assert_called_once_with(
             fake.NETWORK_INFO['security_services'], vserver_client,
             vserver_name)
+        self.library._get_flexgroup_aggr_set.assert_called_once_with()
 
     @ddt.data(None, fake.IPSPACE)
     def test_create_vserver_dp_destination(self, existing_ipspace):
@@ -584,6 +660,9 @@ class NetAppFileStorageLibraryTestCase(test.TestCase):
         self.mock_object(self.library,
                          '_find_matching_aggregates',
                          mock.Mock(return_value=fake.AGGREGATES))
+        self.mock_object(self.library,
+                         '_get_flexgroup_aggr_set',
+                         mock.Mock(return_value=fake.AGGREGATES))
         self.mock_object(self.library,
                          '_create_ipspace',
                          mock.Mock(return_value=fake.IPSPACE))
@@ -609,6 +688,7 @@ class NetAppFileStorageLibraryTestCase(test.TestCase):
             vserver_name, fake.AGGREGATES, fake.IPSPACE)
         self.library._create_port_and_broadcast_domain.assert_called_once_with(
             fake.IPSPACE, fake.NETWORK_INFO)
+        self.library._get_flexgroup_aggr_set.assert_not_called()
 
     def test_create_vserver_already_present(self):
 
@@ -665,6 +745,9 @@ class NetAppFileStorageLibraryTestCase(test.TestCase):
         self.mock_object(self.library,
                          '_find_matching_aggregates',
                          mock.Mock(return_value=fake.AGGREGATES))
+        self.mock_object(self.library,
+                         '_get_flexgroup_aggr_set',
+                         mock.Mock(return_value=fake.AGGREGATES))
         self.mock_object(self.library._client,
                          'get_ipspace_name_for_vlan_port',
                          mock.Mock(return_value=existing_ipspace))
@@ -1264,7 +1347,7 @@ class NetAppFileStorageLibraryTestCase(test.TestCase):
             self.fake_vserver, self.fake_new_vserver_name
         )
 
-    def test_create_share_from_snaphot(self):
+    def test_create_share_from_snapshot(self):
         fake_parent_share = copy.deepcopy(fake.SHARE)
         fake_parent_share['id'] = fake.SHARE_ID2
         mock_create_from_snap = self.mock_object(
@@ -1280,6 +1363,28 @@ class NetAppFileStorageLibraryTestCase(test.TestCase):
             parent_share=fake_parent_share
         )
 
+    def test_create_share_from_snapshot_group(self):
+        share = copy.deepcopy(fake.SHARE)
+        share['source_share_group_snapshot_member_id'] = (
+            fake.CG_SNAPSHOT_MEMBER_ID1)
+        fake_parent_share = copy.deepcopy(fake.SHARE)
+        fake_parent_share['id'] = fake.SHARE_ID2
+        fake_parent_share['host'] = fake.MANILA_HOST_NAME_2
+        mock_create_from_snap = self.mock_object(
+            lib_base.NetAppCmodeFileStorageLibrary,
+            'create_share_from_snapshot')
+        mock_data_session = self.mock_object(data_motion, 'DataMotionSession')
+
+        self.library.create_share_from_snapshot(
+            None, share, fake.SNAPSHOT, share_server=fake.SHARE_SERVER,
+            parent_share=fake_parent_share)
+
+        mock_create_from_snap.assert_called_once_with(
+            None, share, fake.SNAPSHOT, share_server=fake.SHARE_SERVER,
+            parent_share=fake_parent_share
+        )
+        mock_data_session.assert_not_called()
+
     @ddt.data(
         {'src_cluster_name': fake.CLUSTER_NAME,
          'dest_cluster_name': fake.CLUSTER_NAME, 'has_vserver_peers': None},
@@ -1776,13 +1881,19 @@ class NetAppFileStorageLibraryTestCase(test.TestCase):
 
     def _init_mocks_for_svm_dr_check_compatibility(
             self, src_svm_dr_supported=True, dest_svm_dr_supported=True,
-            check_capacity_result=True):
+            check_capacity_result=True, flexgroup_support=False,
+            is_flexgroup_destination_host=False):
         self.mock_object(self.mock_src_client, 'is_svm_dr_supported',
                          mock.Mock(return_value=src_svm_dr_supported))
         self.mock_object(self.mock_dest_client, 'is_svm_dr_supported',
                          mock.Mock(return_value=dest_svm_dr_supported))
         self.mock_object(self.library, '_check_capacity_compatibility',
                          mock.Mock(return_value=check_capacity_result))
+        self.mock_object(self.mock_src_client, 'is_flexgroup_supported',
+                         mock.Mock(return_value=flexgroup_support))
+        self.mock_object(data_motion, 'DataMotionSession')
+        self.mock_object(self.library, 'is_flexgroup_destination_host',
+                         mock.Mock(return_value=is_flexgroup_destination_host))
 
     def _configure_mocks_share_server_migration_check_compatibility(
             self, have_cluster_creds=True,
@@ -1790,6 +1901,7 @@ class NetAppFileStorageLibraryTestCase(test.TestCase):
             dest_cluster_name=fake.CLUSTER_NAME_2,
             pools=fake.POOLS, is_svm_dr=True, failure_scenario=False):
         migration_method = 'svm_dr' if is_svm_dr else 'svm_migrate'
+
         self.library._have_cluster_creds = have_cluster_creds
         self.mock_object(self.library, '_get_vserver',
                          mock.Mock(return_value=(self.fake_src_vserver,
@@ -1841,34 +1953,49 @@ class NetAppFileStorageLibraryTestCase(test.TestCase):
     @ddt.data(
         {'src_svm_dr_supported': False,
          'dest_svm_dr_supported': False,
-         'check_capacity_result': False
+         'check_capacity_result': False,
+         'is_flexgroup_destination_host': False,
          },
         {'src_svm_dr_supported': True,
          'dest_svm_dr_supported': True,
-         'check_capacity_result': False
+         'check_capacity_result': False,
+         'is_flexgroup_destination_host': False,
+         },
+        {'src_svm_dr_supported': True,
+         'dest_svm_dr_supported': True,
+         'check_capacity_result': True,
+         'is_flexgroup_destination_host': True,
          },
     )
     @ddt.unpack
     def test__check_compatibility_svm_dr_not_compatible(
             self, src_svm_dr_supported, dest_svm_dr_supported,
-            check_capacity_result):
+            check_capacity_result, is_flexgroup_destination_host):
         server_total_size = (fake.SHARE_REQ_SPEC.get('shares_size', 0) +
                              fake.SHARE_REQ_SPEC.get('snapshots_size', 0))
 
         self._init_mocks_for_svm_dr_check_compatibility(
             src_svm_dr_supported=src_svm_dr_supported,
             dest_svm_dr_supported=dest_svm_dr_supported,
-            check_capacity_result=check_capacity_result)
+            check_capacity_result=check_capacity_result,
+            flexgroup_support=is_flexgroup_destination_host,
+            is_flexgroup_destination_host=is_flexgroup_destination_host)
 
         method, result = self.library._check_compatibility_using_svm_dr(
-            self.mock_src_client, self.mock_dest_client, fake.SHARE_REQ_SPEC,
+            self.mock_src_client, self.mock_dest_client,
-            fake.POOLS)
+            fake.SERVER_MIGRATION_REQUEST_SPEC, fake.POOLS)
 
         self.assertEqual(method, 'svm_dr')
         self.assertEqual(result, False)
         self.assertTrue(self.mock_src_client.is_svm_dr_supported.called)
 
-        if check_capacity_result and not src_svm_dr_supported:
+        if (src_svm_dr_supported and dest_svm_dr_supported and
+                is_flexgroup_destination_host):
+            self.assertTrue(self.mock_src_client.is_flexgroup_supported.called)
+            self.assertTrue(self.library.is_flexgroup_destination_host.called)
+
+        if (check_capacity_result and not is_flexgroup_destination_host and
+                not src_svm_dr_supported):
             self.assertFalse(self.mock_dest_client.is_svm_dr_supported.called)
             self.library._check_capacity_compatibility.assert_called_once_with(
                 fake.POOLS, True, server_total_size)

manila/tests/share/drivers/netapp/dataontap/cluster_mode/test_lib_single_svm.py

@@ -64,6 +64,16 @@ class NetAppFileStorageLibraryTestCase(test.TestCase):
 
         self.library._client.vserver_exists.return_value = True
         self.library._have_cluster_creds = True
+        mock_client = mock.Mock()
+        mock_client.list_vserver_aggregates.return_value = fake.AGGREGATES
+        self.mock_object(self.library,
+                         '_get_api_client',
+                         mock.Mock(return_value=mock_client))
+        mock_init_flexgroup = self.mock_object(self.library,
+                                               '_initialize_flexgroup_pools')
+        self.mock_object(self.library,
+                         'is_flexvol_pool_configured',
+                         mock.Mock(return_value=True))
         self.mock_object(self.library,
                          '_find_matching_aggregates',
                          mock.Mock(return_value=fake.AGGREGATES))
@@ -74,6 +84,10 @@ class NetAppFileStorageLibraryTestCase(test.TestCase):
 
         self.assertTrue(lib_single_svm.LOG.info.called)
         mock_super.assert_called_once_with()
+        mock_client.list_vserver_aggregates.assert_called_once_with()
+        self.assertTrue(self.library._get_api_client.called)
+        mock_init_flexgroup.assert_called_once_with(set(fake.AGGREGATES))
+        self.assertTrue(self.library.is_flexvol_pool_configured.called)
         self.assertTrue(self.library._find_matching_aggregates.called)
 
     def test_check_for_setup_error_no_vserver(self):
@@ -92,6 +106,16 @@ class NetAppFileStorageLibraryTestCase(test.TestCase):
         self.library._client.vserver_exists.return_value = True
         self.library._have_cluster_creds = False
         self.library._client.list_vservers.return_value = [fake.VSERVER1]
+        mock_client = mock.Mock()
+        mock_client.list_vserver_aggregates.return_value = fake.AGGREGATES
+        self.mock_object(self.library,
+                         '_get_api_client',
+                         mock.Mock(return_value=mock_client))
+        mock_init_flexgroup = self.mock_object(self.library,
+                                               '_initialize_flexgroup_pools')
+        self.mock_object(self.library,
+                         'is_flexvol_pool_configured',
+                         mock.Mock(return_value=True))
         self.mock_object(self.library,
                          '_find_matching_aggregates',
                          mock.Mock(return_value=fake.AGGREGATES))
@@ -101,6 +125,31 @@ class NetAppFileStorageLibraryTestCase(test.TestCase):
         self.library.check_for_setup_error()
 
         mock_super.assert_called_once_with()
+        mock_client.list_vserver_aggregates.assert_called_once_with()
+        self.assertTrue(self.library._get_api_client.called)
+        mock_init_flexgroup.assert_called_once_with(set(fake.AGGREGATES))
+        self.assertTrue(self.library.is_flexvol_pool_configured.called)
+        self.assertTrue(self.library._find_matching_aggregates.called)
+
+    def test_check_for_setup_error_no_aggregates_no_flexvol_pool(self):
+        self.library._client.vserver_exists.return_value = True
+        self.library._have_cluster_creds = True
+        mock_client = mock.Mock()
+        mock_client.list_vserver_aggregates.return_value = fake.AGGREGATES
+        self.mock_object(self.library,
+                         '_get_api_client',
+                         mock.Mock(return_value=mock_client))
+        self.mock_object(self.library, '_initialize_flexgroup_pools')
+        self.mock_object(self.library,
+                         'is_flexvol_pool_configured',
+                         mock.Mock(return_value=False))
+        self.mock_object(self.library,
+                         '_find_matching_aggregates',
+                         mock.Mock(return_value=[]))
+
+        self.library.check_for_setup_error()
+
+        self.assertTrue(self.library.is_flexvol_pool_configured.called)
         self.assertTrue(self.library._find_matching_aggregates.called)
 
     def test_check_for_setup_error_cluster_creds_vserver_mismatch(self):
@@ -114,12 +163,22 @@ class NetAppFileStorageLibraryTestCase(test.TestCase):
     def test_check_for_setup_error_no_aggregates(self):
         self.library._client.vserver_exists.return_value = True
         self.library._have_cluster_creds = True
+        mock_client = mock.Mock()
+        mock_client.list_vserver_aggregates.return_value = fake.AGGREGATES
+        self.mock_object(self.library,
+                         '_get_api_client',
+                         mock.Mock(return_value=mock_client))
+        self.mock_object(self.library, '_initialize_flexgroup_pools')
+        self.mock_object(self.library,
+                         'is_flexvol_pool_configured',
+                         mock.Mock(return_value=True))
         self.mock_object(self.library,
                          '_find_matching_aggregates',
                          mock.Mock(return_value=[]))
 
         self.assertRaises(exception.NetAppException,
                           self.library.check_for_setup_error)
+        self.assertTrue(self.library.is_flexvol_pool_configured.called)
         self.assertTrue(self.library._find_matching_aggregates.called)
 
     def test_get_vserver(self):
@@ -155,6 +214,7 @@ class NetAppFileStorageLibraryTestCase(test.TestCase):
         self.mock_object(self.library,
                          '_find_matching_aggregates',
                          mock.Mock(return_value=['aggr1', 'aggr2']))
+        self.library._flexgroup_pools = {'fg': ['aggr1', 'aggr2']}
 
         result = self.library._get_ems_pool_info()
 
@@ -162,6 +222,7 @@ class NetAppFileStorageLibraryTestCase(test.TestCase):
             'pools': {
                 'vserver': fake.VSERVER1,
                 'aggregates': ['aggr1', 'aggr2'],
+                'flexgroup_aggregates': {'fg': ['aggr1', 'aggr2']},
             },
         }
         self.assertEqual(expected, result)
@@ -189,6 +250,9 @@ class NetAppFileStorageLibraryTestCase(test.TestCase):
     @ddt.data(True, False)
     def test_find_matching_aggregates(self, have_cluster_creds):
 
+        self.mock_object(self.library,
+                         'is_flexvol_pool_configured',
+                         mock.Mock(return_value=True))
         self.library._have_cluster_creds = have_cluster_creds
         aggregates = fake.AGGREGATES + fake.ROOT_AGGREGATES
         mock_vserver_client = mock.Mock()
@@ -213,6 +277,16 @@ class NetAppFileStorageLibraryTestCase(test.TestCase):
                                  result)
             self.assertFalse(mock_client.list_root_aggregates.called)
 
+    def test_find_matching_aggregates_no_flexvol_pool(self):
+
+        self.mock_object(self.library,
+                         'is_flexvol_pool_configured',
+                         mock.Mock(return_value=False))
+
+        result = self.library._find_matching_aggregates()
+
+        self.assertListEqual([], result)
+
     def test_get_network_allocations_number(self):
         self.assertEqual(0, self.library.get_network_allocations_number())
 

manila/tests/share/drivers/netapp/dataontap/cluster_mode/test_performance.py

@@ -57,7 +57,11 @@ class PerformanceLibraryTestCase(test.TestCase):
         self.fake_aggregates = {
             'pool4': {
                 'netapp_aggregate': 'aggr3',
-            }
+            },
+            'flexgroup_pool': {
+                'netapp_aggregate': 'aggr1 aggr2',
+                'netapp_flexgroup': True,
+            },
         }
 
         self.fake_aggr_names = ['aggr1', 'aggr2', 'aggr3']
@@ -191,6 +195,7 @@ class PerformanceLibraryTestCase(test.TestCase):
             'pool2': 75,
             'pool3': 75,
             'pool4': 75,
+            'flexgroup_pool': performance.DEFAULT_UTILIZATION,
         }
         self.assertEqual(expected_pool_utilization,
                          self.perf_library.pool_utilization)
@@ -232,6 +237,7 @@ class PerformanceLibraryTestCase(test.TestCase):
             'pool2': performance.DEFAULT_UTILIZATION,
             'pool3': performance.DEFAULT_UTILIZATION,
             'pool4': performance.DEFAULT_UTILIZATION,
+            'flexgroup_pool': performance.DEFAULT_UTILIZATION,
         }
         self.assertEqual(expected_pool_utilization,
                          self.perf_library.pool_utilization)
@@ -271,6 +277,7 @@ class PerformanceLibraryTestCase(test.TestCase):
             'pool2': performance.DEFAULT_UTILIZATION,
             'pool3': performance.DEFAULT_UTILIZATION,
             'pool4': performance.DEFAULT_UTILIZATION,
+            'flexgroup_pool': performance.DEFAULT_UTILIZATION,
         }
         self.assertEqual(expected_pool_utilization,
                          self.perf_library.pool_utilization)
@@ -310,6 +317,7 @@ class PerformanceLibraryTestCase(test.TestCase):
             'pool2': performance.DEFAULT_UTILIZATION,
             'pool3': performance.DEFAULT_UTILIZATION,
             'pool4': performance.DEFAULT_UTILIZATION,
+            'flexgroup_pool': performance.DEFAULT_UTILIZATION,
         }
         self.assertEqual(expected_pool_utilization,
                          self.perf_library.pool_utilization)

manila/tests/share/drivers/netapp/dataontap/fakes.py

@@ -28,6 +28,7 @@ DRIVER_NAME = 'fake_driver_name'
 APP_VERSION = 'fake_app_vsersion'
 HOST_NAME = 'fake_host'
 POOL_NAME = 'fake_pool'
+FLEXGROUP_STYLE_EXTENDED = 'flexgroup'
 POOL_NAME_2 = 'fake_pool_2'
 VSERVER1 = 'fake_vserver_1'
 VSERVER2 = 'fake_vserver_2'
@@ -37,6 +38,7 @@ VOLUME_NAME_TEMPLATE = 'share_%(share_id)s'
 VSERVER_NAME_TEMPLATE = 'os_%s'
 AGGREGATE_NAME_SEARCH_PATTERN = '(.*)'
 SHARE_NAME = 'share_7cf7c200_d3af_4e05_b87e_9167c95dfcad'
+SHARE_NAME2 = 'share_d24e7257_124e_4fb6_b05b_d384f660bc85'
 SHARE_INSTANCE_NAME = 'share_d24e7257_124e_4fb6_b05b_d384f660bc85'
 FLEXVOL_NAME = 'fake_volume'
 JUNCTION_PATH = '/%s' % FLEXVOL_NAME
@@ -103,6 +105,9 @@ FPOLICY_EXT_TO_INCLUDE_LIST = ['avi']
 FPOLICY_EXT_TO_EXCLUDE = 'jpg,mp3'
 FPOLICY_EXT_TO_EXCLUDE_LIST = ['jpg', 'mp3']
 
+JOB_ID = '123'
+JOB_STATE = 'success'
+
 CLIENT_KWARGS = {
     'username': 'admin',
     'trace': False,
@@ -132,6 +137,7 @@ SHARE = {
     'status': constants.STATUS_AVAILABLE,
     'share_server': None,
     'encrypt': False,
+    'share_id': SHARE_ID,
 }
 
 SHARE_INSTANCE = {
@@ -156,6 +162,7 @@ FLEXVOL_TO_MANAGE = {
     'type': 'rw',
     'style': 'flex',
     'size': '1610612736',  # rounds up to 2 GB
+    'style-extended': FLEXGROUP_STYLE_EXTENDED,
 }
 
 FLEXVOL_WITHOUT_QOS = copy.deepcopy(FLEXVOL_TO_MANAGE)
@@ -863,6 +870,46 @@ AGGREGATE_CAPACITIES = {
     }
 }
 
+FLEXGROUP_POOL_NAME = 'flexgroup_pool'
+
+FLEXGROUP_POOL_AGGR = [AGGREGATES[0], AGGREGATES[1]]
+
+FLEXGROUP_POOL_OPT = {
+    FLEXGROUP_POOL_NAME: FLEXGROUP_POOL_AGGR,
+}
+
+FLEXGROUP_POOL_OPT_RAW = {
+    FLEXGROUP_POOL_NAME: '%s %s' % (AGGREGATES[0], AGGREGATES[1]),
+}
+
+FLEXGROUP_POOL = {
+    'pool_name': FLEXGROUP_POOL_NAME,
+    'netapp_aggregate': '%s %s' % (AGGREGATES[0], AGGREGATES[1]),
+    'total_capacity_gb': 6.6,
+    'free_capacity_gb': 2.2,
+    'allocated_capacity_gb': 4.39,
+    'reserved_percentage': 5,
+    'max_over_subscription_ratio': 2.0,
+    'dedupe': [True, False],
+    'compression': [True, False],
+    'thin_provisioning': [True, False],
+    'netapp_flexvol_encryption': True,
+    'netapp_raid_type': 'raid4 raid_dp',
+    'netapp_disk_type': ['FCAL', 'SATA', 'SSD'],
+    'netapp_hybrid_aggregate': 'false true',
+    'utilization': 50.0,
+    'filter_function': FLEXGROUP_POOL_NAME,
+    'goodness_function': 'goodness',
+    'snapshot_support': True,
+    'create_share_from_snapshot_support': True,
+    'revert_to_snapshot_support': True,
+    'qos': True,
+    'security_service_update_support': True,
+    'netapp_flexgroup': True,
+}
+
+FLEXGROUP_AGGR_SET = set(FLEXGROUP_POOL_OPT[FLEXGROUP_POOL_NAME])
+
 AGGREGATE_CAPACITIES_VSERVER_CREDS = {
     AGGREGATES[0]: {
         'available': 1181116007,  # 1.1 GB
@@ -878,21 +925,38 @@ SSC_INFO = {
         'netapp_disk_type': 'FCAL',
         'netapp_hybrid_aggregate': 'false',
         'netapp_aggregate': AGGREGATES[0],
+        'netapp_flexgroup': False,
     },
     AGGREGATES[1]: {
         'netapp_raid_type': 'raid_dp',
         'netapp_disk_type': ['SATA', 'SSD'],
         'netapp_hybrid_aggregate': 'true',
         'netapp_aggregate': AGGREGATES[1],
+        'netapp_flexgroup': False,
+    }
+}
+
+SSC_INFO_MAP = {
+    AGGREGATES[0]: {
+        'netapp_raid_type': 'raid4',
+        'netapp_disk_type': ['FCAL'],
+        'netapp_hybrid_aggregate': 'false',
+    },
+    AGGREGATES[1]: {
+        'netapp_raid_type': 'raid_dp',
+        'netapp_disk_type': ['SATA', 'SSD'],
+        'netapp_hybrid_aggregate': 'true',
     }
 }
 
 SSC_INFO_VSERVER_CREDS = {
     AGGREGATES[0]: {
         'netapp_aggregate': AGGREGATES[0],
+        'netapp_flexgroup': False,
     },
     AGGREGATES[1]: {
         'netapp_aggregate': AGGREGATES[1],
+        'netapp_flexgroup': False,
     }
 }
 
@@ -921,6 +985,7 @@ POOLS = [
         'revert_to_snapshot_support': True,
         'qos': True,
         'security_service_update_support': True,
+        'netapp_flexgroup': False,
     },
     {
         'pool_name': AGGREGATES[1],
@@ -946,12 +1011,15 @@ POOLS = [
         'revert_to_snapshot_support': True,
         'qos': True,
         'security_service_update_support': True,
+        'netapp_flexgroup': False,
     },
 ]
 
 POOLS_VSERVER_CREDS = [
     {
         'pool_name': AGGREGATES[0],
+        'filter_function': None,
+        'goodness_function': None,
         'netapp_aggregate': AGGREGATES[0],
         'total_capacity_gb': 'unknown',
         'free_capacity_gb': 1.1,
@@ -964,13 +1032,12 @@ POOLS_VSERVER_CREDS = [
         'thin_provisioning': [True, False],
         'netapp_flexvol_encryption': True,
         'utilization': 50.0,
-        'filter_function': None,
-        'goodness_function': None,
         'snapshot_support': True,
         'create_share_from_snapshot_support': True,
         'revert_to_snapshot_support': True,
         'qos': False,
         'security_service_update_support': True,
+        'netapp_flexgroup': False,
     },
     {
         'pool_name': AGGREGATES[1],
@@ -986,13 +1053,12 @@ POOLS_VSERVER_CREDS = [
         'thin_provisioning': [True, False],
         'netapp_flexvol_encryption': True,
         'utilization': 50.0,
-        'filter_function': None,
-        'goodness_function': None,
         'snapshot_support': True,
         'create_share_from_snapshot_support': True,
         'revert_to_snapshot_support': True,
         'qos': False,
         'security_service_update_support': True,
+        'netapp_flexgroup': False,
     },
 ]
 
@@ -1730,3 +1796,7 @@ def get_network_info(user_network_allocation, admin_network_allocation):
     net_info['admin_network_allocations'] = admin_network_allocation
 
     return net_info
+
+
+def fake_get_filter_function(pool=None):
+    return pool if pool else 'filter'

manila/tests/share/drivers/netapp/dataontap/protocols/test_nfs_cmode.py

@@ -44,14 +44,19 @@ class NetAppClusteredNFSHelperTestCase(test.TestCase):
     def test__escaped_address(self, raw, escaped):
         self.assertEqual(escaped, self.helper._escaped_address(raw))
 
-    def test_create_share(self):
+    @ddt.data(True, False)
+    def test_create_share(self, is_flexgroup):
 
         mock_ensure_export_policy = self.mock_object(self.helper,
                                                      '_ensure_export_policy')
         self.mock_client.get_volume_junction_path.return_value = (
             fake.NFS_SHARE_PATH)
+        self.mock_client.get_volume.return_value = {
+            'junction-path': fake.NFS_SHARE_PATH,
+        }
 
-        result = self.helper.create_share(fake.NFS_SHARE, fake.SHARE_NAME)
+        result = self.helper.create_share(fake.NFS_SHARE, fake.SHARE_NAME,
+                                          is_flexgroup=is_flexgroup)
 
         export_addresses = [fake.SHARE_ADDRESS_1, fake.SHARE_ADDRESS_2]
         export_paths = [result(address) for address in export_addresses]
@@ -63,6 +68,10 @@ class NetAppClusteredNFSHelperTestCase(test.TestCase):
         (self.mock_client.clear_nfs_export_policy_for_volume.
             assert_called_once_with(fake.SHARE_NAME))
         self.assertTrue(mock_ensure_export_policy.called)
+        if is_flexgroup:
+            self.assertTrue(self.mock_client.get_volume.called)
+        else:
+            self.assertTrue(self.mock_client.get_volume_junction_path.called)
 
     def test_delete_share(self):
 

manila/tests/share/drivers/netapp/test_utils.py

@@ -26,6 +26,7 @@ from oslo_log import log
 from manila import exception
 from manila.share.drivers.netapp import utils as na_utils
 from manila import test
+from manila.tests.share.drivers.netapp.dataontap import fakes as fake
 from manila import version
 
 
@@ -152,6 +153,74 @@ class NetAppDriverUtilsTestCase(test.TestCase):
             sorted(na_utils.convert_to_list({'key1': 'value1',
                                              'key2': 'value2'})))
 
+    @ddt.data({'is_fg': True, 'type': na_utils.EXTENDED_DATA_PROTECTION_TYPE},
+              {'is_fg': False, 'type': na_utils.DATA_PROTECTION_TYPE})
+    @ddt.unpack
+    def test_get_relationship_type(self, is_fg, type):
+        relationship_type = na_utils.get_relationship_type(is_fg)
+
+        self.assertEqual(type, relationship_type)
+
+    @ddt.data({'is_style': True, 'style': na_utils.FLEXGROUP_STYLE_EXTENDED},
+              {'is_style': False, 'style': na_utils.FLEXVOL_STYLE_EXTENDED})
+    @ddt.unpack
+    def test_is_style_extended_flexgroup(self, is_style, style):
+        res = na_utils.is_style_extended_flexgroup(style)
+
+        self.assertEqual(is_style, res)
+
+    @ddt.data(True, False)
+    def test_parse_flexgroup_pool_config(self, check):
+
+        result = na_utils.parse_flexgroup_pool_config(
+            [fake.FLEXGROUP_POOL_OPT_RAW],
+            cluster_aggr_set=set(fake.FLEXGROUP_POOL_AGGR),
+            check=check)
+
+        self.assertEqual(fake.FLEXGROUP_POOL_OPT, result)
+
+    def test_parse_flexgroup_pool_config_raise_invalid_aggr(self):
+
+        self.assertRaises(exception.NetAppException,
+                          na_utils.parse_flexgroup_pool_config,
+                          [fake.FLEXGROUP_POOL_OPT_RAW],
+                          cluster_aggr_set=set(),
+                          check=True)
+
+    def test_parse_flexgroup_pool_config_raise_duplicated_pool(self):
+
+        fake_pool = {
+            'flexgroup1': fake.FLEXGROUP_POOL_AGGR[0],
+            'flexgroup2': fake.FLEXGROUP_POOL_AGGR[0],
+        }
+
+        self.assertRaises(exception.NetAppException,
+                          na_utils.parse_flexgroup_pool_config,
+                          [fake_pool],
+                          cluster_aggr_set=set(fake.FLEXGROUP_POOL_AGGR),
+                          check=True)
+
+    def test_parse_flexgroup_pool_config_raise_repeated_aggr(self):
+
+        aggr_pool = '%s %s' % (fake.FLEXGROUP_POOL_AGGR[0],
+                               fake.FLEXGROUP_POOL_AGGR[0])
+
+        self.assertRaises(exception.NetAppException,
+                          na_utils.parse_flexgroup_pool_config,
+                          [{'flexgroup1': aggr_pool}],
+                          cluster_aggr_set=set(fake.FLEXGROUP_POOL_AGGR),
+                          check=True)
+
+    def test_parse_flexgroup_pool_config_raise_invalid_pool_name(self):
+
+        aggr_pool = '%s %s' % (fake.FLEXGROUP_POOL_AGGR[0],
+                               fake.FLEXGROUP_POOL_AGGR[0])
+        self.assertRaises(exception.NetAppException,
+                          na_utils.parse_flexgroup_pool_config,
+                          [{fake.FLEXGROUP_POOL_AGGR[0]: aggr_pool}],
+                          cluster_aggr_set=set(fake.FLEXGROUP_POOL_AGGR),
+                          check=True)
+
 
 class OpenstackInfoTestCase(test.TestCase):
 

releasenotes/notes/netapp-flexgroup-support-9b3f30afb94d3a86.yaml

@@ -0,0 +1,40 @@
+---
+features:
+  - |
+    The NetApp driver has been working  with FlexVol ONTAP volumes.
+    The driver does not support scaling FlexVol volumes higher than
+    100 TiB, which was a theoretical limit for the large namespace that
+    these containers were meant to handle. ONTAP's Flexgroup volumes
+    eliminate such limitations. So, added the support for provisioning
+    share as FlexGroup in the NetApp driver.
+
+    The FlexGroup provision is enabled by new option
+    ``netapp_enable_flexgroup``, which will make the driver report a single
+    pool represeting all aggregates. The selection on which aggregates the
+    FlexGroup share will reside is up to ONTAP. If the administrator desires
+    to control that selection through Manila scheduler, the configuration
+    option ``netapp_flexgroup_pools`` can be used to tune the storage pool
+    layout.
+
+    When enabling FlexGroup, the FlexVol pools continue enabled by default.
+    For having only FlexGroup, the new option ``netapp_flexgroup_pool_only``
+    must be set to `True`.
+
+    Now, each NetApp pool will report the capability: `netapp_flexgroup` informing
+    which type of share resides there (FlexGroup or FlexVol).
+
+    The following operations are allowed with FlexGroup shares (DHSS
+    True/False and NFS/CIFS):
+
+    - Create/Delete share;
+    - Shrink/Extend share;
+    - Create/Delete snapshot;
+    - Revert to snapshot;
+    - Manage/Unmanage snapshots;
+    - Create from snapshot;
+    - Replication;
+    - Manage/Unmanage shares;
+
+    FlexGroup feature requires ONTAP version 9.8 or newer.
+    Replication with more than one non-active replica per share requires
+    ONTAP 9.9.1 or newer.