Enhance Shaker to provide the ability to define support templates

Issue: At AT&T we have large complex test stacks that make putting
everything into a single heat template and environment file very
cumbersome. Large monolithic templates make it harder to debug failures,
maintain, extend, and organize these tests. In order to solve this issue
we have enhanced Shaker to support specifying support templates with
environment files.

This commit enhances Shaker to add the ability to define
support_templates with env_files in test definitions.

Support templates spin up "support type" resources before the actual
test stack is spun up. This could range from networks, to volumes, to
anything Heat can create. The support resources do not have any reliance
on resources created in the test stack, they set up a "foundation" for
the test stack. The test stack can then reference these resources by
name. (e.g. assume they exist by the time the test stack is spun up)

While the example provided with this commit is simple, and the support
networks that get created are not directly used in the test, it
shows the basic principles of how support templates work.

As a real-world example and to give an idea of the complexity this
enhancement is trying to solve, we have a test definition that looks
like this:

  support_templates:
  -
    Base:
      template: templates/module_1_base.yaml
      env_file: env/module_1_base.env
  -
   SI_L2:
      template: templates/module_2_si_l2.yaml
      env_file: env/module_2_si_l2.env
  -
   SI_L3:
      template: templates/module_3_si_l3.yaml
      env_file: env/module_3_si_l3.env

  template: templates/module_4_master_servant.yaml
  env_file: env/module_4_master_servant.env

The first support stack (module_1) sets up some "base" network
resources. This stack provides some network resources used by the SI_L2
and SI_L3 support stacks.

SI_L2 is a support stack with 2 VMs that do Contrail service chaining
on an L2 network.

SI_L3 is a support stack with 2 VMs that do Contrail service chaining
on an L3 network.

Then the test stack (module 4) gets spun up on N amount of computes and
runs traffic across the SI_L2 and SI_L3 service chained networks.

After the test run all stacks are cleaned up

Using the concept of support stacks allows us to beter organize and
maintain our complex tests and allows for faster debugging due to the
"layered" nature of the setup.

Support templates also allow us to spin up more Shaker test threads
that use the same support templates simultaneously to better simulate
real-world network traffic. It also reduces the set up time of certain
tests we have since the support stacks already exist.

This enhancement does not alter existing Shaker functionality and
is fully backwards compatible.

Change-Id: Ife51bc55874c6ec4faac221bab8f9f0eea175fdc

doc/source/catalog.rst

@@ -414,6 +414,20 @@ traffic goes within the tenant network (L2 domain)
 To use this scenario specify parameter ``--scenario test/sample_with_env``.
 Scenario source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/test/sample_with_env.yaml
 
+.. _scenario_sample_tcp_test_with_support_stacks:
+
+Sample TCP Test with Support Stacks
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+This test definition demonstrates the use of support stacks In this scenario
+Shaker launches pairs of instances in the same tenant network. Each test VM is
+also connected to a previously launched support network. The support neworks
+are part of their own support heat stack. Every instance is hosted on a
+separate compute node, 1 compute node is utilized. The traffic goes within the
+tenant network (L2 domain)
+
+To use this scenario specify parameter ``--scenario test/sample_with_support_stacks``.
+Scenario source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/test/sample_with_support_stacks.yaml
+
 .. _scenario_static_agents:
 
 Static agents
@@ -553,3 +567,23 @@ the same L2 domain.
 
 Template source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/test/l2_with_env.hot
 
+.. _template_test_templates_l2_with_support:
+
+test/templates/l2_with_support
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+This Heat template creates a new Neutron network, a router to the external
+network and plugs instances into this new network. All instances are located in
+the same L2 domain. The VMs are also connected to support networks that should
+exist before this template is spun up.
+
+Template source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/test/templates/l2_with_support.hot
+
+.. _template_test_templates_support_network:
+
+test/templates/support_network
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+This Heat template creates a new Neutron network. This is used to demonstrate a
+support stack in Shaker.
+
+Template source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/test/templates/support_network.hot
+

doc/source/tools/shaker-all-in-one.txt

@@ -198,7 +198,8 @@ optional arguments:
                         "interactive".
   --scenario SCENARIO   Comma-separated list of scenarios to play. Each entity
                         can be a file name or one of aliases:
-                        "misc/instance_metadata",
+                        "misc/instance_metadata", "misc/static_agent",
+                        "misc/static_agents_pair",
                         "openstack/cross_az/full_l2",
                         "openstack/cross_az/full_l3_east_west",
                         "openstack/cross_az/full_l3_north_south",

doc/source/tools/shaker-spot.txt

@@ -77,7 +77,8 @@ optional arguments:
                         "interactive".
   --scenario SCENARIO   Comma-separated list of scenarios to play. Each entity
                         can be a file name or one of aliases:
-                        "misc/instance_metadata",
+                        "misc/instance_metadata", "misc/static_agent",
+                        "misc/static_agents_pair",
                         "openstack/cross_az/full_l2",
                         "openstack/cross_az/full_l3_east_west",
                         "openstack/cross_az/full_l3_north_south",

doc/source/tools/shaker.txt

@@ -158,7 +158,8 @@ optional arguments:
                         "interactive".
   --scenario SCENARIO   Comma-separated list of scenarios to play. Each entity
                         can be a file name or one of aliases:
-                        "misc/instance_metadata",
+                        "misc/instance_metadata", "misc/static_agent",
+                        "misc/static_agents_pair",
                         "openstack/cross_az/full_l2",
                         "openstack/cross_az/full_l3_east_west",
                         "openstack/cross_az/full_l3_north_south",

etc/shaker.conf

@@ -192,7 +192,8 @@
 #cleanup_on_error = true
 
 # Comma-separated list of scenarios to play. Each entity can be a file name or
-# one of aliases: "misc/instance_metadata", "openstack/cross_az/full_l2",
+# one of aliases: "misc/instance_metadata", "misc/static_agent",
+# "misc/static_agents_pair", "openstack/cross_az/full_l2",
 # "openstack/cross_az/full_l3_east_west",
 # "openstack/cross_az/full_l3_north_south", "openstack/cross_az/perf_l2",
 # "openstack/cross_az/perf_l3_east_west",

requirements.txt

@@ -25,3 +25,4 @@ python-subunit>=0.0.18 # Apache-2.0/BSD
 PyYAML>=3.10.0 # MIT
 pyzmq>=16.0 # LGPL+BSD
 six>=1.9.0 # MIT
+timeout-decorator>=0.4.0 # MIT

shaker/engine/deploy.py

@@ -16,6 +16,7 @@
 import collections
 import functools
 import random
+import sys
 
 import jinja2
 from oslo_config import cfg
@@ -27,7 +28,6 @@ from shaker.openstack.clients import neutron
 from shaker.openstack.clients import nova
 from shaker.openstack.clients import openstack
 
-
 LOG = logging.getLogger(__name__)
 
 
@@ -230,9 +230,14 @@ def normalize_accommodation(accommodation):
 class Deployment(object):
     def __init__(self):
         self.openstack_client = None
-        self.has_stack = False
+        self.stack_id = None
         self.privileged_mode = True
 
+        # The current run "owns" the support stacks, it is tracked
+        # so it can be deleted later.
+        self.support_stacks = []
+        self.TrackStack = collections.namedtuple('TrackStack', 'name id')
+
     def connect_to_openstack(self, openstack_params, flavor_name, image_name,
                              external_net, dns_nameservers):
         LOG.debug('Connecting to OpenStack')
@@ -300,19 +305,24 @@ class Deployment(object):
             LOG.error('Failed to gather required parameters to create '
                       'heat stack: %s', e)
             exit(1)
+
         merged_parameters.update(specification.get('template_parameters', {}))
 
         env_file = specification.get('env_file', None)
         if env_file is not None:
             env_file = self._render_env_template(env_file, base_dir)
 
-        self.has_stack = True
+        support_templates = specification.get('support_templates', None)
-        stack_id = heat.create_stack(
+        if support_templates is not None:
+            self._deploy_support_stacks(support_templates, base_dir)
+
+        self.stack_id = heat.create_stack(
             self.openstack_client.heat, self.stack_name, rendered_template,
             merged_parameters, env_file)
 
         # get info about deployed objects
-        outputs = heat.get_stack_outputs(self.openstack_client.heat, stack_id)
+        outputs = heat.get_stack_outputs(self.openstack_client.heat,
+                                         self.stack_id)
         override = self._get_override(specification.get('override'))
 
         agents = filter_agents(agents, outputs, override)
@@ -324,6 +334,45 @@ class Deployment(object):
 
         return agents
 
+    def _deploy_support_stacks(self, support_templates, base_dir):
+        for stack in support_templates:
+            try:
+                support_name = stack['name']
+                support_template = utils.read_file(stack['template'],
+                                                   base_dir=base_dir)
+
+                support_env_file = stack.get('env_file', None)
+                if support_env_file is not None:
+                    support_env_file = self._render_env_template(
+                        support_env_file, base_dir)
+
+                # user should set default values in supoort template
+                # or provide a heat environment file to update
+                # parameters for support templates
+                support_template_params = {}
+
+                support_id = heat.create_stack(
+                    self.openstack_client.heat, support_name,
+                    support_template, support_template_params,
+                    support_env_file)
+
+                # track support stacks for cleanup
+                current_stack = self.TrackStack(name=support_name,
+                                                id=support_id)
+                self.support_stacks.append(current_stack)
+                LOG.debug('Tracking support stacks: %s',
+                          self.support_stacks)
+
+            except heat.exc.Conflict as err:
+                # continue even if support stack already exists. This
+                # allows re-use of existing support stacks if multiple
+                # runs reference the same support stack.
+                LOG.info('Ignoring stack exists errors: %s', err)
+                # clear the exception so polling heat later doesn't
+                # continue to show the exception in the logs
+                if sys.version_info < (3, 0):
+                    sys.exc_clear()
+
     def _get_override(self, override_spec):
         def override_ip(agent, ip_type):
             return dict(ip=nova.get_server_ip(
@@ -373,6 +422,20 @@ class Deployment(object):
         return agents
 
     def cleanup(self):
-        if self.has_stack and cfg.CONF.cleanup_on_error:
+        # cleanup the test stack first since it could be referencing resources
-            LOG.debug('Cleaning up the stack: %s', self.stack_name)
+        # in a support stack, and it was the last stack created.
-            self.openstack_client.heat.stacks.delete(self.stack_name)
+        if self.stack_id is not None and cfg.CONF.cleanup_on_error:
+            LOG.debug('Cleaning up the test stack: %s with id: %s',
+                      self.stack_name, self.stack_id)
+            heat.wait_stack_deletion(self.openstack_client.heat, self.stack_id)
+
+        # cleanup support stacks; reverse the order to prevent deletion errors
+        # due to possible dependencies between the support stacks.
+        # Only stacks tracked during the run are deleted. e.g. if a support
+        # stack already existed, the current run doesn't "own" it so it
+        # won't be cleaned up.
+        if len(self.support_stacks) > 0 and cfg.CONF.cleanup_on_error:
+            for stack in reversed(self.support_stacks):
+                LOG.debug('Cleaning up the support stack: %s with id: %s',
+                          stack.name, stack.id)
+                heat.wait_stack_deletion(self.openstack_client.heat, stack.id)

shaker/openstack/clients/heat.py

@@ -13,11 +13,13 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+import sys
 import time
 
 from heatclient import exc
 from oslo_log import log as logging
-
+from timeout_decorator import timeout
+from timeout_decorator import TimeoutError
 
 LOG = logging.getLogger(__name__)
 
@@ -76,6 +78,31 @@ def wait_stack_completion(heat_client, stack_id):
                         dict(id=stack_id, status=status, reason=reason))
 
 
+# set the timeout for this method so we don't get stuck polling indefinitely
+# waiting for a delete
+@timeout(600)
+def wait_stack_deletion(heat_client, stack_id):
+    try:
+        heat_client.stacks.delete(stack_id)
+        while True:
+            status, reason = get_stack_status(heat_client, stack_id)
+            LOG.debug('Stack status: %s Stack reason: %s', status, reason)
+            if status == 'FAILED':
+                raise exc.StackFailure('Failed to delete stack %s' % stack_id)
+
+            time.sleep(5)
+
+    except TimeoutError:
+        LOG.error('Timed out waiting for deletion of stack %s' % stack_id)
+
+    except exc.HTTPNotFound:
+        # once the stack is gone we can assume it was successfully deleted
+        # clear the exception so it doesn't confuse the logs
+        if sys.version_info < (3, 0):
+            sys.exc_clear()
+        LOG.info('Stack %s was successfully deleted', stack_id)
+
+
 def get_stack_outputs(heat_client, stack_id):
     # try to use optimized way to retrieve outputs, fallback otherwise
     if hasattr(heat_client.stacks, 'output_list'):

shaker/resources/schemas/scenario.yaml

@@ -9,6 +9,17 @@ mapping:
   deployment:
     type: map
     mapping:
+      support_templates:
+        type: seq
+        sequence:
+          - type: map
+            mapping:
+              name:
+                type: str
+              template:
+                type: str
+              env_file:
+                type: str
       template:
         type: str
       env_file:

shaker/scenarios/test/env/l2_with_support.env

@@ -0,0 +1,3 @@
+parameters:
+  support_network_1_name: 'support_network_1'
+  support_network_2_name: 'support_network_2'

shaker/scenarios/test/env/support_1.env

@@ -0,0 +1,3 @@
+parameters:
+  support_cidr: '20.0.0.0/16'
+  network_name: 'support_network_1'

shaker/scenarios/test/env/support_2.env

@@ -0,0 +1,3 @@
+parameters:
+  support_cidr: '30.0.0.0/16'
+  network_name: 'support_network_2'

shaker/scenarios/test/sample_with_support_stacks.yaml

@@ -0,0 +1,35 @@
+title: Sample TCP Test with Support Stacks
+
+description:
+  This test definition demonstrates the use of support stacks
+
+  In this scenario Shaker launches pairs of instances in the same tenant
+  network. Each test VM is also connected to a previously launched
+  support network. The support neworks are part of their own support
+  heat stack. Every instance is hosted on a separate compute node,
+  1 compute node is utilized. The traffic goes within the tenant network
+  (L2 domain)
+
+deployment:
+  support_templates:
+  - name: support_1
+    template: templates/support_network.hot
+    env_file: env/support_1.env
+
+  - name: support_2
+    template: templates/support_network.hot
+    env_file: env/support_2.env
+
+  template: templates/l2_with_support.hot
+  env_file: env/l2_with_support.env
+
+  accommodation: [pair, compute_nodes: 1]
+
+execution:
+  tests:
+  -
+    title: tcp
+    class: iperf3
+
+    sla:
+    - "[type == 'agent'] >> (stats.bandwidth.avg > 5000)"

shaker/scenarios/test/templates/l2_with_support.hot

@@ -0,0 +1,112 @@
+heat_template_version: 2013-05-23
+
+description:
+  This Heat template creates a new Neutron network, a router to the external
+  network and plugs instances into this new network. All instances are located
+  in the same L2 domain. The VMs are also connected to support networks
+  that should exist before this template is spun up.
+
+parameters:
+  image:
+    type: string
+    description: Name of image to use for servers
+  flavor:
+    type: string
+    description: Flavor to use for servers
+  external_net:
+    type: string
+    description: ID or name of external network
+  server_endpoint:
+    type: string
+    description: Server endpoint address
+  dns_nameservers:
+    type: comma_delimited_list
+    description: DNS nameservers for the subnet
+  support_network_1_name:
+    type: string
+    description: name of first support network
+  support_network_2_name:
+    type: string
+    description: name of second support network
+
+resources:
+  private_net:
+    type: OS::Neutron::Net
+    properties:
+      name: {{ unique }}_net
+
+  private_subnet:
+    type: OS::Neutron::Subnet
+    properties:
+      network_id: { get_resource: private_net }
+      cidr: 10.0.0.0/16
+      dns_nameservers: { get_param: dns_nameservers }
+
+  router:
+    type: OS::Neutron::Router
+    properties:
+      external_gateway_info:
+        network: { get_param: external_net }
+
+  router_interface:
+    type: OS::Neutron::RouterInterface
+    properties:
+      router_id: { get_resource: router }
+      subnet_id: { get_resource: private_subnet }
+
+  server_security_group:
+    type: OS::Neutron::SecurityGroup
+    properties:
+      rules: [
+        {remote_ip_prefix: 0.0.0.0/0,
+        protocol: tcp,
+        port_range_min: 1,
+        port_range_max: 65535},
+        {remote_ip_prefix: 0.0.0.0/0,
+        protocol: udp,
+        port_range_min: 1,
+        port_range_max: 65535},
+        {remote_ip_prefix: 0.0.0.0/0,
+        protocol: icmp}]
+
+{% for agent in agents.values() %}
+
+  {{ agent.id }}:
+    type: OS::Nova::Server
+    properties:
+      name: {{ agent.id }}
+      image: { get_param: image }
+      flavor: { get_param: flavor }
+      availability_zone: "{{ agent.availability_zone }}"
+      networks:
+        - port: { get_resource: {{ agent.id }}_port }
+        - network: { get_param: support_network_1_name }
+        - network: { get_param: support_network_2_name }
+      user_data_format: RAW
+      user_data:
+        str_replace:
+          template: |
+            #!/bin/sh
+            screen -dmS shaker-agent-screen shaker-agent --server-endpoint=$SERVER_ENDPOINT --agent-id=$AGENT_ID
+          params:
+            "$SERVER_ENDPOINT": { get_param: server_endpoint }
+            "$AGENT_ID": {{ agent.id }}
+
+  {{ agent.id }}_port:
+    type: OS::Neutron::Port
+    properties:
+      network_id: { get_resource: private_net }
+      fixed_ips:
+        - subnet_id: { get_resource: private_subnet }
+      security_groups: [{ get_resource: server_security_group }]
+
+{% endfor %}
+
+outputs:
+{% for agent in agents.values() %}
+  {{ agent.id }}_instance_name:
+    value: { get_attr: [ {{ agent.id }}, instance_name ] }
+  {{ agent.id }}_ip:
+    value: { get_attr: [ {{ agent.id }}, networks, { get_attr: [private_net, name] }, 0 ] }
+{% endfor %}
+

shaker/scenarios/test/templates/support_network.hot

@@ -0,0 +1,26 @@
+heat_template_version: 2013-05-23
+
+description:
+  This Heat template creates a new Neutron network. This is used to demonstrate
+  a support stack in Shaker.
+
+parameters:
+  support_cidr:
+    type: string
+    description: set a cidr from the env file
+  network_name:
+    type: string
+    description: set the support network name
+
+resources:
+  private_net:
+    type: OS::Neutron::Net
+    properties:
+      name: { get_param: network_name }
+
+  private_subnet:
+    type: OS::Neutron::Subnet
+    properties:
+      network_id: { get_resource: private_net }
+      cidr: { get_param: support_cidr }
+

shaker/tests/test_deploy.py

@@ -20,14 +20,17 @@ import mock
 import os
 import re
 import testtools
+import uuid
 
 from oslo_config import cfg
 from oslo_config import fixture as config_fixture_pkg
 from shaker.engine import config
 from shaker.engine import deploy
 from shaker.engine import utils
+from shaker.openstack.clients import heat
 from shaker.openstack.clients import nova
 from shaker.tests import fakes
+from timeout_decorator import TimeoutError
 
 ZONE = 'zone'
 
@@ -576,13 +579,13 @@ class TestDeploy(testtools.TestCase):
         absolute_path = utils.resolve_relative_path(test_file)
         scenario = utils.read_yaml_file(absolute_path)
 
-        heat_id = 'shaker_abcdefg'
+        stack_name = 'shaker_abcdefg'
 
         server_endpoint = "127.0.0.01"
         base_dir = os.path.dirname(absolute_path)
 
         deployment = deploy.Deployment()
-        deployment.stack_name = heat_id
+        deployment.stack_name = stack_name
         deployment.external_net = 'test-external_net'
         deployment.image_name = 'test-image'
         deployment.flavor_name = 'test-flavor'
@@ -597,13 +600,13 @@ class TestDeploy(testtools.TestCase):
 
         nova_nodes_mock.return_value = [{'host': 'host-1', 'zone': 'nova'}]
 
-        create_stack_mock.create_stack.return_value = heat_id
+        create_stack_mock.return_value = uuid.uuid4()
 
         heat_outputs = {
-            heat_id + '_master_0_instance_name': 'instance-0000052f',
+            stack_name + '_master_0_instance_name': 'instance-0000052f',
-            heat_id + '_master_0_ip': '192.0.0.3',
+            stack_name + '_master_0_ip': '192.0.0.3',
-            heat_id + '_slave_0_ip': '192.0.0.4',
+            stack_name + '_slave_0_ip': '192.0.0.4',
-            heat_id + '_slave_0_instance_name': 'instance-0000052c'}
+            stack_name + '_slave_0_instance_name': 'instance-0000052c'}
 
         stack_output_mock.return_value = heat_outputs
 
@@ -629,6 +632,139 @@ class TestDeploy(testtools.TestCase):
 
         self.assertEqual(expected, agents)
 
+    @mock.patch('shaker.openstack.clients.heat.get_stack_outputs')
+    @mock.patch('shaker.openstack.clients.heat.create_stack')
+    @mock.patch('shaker.openstack.clients.openstack.OpenStackClient')
+    @mock.patch('shaker.engine.deploy.Deployment._get_compute_nodes')
+    def test_deploy_from_hot_with_support_stacks(self, nova_nodes_mock,
+                                                 openstack_mock,
+                                                 create_stack_mock,
+                                                 stack_output_mock):
+        test_file = 'shaker/scenarios/test/sample_with_support_stacks.yaml'
+        absolute_path = utils.resolve_relative_path(test_file)
+        scenario = utils.read_yaml_file(absolute_path)
+
+        stack_name = 'shaker_abcdefg'
+
+        server_endpoint = "127.0.0.01"
+        base_dir = os.path.dirname(absolute_path)
+
+        deployment = deploy.Deployment()
+        deployment.stack_name = stack_name
+        deployment.external_net = 'test-external_net'
+        deployment.image_name = 'test-image'
+        deployment.flavor_name = 'test-flavor'
+        deployment.dns_nameservers = '8.8.8.8'
+        deployment.openstack_client = openstack_mock
+
+        nova_nodes_mock.return_value = [{'host': 'host-1', 'zone': 'nova'}]
+
+        create_stack_mock.return_value = uuid.uuid4()
+
+        heat_outputs = {
+            stack_name + '_master_0_instance_name': 'instance-0000052f',
+            stack_name + '_master_0_ip': '10.0.0.3',
+            stack_name + '_slave_0_ip': '10.0.0.4',
+            stack_name + '_slave_0_instance_name': 'instance-0000052c'}
+
+        stack_output_mock.return_value = heat_outputs
+
+        expected = {
+            'shaker_abcdefg_master_0': {'availability_zone': 'nova:host-1',
+                                        'id': 'shaker_abcdefg_master_0',
+                                        'ip': '10.0.0.3',
+                                        'mode': 'master',
+                                        'node': 'host-1',
+                                        'slave_id': 'shaker_abcdefg_slave_0',
+                                        'zone': 'nova'},
+            'shaker_abcdefg_slave_0': {'availability_zone': 'nova:host-1',
+                                       'id': 'shaker_abcdefg_slave_0',
+                                       'ip': '10.0.0.4',
+                                       'master_id': 'shaker_abcdefg_master_0',
+                                       'mode': 'slave',
+                                       'node': 'host-1',
+                                       'zone': 'nova'}}
+
+        agents = deployment._deploy_from_hot(scenario['deployment'],
+                                             server_endpoint,
+                                             base_dir=base_dir)
+
+        self.assertEqual(create_stack_mock.call_count, 3)
+        self.assertEqual(expected, agents)
+
+    @mock.patch('shaker.openstack.clients.heat.create_stack')
+    @mock.patch('shaker.openstack.clients.openstack.OpenStackClient')
+    def test_deploy_support_stacks(self, openstack_mock, create_stack_mock):
+        test_file = 'shaker/scenarios/test/sample_with_support_stacks.yaml'
+        absolute_path = utils.resolve_relative_path(test_file)
+        scenario = utils.read_yaml_file(absolute_path)
+
+        support_stacks = scenario['deployment']['support_templates']
+        base_dir = os.path.dirname(absolute_path)
+
+        deployment = deploy.Deployment()
+        deployment.openstack_client = openstack_mock
+
+        support_stack_1 = uuid.uuid4()
+        support_stack_2 = uuid.uuid4()
+
+        create_stack_mock.side_effect = (support_stack_1, support_stack_2)
+
+        deployment._deploy_support_stacks(support_stacks, base_dir)
+
+        self.assertEqual(support_stack_1, deployment.support_stacks[0].id)
+        self.assertEqual(support_stack_2, deployment.support_stacks[1].id)
+
+    @mock.patch('shaker.openstack.clients.heat.create_stack')
+    @mock.patch('shaker.openstack.clients.openstack.OpenStackClient')
+    def test_deploy_support_stacks_with_conflict(self, openstack_mock,
+                                                 create_stack_mock):
+        test_file = 'shaker/scenarios/test/sample_with_support_stacks.yaml'
+        absolute_path = utils.resolve_relative_path(test_file)
+        scenario = utils.read_yaml_file(absolute_path)
+
+        support_stacks = scenario['deployment']['support_templates']
+        base_dir = os.path.dirname(absolute_path)
+
+        deployment = deploy.Deployment()
+        deployment.openstack_client = openstack_mock
+
+        support_stack_1 = heat.exc.Conflict
+        support_stack_2 = uuid.uuid4()
+
+        create_stack_mock.side_effect = (support_stack_1, support_stack_2)
+
+        deployment._deploy_support_stacks(support_stacks, base_dir)
+
+        self.assertEqual(support_stack_2, deployment.support_stacks[0].id)
+        self.assertEqual(create_stack_mock.call_count, 2)
+
+    @mock.patch('shaker.openstack.clients.heat.get_stack_status')
+    @mock.patch('shaker.openstack.clients.openstack.OpenStackClient')
+    def test_wait_stack_deletion(self, openstack_mock, get_status_mock):
+        get_status_mock.side_effect = heat.exc.HTTPNotFound
+        stack_id = uuid.uuid4()
+
+        self.assertIsNone(heat.wait_stack_deletion(openstack_mock, stack_id))
+
+    @mock.patch('shaker.openstack.clients.heat.get_stack_status')
+    @mock.patch('shaker.openstack.clients.openstack.OpenStackClient')
+    def test_wait_stack_deletion_failed_stack(self, openstack_mock,
+                                              get_status_mock):
+        get_status_mock.return_value = ('FAILED', 'some_reason')
+        stack_id = uuid.uuid4()
+        self.assertRaises(heat.exc.StackFailure, heat.wait_stack_deletion,
+                          openstack_mock, stack_id)
+
+    @mock.patch('shaker.openstack.clients.heat.get_stack_status')
+    @mock.patch('shaker.openstack.clients.openstack.OpenStackClient')
+    def test_wait_stack_deletion_timeout(self, openstack_mock,
+                                         get_status_mock):
+        get_status_mock.side_effect = TimeoutError
+        stack_id = uuid.uuid4()
+
+        self.assertIsNone(heat.wait_stack_deletion(openstack_mock, stack_id))
+
     @mock.patch('shaker.openstack.clients.openstack.OpenStackClient')
     def test_get_compute_nodes_flavor_no_extra_specs(self,
                                                      nova_client_mock):