Use bundletester for amulet test execution

Switch to using bundletester for execution of functional tests,
leveraging tox to build out test virtualenvs.

Rename amulet tests inline with gate-*, dev-* and dfs-*
naming standards.

Update README to refer to functional testing section of the charm
guide.

Change-Id: I618219551fc116c151cda4790c01aa9e86fa462a

Makefile

@@ -10,8 +10,7 @@ test:
 
 functional_test:
 	@echo Starting Amulet tests...
-	@tests/setup/00-setup
+	@tox -e func27
-	@juju test -v -p AMULET_HTTP_PROXY,AMULET_OS_VIP --timeout 2700
 
 bin/charm_helpers_sync.py:
 	@mkdir -p bin

test-requirements.txt

@@ -7,3 +7,19 @@ flake8>=2.2.4,<=2.4.1
 os-testr>=0.4.1
 charm-tools>=2.0.0
 requests==2.6.0
+# BEGIN: Amulet OpenStack Charm Helper Requirements
+# Liberty client lower constraints
+amulet>=1.14.3,<2.0
+bundletester>=0.6.1,<1.0
+python-ceilometerclient>=1.5.0,<2.0
+python-cinderclient>=1.4.0,<2.0
+python-glanceclient>=1.1.0,<2.0
+python-heatclient>=0.8.0,<1.0
+python-keystoneclient>=1.7.1,<2.0
+python-neutronclient>=3.1.0,<4.0
+python-novaclient>=2.30.1,<3.0
+python-openstackclient>=1.7.0,<2.0
+python-swiftclient>=2.6.0,<3.0
+pika>=0.10.0,<1.0
+distro-info
+# END: Amulet OpenStack Charm Helper Requirements

tests/README

@@ -1,113 +0,0 @@
-This directory provides Amulet tests to verify basic deployment functionality
-from the perspective of this charm, its requirements and its features, as
-exercised in a subset of the full OpenStack deployment test bundle topology.
-
-Reference:  lp:openstack-charm-testing for full test bundles.
-
-A single topology and configuration is defined and deployed, once for each of
-the defined Ubuntu:OpenStack release combos.  The ongoing goal is for this
-charm to always possess tests and combo definitions for all currently-supported
-release combinations of U:OS.
-
-test_* methods are called in lexical sort order, as with most runners.  However,
-each individual test method should be idempotent and expected to pass regardless
-of run order or Ubuntu:OpenStack combo.  When writing or modifying tests,
-ensure that every individual test is not dependent on another test_ method.
-
-Test naming convention, purely for code organization purposes:
-    1xx service and endpoint checks
-    2xx relation checks
-    3xx config checks
-    4xx functional checks
-    9xx restarts, config changes, actions and other final checks
-
-In order to run tests, charm-tools and juju must be installed:
-    sudo add-apt-repository ppa:juju/stable
-    sudo apt-get update
-    sudo apt-get install charm-tools juju juju-deployer amulet
-
-Alternatively, tests may be exercised with proposed or development versions
-of juju and related tools:
-
-    # juju proposed version
-    sudo add-apt-repository ppa:juju/proposed
-    sudo apt-get update
-    sudo apt-get install charm-tools juju juju-deployer
-
-    # juju development version
-    sudo add-apt-repository ppa:juju/devel
-    sudo apt-get update
-    sudo apt-get install charm-tools juju juju-deployer
-
-Some tests may need to download files. If a web proxy server is required in
-the environment, the AMULET_HTTP_PROXY environment variable must be set and
-passed into the juju test command.  This is unrelated to juju's http proxy
-settings or behavior.
-
-The following examples demonstrate different ways that tests can be executed.
-All examples are run from the charm's root directory.
-
-  * To run all +x tests in the tests directory:
-
-      bzr branch lp:charms/trusty/foo
-      cd foo
-      make functional_test
-
-  * To run the tests against a specific release combo as defined in tests/:
-
-      bzr branch lp:charms/trusty/foo
-      cd foo
-      juju test -v -p AMULET_HTTP_PROXY --timeout 2700 015-basic-trusty-icehouse
-
-  * To run tests and keep the juju environment deployed after a failure:
-
-      bzr branch lp:charms/trusty/foo
-      cd foo
-      juju test --set-e -v -p AMULET_HTTP_PROXY --timeout 2700 015-basic-trusty-icehouse
-
-  * To re-run a test module against an already deployed environment (one
-    that was deployed by a previous call to 'juju test --set-e'):
-
-      ./tests/015-basic-trusty-icehouse
-
-  * Even with --set-e, `juju test` will tear down the deployment when all
-    tests pass. The following work flow may be more effective when
-    iterating on test writing.
-
-      bzr branch lp:charms/trusty/foo
-      cd foo
-      ./tests/setup/00-setup
-      juju bootstrap
-      ./tests/015-basic-trusty-icehouse
-      # make some changes, run tests again
-      ./tests/015-basic-trusty-icehouse
-      # make some changes, run tests again
-      ./tests/015-basic-trusty-icehouse
-
-  * There may be test definitions in the tests/ dir which are not set +x
-    executable.  This is generally true for deprecated releases, or for
-    upcoming releases which are not yet validated and enabled.  To enable
-    and run these tests:
-      bzr branch lp:charms/trusty/foo
-      cd foo
-      ls tests
-      chmod +x tests/017-basic-trusty-kilo
-      ./tests/setup/00-setup
-      juju bootstrap
-      ./tests/017-basic-trusty-kilo
-
-
-Additional notes:
-
-  * Use DEBUG to turn on debug logging, use ERROR otherwise.
-      u = OpenStackAmuletUtils(ERROR)
-      u = OpenStackAmuletUtils(DEBUG)
-
-  * To interact with the deployed environment:
-      export OS_USERNAME=admin
-      export OS_PASSWORD=openstack
-      export OS_TENANT_NAME=admin
-      export OS_REGION_NAME=RegionOne
-      export OS_AUTH_URL=${OS_AUTH_PROTOCOL:-http}://`juju-deployer -e trusty -f keystone`:5000/v2.0
-      keystone user-list
-      glance image-list

tests/README.md

@@ -0,0 +1,9 @@
+# Overview
+
+This directory provides Amulet tests to verify basic deployment functionality
+from the perspective of this charm, its requirements and its features, as
+exercised in a subset of the full OpenStack deployment test bundle topology.
+
+For full details on functional testing of OpenStack charms please refer to
+the [functional testing](http://docs.openstack.org/developer/charm-guide/testing.html#functional-testing)
+section of the OpenStack Charm Guide.

tests/basic_deployment.py

@@ -1,4 +1,4 @@
-#!/usr/bin/python
+#!/usr/bin/env python
 #
 # Copyright 2016 Canonical Ltd
 #
@@ -120,18 +120,18 @@ class CephOsdBasicDeployment(OpenStackAmuletDeployment):
     def _initialize_tests(self):
         """Perform final initialization before tests get run."""
         # Access the sentries for inspecting service units
-        self.mysql_sentry = self.d.sentry.unit['mysql/0']
+        self.mysql_sentry = self.d.sentry['mysql'][0]
-        self.keystone_sentry = self.d.sentry.unit['keystone/0']
+        self.keystone_sentry = self.d.sentry['keystone'][0]
-        self.rabbitmq_sentry = self.d.sentry.unit['rabbitmq-server/0']
+        self.rabbitmq_sentry = self.d.sentry['rabbitmq-server'][0]
-        self.nova_sentry = self.d.sentry.unit['nova-compute/0']
+        self.nova_sentry = self.d.sentry['nova-compute'][0]
-        self.glance_sentry = self.d.sentry.unit['glance/0']
+        self.glance_sentry = self.d.sentry['glance'][0]
-        self.cinder_sentry = self.d.sentry.unit['cinder/0']
+        self.cinder_sentry = self.d.sentry['cinder'][0]
-        self.ceph0_sentry = self.d.sentry.unit['ceph-mon/0']
+        self.ceph0_sentry = self.d.sentry['ceph-mon'][0]
-        self.ceph1_sentry = self.d.sentry.unit['ceph-mon/1']
+        self.ceph1_sentry = self.d.sentry['ceph-mon'][1]
-        self.ceph2_sentry = self.d.sentry.unit['ceph-mon/2']
+        self.ceph2_sentry = self.d.sentry['ceph-mon'][2]
-        self.ceph_osd_sentry = self.d.sentry.unit['ceph-osd/0']
+        self.ceph_osd_sentry = self.d.sentry['ceph-osd'][0]
-        self.ceph_osd1_sentry = self.d.sentry.unit['ceph-osd/1']
+        self.ceph_osd1_sentry = self.d.sentry['ceph-osd'][1]
-        self.ceph_osd2_sentry = self.d.sentry.unit['ceph-osd/2']
+        self.ceph_osd2_sentry = self.d.sentry['ceph-osd'][2]
         u.log.debug('openstack release val: {}'.format(
             self._get_openstack_release()))
         u.log.debug('openstack release str: {}'.format(

tests/dev-basic-xenial-newton

@@ -1,4 +1,4 @@
-#!/usr/bin/python
+#!/usr/bin/env python
 #
 # Copyright 2016 Canonical Ltd
 #
@@ -14,12 +14,12 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-"""Amulet tests on a basic ceph-osd deployment on trusty-juno."""
+"""Amulet tests on a basic ceph-osd deployment on xenial-newton."""
 
 from basic_deployment import CephOsdBasicDeployment
 
 if __name__ == '__main__':
-    deployment = CephOsdBasicDeployment(series='trusty',
+    deployment = CephOsdBasicDeployment(series='xenial',
-                                     openstack='cloud:trusty-juno',
+                                        openstack='cloud:xenial-newton',
-                                     source='cloud:trusty-updates/juno')
+                                        source='cloud:xenial-updates/newton')
     deployment.run_tests()

tests/dev-basic-yakkety-newton

@@ -1,4 +1,4 @@
-#!/usr/bin/python
+#!/usr/bin/env python
 #
 # Copyright 2016 Canonical Ltd
 #
@@ -14,10 +14,10 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-"""Amulet tests on a basic ceph-osd deployment on wily-liberty."""
+"""Amulet tests on a basic ceph-osd deployment on yakkety-newton."""
 
 from basic_deployment import CephOsdBasicDeployment
 
 if __name__ == '__main__':
-    deployment = CephOsdBasicDeployment(series='wily')
+    deployment = CephOsdBasicDeployment(series='yakkety')
     deployment.run_tests()

tests/gate-basic-precise-icehouse

@@ -1,4 +1,4 @@
-#!/usr/bin/python
+#!/usr/bin/env python
 #
 # Copyright 2016 Canonical Ltd
 #

tests/gate-basic-trusty-icehouse

@@ -1,4 +1,4 @@
-#!/usr/bin/python
+#!/usr/bin/env python
 #
 # Copyright 2016 Canonical Ltd
 #

tests/gate-basic-trusty-kilo

@@ -1,4 +1,4 @@
-#!/usr/bin/python
+#!/usr/bin/env python
 #
 # Copyright 2016 Canonical Ltd
 #

tests/gate-basic-trusty-liberty

@@ -1,4 +1,4 @@
-#!/usr/bin/python
+#!/usr/bin/env python
 #
 # Copyright 2016 Canonical Ltd
 #

tests/gate-basic-trusty-mitaka

@@ -1,4 +1,4 @@
-#!/usr/bin/python
+#!/usr/bin/env python
 #
 # Copyright 2016 Canonical Ltd
 #

tests/gate-basic-xenial-mitaka

@@ -1,4 +1,4 @@
-#!/usr/bin/python
+#!/usr/bin/env python
 #
 # Copyright 2016 Canonical Ltd
 #

tests/setup/00-setup

@@ -1,17 +0,0 @@
-#!/bin/bash
-
-set -ex
-
-sudo add-apt-repository --yes ppa:juju/stable
-sudo apt-get update --yes
-sudo apt-get install --yes amulet \
-                           distro-info-data \
-                           python-cinderclient \
-                           python-distro-info \
-                           python-glanceclient \
-                           python-heatclient \
-                           python-keystoneclient \
-                           python-neutronclient \
-                           python-novaclient \
-                           python-pika \
-                           python-swiftclient

tests/tests.yaml

@@ -1,21 +1,17 @@
-bootstrap: true
+# Bootstrap the model if necessary.
-reset: false
+bootstrap: True
-virtualenv: true
+# Re-use bootstrap node instead of destroying/re-bootstrapping.
-makefile:
+reset: True
-    - lint
+# Use tox/requirements to drive the venv instead of bundletester's venv feature.
-    - test
+virtualenv: False
-sources:
+# Leave makefile empty, otherwise unit/lint tests will rerun ahead of amulet.
-    - ppa:juju/stable
+makefile: []
-packages:
+# Do not specify juju PPA sources.  Juju is presumed to be pre-installed
-    - amulet
+# and configured in all test runner environments.
-    - distro-info-data
+#sources:
-    - python-ceilometerclient
+# Do not specify or rely on system packages.
-    - python-cinderclient
+#packages:
-    - python-distro-info
+# Do not specify python packages here.  Use test-requirements.txt
-    - python-glanceclient
+# and tox instead.  ie. The venv is constructed before bundletester
-    - python-heatclient
+# is invoked.
-    - python-keystoneclient
+#python-packages:
-    - python-neutronclient
-    - python-novaclient
-    - python-pika
-    - python-swiftclient

tox.ini

@@ -5,6 +5,8 @@ skipsdist = True
 [testenv]
 setenv = VIRTUAL_ENV={envdir}
          PYTHONHASHSEED=0
+         AMULET_SETUP_TIMEOUT=2700
+passenv = HOME TERM AMULET_HTTP_PROXY AMULET_OS_VIP
 install_command =
   pip install --allow-unverified python-apt {opts} {packages}
 commands = ostestr {posargs}
@@ -18,12 +20,56 @@ deps = -r{toxinidir}/requirements.txt
 basepython = python2.7
 deps = -r{toxinidir}/requirements.txt
        -r{toxinidir}/test-requirements.txt
-commands = flake8 {posargs} actions hooks unit_tests tests
+commands = flake8 {posargs} --exclude */charmhelpers hooks unit_tests tests actions
            charm-proof
 
 [testenv:venv]
 commands = {posargs}
 
+[testenv:func27-noop]
+# DRY RUN - For Debug
+basepython = python2.7
+deps = -r{toxinidir}/requirements.txt
+       -r{toxinidir}/test-requirements.txt
+commands =
+    bundletester -vl DEBUG -r json -o func-results.json --test-pattern "gate-*" -n --no-destroy
+
+[testenv:func27]
+# Charm Functional Test
+# Run all gate tests which are +x (expected to always pass)
+basepython = python2.7
+deps = -r{toxinidir}/requirements.txt
+       -r{toxinidir}/test-requirements.txt
+commands =
+    bundletester -vl DEBUG -r json -o func-results.json --test-pattern "gate-*" --no-destroy
+
+[testenv:func27-smoke]
+# Charm Functional Test
+# Run a specific test as an Amulet smoke test (expected to always pass)
+basepython = python2.7
+deps = -r{toxinidir}/requirements.txt
+       -r{toxinidir}/test-requirements.txt
+commands =
+    bundletester -vl DEBUG -r json -o func-results.json gate-basic-xenial-mitaka --no-destroy
+
+[testenv:func27-dfs]
+# Charm Functional Test
+# Run all deploy-from-source tests which are +x (may not always pass!)
+basepython = python2.7
+deps = -r{toxinidir}/requirements.txt
+       -r{toxinidir}/test-requirements.txt
+commands =
+    bundletester -vl DEBUG -r json -o func-results.json --test-pattern "dfs-*" --no-destroy
+
+[testenv:func27-dev]
+# Charm Functional Test
+# Run all development test targets which are +x (may not always pass!)
+basepython = python2.7
+deps = -r{toxinidir}/requirements.txt
+       -r{toxinidir}/test-requirements.txt
+commands =
+    bundletester -vl DEBUG -r json -o func-results.json --test-pattern "dev-*" --no-destroy
+
 [flake8]
 ignore = E402,E226
 exclude = hooks/charmhelpers