Update tests/README

tests/README

@@ -1,53 +1,113 @@
-This directory provides Amulet tests that focus on verification of
-ceph-radosgw deployments.
+This directory provides Amulet tests to verify basic deployment functionality
+from the perspective of this charm, its requirements and its features, as
+exercised in a subset of the full OpenStack deployment test bundle topology.
 
-In order to run tests, you'll need charm-tools installed (in addition to
-juju, of course):
+Reference:  lp:openstack-charm-testing for full test bundles.
+
+A single topology and configuration is defined and deployed, once for each of
+the defined Ubuntu:OpenStack release combos.  The ongoing goal is for this
+charm to always possess tests and combo definitions for all currently-supported
+release combinations of U:OS.
+
+test_* methods are called in lexical sort order, as with most runners.  However,
+each individual test method should be idempotent and expected to pass regardless
+of run order or Ubuntu:OpenStack combo.  When writing or modifying tests,
+ensure that every individual test is not dependent on another test_ method.
+
+Test naming convention, purely for code organization purposes:
+    1xx service and endpoint checks
+    2xx relation checks
+    3xx config checks
+    4xx functional checks
+    9xx restarts, config changes, actions and other final checks
+
+In order to run tests, charm-tools and juju must be installed:
     sudo add-apt-repository ppa:juju/stable
     sudo apt-get update
-    sudo apt-get install charm-tools
+    sudo apt-get install charm-tools juju juju-deployer amulet
 
-If you use a web proxy server to access the web, you'll need to set the
-AMULET_HTTP_PROXY environment variable to the http URL of the proxy server.
+Alternatively, tests may be exercised with proposed or development versions
+of juju and related tools:
+
+    # juju proposed version
+    sudo add-apt-repository ppa:juju/proposed
+    sudo apt-get update
+    sudo apt-get install charm-tools juju juju-deployer
+
+    # juju development version
+    sudo add-apt-repository ppa:juju/devel
+    sudo apt-get update
+    sudo apt-get install charm-tools juju juju-deployer
+
+Some tests may need to download files. If a web proxy server is required in
+the environment, the AMULET_HTTP_PROXY environment variable must be set and
+passed into the juju test command.  This is unrelated to juju's http proxy
+settings or behavior.
 
 The following examples demonstrate different ways that tests can be executed.
 All examples are run from the charm's root directory.
 
-  * To run all tests (starting with 00-setup):
+  * To run all +x tests in the tests directory:
 
-      make test
+      bzr branch lp:charms/trusty/foo
+      cd foo
+      make functional_test
 
-  * To run a specific test module (or modules):
+  * To run the tests against a specific release combo as defined in tests/:
 
-      juju test -v -p AMULET_HTTP_PROXY 15-basic-trusty-icehouse
+      bzr branch lp:charms/trusty/foo
+      cd foo
+      juju test -v -p AMULET_HTTP_PROXY 015-basic-trusty-icehouse
 
-  * To run a specific test module (or modules), and keep the environment
-    deployed after a failure:
+  * To run tests and keep the juju environment deployed after a failure:
 
-      juju test --set-e -v -p AMULET_HTTP_PROXY 15-basic-trusty-icehouse
+      bzr branch lp:charms/trusty/foo
+      cd foo
+      juju test --set-e -v -p AMULET_HTTP_PROXY 015-basic-trusty-icehouse
 
   * To re-run a test module against an already deployed environment (one
     that was deployed by a previous call to 'juju test --set-e'):
 
-      ./tests/15-basic-trusty-icehouse
+      ./tests/015-basic-trusty-icehouse
 
-For debugging and test development purposes, all code should be idempotent.
-In other words, the code should have the ability to be re-run without changing
-the results beyond the initial run.  This enables editing and re-running of a
-test module against an already deployed environment, as described above.
+  * Even with --set-e, `juju test` will tear down the deployment when all
+    tests pass. The following work flow may be more effective when
+    iterating on test writing.
 
-Manual debugging tips:
+      bzr branch lp:charms/trusty/foo
+      cd foo
+      ./tests/setup/00-setup
+      juju bootstrap
+      ./tests/015-basic-trusty-icehouse
+      # make some changes, run tests again
+      ./tests/015-basic-trusty-icehouse
+      # make some changes, run tests again
+      ./tests/015-basic-trusty-icehouse
 
-  * Set the following env vars before using the OpenStack CLI as admin:
-      export OS_AUTH_URL=http://`juju-deployer -f keystone 2>&1 | tail -n 1`:5000/v2.0
-      export OS_TENANT_NAME=admin
+  * There may be test definitions in the tests/ dir which are not set +x
+    executable.  This is generally true for deprecated releases, or for
+    upcoming releases which are not yet validated and enabled.  To enable
+    and run these tests:
+      bzr branch lp:charms/trusty/foo
+      cd foo
+      ls tests
+      chmod +x tests/017-basic-trusty-kilo
+      ./tests/setup/00-setup
+      juju bootstrap
+      ./tests/017-basic-trusty-kilo
+
+
+Additional notes:
+
+  * Use DEBUG to turn on debug logging, use ERROR otherwise.
+      u = OpenStackAmuletUtils(ERROR)
+      u = OpenStackAmuletUtils(DEBUG)
+
+  * To interact with the deployed environment:
       export OS_USERNAME=admin
       export OS_PASSWORD=openstack
+      export OS_TENANT_NAME=admin
       export OS_REGION_NAME=RegionOne
-
-  * Set the following env vars before using the OpenStack CLI as demoUser:
-      export OS_AUTH_URL=http://`juju-deployer -f keystone 2>&1 | tail -n 1`:5000/v2.0
-      export OS_TENANT_NAME=demoTenant
-      export OS_USERNAME=demoUser
-      export OS_PASSWORD=password
-      export OS_REGION_NAME=RegionOne
+      export OS_AUTH_URL=${OS_AUTH_PROTOCOL:-http}://`juju-deployer -e trusty -f keystone`:5000/v2.0
+      keystone user-list
+      glance image-list