Merge "Add test type clarification to devref"
This commit is contained in:
Jenkins
Gerrit Code Review
commit
1cd90c646a
@ -1,35 +0,0 @@
|
|||||||
=====================================
|
|
||||||
OpenStack Cinder Testing Infrastructure
|
|
||||||
=====================================
|
|
||||||
|
|
||||||
A note of clarification is in order, to help those who are new to testing in
|
|
||||||
OpenStack cinder:
|
|
||||||
|
|
||||||
- actual unit tests are created in the "tests" directory;
|
|
||||||
- the "testing" directory is used to house the infrastructure needed to support
|
|
||||||
testing in OpenStack Cinder.
|
|
||||||
|
|
||||||
This README file attempts to provide current and prospective contributors with
|
|
||||||
everything they need to know in order to start creating unit tests and
|
|
||||||
utilizing the convenience code provided in cinder.testing.
|
|
||||||
|
|
||||||
For more detailed information on cinder unit tests visit:
|
|
||||||
http://docs.openstack.org/developer/cinder/devref/unit_tests.html
|
|
||||||
|
|
||||||
Running Tests
|
|
||||||
-----------------------------------------------
|
|
||||||
|
|
||||||
In the root of the cinder source code run the run_tests.sh script. This will
|
|
||||||
offer to create a virtual environment and populate it with dependencies.
|
|
||||||
If you don't have dependencies installed that are needed for compiling cinder's
|
|
||||||
direct dependencies, you'll have to use your operating system's method of
|
|
||||||
installing extra dependencies. To get help using this script execute it with
|
|
||||||
the -h parameter to get options `./run_tests.sh -h`
|
|
||||||
|
|
||||||
Writing Unit Tests
|
|
||||||
------------------
|
|
||||||
|
|
||||||
- All new unit tests are to be written in python-mock.
|
|
||||||
- Old tests that are still written in mox should be updated to use python-mock.
|
|
||||||
Usage of mox has been deprecated for writing Cinder unit tests.
|
|
||||||
- use addCleanup in favor of tearDown
|
|
||||||
@ -94,17 +94,12 @@ Grab the code::
|
|||||||
|
|
||||||
Running unit tests
|
Running unit tests
|
||||||
------------------
|
------------------
|
||||||
The unit tests will run by default inside a virtualenv in the ``.venv``
|
Run the unit tests by doing::
|
||||||
directory. Run the unit tests by doing::
|
|
||||||
|
|
||||||
./run_tests.sh
|
tox -e py34
|
||||||
|
tox -e py27
|
||||||
|
|
||||||
The first time you run them, you will be asked if you want to create a virtual
|
See :doc:`testing` for more details.
|
||||||
environment (hit "y")::
|
|
||||||
|
|
||||||
No virtual environment found...create one? (Y/n)
|
|
||||||
|
|
||||||
See :doc:`unit_tests` for more details.
|
|
||||||
|
|
||||||
.. _virtualenv:
|
.. _virtualenv:
|
||||||
|
|
||||||
|
|||||||
@ -18,7 +18,8 @@
|
|||||||
Developer Guide
|
Developer Guide
|
||||||
===============
|
===============
|
||||||
|
|
||||||
In this section you will find information on Cinder's lower level programming APIs.
|
In this section you will find information on Cinder's lower level programming
|
||||||
|
APIs.
|
||||||
|
|
||||||
|
|
||||||
Programming HowTos and Tutorials
|
Programming HowTos and Tutorials
|
||||||
@ -30,7 +31,7 @@ Programming HowTos and Tutorials
|
|||||||
api_microversion_dev
|
api_microversion_dev
|
||||||
api_conditional_updates
|
api_conditional_updates
|
||||||
api_microversion_history
|
api_microversion_history
|
||||||
unit_tests
|
testing
|
||||||
addmethod.openstackapi
|
addmethod.openstackapi
|
||||||
drivers
|
drivers
|
||||||
gmr
|
gmr
|
||||||
|
|||||||
158
doc/source/devref/testing.rst
Normal file
158
doc/source/devref/testing.rst
Normal file
@ -0,0 +1,158 @@
|
|||||||
|
Testing
|
||||||
|
=======
|
||||||
|
|
||||||
|
Cinder contains a few different test suites in the cinder/tests/ directory. The
|
||||||
|
different test suites are Unit Tests, Functional Tests, and Tempest Tests.
|
||||||
|
|
||||||
|
Test Types
|
||||||
|
----------
|
||||||
|
|
||||||
|
|
||||||
|
Unit Tests
|
||||||
|
~~~~~~~~~~
|
||||||
|
|
||||||
|
Unit tests are tests for individual methods, with at most a small handful of
|
||||||
|
modules involved. Mock should be used to remove any external dependencies.
|
||||||
|
|
||||||
|
All significant code changes should have unit test coverage validating the code
|
||||||
|
happy path and any failure paths.
|
||||||
|
|
||||||
|
Any proposed code change will be automatically rejected by the OpenStack
|
||||||
|
Jenkins server [#f1]_ if the change causes unit test failures.
|
||||||
|
|
||||||
|
Functional Tests
|
||||||
|
~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
Functional tests validate a code path within Cinder. These tests should
|
||||||
|
validate the interaction of various modules within the project to verify the
|
||||||
|
code is logically correct.
|
||||||
|
|
||||||
|
Functional tests run with a database present and may start Cinder services to
|
||||||
|
accept requests. These tests should not need to access an other OpenStack
|
||||||
|
non-Cinder services.
|
||||||
|
|
||||||
|
Tempest Tests
|
||||||
|
~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
The tempest tests in the Cinder tree validate the operational correctness
|
||||||
|
between Cinder and external components such as Nova, Glance, etc. These are
|
||||||
|
integration tests driven via public APIs to verify actual end user usage
|
||||||
|
scenarios.
|
||||||
|
|
||||||
|
Running the tests
|
||||||
|
-----------------
|
||||||
|
|
||||||
|
There are a number of ways to run tests currently, and there's a combination of
|
||||||
|
frameworks used depending on what commands you use. The preferred method is to
|
||||||
|
use tox, which calls ostestr via the tox.ini file.
|
||||||
|
|
||||||
|
Unit Tests
|
||||||
|
~~~~~~~~~~
|
||||||
|
|
||||||
|
To run all unit tests simply run::
|
||||||
|
|
||||||
|
tox
|
||||||
|
|
||||||
|
This will create a virtual environment, load all the packages from
|
||||||
|
test-requirements.txt and run all unit tests as well as run flake8 and hacking
|
||||||
|
checks against the code.
|
||||||
|
|
||||||
|
You may run individual test targets, for example only py27 tests, by running::
|
||||||
|
|
||||||
|
tox -e py27
|
||||||
|
|
||||||
|
Note that you can inspect the tox.ini file to get more details on the available
|
||||||
|
options and what the test run does by default.
|
||||||
|
|
||||||
|
Functional Tests
|
||||||
|
~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
To run all functional tests, run::
|
||||||
|
|
||||||
|
tox -e functional
|
||||||
|
|
||||||
|
Tempest Tests
|
||||||
|
~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
Tempest tests in the Cinder tree are "plugged in" to the normal tempest test
|
||||||
|
execution. To ensure the Cinder tests are picked up when running tempest, run::
|
||||||
|
|
||||||
|
cd /opt/stack/tempest
|
||||||
|
tox -e all-plugin
|
||||||
|
|
||||||
|
More information about tempest can be found in the `Tempest Documentation
|
||||||
|
<http://docs.openstack.org/developer/tempest/overview.html>`_.
|
||||||
|
|
||||||
|
Running a subset of tests using tox
|
||||||
|
-----------------------------------
|
||||||
|
One common activity is to just run a single test, you can do this with tox
|
||||||
|
simply by specifying to just run py27 or py34 tests against a single test::
|
||||||
|
|
||||||
|
tox -epy27 -- -n cinder.tests.unit.test_volume.AvailabilityZoneTestCase.test_list_availability_zones_cached
|
||||||
|
|
||||||
|
Or all tests in the test_volume.py file::
|
||||||
|
|
||||||
|
tox -epy27 -- -n cinder.tests.unit.test_volume
|
||||||
|
|
||||||
|
You may also use regular expressions to run any matching tests::
|
||||||
|
|
||||||
|
tox -epy27 -- -r test_volume
|
||||||
|
|
||||||
|
For more information on these options and how to run tests, please see the
|
||||||
|
`ostestr documentation <http://docs.openstack.org/developer/os-testr/>`_.
|
||||||
|
|
||||||
|
Gotchas
|
||||||
|
-------
|
||||||
|
|
||||||
|
**Running Tests from Shared Folders**
|
||||||
|
|
||||||
|
If you are running the unit tests from a shared folder, you may see tests start
|
||||||
|
to fail or stop completely as a result of Python lockfile issues. You
|
||||||
|
can get around this by manually setting or updating the following line in
|
||||||
|
``cinder/tests/conf_fixture.py``::
|
||||||
|
|
||||||
|
CONF['lock_path'].SetDefault('/tmp')
|
||||||
|
|
||||||
|
Note that you may use any location (not just ``/tmp``!) as long as it is not
|
||||||
|
a shared folder.
|
||||||
|
|
||||||
|
**Running py34 tests**
|
||||||
|
|
||||||
|
You will need to install python3-dev in order to get py34 tests to run. If you
|
||||||
|
do not have this, you will get the following::
|
||||||
|
|
||||||
|
netifaces.c:1:20: fatal error: Python.h: No such file or directory
|
||||||
|
#include <Python.h>
|
||||||
|
^
|
||||||
|
compilation terminated.
|
||||||
|
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
|
||||||
|
|
||||||
|
----------------------------------------
|
||||||
|
<snip>
|
||||||
|
ERROR: could not install deps [-r/opt/stack/cinder/test-requirements.txt,
|
||||||
|
oslo.versionedobjects[fixtures]]; v = InvocationError('/opt/stack/cinder/
|
||||||
|
.tox/py34/bin/pip install -r/opt/stack/cinder/test-requirements.txt
|
||||||
|
oslo.versionedobjects[fixtures] (see /opt/stack/cinder/.tox/py34/log/py34-1.log)', 1)
|
||||||
|
_______________________________________________________________ summary _______________________________________________________________
|
||||||
|
ERROR: py34: could not install deps [-r/opt/stack/cinder/test-requirements.txt,
|
||||||
|
oslo.versionedobjects[fixtures]]; v = InvocationError('/opt/stack/cinder/
|
||||||
|
.tox/py34/bin/pip install -r/opt/stack/cinder/test-requirements.txt
|
||||||
|
oslo.versionedobjects[fixtures] (see /opt/stack/cinder/.tox/py34/log/py34-1.log)', 1)
|
||||||
|
|
||||||
|
To Fix:
|
||||||
|
|
||||||
|
- On Ubuntu/Debian::
|
||||||
|
|
||||||
|
sudo apt-get install python3-dev
|
||||||
|
|
||||||
|
- On Fedora 21/RHEL7/CentOS7::
|
||||||
|
|
||||||
|
sudo yum install python3-devel
|
||||||
|
|
||||||
|
- On Fedora 22 and higher::
|
||||||
|
|
||||||
|
sudo dnf install python3-devel
|
||||||
|
|
||||||
|
.. rubric:: Footnotes
|
||||||
|
|
||||||
|
.. [#f1] See :doc:`jenkins`.
|
||||||
@ -1,209 +0,0 @@
|
|||||||
Unit Tests
|
|
||||||
==========
|
|
||||||
|
|
||||||
Cinder contains a suite of unit tests, in the cinder/tests/unit directory.
|
|
||||||
|
|
||||||
Any proposed code change will be automatically rejected by the OpenStack
|
|
||||||
Jenkins server [#f1]_ if the change causes unit test failures.
|
|
||||||
|
|
||||||
Running the tests
|
|
||||||
-----------------
|
|
||||||
There are a number of ways to run unit tests currently, and there's a combination
|
|
||||||
of frameworks used depending on what commands you use. The preferred method
|
|
||||||
is to use tox, which calls ostestr via the tox.ini file. To run all tests simply run::
|
|
||||||
|
|
||||||
tox
|
|
||||||
|
|
||||||
This will create a virtual environment, load all the packages from test-requirements.txt
|
|
||||||
and run all unit tests as well as run flake8 and hacking checks against the code.
|
|
||||||
|
|
||||||
Note that you can inspect the tox.ini file to get more details on the available options
|
|
||||||
and what the test run does by default.
|
|
||||||
|
|
||||||
Running a subset of tests using tox
|
|
||||||
-----------------------------------
|
|
||||||
One common activity is to just run a single test, you can do this with tox simply by
|
|
||||||
specifying to just run py27 or py34 tests against a single test::
|
|
||||||
|
|
||||||
tox -epy27 -- -n cinder.tests.unit.test_volume.AvailabilityZoneTestCase.test_list_availability_zones_cached
|
|
||||||
|
|
||||||
Or all tests in the test_volume.py file::
|
|
||||||
|
|
||||||
tox -epy27 -- -n cinder.tests.unit.test_volume
|
|
||||||
|
|
||||||
For more information on these options and how to run tests, please see the `ostestr
|
|
||||||
documentation <http://docs.openstack.org/developer/os-testr/>`_.
|
|
||||||
|
|
||||||
Run tests wrapper script
|
|
||||||
------------------------
|
|
||||||
|
|
||||||
In addition you can also use the wrapper script run_tests.sh by simply executing::
|
|
||||||
|
|
||||||
./run_tests.sh
|
|
||||||
|
|
||||||
This script is a wrapper around the testr testrunner and the flake8 checker. Note that
|
|
||||||
there has been talk around deprecating this wrapper and this method of testing, it's currently
|
|
||||||
available still but it may be good to get used to using tox or even ostestr directly.
|
|
||||||
|
|
||||||
Documentation is left in place for those that still use it.
|
|
||||||
|
|
||||||
Flags
|
|
||||||
-----
|
|
||||||
|
|
||||||
The ``run_tests.sh`` script supports several flags. You can view a list of
|
|
||||||
flags by doing::
|
|
||||||
|
|
||||||
run_tests.sh -h
|
|
||||||
|
|
||||||
This will show the following help information::
|
|
||||||
|
|
||||||
Usage: ./run_tests.sh [OPTION]...
|
|
||||||
Run Cinder's test suite(s)
|
|
||||||
|
|
||||||
-V, --virtual-env Always use virtualenv. Install automatically if not present
|
|
||||||
-N, --no-virtual-env Don't use virtualenv. Run tests in local environment
|
|
||||||
-s, --no-site-packages Isolate the virtualenv from the global Python environment
|
|
||||||
-r, --recreate-db Recreate the test database (deprecated, as this is now the default).
|
|
||||||
-n, --no-recreate-db Don't recreate the test database.
|
|
||||||
-x, --stop Stop running tests after the first error or failure.
|
|
||||||
-f, --force Force a clean re-build of the virtual environment. Useful when dependencies have been added.
|
|
||||||
-p, --pep8 Just run pep8
|
|
||||||
-P, --no-pep8 Don't run pep8
|
|
||||||
-c, --coverage Generate coverage report
|
|
||||||
-h, --help Print this usage message
|
|
||||||
--hide-elapsed Don't print the elapsed time for each test along with slow test list
|
|
||||||
|
|
||||||
Because ``run_tests.sh`` is a wrapper around testr, it also accepts the same
|
|
||||||
flags as testr. See the `testr documentation`_ for details about
|
|
||||||
these additional flags.
|
|
||||||
|
|
||||||
.. _testr documentation: https://testrepository.readthedocs.org/en/latest/
|
|
||||||
.. _nose options documentation: http://readthedocs.org/docs/nose/en/latest/usage.html#options
|
|
||||||
|
|
||||||
Running a subset of tests
|
|
||||||
-------------------------
|
|
||||||
|
|
||||||
Instead of running all tests, you can specify an individual directory, file,
|
|
||||||
class, or method that contains test code.
|
|
||||||
|
|
||||||
To run the tests in the ``cinder/tests/scheduler`` directory::
|
|
||||||
|
|
||||||
./run_tests.sh scheduler
|
|
||||||
|
|
||||||
To run the tests in the ``cinder/tests/test_libvirt.py`` file::
|
|
||||||
|
|
||||||
./run_tests.sh test_libvirt
|
|
||||||
|
|
||||||
To run the tests in the `HostStateTestCase` class in
|
|
||||||
``cinder/tests/test_libvirt.py``::
|
|
||||||
|
|
||||||
./run_tests.sh test_libvirt.HostStateTestCase
|
|
||||||
|
|
||||||
To run the `ToPrimitiveTestCase.test_dict` test method in
|
|
||||||
``cinder/tests/test_utils.py``::
|
|
||||||
|
|
||||||
./run_tests.sh test_utils.ToPrimitiveTestCase.test_dict
|
|
||||||
|
|
||||||
|
|
||||||
Virtualenv
|
|
||||||
----------
|
|
||||||
|
|
||||||
By default, the tests use the Python packages installed inside a
|
|
||||||
virtualenv [#f2]_. (This is equivalent to using the ``-V, --virtualenv`` flag).
|
|
||||||
If the virtualenv does not exist, it will be created the first time the tests are run.
|
|
||||||
|
|
||||||
If you wish to recreate the virtualenv, call ``run_tests.sh`` with the flag::
|
|
||||||
|
|
||||||
-f, --force
|
|
||||||
|
|
||||||
Recreating the virtualenv is useful if the package dependencies have changed
|
|
||||||
since the virtualenv was last created. If the ``requirements.txt`` or
|
|
||||||
``tools/install_venv.py`` files have changed, it's a good idea to recreate the
|
|
||||||
virtualenv.
|
|
||||||
|
|
||||||
By default, the unit tests will see both the packages in the virtualenv and
|
|
||||||
the packages that have been installed in the Python global environment. In
|
|
||||||
some cases, the packages in the Python global environment may cause a conflict
|
|
||||||
with the packages in the virtualenv. If this occurs, you can isolate the
|
|
||||||
virtualenv from the global environment by using the flag::
|
|
||||||
|
|
||||||
-s, --no-site packages
|
|
||||||
|
|
||||||
If you do not wish to use a virtualenv at all, use the flag::
|
|
||||||
|
|
||||||
-N, --no-virtual-env
|
|
||||||
|
|
||||||
Database
|
|
||||||
--------
|
|
||||||
|
|
||||||
Some of the unit tests make queries against an sqlite database. By
|
|
||||||
default, the test database (``tests.sqlite``) is deleted and recreated each
|
|
||||||
time ``run_tests.sh`` is invoked (This is equivalent to using the
|
|
||||||
``-r, --recreate-db`` flag). To reduce testing time if a database already
|
|
||||||
exists it can be reused by using the flag::
|
|
||||||
|
|
||||||
-n, --no-recreate-db
|
|
||||||
|
|
||||||
Reusing an existing database may cause tests to fail if the schema has
|
|
||||||
changed. If any files in the ``cinder/db/sqlalchemy`` have changed, it's a good
|
|
||||||
idea to recreate the test database.
|
|
||||||
|
|
||||||
Gotchas
|
|
||||||
-------
|
|
||||||
|
|
||||||
**Running Tests from Shared Folders**
|
|
||||||
|
|
||||||
If you are running the unit tests from a shared folder, you may see tests start
|
|
||||||
to fail or stop completely as a result of Python lockfile issues. You
|
|
||||||
can get around this by manually setting or updating the following line in
|
|
||||||
``cinder/tests/conf_fixture.py``::
|
|
||||||
|
|
||||||
CONF['lock_path'].SetDefault('/tmp')
|
|
||||||
|
|
||||||
Note that you may use any location (not just ``/tmp``!) as long as it is not
|
|
||||||
a shared folder.
|
|
||||||
|
|
||||||
.. rubric:: Footnotes
|
|
||||||
|
|
||||||
.. [#f1] See :doc:`jenkins`.
|
|
||||||
|
|
||||||
.. [#f2] See :doc:`development.environment` for more details about the use of
|
|
||||||
virtualenv.
|
|
||||||
|
|
||||||
**Running py34 tests**
|
|
||||||
|
|
||||||
You will need to install:
|
|
||||||
python3-dev
|
|
||||||
in order to get py34 tests to run. If you do not have this, you will get the following::
|
|
||||||
|
|
||||||
netifaces.c:1:20: fatal error: Python.h: No such file or directory
|
|
||||||
#include <Python.h>
|
|
||||||
^
|
|
||||||
compilation terminated.
|
|
||||||
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
|
|
||||||
|
|
||||||
----------------------------------------
|
|
||||||
<snip>
|
|
||||||
ERROR: could not install deps [-r/opt/stack/cinder/test-requirements.txt,
|
|
||||||
oslo.versionedobjects[fixtures]]; v = InvocationError('/opt/stack/cinder/
|
|
||||||
.tox/py34/bin/pip install -r/opt/stack/cinder/test-requirements.txt
|
|
||||||
oslo.versionedobjects[fixtures] (see /opt/stack/cinder/.tox/py34/log/py34-1.log)', 1)
|
|
||||||
_______________________________________________________________ summary _______________________________________________________________
|
|
||||||
ERROR: py34: could not install deps [-r/opt/stack/cinder/test-requirements.txt,
|
|
||||||
oslo.versionedobjects[fixtures]]; v = InvocationError('/opt/stack/cinder/
|
|
||||||
.tox/py34/bin/pip install -r/opt/stack/cinder/test-requirements.txt
|
|
||||||
oslo.versionedobjects[fixtures] (see /opt/stack/cinder/.tox/py34/log/py34-1.log)', 1)
|
|
||||||
|
|
||||||
To Fix:
|
|
||||||
|
|
||||||
- On Ubuntu/Debian::
|
|
||||||
|
|
||||||
sudo apt-get install python3-dev
|
|
||||||
|
|
||||||
- On Fedora 21/RHEL7/CentOS7::
|
|
||||||
|
|
||||||
sudo yum install python3-devel
|
|
||||||
|
|
||||||
- On Fedora 22 and higher::
|
|
||||||
|
|
||||||
sudo dnf install python3-devel
|
|
||||||
Loading…
Reference in New Issue
Block a user