Merge "Add test type clarification to devref"
This commit is contained in:
Jenkins
Gerrit Code Review
commit
1cd90c646a
@ -1,35 +0,0 @@
|
||||
=====================================
|
||||
OpenStack Cinder Testing Infrastructure
|
||||
=====================================
|
||||
|
||||
A note of clarification is in order, to help those who are new to testing in
|
||||
OpenStack cinder:
|
||||
|
||||
- actual unit tests are created in the "tests" directory;
|
||||
- the "testing" directory is used to house the infrastructure needed to support
|
||||
testing in OpenStack Cinder.
|
||||
|
||||
This README file attempts to provide current and prospective contributors with
|
||||
everything they need to know in order to start creating unit tests and
|
||||
utilizing the convenience code provided in cinder.testing.
|
||||
|
||||
For more detailed information on cinder unit tests visit:
|
||||
http://docs.openstack.org/developer/cinder/devref/unit_tests.html
|
||||
|
||||
Running Tests
|
||||
-----------------------------------------------
|
||||
|
||||
In the root of the cinder source code run the run_tests.sh script. This will
|
||||
offer to create a virtual environment and populate it with dependencies.
|
||||
If you don't have dependencies installed that are needed for compiling cinder's
|
||||
direct dependencies, you'll have to use your operating system's method of
|
||||
installing extra dependencies. To get help using this script execute it with
|
||||
the -h parameter to get options `./run_tests.sh -h`
|
||||
|
||||
Writing Unit Tests
|
||||
------------------
|
||||
|
||||
- All new unit tests are to be written in python-mock.
|
||||
- Old tests that are still written in mox should be updated to use python-mock.
|
||||
Usage of mox has been deprecated for writing Cinder unit tests.
|
||||
- use addCleanup in favor of tearDown
|
||||
@ -94,17 +94,12 @@ Grab the code::
|
||||
|
||||
Running unit tests
|
||||
------------------
|
||||
The unit tests will run by default inside a virtualenv in the ``.venv``
|
||||
directory. Run the unit tests by doing::
|
||||
Run the unit tests by doing::
|
||||
|
||||
./run_tests.sh
|
||||
tox -e py34
|
||||
tox -e py27
|
||||
|
||||
The first time you run them, you will be asked if you want to create a virtual
|
||||
environment (hit "y")::
|
||||
|
||||
No virtual environment found...create one? (Y/n)
|
||||
|
||||
See :doc:`unit_tests` for more details.
|
||||
See :doc:`testing` for more details.
|
||||
|
||||
.. _virtualenv:
|
||||
|
||||
|
||||
@ -18,7 +18,8 @@
|
||||
Developer Guide
|
||||
===============
|
||||
|
||||
In this section you will find information on Cinder's lower level programming APIs.
|
||||
In this section you will find information on Cinder's lower level programming
|
||||
APIs.
|
||||
|
||||
|
||||
Programming HowTos and Tutorials
|
||||
@ -30,7 +31,7 @@ Programming HowTos and Tutorials
|
||||
api_microversion_dev
|
||||
api_conditional_updates
|
||||
api_microversion_history
|
||||
unit_tests
|
||||
testing
|
||||
addmethod.openstackapi
|
||||
drivers
|
||||
gmr
|
||||
|
||||
158
doc/source/devref/testing.rst
Normal file
158
doc/source/devref/testing.rst
Normal file
@ -0,0 +1,158 @@
|
||||
Testing
|
||||
=======
|
||||
|
||||
Cinder contains a few different test suites in the cinder/tests/ directory. The
|
||||
different test suites are Unit Tests, Functional Tests, and Tempest Tests.
|
||||
|
||||
Test Types
|
||||
----------
|
||||
|
||||
|
||||
Unit Tests
|
||||
~~~~~~~~~~
|
||||
|
||||
Unit tests are tests for individual methods, with at most a small handful of
|
||||
modules involved. Mock should be used to remove any external dependencies.
|
||||
|
||||
All significant code changes should have unit test coverage validating the code
|
||||
happy path and any failure paths.
|
||||
|
||||
Any proposed code change will be automatically rejected by the OpenStack
|
||||
Jenkins server [#f1]_ if the change causes unit test failures.
|
||||
|
||||
Functional Tests
|
||||
~~~~~~~~~~~~~~~~
|
||||
|
||||
Functional tests validate a code path within Cinder. These tests should
|
||||
validate the interaction of various modules within the project to verify the
|
||||
code is logically correct.
|
||||
|
||||
Functional tests run with a database present and may start Cinder services to
|
||||
accept requests. These tests should not need to access an other OpenStack
|
||||
non-Cinder services.
|
||||
|
||||
Tempest Tests
|
||||
~~~~~~~~~~~~~
|
||||
|
||||
The tempest tests in the Cinder tree validate the operational correctness
|
||||
between Cinder and external components such as Nova, Glance, etc. These are
|
||||
integration tests driven via public APIs to verify actual end user usage
|
||||
scenarios.
|
||||
|
||||
Running the tests
|
||||
-----------------
|
||||
|
||||
There are a number of ways to run tests currently, and there's a combination of
|
||||
frameworks used depending on what commands you use. The preferred method is to
|
||||
use tox, which calls ostestr via the tox.ini file.
|
||||
|
||||
Unit Tests
|
||||
~~~~~~~~~~
|
||||
|
||||
To run all unit tests simply run::
|
||||
|
||||
tox
|
||||
|
||||
This will create a virtual environment, load all the packages from
|
||||
test-requirements.txt and run all unit tests as well as run flake8 and hacking
|
||||
checks against the code.
|
||||
|
||||
You may run individual test targets, for example only py27 tests, by running::
|
||||
|
||||
tox -e py27
|
||||
|
||||
Note that you can inspect the tox.ini file to get more details on the available
|
||||
options and what the test run does by default.
|
||||
|
||||
Functional Tests
|
||||
~~~~~~~~~~~~~~~~
|
||||
|
||||
To run all functional tests, run::
|
||||
|
||||
tox -e functional
|
||||
|
||||
Tempest Tests
|
||||
~~~~~~~~~~~~~
|
||||
|
||||
Tempest tests in the Cinder tree are "plugged in" to the normal tempest test
|
||||
execution. To ensure the Cinder tests are picked up when running tempest, run::
|
||||
|
||||
cd /opt/stack/tempest
|
||||
tox -e all-plugin
|
||||
|
||||
More information about tempest can be found in the `Tempest Documentation
|
||||
<http://docs.openstack.org/developer/tempest/overview.html>`_.
|
||||
|
||||
Running a subset of tests using tox
|
||||
-----------------------------------
|
||||
One common activity is to just run a single test, you can do this with tox
|
||||
simply by specifying to just run py27 or py34 tests against a single test::
|
||||
|
||||
tox -epy27 -- -n cinder.tests.unit.test_volume.AvailabilityZoneTestCase.test_list_availability_zones_cached
|
||||
|
||||
Or all tests in the test_volume.py file::
|
||||
|
||||
tox -epy27 -- -n cinder.tests.unit.test_volume
|
||||
|
||||
You may also use regular expressions to run any matching tests::
|
||||
|
||||
tox -epy27 -- -r test_volume
|
||||
|
||||
For more information on these options and how to run tests, please see the
|
||||
`ostestr documentation <http://docs.openstack.org/developer/os-testr/>`_.
|
||||
|
||||
Gotchas
|
||||
-------
|
||||
|
||||
**Running Tests from Shared Folders**
|
||||
|
||||
If you are running the unit tests from a shared folder, you may see tests start
|
||||
to fail or stop completely as a result of Python lockfile issues. You
|
||||
can get around this by manually setting or updating the following line in
|
||||
``cinder/tests/conf_fixture.py``::
|
||||
|
||||
CONF['lock_path'].SetDefault('/tmp')
|
||||
|
||||
Note that you may use any location (not just ``/tmp``!) as long as it is not
|
||||
a shared folder.
|
||||
|
||||
**Running py34 tests**
|
||||
|
||||
You will need to install python3-dev in order to get py34 tests to run. If you
|
||||
do not have this, you will get the following::
|
||||
|
||||
netifaces.c:1:20: fatal error: Python.h: No such file or directory
|
||||
#include <Python.h>
|
||||
^
|
||||
compilation terminated.
|
||||
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
|
||||
|
||||
----------------------------------------
|
||||
<snip>
|
||||
ERROR: could not install deps [-r/opt/stack/cinder/test-requirements.txt,
|
||||
oslo.versionedobjects[fixtures]]; v = InvocationError('/opt/stack/cinder/
|
||||
.tox/py34/bin/pip install -r/opt/stack/cinder/test-requirements.txt
|
||||
oslo.versionedobjects[fixtures] (see /opt/stack/cinder/.tox/py34/log/py34-1.log)', 1)
|
||||
_______________________________________________________________ summary _______________________________________________________________
|
||||
ERROR: py34: could not install deps [-r/opt/stack/cinder/test-requirements.txt,
|
||||
oslo.versionedobjects[fixtures]]; v = InvocationError('/opt/stack/cinder/
|
||||
.tox/py34/bin/pip install -r/opt/stack/cinder/test-requirements.txt
|
||||
oslo.versionedobjects[fixtures] (see /opt/stack/cinder/.tox/py34/log/py34-1.log)', 1)
|
||||
|
||||
To Fix:
|
||||
|
||||
- On Ubuntu/Debian::
|
||||
|
||||
sudo apt-get install python3-dev
|
||||
|
||||
- On Fedora 21/RHEL7/CentOS7::
|
||||
|
||||
sudo yum install python3-devel
|
||||
|
||||
- On Fedora 22 and higher::
|
||||
|
||||
sudo dnf install python3-devel
|
||||
|
||||
.. rubric:: Footnotes
|
||||
|
||||
.. [#f1] See :doc:`jenkins`.
|
||||
@ -1,209 +0,0 @@
|
||||
Unit Tests
|
||||
==========
|
||||
|
||||
Cinder contains a suite of unit tests, in the cinder/tests/unit directory.
|
||||
|
||||
Any proposed code change will be automatically rejected by the OpenStack
|
||||
Jenkins server [#f1]_ if the change causes unit test failures.
|
||||
|
||||
Running the tests
|
||||
-----------------
|
||||
There are a number of ways to run unit tests currently, and there's a combination
|
||||
of frameworks used depending on what commands you use. The preferred method
|
||||
is to use tox, which calls ostestr via the tox.ini file. To run all tests simply run::
|
||||
|
||||
tox
|
||||
|
||||
This will create a virtual environment, load all the packages from test-requirements.txt
|
||||
and run all unit tests as well as run flake8 and hacking checks against the code.
|
||||
|
||||
Note that you can inspect the tox.ini file to get more details on the available options
|
||||
and what the test run does by default.
|
||||
|
||||
Running a subset of tests using tox
|
||||
-----------------------------------
|
||||
One common activity is to just run a single test, you can do this with tox simply by
|
||||
specifying to just run py27 or py34 tests against a single test::
|
||||
|
||||
tox -epy27 -- -n cinder.tests.unit.test_volume.AvailabilityZoneTestCase.test_list_availability_zones_cached
|
||||
|
||||
Or all tests in the test_volume.py file::
|
||||
|
||||
tox -epy27 -- -n cinder.tests.unit.test_volume
|
||||
|
||||
For more information on these options and how to run tests, please see the `ostestr
|
||||
documentation <http://docs.openstack.org/developer/os-testr/>`_.
|
||||
|
||||
Run tests wrapper script
|
||||
------------------------
|
||||
|
||||
In addition you can also use the wrapper script run_tests.sh by simply executing::
|
||||
|
||||
./run_tests.sh
|
||||
|
||||
This script is a wrapper around the testr testrunner and the flake8 checker. Note that
|
||||
there has been talk around deprecating this wrapper and this method of testing, it's currently
|
||||
available still but it may be good to get used to using tox or even ostestr directly.
|
||||
|
||||
Documentation is left in place for those that still use it.
|
||||
|
||||
Flags
|
||||
-----
|
||||
|
||||
The ``run_tests.sh`` script supports several flags. You can view a list of
|
||||
flags by doing::
|
||||
|
||||
run_tests.sh -h
|
||||
|
||||
This will show the following help information::
|
||||
|
||||
Usage: ./run_tests.sh [OPTION]...
|
||||
Run Cinder's test suite(s)
|
||||
|
||||
-V, --virtual-env Always use virtualenv. Install automatically if not present
|
||||
-N, --no-virtual-env Don't use virtualenv. Run tests in local environment
|
||||
-s, --no-site-packages Isolate the virtualenv from the global Python environment
|
||||
-r, --recreate-db Recreate the test database (deprecated, as this is now the default).
|
||||
-n, --no-recreate-db Don't recreate the test database.
|
||||
-x, --stop Stop running tests after the first error or failure.
|
||||
-f, --force Force a clean re-build of the virtual environment. Useful when dependencies have been added.
|
||||
-p, --pep8 Just run pep8
|
||||
-P, --no-pep8 Don't run pep8
|
||||
-c, --coverage Generate coverage report
|
||||
-h, --help Print this usage message
|
||||
--hide-elapsed Don't print the elapsed time for each test along with slow test list
|
||||
|
||||
Because ``run_tests.sh`` is a wrapper around testr, it also accepts the same
|
||||
flags as testr. See the `testr documentation`_ for details about
|
||||
these additional flags.
|
||||
|
||||
.. _testr documentation: https://testrepository.readthedocs.org/en/latest/
|
||||
.. _nose options documentation: http://readthedocs.org/docs/nose/en/latest/usage.html#options
|
||||
|
||||
Running a subset of tests
|
||||
-------------------------
|
||||
|
||||
Instead of running all tests, you can specify an individual directory, file,
|
||||
class, or method that contains test code.
|
||||
|
||||
To run the tests in the ``cinder/tests/scheduler`` directory::
|
||||
|
||||
./run_tests.sh scheduler
|
||||
|
||||
To run the tests in the ``cinder/tests/test_libvirt.py`` file::
|
||||
|
||||
./run_tests.sh test_libvirt
|
||||
|
||||
To run the tests in the `HostStateTestCase` class in
|
||||
``cinder/tests/test_libvirt.py``::
|
||||
|
||||
./run_tests.sh test_libvirt.HostStateTestCase
|
||||
|
||||
To run the `ToPrimitiveTestCase.test_dict` test method in
|
||||
``cinder/tests/test_utils.py``::
|
||||
|
||||
./run_tests.sh test_utils.ToPrimitiveTestCase.test_dict
|
||||
|
||||
|
||||
Virtualenv
|
||||
----------
|
||||
|
||||
By default, the tests use the Python packages installed inside a
|
||||
virtualenv [#f2]_. (This is equivalent to using the ``-V, --virtualenv`` flag).
|
||||
If the virtualenv does not exist, it will be created the first time the tests are run.
|
||||
|
||||
If you wish to recreate the virtualenv, call ``run_tests.sh`` with the flag::
|
||||
|
||||
-f, --force
|
||||
|
||||
Recreating the virtualenv is useful if the package dependencies have changed
|
||||
since the virtualenv was last created. If the ``requirements.txt`` or
|
||||
``tools/install_venv.py`` files have changed, it's a good idea to recreate the
|
||||
virtualenv.
|
||||
|
||||
By default, the unit tests will see both the packages in the virtualenv and
|
||||
the packages that have been installed in the Python global environment. In
|
||||
some cases, the packages in the Python global environment may cause a conflict
|
||||
with the packages in the virtualenv. If this occurs, you can isolate the
|
||||
virtualenv from the global environment by using the flag::
|
||||
|
||||
-s, --no-site packages
|
||||
|
||||
If you do not wish to use a virtualenv at all, use the flag::
|
||||
|
||||
-N, --no-virtual-env
|
||||
|
||||
Database
|
||||
--------
|
||||
|
||||
Some of the unit tests make queries against an sqlite database. By
|
||||
default, the test database (``tests.sqlite``) is deleted and recreated each
|
||||
time ``run_tests.sh`` is invoked (This is equivalent to using the
|
||||
``-r, --recreate-db`` flag). To reduce testing time if a database already
|
||||
exists it can be reused by using the flag::
|
||||
|
||||
-n, --no-recreate-db
|
||||
|
||||
Reusing an existing database may cause tests to fail if the schema has
|
||||
changed. If any files in the ``cinder/db/sqlalchemy`` have changed, it's a good
|
||||
idea to recreate the test database.
|
||||
|
||||
Gotchas
|
||||
-------
|
||||
|
||||
**Running Tests from Shared Folders**
|
||||
|
||||
If you are running the unit tests from a shared folder, you may see tests start
|
||||
to fail or stop completely as a result of Python lockfile issues. You
|
||||
can get around this by manually setting or updating the following line in
|
||||
``cinder/tests/conf_fixture.py``::
|
||||
|
||||
CONF['lock_path'].SetDefault('/tmp')
|
||||
|
||||
Note that you may use any location (not just ``/tmp``!) as long as it is not
|
||||
a shared folder.
|
||||
|
||||
.. rubric:: Footnotes
|
||||
|
||||
.. [#f1] See :doc:`jenkins`.
|
||||
|
||||
.. [#f2] See :doc:`development.environment` for more details about the use of
|
||||
virtualenv.
|
||||
|
||||
**Running py34 tests**
|
||||
|
||||
You will need to install:
|
||||
python3-dev
|
||||
in order to get py34 tests to run. If you do not have this, you will get the following::
|
||||
|
||||
netifaces.c:1:20: fatal error: Python.h: No such file or directory
|
||||
#include <Python.h>
|
||||
^
|
||||
compilation terminated.
|
||||
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
|
||||
|
||||
----------------------------------------
|
||||
<snip>
|
||||
ERROR: could not install deps [-r/opt/stack/cinder/test-requirements.txt,
|
||||
oslo.versionedobjects[fixtures]]; v = InvocationError('/opt/stack/cinder/
|
||||
.tox/py34/bin/pip install -r/opt/stack/cinder/test-requirements.txt
|
||||
oslo.versionedobjects[fixtures] (see /opt/stack/cinder/.tox/py34/log/py34-1.log)', 1)
|
||||
_______________________________________________________________ summary _______________________________________________________________
|
||||
ERROR: py34: could not install deps [-r/opt/stack/cinder/test-requirements.txt,
|
||||
oslo.versionedobjects[fixtures]]; v = InvocationError('/opt/stack/cinder/
|
||||
.tox/py34/bin/pip install -r/opt/stack/cinder/test-requirements.txt
|
||||
oslo.versionedobjects[fixtures] (see /opt/stack/cinder/.tox/py34/log/py34-1.log)', 1)
|
||||
|
||||
To Fix:
|
||||
|
||||
- On Ubuntu/Debian::
|
||||
|
||||
sudo apt-get install python3-dev
|
||||
|
||||
- On Fedora 21/RHEL7/CentOS7::
|
||||
|
||||
sudo yum install python3-devel
|
||||
|
||||
- On Fedora 22 and higher::
|
||||
|
||||
sudo dnf install python3-devel
|
||||
Loading…
Reference in New Issue
Block a user