diff --git a/cinder/testing/README.rst b/cinder/testing/README.rst deleted file mode 100644 index e83a7cea4dc..00000000000 --- a/cinder/testing/README.rst +++ /dev/null @@ -1,35 +0,0 @@ -===================================== -OpenStack Cinder Testing Infrastructure -===================================== - -A note of clarification is in order, to help those who are new to testing in -OpenStack cinder: - -- actual unit tests are created in the "tests" directory; -- the "testing" directory is used to house the infrastructure needed to support - testing in OpenStack Cinder. - -This README file attempts to provide current and prospective contributors with -everything they need to know in order to start creating unit tests and -utilizing the convenience code provided in cinder.testing. - -For more detailed information on cinder unit tests visit: -http://docs.openstack.org/developer/cinder/devref/unit_tests.html - -Running Tests ------------------------------------------------ - -In the root of the cinder source code run the run_tests.sh script. This will -offer to create a virtual environment and populate it with dependencies. -If you don't have dependencies installed that are needed for compiling cinder's -direct dependencies, you'll have to use your operating system's method of -installing extra dependencies. To get help using this script execute it with -the -h parameter to get options `./run_tests.sh -h` - -Writing Unit Tests ------------------- - -- All new unit tests are to be written in python-mock. -- Old tests that are still written in mox should be updated to use python-mock. - Usage of mox has been deprecated for writing Cinder unit tests. -- use addCleanup in favor of tearDown diff --git a/doc/source/devref/development.environment.rst b/doc/source/devref/development.environment.rst index fedbbb7d6f2..71553434bde 100644 --- a/doc/source/devref/development.environment.rst +++ b/doc/source/devref/development.environment.rst @@ -94,17 +94,12 @@ Grab the code:: Running unit tests ------------------ -The unit tests will run by default inside a virtualenv in the ``.venv`` -directory. Run the unit tests by doing:: +Run the unit tests by doing:: - ./run_tests.sh + tox -e py34 + tox -e py27 -The first time you run them, you will be asked if you want to create a virtual -environment (hit "y"):: - - No virtual environment found...create one? (Y/n) - -See :doc:`unit_tests` for more details. +See :doc:`testing` for more details. .. _virtualenv: diff --git a/doc/source/devref/index.rst b/doc/source/devref/index.rst index c29e6fc93f1..4737a8bd57e 100644 --- a/doc/source/devref/index.rst +++ b/doc/source/devref/index.rst @@ -18,7 +18,8 @@ Developer Guide =============== -In this section you will find information on Cinder's lower level programming APIs. +In this section you will find information on Cinder's lower level programming +APIs. Programming HowTos and Tutorials @@ -30,7 +31,7 @@ Programming HowTos and Tutorials api_microversion_dev api_conditional_updates api_microversion_history - unit_tests + testing addmethod.openstackapi drivers gmr diff --git a/doc/source/devref/testing.rst b/doc/source/devref/testing.rst new file mode 100644 index 00000000000..f8f6a6b3536 --- /dev/null +++ b/doc/source/devref/testing.rst @@ -0,0 +1,158 @@ +Testing +======= + +Cinder contains a few different test suites in the cinder/tests/ directory. The +different test suites are Unit Tests, Functional Tests, and Tempest Tests. + +Test Types +---------- + + +Unit Tests +~~~~~~~~~~ + +Unit tests are tests for individual methods, with at most a small handful of +modules involved. Mock should be used to remove any external dependencies. + +All significant code changes should have unit test coverage validating the code +happy path and any failure paths. + +Any proposed code change will be automatically rejected by the OpenStack +Jenkins server [#f1]_ if the change causes unit test failures. + +Functional Tests +~~~~~~~~~~~~~~~~ + +Functional tests validate a code path within Cinder. These tests should +validate the interaction of various modules within the project to verify the +code is logically correct. + +Functional tests run with a database present and may start Cinder services to +accept requests. These tests should not need to access an other OpenStack +non-Cinder services. + +Tempest Tests +~~~~~~~~~~~~~ + +The tempest tests in the Cinder tree validate the operational correctness +between Cinder and external components such as Nova, Glance, etc. These are +integration tests driven via public APIs to verify actual end user usage +scenarios. + +Running the tests +----------------- + +There are a number of ways to run tests currently, and there's a combination of +frameworks used depending on what commands you use. The preferred method is to +use tox, which calls ostestr via the tox.ini file. + +Unit Tests +~~~~~~~~~~ + +To run all unit tests simply run:: + + tox + +This will create a virtual environment, load all the packages from +test-requirements.txt and run all unit tests as well as run flake8 and hacking +checks against the code. + +You may run individual test targets, for example only py27 tests, by running:: + + tox -e py27 + +Note that you can inspect the tox.ini file to get more details on the available +options and what the test run does by default. + +Functional Tests +~~~~~~~~~~~~~~~~ + +To run all functional tests, run:: + + tox -e functional + +Tempest Tests +~~~~~~~~~~~~~ + +Tempest tests in the Cinder tree are "plugged in" to the normal tempest test +execution. To ensure the Cinder tests are picked up when running tempest, run:: + + cd /opt/stack/tempest + tox -e all-plugin + +More information about tempest can be found in the `Tempest Documentation +`_. + +Running a subset of tests using tox +----------------------------------- +One common activity is to just run a single test, you can do this with tox +simply by specifying to just run py27 or py34 tests against a single test:: + + tox -epy27 -- -n cinder.tests.unit.test_volume.AvailabilityZoneTestCase.test_list_availability_zones_cached + +Or all tests in the test_volume.py file:: + + tox -epy27 -- -n cinder.tests.unit.test_volume + +You may also use regular expressions to run any matching tests:: + + tox -epy27 -- -r test_volume + +For more information on these options and how to run tests, please see the +`ostestr documentation `_. + +Gotchas +------- + +**Running Tests from Shared Folders** + +If you are running the unit tests from a shared folder, you may see tests start +to fail or stop completely as a result of Python lockfile issues. You +can get around this by manually setting or updating the following line in +``cinder/tests/conf_fixture.py``:: + + CONF['lock_path'].SetDefault('/tmp') + +Note that you may use any location (not just ``/tmp``!) as long as it is not +a shared folder. + +**Running py34 tests** + +You will need to install python3-dev in order to get py34 tests to run. If you +do not have this, you will get the following:: + + netifaces.c:1:20: fatal error: Python.h: No such file or directory + #include + ^ + compilation terminated. + error: command 'x86_64-linux-gnu-gcc' failed with exit status 1 + + ---------------------------------------- + + ERROR: could not install deps [-r/opt/stack/cinder/test-requirements.txt, + oslo.versionedobjects[fixtures]]; v = InvocationError('/opt/stack/cinder/ + .tox/py34/bin/pip install -r/opt/stack/cinder/test-requirements.txt + oslo.versionedobjects[fixtures] (see /opt/stack/cinder/.tox/py34/log/py34-1.log)', 1) + _______________________________________________________________ summary _______________________________________________________________ + ERROR: py34: could not install deps [-r/opt/stack/cinder/test-requirements.txt, + oslo.versionedobjects[fixtures]]; v = InvocationError('/opt/stack/cinder/ + .tox/py34/bin/pip install -r/opt/stack/cinder/test-requirements.txt + oslo.versionedobjects[fixtures] (see /opt/stack/cinder/.tox/py34/log/py34-1.log)', 1) + +To Fix: + +- On Ubuntu/Debian:: + + sudo apt-get install python3-dev + +- On Fedora 21/RHEL7/CentOS7:: + + sudo yum install python3-devel + +- On Fedora 22 and higher:: + + sudo dnf install python3-devel + +.. rubric:: Footnotes + +.. [#f1] See :doc:`jenkins`. diff --git a/doc/source/devref/unit_tests.rst b/doc/source/devref/unit_tests.rst deleted file mode 100644 index 495ef5fa273..00000000000 --- a/doc/source/devref/unit_tests.rst +++ /dev/null @@ -1,209 +0,0 @@ -Unit Tests -========== - -Cinder contains a suite of unit tests, in the cinder/tests/unit directory. - -Any proposed code change will be automatically rejected by the OpenStack -Jenkins server [#f1]_ if the change causes unit test failures. - -Running the tests ------------------ -There are a number of ways to run unit tests currently, and there's a combination -of frameworks used depending on what commands you use. The preferred method -is to use tox, which calls ostestr via the tox.ini file. To run all tests simply run:: - - tox - -This will create a virtual environment, load all the packages from test-requirements.txt -and run all unit tests as well as run flake8 and hacking checks against the code. - -Note that you can inspect the tox.ini file to get more details on the available options -and what the test run does by default. - -Running a subset of tests using tox ------------------------------------ -One common activity is to just run a single test, you can do this with tox simply by -specifying to just run py27 or py34 tests against a single test:: - - tox -epy27 -- -n cinder.tests.unit.test_volume.AvailabilityZoneTestCase.test_list_availability_zones_cached - -Or all tests in the test_volume.py file:: - - tox -epy27 -- -n cinder.tests.unit.test_volume - -For more information on these options and how to run tests, please see the `ostestr -documentation `_. - -Run tests wrapper script ------------------------- - -In addition you can also use the wrapper script run_tests.sh by simply executing:: - - ./run_tests.sh - -This script is a wrapper around the testr testrunner and the flake8 checker. Note that -there has been talk around deprecating this wrapper and this method of testing, it's currently -available still but it may be good to get used to using tox or even ostestr directly. - -Documentation is left in place for those that still use it. - -Flags ------ - -The ``run_tests.sh`` script supports several flags. You can view a list of -flags by doing:: - - run_tests.sh -h - -This will show the following help information:: - - Usage: ./run_tests.sh [OPTION]... - Run Cinder's test suite(s) - - -V, --virtual-env Always use virtualenv. Install automatically if not present - -N, --no-virtual-env Don't use virtualenv. Run tests in local environment - -s, --no-site-packages Isolate the virtualenv from the global Python environment - -r, --recreate-db Recreate the test database (deprecated, as this is now the default). - -n, --no-recreate-db Don't recreate the test database. - -x, --stop Stop running tests after the first error or failure. - -f, --force Force a clean re-build of the virtual environment. Useful when dependencies have been added. - -p, --pep8 Just run pep8 - -P, --no-pep8 Don't run pep8 - -c, --coverage Generate coverage report - -h, --help Print this usage message - --hide-elapsed Don't print the elapsed time for each test along with slow test list - -Because ``run_tests.sh`` is a wrapper around testr, it also accepts the same -flags as testr. See the `testr documentation`_ for details about -these additional flags. - -.. _testr documentation: https://testrepository.readthedocs.org/en/latest/ -.. _nose options documentation: http://readthedocs.org/docs/nose/en/latest/usage.html#options - -Running a subset of tests -------------------------- - -Instead of running all tests, you can specify an individual directory, file, -class, or method that contains test code. - -To run the tests in the ``cinder/tests/scheduler`` directory:: - - ./run_tests.sh scheduler - -To run the tests in the ``cinder/tests/test_libvirt.py`` file:: - - ./run_tests.sh test_libvirt - -To run the tests in the `HostStateTestCase` class in -``cinder/tests/test_libvirt.py``:: - - ./run_tests.sh test_libvirt.HostStateTestCase - -To run the `ToPrimitiveTestCase.test_dict` test method in -``cinder/tests/test_utils.py``:: - - ./run_tests.sh test_utils.ToPrimitiveTestCase.test_dict - - -Virtualenv ----------- - -By default, the tests use the Python packages installed inside a -virtualenv [#f2]_. (This is equivalent to using the ``-V, --virtualenv`` flag). -If the virtualenv does not exist, it will be created the first time the tests are run. - -If you wish to recreate the virtualenv, call ``run_tests.sh`` with the flag:: - - -f, --force - -Recreating the virtualenv is useful if the package dependencies have changed -since the virtualenv was last created. If the ``requirements.txt`` or -``tools/install_venv.py`` files have changed, it's a good idea to recreate the -virtualenv. - -By default, the unit tests will see both the packages in the virtualenv and -the packages that have been installed in the Python global environment. In -some cases, the packages in the Python global environment may cause a conflict -with the packages in the virtualenv. If this occurs, you can isolate the -virtualenv from the global environment by using the flag:: - - -s, --no-site packages - -If you do not wish to use a virtualenv at all, use the flag:: - - -N, --no-virtual-env - -Database --------- - -Some of the unit tests make queries against an sqlite database. By -default, the test database (``tests.sqlite``) is deleted and recreated each -time ``run_tests.sh`` is invoked (This is equivalent to using the -``-r, --recreate-db`` flag). To reduce testing time if a database already -exists it can be reused by using the flag:: - - -n, --no-recreate-db - -Reusing an existing database may cause tests to fail if the schema has -changed. If any files in the ``cinder/db/sqlalchemy`` have changed, it's a good -idea to recreate the test database. - -Gotchas -------- - -**Running Tests from Shared Folders** - -If you are running the unit tests from a shared folder, you may see tests start -to fail or stop completely as a result of Python lockfile issues. You -can get around this by manually setting or updating the following line in -``cinder/tests/conf_fixture.py``:: - - CONF['lock_path'].SetDefault('/tmp') - -Note that you may use any location (not just ``/tmp``!) as long as it is not -a shared folder. - -.. rubric:: Footnotes - -.. [#f1] See :doc:`jenkins`. - -.. [#f2] See :doc:`development.environment` for more details about the use of - virtualenv. - -**Running py34 tests** - -You will need to install: -python3-dev -in order to get py34 tests to run. If you do not have this, you will get the following:: - - netifaces.c:1:20: fatal error: Python.h: No such file or directory - #include - ^ - compilation terminated. - error: command 'x86_64-linux-gnu-gcc' failed with exit status 1 - - ---------------------------------------- - - ERROR: could not install deps [-r/opt/stack/cinder/test-requirements.txt, - oslo.versionedobjects[fixtures]]; v = InvocationError('/opt/stack/cinder/ - .tox/py34/bin/pip install -r/opt/stack/cinder/test-requirements.txt - oslo.versionedobjects[fixtures] (see /opt/stack/cinder/.tox/py34/log/py34-1.log)', 1) - _______________________________________________________________ summary _______________________________________________________________ - ERROR: py34: could not install deps [-r/opt/stack/cinder/test-requirements.txt, - oslo.versionedobjects[fixtures]]; v = InvocationError('/opt/stack/cinder/ - .tox/py34/bin/pip install -r/opt/stack/cinder/test-requirements.txt - oslo.versionedobjects[fixtures] (see /opt/stack/cinder/.tox/py34/log/py34-1.log)', 1) - -To Fix: - -- On Ubuntu/Debian:: - - sudo apt-get install python3-dev - -- On Fedora 21/RHEL7/CentOS7:: - - sudo yum install python3-devel - -- On Fedora 22 and higher:: - - sudo dnf install python3-devel