Commit Graph

344 Commits

Author SHA1 Message Date
Sean McGinnis
5d6ec316d6
Drop py35 jobs
Python 3.5 was the target runtime for the Rocky release. The current
target py3 runtime for Stein is Python 3.6, so there is no reason to
keep testing against the older version.

https://governance.openstack.org/tc/reference/runtimes/stein.html#python-runtime-for-stein

Change-Id: If810570859e4c4a942f44097717170d5a0affbd0
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
2019-02-26 11:08:40 -06:00
Sean McGinnis
d5b539be36
Doc8: Stop skipping D001: Line too long
This cleans up the cases where we had D001 violations so we can stop
skipping that check in doc8 runs.

Change-Id: Ie52f6ecac1a645fcbcc643b9ca63e033b622d830
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
2019-02-19 16:51:56 -06:00
Sean McGinnis
b6f9932f9c
Include .inc files in doc8 linting
We use *.inc files extensively, especially in the api-ref, but these
files are not covered by default with our doc8 validation.

This adds support for those files and fixes a few issues that were
identified after enabling coverage.

Change-Id: I807823f3cb44274c3940db55e901d434304a09f5
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
2019-02-19 11:33:05 -06:00
Eric Harney
e8fe5aaf6a fast8 improvements
Determines how many commits to check based
on the $FAST8_NUM_COMMITS env variable.

If set to "smart", it uses git to try to run
against all unsubmitted commits.  This allows
fast8 to be more useful when actively developing
a series of patches.

Change-Id: I0adfba626ba2ea3479faf0f6ade712f14d3080d3
2019-01-22 14:36:23 -05:00
Zuul
806fac5606 Merge "Automate generation of api-ref samples" 2018-12-14 08:03:34 +00:00
whoami-rajat
67150217c9 Automate generation of api-ref samples
This patch adds the following functionality:

1) Base framework for test cases
2) Volume type tests

Change-Id: I0e51bcc8c634923ca6fcff99a9f40b786933caa5
Partially-Implements: blueprint generating-sample-files
2018-12-06 12:19:55 +05:30
Yikun Jiang
3c2a22f75c Make lower constraints CI works well
The actual target of openstack-tox-lower-constraints is using the lower
version package to run unit test.

But in current Cinder CI,there was a hard code to specify the
upper-constraints.txt, that means we install the package in
upper-constraints.txt, but the lower-constraints.txt doesn't work well.

This patch try to fix this error, move the hard code to 'deps'.

We also refresh the lower-constraints.txt and requirements.txt to make
CI works well:
1. Add libxml2-dev, libxslt-devel, libxslt1-dev to bindep.txt to make
lxml can be built successfully, like what we do in other project. [0]
2. Upgrade oslo.messaging to 6.4.0 to avoid "ACCESS_REFUSED
- Login was refused using authentication mechanism AMQPLAIN." error.
3. Upgrade oslo.utils to 3.34.0 to add a new method escape_ipv6 [1] to
oslo_utils.netutils, which is needed by [2].
4. Upgrade keystoneauth1 to 3.7.0 to fix "no such option split_loggers
in group [service_user]" [3]

[0] https://github.com/openstack/nova/blob/bb3ded3/bindep.txt#L17-L19
[1] https://review.openstack.org/#/c/495166/
[2] https://github.com/openstack/cinder/blob/fe421fe/cinder/volume/drivers/netapp/utils.py#L172
[3] https://review.openstack.org/#/c/578008/

Change-Id: Ide84d49c283c0ad179a5cfdb0392d34ee73d97ea
Closes-bug: #1801239
2018-11-06 20:29:53 +08:00
Sean McGinnis
50a1ed71cb
Remove setup.py check from pep8 job
Using "python setup.py check -r -s" method of checking the package has
been deprecated with the new recommendation to build the sdist and
wheel, then running "twine check" against the output.

Luckily, there is already a job that covers this that only runs when the
README, setup.py, or setup.cfg files change, making running this in the
pep8 job redundant. This covered by the test-release-openstack-python3
that is defined in the publish-to-pypi-python3 template.

More details can be found in this mailing list post:

http://lists.openstack.org/pipermail/openstack-dev/2018-October/136136.html

Change-Id: Ia981b27852b9061c8a406e68388c60595aa528f8
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
2018-10-30 10:11:48 -05:00
Eric Harney
e82a842616 Add test-requirements to pylint reqs
Need this for ddt, mock, etc.  pylint runs
currently throw ImportErrors related to this.

Change-Id: Ia906cd89acc9c12e4b731751763b4e9331aa80d2
2018-10-15 10:22:20 -04:00
imacdonn
8ca71f5b5b Don't quote {posargs} in tox.ini
Quotes around {posargs} cause the entire string to be combined into one
arg that gets passed to stestr. This prevents passing multiple args
(e.g. '--concurrency=16 some-regex')

Change-Id: I8ce3c403904ea1c6990f88d52c1c1a068aaed75a
2018-10-08 19:44:04 +00:00
Chuck Short
adcfb4f97e Simplify running pylint
This commit does several things:

- Setup and run pylint directly rather than running
  through a script. This allows the user to see what is happening
  while the user is running through pylint.
- Allow the user to either run pylint on a particular changeset,
  or the entire cinder tree.
- Allow the user to run on a particular changeset. Using like
  HEAD~1, etc.
- Since the pylint gate check I disabled the tests that were
  reported by pylint. The thought here would be go through
  the failures and correct them.
- Update pylint to 2.1.1.
- Removed old pylintrc.

Change-Id: I708c93843d991e8c0ac114b0b2d6ede373b96d0a
Signed-off-by: Chuck Short <chucks@redhat.com>
2018-09-06 10:00:07 -04:00
Sean McGinnis
5a2591051f Enable running py36 unit tests
This adds a tox setting for skipping default targets if the
specified version is not available. This allows tests to run
using python 3.6 if available, but conversely also allows
skipping python 3.5 or even 2.7 if those runtimes are not
present.

This only affects local default tox runs and does not change
what is tested in the gate.

Change-Id: Ia34fd32ab8d4672a2725468ee3d37697f51d5b3a
2018-06-28 11:11:27 -05:00
Sean McGinnis
5542de51bc Be explicit about tox python version
We want to default all jobs to use python 3 except for
the ones that use explicit versions. Unfortunately, the
current tox behavior for setting basepython to python3
is not intuitive.

If the system default is python 3.5, then when basepython
is set to python3, both the py35 and py36 targets will
end up running with python 3.5.

To get around this we need to not set a default base
version and explicitly set each testenv to python3 that
we want to run using 3 instead of 2 but do not really
care which minor version is used.

Change-Id: I3971eeafbd1e732a584b0c52d2eb6c2175b709aa
2018-06-13 16:54:35 -05:00
Zuul
b711c8da16 Merge "Run pylint job under py3" 2018-06-12 16:00:08 +00:00
Zuul
a8578bd488 Merge "Default tox jobs to python3" 2018-06-12 15:43:38 +00:00
junboli
1268872774 Add check to the flake8 job
Follow up recommendation of [1].
[1] http://lists.openstack.org/pipermail/openstack-dev/2018-June/131233.html

Change-Id: Id268be8b32ce870f2f04fa8a52f4f1854a613fa7
2018-06-07 19:48:57 +08:00
Sean McGinnis
d675de18de Run pylint job under py3
This switches the pylint target to run under python3. In
order to work right, it also raises the version of pylint
used to a newer version.

Related-bug: #1761800

Change-Id: I0bc68a5aee24a53b9d3a08c20549b89d08989678
2018-06-06 16:28:34 -05:00
Sean McGinnis
37510ac034 Default tox jobs to python3
The long term goal for OpenStack is to support python3 for
all things except explicit python2 testing by the T release.
Part of preparing for that is ensuring all ancillary things
like docs jobs and other tooling work with python3.

This switches the default for our docs jobs to python3 so
anything that does not specify another version will end up
using it.

Further work is necessary for the pylint job due to changes
between the runtimes. That will be done in a follow up patch.

There was also a difference in behavior with the genopts job
where it ends up trying to recreate the venv under which it
is currently running under, resulting in a corrupted venv
and a failure. This cleans up that script and changes it so
rather than a tox job calling a tox job it just runs the
script.

Change-Id: Ie79528b33070e6c60f9f5e1f2219084a2f0cb039
2018-06-05 14:16:34 -05:00
Zuul
1ca6ab6ffa Merge "Remove stale pip-missing-reqs tox test" 2018-05-10 18:45:59 +00:00
Zuul
603589e54f Merge "Default pep8/fast8 to run under python 3" 2018-05-10 18:40:27 +00:00
fpxie
99119568b0 Remove stale pip-missing-reqs tox test
pip_missing_reqs tool is no longer maintained and has broken with release 10 of pip

Refer to:
  http://lists.openstack.org/pipermail/openstack-dev/2018-April/130027.html

Change-Id: Iaa65b1789590e85602e7a63ca4e80cd087b05693
2018-05-07 15:20:18 +08:00
Sean McGinnis
7745ff3944 Test running jobs under py36/bionic
Change-Id: I2036f9be98f76cba21180d74dbde1050c3359234
2018-04-26 08:56:55 -05:00
Zuul
08fb43e805 Merge "add lower-constraints job" 2018-04-10 20:59:01 +00:00
Zuul
b9a0a64211 Merge "Use sphinxcontrib.apidoc for module docs" 2018-04-10 05:20:37 +00:00
Eric Harney
bae574d0c9 Add doc to pep8 check_exec
Detect executable files in doc/source.

Change-Id: I43894faacdec839cc61bfb7db675f33cbc5d454d
2018-04-09 14:42:05 -04:00
Doug Hellmann
6a8ebbc1ae add lower-constraints job
Create a tox environment for running the unit tests against the lower
bounds of the dependencies.

Create a lower-constraints.txt to be used to enforce the lower bounds
in those tests.

Add openstack-tox-lower-constraints job to the zuul configuration.

See http://lists.openstack.org/pipermail/openstack-dev/2018-March/128352.html
for more details.

Change-Id: I329e6364d537f2acac796e389f15f3c7162a9244
Depends-On: https://review.openstack.org/555034
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
2018-04-07 21:23:14 -04:00
Sean McGinnis
cb1a05a2c2 Default pep8/fast8 to run under python 3
There are slight differences with pep8 when run under python 3.
It is a little more strict on certain things, plus we want to
get to just supporting python 3 eventually, so this switches
over the pep8 and fast8 targets to use python 3.

Change-Id: I3212bfb2dd173e79dae6d548827a803f68923da9
2018-04-05 19:47:50 -05:00
Sean McGinnis
3b34019f0b Use sphinxcontrib.apidoc for module docs
Now that sphinxcontrib.apidoc is available, we should get rid of
the custom code we have for generating this information ourselves.
This also cleans up some really stale docs that are no longer
needed.

Change-Id: Iaa9fecc2478326b45b67a2cfd98de5f93f537efe
2018-04-05 14:36:03 -05:00
Sean McGinnis
e68fba2b09 Fix doc format errors in module docstrings
This fixes some formatting issues in docstrings to allow us
to enable the warnings-as-errors flag for our docs build.

Co-authored by: Jay S. Bryant <jungleboyj@electronicjungle.net>

Change-Id: I9567e232686d8775ff67418e8333c164f8acd8f3
2018-03-28 17:05:07 -05:00
Zuul
c0b8de28e4 Merge "Enable hash randomization in unit tests" 2018-02-22 04:56:52 +00:00
Eric Harney
52965d8974 Enable hash randomization in unit tests
This is needed to ensure that unit tests match how
deployed environments run.

Related-Bug: #1348818
Related-Bug: #1658725
Related-Bug: #1715915

Depends-On: I91a53fdef98c9a14ccc536403063e228e99d6971
Depends-On: Ie548c90f5657e5270fd966207f453d5211e21867
Depends-On: I754d48d8bc4081b96fdf90465c4897d0fadd6262
Depends-On: Ia8762aff6fd9954d350310b4c68ee4fe94de68bf
Depends-On: I6fe0ae5da270d9c77cb248551dcb32a5c063977a
Depends-On: I96b9b413387f2ef02ea7ccfb6e676cad0b87c6f1
Depends-On: Icdde5e3f213adfe4893d1b9e8f2bfe4f61704037
Depends-On: I29a874ebe0ebaeca58b428faa0bab50482baf6f1
Depends-On: I30817ae767e08c45f9540b19e922864fe2f7a9c2
Depends-On: I281ad59fa518f936aa8d1b518acf61800ba949f5
Depends-On: I0abd0415306c0c0544eb5d9a6ed08640ff6bb0cf
Depends-On: I599b585444cbd51182d9f17fddd97bd91637e1e3
Depends-On: I6e4093ca8e79128ed23a4d5068e5d2870e16332c
Depends-On: I574dca83a971e746a54895a87e394befda0ccdcf
Depends-On: Id5a961c66a62e289a96daa7437374768ef407bb6
Depends-On: Ibc6c20e6ffce58c6bbef3155f73e54909d76e132
Change-Id: I4c7396f9d8e58ac8420754503f8ed60cd7473426
2018-02-20 12:20:58 -05:00
Eric Harney
0368192a1f Fix find -delete race in tox
If you launch py35 and py27 envs from the same
directory simultaneously, the find -delete commands
here will race against each other and cause a
failure.

Set the find option to ignore readdir races.

Change-Id: I4977ad687150b5a1d6f09a961fca4db5d768ef71
2018-02-19 13:07:39 -05:00
Eric Harney
12b6ccc804 Run doc8 first
For "tox -e docs", run doc8 first.

doc8 runs much faster than sphinx builds and
will provide quick feedback on syntax issues.

Change-Id: Id4f77020a70acd8eb9df513d43f86d9fe78f7e22
2018-01-23 17:06:53 +00:00
jiansong
f877ec2917 Follow the new PTI for document build
For compliance with the Project Testing Interface as described in:

https://governance.openstack.org/tc/reference/project-testing-interface.html

For more detials information, please refer to:

http://lists.openstack.org/pipermail/openstack-dev/2017-December/125710.html

Change-Id: I84e614346cfa4d302f919d12bea9680c5e527475
2017-12-20 18:24:11 -08:00
Sean McGinnis
6d36629294 Add output of slowest tests to UT runs
With the switch to stestr we lost the automatic output of the slowest
tests. We can still get that output, but it is a separate command to
run after the tests. Add that to our tox.ini command for test execution.

Change-Id: I37937c1ebe4a10b896f1deb77f64b520ceba2830
2017-12-11 10:25:29 -06:00
Zuul
8751b5dc8a Merge "Remove doc/build before running docs job" 2017-10-27 00:19:44 +00:00
junboli
b7048784b1 Remove doc/build before running docs job
Clear doc/build directory before running docs job may better guarantee
each building output the newest docs.

Change-Id: I17537742a4c1421f2bd0561d2c48c5ae56d89026
2017-10-25 08:21:05 +00:00
TommyLike
988f79479e Add policy documentation and sample file
This patch adds documentation and sample
file for default policy in code feature.

Change-Id: I597971a29ec61a1bf8c991b2715ec7644b2e2692
Partial-Implements: blueprint policy-in-code
2017-10-20 10:47:34 +08:00
j-griffith
f93bc791bf Switch from ostestr to stestr
This change just updates the tox.ini file to call stestr directly
instead of going through ostestr.

For more background you can read the ML posting:
    https://goo.gl/TFvcxy

So now you won't need `--n` to run single tests, and you can also
supply regex directly without even the `--`.  For example:
  `tox -epy27 $regex`

Change-Id: I75d4ca19f87fe1930cf54b554b7e232632a4a21c
2017-10-15 15:02:55 +00:00
Sean McGinnis
676195e009 Add .stestr.conf configuration
os-testr has moved over to use stestr instead of testr. While this
is usually compaible with existing settings, there is a warning
that is emitted when .stestr.conf is not present. It is usually
able to fall back to parsing the .testr.conf file, but to be more
correct and to prevent future problems we should update the config.

Change-Id: I11e43cff87cad20b9c880e276e20a04123f830fb
2017-10-10 00:46:42 +00:00
Jenkins
e912ced6e1 Merge "[policy in code] Add support for attachment resource" 2017-10-08 11:01:53 +00:00
TommyLike
43a3152581 [policy in code] Add support for attachment resource
This is the basic patch which consits of the framework
code for default policy in code feature as well as
support in attachment resource.

Change-Id: Ie3ff068e61ea8e0e8fff78deb732e183e036a10c
Partial-Implements: blueprint policy-in-code
2017-09-26 03:19:20 +00:00
Sean McGinnis
604529a160 Clear cached autogenerated docs before docs build
Leftover data in the autogenerated docs can result in build
failure. Remove these files before every build to be safe.

Change-Id: Iec3ff7334b7af80cb34019590544fab0b1c95240
Closes-bug: #1716478
2017-09-20 12:36:30 -05:00
Eric Harney
d3abafdee4 Move config-generator to tools
Moving this out of cinder/ separates it from the
actual run-time Cinder code.

Change-Id: I056f40120fdef60a0dc1a5926729748fd2835938
2017-08-16 16:18:48 -04:00
Eric Harney
6460050c5c Add releasenotes to check_exec list
Prevent addition of executable files in
the releasenotes/notes directory.

Change-Id: I80ac0b7fce7b16a453fe9139f0dacf88325af8e7
2017-08-16 15:50:44 -04:00
Jay S. Bryant
429281bee2 Ignore all .egg-info directories in doc8 check
Doc builds can fail if you have .egg-info directories
in your repo.  This change makes sure that those
directories are ignored in the doc8 check.

Change-Id: I8e713651919f43feb80fca59b74c103b3255dd75
2017-07-13 10:43:56 -05:00
Ngo Quoc Cuong
8ca93ac3f7 Enable H904 check
H904  String interpolation should be delayed to be handled by the
logging code, rather than being done at the point of the logging call.
Use ',' instead of '%'.

See: https://docs.openstack.org/oslo.i18n/latest/user/guidelines.html#adding-variables-to-log-messages

Change-Id: Ib5781b837ba60028dce6dddd5b198647e002c8f1
2017-07-05 21:27:05 -04:00
jeremy.zhang
85a1b8d40b Enable some off-by-default checks
Some of the available checks are disabled by default in flake8, like:
[H106] Don’t put vim configuration in source files
[H203] Use assertIs(Not)None to check for None

This patch is to enable the H106 and H203 checks in Cinder project.
The C312 hacking rule will be removed when turn on H203.

Change-Id: I2e883c301b64d5977bbb907b63c9c144bc6f959d
2017-06-22 02:17:54 +00:00
Dirk Mueller
2db0497fde Add a local bindep.txt override
This avoids falling back to the global bindep-fallback.txt which
installs a pretty big environment. Without this change, a lot
of additional but unnecessary packages are installed on the nodepool
workers.

Change-Id: I8c581225eada929bc947b13f851bb228000f32fe
2017-05-25 23:27:10 +02:00
Jenkins
67fc1f76b4 Merge "Add bandit-baseline to tox.ini" 2017-03-29 22:29:40 +00:00
Nicholas Jones
2bec754d50 Add bandit-baseline to tox.ini
To add a bandit gate, a new tox environment is needed

Change-Id: If247add30d8dc4761dacb82c8d02ea4bd71546f0
2017-03-29 10:55:25 -05:00
Eric Harney
52e0f003fb Revert "Remove v1 API reference"
The v1 API is still in the tree, even if it is off by
default and deprecated.  We need to keep documentation on
how it works at least until we remove the code.  This was
published to [1] which was the primary way to find this
information, and it is now missing.

[1] https://developer.openstack.org/api-ref/block-storage/v1/

This marks v1 as deprecated in the toctree.

This reverts commit 8139f7f108.

Change-Id: Ic9d0fcce5ca0f8455c212292664b8f9694edfce3
2017-03-15 17:04:22 -04:00
Jenkins
d9278b5a60 Merge "Remove support for py34" 2017-02-17 10:29:04 +00:00
Danny Al-Gaaf
e3e749fa30 doc: verify all rst files
Make use of doc8 to verify all rst files which are not
autogenerated for errors and fail if there are any issues
found. The doc8 checks are now part of the tox 'docs'
environment and ran automatically. Checks can also be called
direcly via 'tox -e docs'.

Fix all issues found by doc8.

Closes-Bug: #1566765

Change-Id: I2b25247030a1aadcfe029c9e071ef17f2f72046b
Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>
2017-02-14 12:36:57 +01:00
yanjun.fu
69c24cd5f4 Remove support for py34
The gating on python 3.4 is restricted to <= Mitaka. This is due to
the change from Ubuntu Trusty to Xenial, where only python3.5 is
available. There is no need to continue to keep these settings.

Change-Id: Ief6f4b8b37113f08dd0ae767f3dd2b5b2177e3ce
2017-02-08 17:32:00 +08:00
xianming mao
dbf61cdb4a Enable DeprecationWarning in test environments
Many deprecations are triggered early (on imports, for example)
before the warnings are enabled by the WarningsFixture in the
base test class.
To make sure all DeprecationWarning messages are emitted we enable
them via the PYTHONWARNINGS environment variable.

Note: this copy from 1be35ff039d67c50866f2556ef847b048cbf3578

Change-Id: I7c6106149832d048d4d3bd2701c2c0db81bd87cb
2017-01-05 12:22:22 +00:00
Jeremy Liu
45bfe2ecd5 Fix warning when running tox -e docs
We get a warning "WARNING:test command found but not installed in testenv"
when building docs. This patch fix it.

Change-Id: Ib9798e59c55099d5a4bf8989401b63487b0c6338
2016-12-13 17:30:47 +08:00
Sean McGinnis
ca0786778c Switch default py3x to py35
Per our conversation in the Cinder weekly meeting [1] we will
now switch to just running local tests against python 3.5 as
that is the more recent and common 3.x version available on new
distros. Other versions can still be run explicitly, but this
changes the default tox run to only use 3.5.

[1] http://eavesdrop.openstack.org/meetings/cinder/2016/cinder.2016-11-30-16.00.log.html#l-54

Change-Id: If2226cb229f016e36c4dfab6224d5ae353b926bb
2016-12-09 14:22:28 -06:00
Michał Dulko
792108f771 Remove ovo[fixtures] dependency from tox.ini
Looks like we should now be able to add oslo.versionedobjects[fixtures]
to test-requirements.txt and remove it from tox.ini, so this commit does
that.

Change-Id: I420ea16d8ad6d883e7c2c8de80f912ea35500367
2016-09-28 18:31:49 +02:00
Jenkins
bd63288029 Merge "Let setup.py compile_catalog process all language files" 2016-09-08 05:01:36 +00:00
Jenkins
aeabde14f3 Merge "Use constraints for api-ref environment" 2016-08-30 23:14:39 +00:00
Jenkins
226dd7c8c1 Merge "Add functional-py35 to tox" 2016-08-28 19:23:47 +00:00
Andreas Jaeger
f589c6e2d1 Cleanup api-ref
Remove useage of old os-api-ref version and replace with current version.

Follow other repositories and merge v1 and v2 in a single document.

Note that we need to have a single document so that preview of the
document works properly in our CI.

Change-Id: I82c8b9ca317298b3dc1f7133ce8c0e9b3503730d
2016-08-27 17:13:04 +02:00
Sheel Rana
7f2131e3a1 Update api-ref path
api-ref path needs update to support uploading cinder api-refs to
developer.openstack.org/api-ref/block-storage

update includes
https://github.com/openstack/cinder/tree/master/api-ref/v2/source
to
https://github.com/openstack/cinder/tree/master/api-ref/source/v2

Depends-on: Id38dd0609c77b0d6bcd0d935a9312b0211134802

Implements: bp api-reference-to-rst

Change-Id: If3a058a95f1eaa8481a926183591ff0f4b23e97b
2016-08-27 16:58:10 +02:00
Sven Anderson
0adde01a06 Let setup.py compile_catalog process all language files
Two years ago the translation files have been split into several
files, separating the log messages of different log levels from each
other, like X.pot, X-log-warning.pot, X-log-info.pot, and so on.
However, the setup.py command `compile_catalogs`, that comes from the
babel package and compiles the corresponding .po files into .mo
files, only supported one file per python package.  This means that
during packaging `compile_catalogs` never compiled the X-log-*.po
files, so the corresponding translations were always missing.

Since babel 2.3 the domain can be set to a space separated list of
domains.  This change adds the the additional log level files to the
domain list.

The obsolete check that .po and .pot files are valid is removed from
tox.ini.

Change-Id: I149c2254cb04297e598cfd3ca73b24efd0c8ef18
Closes-Bug: #1536226
2016-08-24 14:56:04 -04:00
Tom Barron
7517be2f5c Use constraints for api-ref environment
Change [1] updated tox.ini to use constraints rather than
specific, non-constrained install commands but neglected
to update the api-ref environment.  In addition to fixing
that omission we also remove a bit of ugly trailing
whitespace.

Co-authored-By: Gorka Eguileor <geguileo@redhat.com>

[1] I879f34129c0d958be0a73bfdc7f641e178d4efe9

Change-Id: I58ec756cd3de2e501c77a4d947e680d0959d22ec
2016-08-23 13:14:59 -04:00
Tom Barron
66023c52f8 Use constraints for all tox environments
Infra is now ready to use constraints everywhere [1], so adapt
releasenotes, coverage, and venv to use the default install command
instead of a specific non-constrained install command.

With this change, all tox environments now use constraints.

[1] http://lists.openstack.org/pipermail/openstack-dev/2016-August/101474.html

Change-Id: I879f34129c0d958be0a73bfdc7f641e178d4efe9
2016-08-15 17:24:40 -04:00
Eric Harney
20d8ff5545 Add functional-py35 to tox
This allows running functional tests in a py35
environment with "tox -e functional-py35".

Change-Id: I0e2a8f2f43ccf06c1e676a52a3843c04056c050a
2016-08-11 10:15:08 -04:00
Jenkins
2197f273d1 Merge "Fix tox pip-missing-reqs" 2016-08-10 17:47:44 +00:00
Gorka Eguileor
5f0f752be8 Prevent doc generation failure on OVO decorators
Currently Sphinx will fail to generate the documentation if we are using
decorators from any OVO in cinder.objects, because the OVOs are only
added to the cinder.objects namespace when the CLI programs are run.

So we need to run `register_all` method during documentation generation
to avoid failures like:

    AttributeError: 'module' object has no attribute 'Volume'

This patch modifies autogenerated doc/source/conf.py file and since now
it is no longer completely autogenerated it is added to PEP8 checks,
which required some minor changes.

Change-Id: Ifeeef61a778f9e3f3daceba8ed05cd2036219499
2016-08-09 18:24:25 +02:00
Eric Harney
7c9139ccd4 Fix tox pip-missing-reqs
tox -e pip-missing-reqs was failing due to a
bug in pip_missing_reqs.  The fixed package
has also been renamed to pip_check_reqs as of
version 2.0.

Change-Id: If91b979fedac8de5c611a87b900a2bcdb80d58b3
2016-08-08 16:25:50 -04:00
Eric Harney
872a46d676 Add functional-py34 to tox
This allows running functional tests in a py3
environment with "tox -e functional-py34".

Change-Id: Ia2cadbcaaea1ea91913a0b6e53009d14c990648b
2016-08-03 12:41:00 -04:00
Sean McGinnis
8148038e92 Add driver list to doc build
We have tox -e gendriverlist that outputs an RST-ish report of all drivers
in the tree. This output can be used in the docs build to automatically
publish the list of drivers to make it easier to find officially supported
drivers.

This effectively removes the existing drivers.html that was generated prior
that did not actually contain any useful information.

Change-Id: I8de78723af76aabcc976733ac4b248db0b8ca16f
2016-07-26 11:09:05 -05:00
Tom Barron
24db697d1b update min tox version to 2.0
The commands used by constraints need at least tox 2.0.  Update to
reflect reality, which should help with local running of constraints
targets.

Note: this is a clone of mriedem's manila change [1].

[1] I27348462a21daab479d76b24b48e3a4e017504cc

Change-Id: I185c8f80285f2e7b4f3d3d45a2e6a7dd5741bc1d
2016-07-12 18:00:40 -04:00
Michał Dulko
348c26094b Delete *.pyc files before running tox tests
When jumping between different review branches .pyc files of newly
introduced .py files are getting left behind. When these are related to
DB migrations all tests start to fail with: "AssertionError: There is no
script for XY version". This is pretty annoying, so why not remove all
*.pyc files before running any tests? This commit implements that.

Change-Id: Iac223fb156af3729b51fc2b11323a7cc30f9c07d
2016-07-11 15:17:02 +02:00
Jenkins
bffc3030da Merge "Added coverage report to tox.ini" 2016-06-27 18:19:34 +00:00
Michał Dulko
c063cbdae9 Don't reuse pep8 env in compliance tox runs
gate-cinder-tox-compliance is broken because run-tox.sh script fails
with ".tox/compliance/bin/pip: No such file or directory". This is
because we're reusing pep8 venv (envdir tox directive), so
.tox/compliance directory never gets created.

To fix hat we can simply stop reusing that venv and let compliance tox
job create it's own venv. It can be done by removing envdir directive
from tox.ini.

Change-Id: Ib809872f5fb1581e1b098775c2ebdc4e724c70e2
Closes-Bug: 1596508
2016-06-27 13:49:03 +02:00
Ryan Selden
2f56ccfb5a Added coverage report to tox.ini
When `tox -e cover` is run, nice coverage reports will be
generated in the cover/ directory.

Change-Id: I8da177475ff50e1ca08c28371ac1f4352ea14736
2016-06-24 23:12:12 +00:00
Sean McGinnis
e7b40242f8 Add driver interface checks
This is the start of an effort to both validate that drivers fully
implement the expected minimum requirements as well as to create a clear
place for driver developers to learn what needs to be implemented and get
documentation explaining what is expected for each method.

This also enables us to create tooling for documenting the available
drivers and their capabilities, to some degree. A follow up patch will
show some of what I'm thinking there, but it will make it possible to write
scripts for different needs.

This is somewhat a cleanup attempt to the ABC work that was started a
while back. This does not aim to replace that effort, but give a
mechanism for some of the things expected out of that effort that ended
up not being possible with how it evolved.

In most cases we do not really care if a driver is inherited from a
certain base class, just that it conforms to the given interface.

The interface/inheritance work really centers around two separate
things:

 * Ensuring drivers conform to an expected interface
 * Allowing code reuse and common implementation

This is really for the first item. Additional work is needed to complete
the ABC work we've done, but that really focuses on the second item, and
is out of scope for the intent of this patch.

Change-Id: I4168225126fe88c31712d94f0a130e9e7ede3446
2016-06-13 15:21:47 +00:00
Jenkins
b230cb743c Merge "Add sample config file to cinder docs" 2016-06-09 23:00:39 +00:00
Sheel Rana
41929d1c7a migrate to os-api-ref
os-api-ref is released now, so we can move to using it instead
of copy pasting in cinder tree.

Change-Id: Id38dd0609c77b0d6bcd0d935a9312b0211134802
2016-06-09 10:35:02 +00:00
Sean McGinnis
fcd7fcea3e Add sample config file to cinder docs
The oslo sphinxconfiggen module was added to the oslo.config
2.3.0 release. This enables config file generation as part of
the sphinx doc generation.

The generated config file will pick up the current config
options from the code base. And as an added bonus, it will
now be published to the docs.openstack.org site for easy
reference or download.

This also puts us inline with what other projects like Nova
are doing for sample config files and is the recommended
method from the Oslo team.

Change-Id: I912a97eb2686d3dc56e50d8641d7bd930179bc18
2016-05-17 08:52:21 -05:00
Daniel Gonzalez
c2f2eeb394 Whitelist 'rm' in api-ref tox environment
Running the api-ref environment currently warns that 'rm' is not
installed in testenv. This patch removes the warning by whitelisting
the rm command.

Change-Id: I4072491713a38b11d9f55b26a8ba0a3cf49445f0
Closes-Bug: #1581778
2016-05-14 11:36:19 +02:00
Sheel Rana
8659ff0ef5 WADL to RST migration in cinder tree
This patch is for converting API Reference to RST and host it
in the Cinder tree.
This patch contains all the RST for cinder to bring over to their
repos to begin building API reference information from within.
This contains .inc files which have all the contents of the .rst files
but are grouped together for easier editing

This is the results of the RST conversion from WADL. It creates a
single index plus a bunch of included files which represent sections
of the API.

Cleaning task will be done once this part is merged.

As we have removed XML API support in Newton release, we need to
remove XML from API reference as well.

But we still have installations which are using XML.
So implementation plan is to first merge this patch in Newton release
keeping XML for now, then I will backport same to stable branches.
After that I will remove same from Newton release to match API
reference with actual API implementation.

Implements: bp api-reference-to-rst
Change-Id: I865ac922538bfa5bd45c24eb4bc49f5e966dc811
2016-05-12 19:07:30 +00:00
Jenkins
01f5f04689 Merge "Run py34 tests with plain 'tox' command" 2016-04-04 23:47:48 +00:00
Christopher J Schaefer
4e8ef51bc7 Remove bandit.yaml in favor of defaults
The most recent version of bandit no longer requires a configuration
file (bandit.yaml) to run the default test suite. Removing the
configuration file and updating the bandit arguments will run all of
the currently available bandit tests in the default test suite.

Change-Id: If144c4017ce9bb58c3e83b62a5e31325353200b5
2016-03-18 11:08:52 -05:00
Tom Barron
1787244db4 Run py34 tests with plain 'tox' command
Now that all cinder unit tests pass on python 3.4 [1], we can run py34
tests by default alongside py27 and pep8.

This commit also addresses the annoyance of py34 tox tests failing with
'db type could not be determined' if py27 tests were initially run in
the workspace.

[1] https://haypo.github.io/openstack_mitaka_python3.html

Change-Id: If60e5d0d3185e78f38fa2bfc7b6bb4840f09d840
Closes-bug: #1489059
2016-03-09 03:07:26 +00:00
Jenkins
077b8593ae Merge "always use pip constraints" 2016-02-29 18:45:29 +00:00
Jenkins
5f2b909ad3 Merge "Add pip-missing-reqs tox env" 2016-02-29 12:59:31 +00:00
Andreas Jaeger
379f272924 always use pip constraints
This flips the tox.ini to always use constraints installation for
all targets. It drops the extra -constraints targets in the process.
This makes it so there is no developer change required to operate in our
new world order.

Change-Id: I2b56835edb2d75b7c871aae030e909578dbb958d
Depends-On: Iecd35d6bc3b1e9e0ed31b48a34e4149d34d55ac2
2016-02-29 13:00:11 +01:00
Victor Stinner
a09143cb31 Enable all unit tests on Python 3.4
* Remove tests-py3.txt: all unit tests now pass on Python 3
* tox.ini: remove specific command for py34 test environment, it now
  inherits testenv commands (same command than Python 2.7)

Partial-Implements: blueprint cinder-python3
Change-Id: I52c68a63740b3ae6097fad419b59a9c29422b525
2016-02-26 14:35:34 +00:00
Ivan Kolodyazhny
7f45046017 Revert changes to use ostestr
ostestr has more user-friendly output including lists of all and failed
tests.

Change-Id: I1f715a5688d702aaf8489928a1b0a39093caa5da
2016-02-17 16:04:40 +02:00
Ivan Kolodyazhny
6bdc836965 Move integrated tests to 'functional' directory
Functional unit tests should not be a part of unit
tests job. This patch adds new tox environment to run
functional tests:
    $ tox -e functional

It also removes logging from functional tests. Some of log calls were
replaced with asserts.

Related Implements: blueprint cinder-integrated-tests

Change-Id: I0ebfef2fe05f502cd5fb08fbc8af09008949e457
2016-02-09 23:28:27 +02:00
Gorka Eguileor
e374a952af Run flake8 also on cinder/common
Even though testenvs pep8 and pep8-constraints included cinder/common
directory to be processed by flake8, global configuration in flake8
secion was excluding common glob preventing the directory from being
checked.

This patch removes glob from flake8 exclude section, unnecessary
cinder/common inclusion when running flake8, and fixes
cinder/common/config.py.

If we had a common directory in our code root and wanted to exclude it,
glob should be ./common instead of just common like we had now.

Change-Id: Iefc64c02335ed9296bda323fdcb5688125fb4e77
2016-01-21 14:17:34 +01:00
Eric Harney
f2a3dc61d6 Add pip-missing-reqs tox env
This allows discovery of dependencies that are
not listed in [test-]requirements.txt.  Taken from
oslo.db.

Change-Id: I66000bb61e433f8d75ed15d600f2e87ef45140b8
2016-01-19 12:01:24 -05:00
David Sariel
01ad4ccebd Re-enabled hacking checks for H105
Following the upgrade to hacking version 0.10.0 the H105 issues were
ignored. There was a note left in tox.ini to clean up the H105 issues
and to re-enable the check.

Updated tox.ini to re-enable the check of H105 and fixed corresponding
pep8 issues. This way Cinder aligns rows with Nova, Neutron, Glance etc
that have H105 enabled.

Closes-Bug: 1531104
Change-Id: I2da70cbeb838f16ec8479284dda9d7c62566dace
2016-01-05 11:22:40 +02:00
Ondřej Nový
fe990676a9 Deprecated tox -downloadcache option removed
Caching is enabled by default from pip version 6.0

More info:
https://testrun.org/tox/latest/config.html#confval-downloadcache=path
https://pip.pypa.io/en/stable/reference/pip_install/#caching

Change-Id: I5376fa4891a1a1f469af36d850b4045d24590aec
2015-12-11 21:10:53 +01:00
Manjeet Singh Bhatia
2c38caefb5 Use wild card for passing env variable
Change-Id: Ia8908941d5619d8c79fa7373c63ff37725fcc4f5
2015-12-10 15:43:59 +00:00
Michał Dulko
88ebf23944 Force releasenotes warnings to be treated as errors
Change-Id: I03a8071114d8d4042ad143b55860e5562d86b34d
2015-11-26 10:15:32 +01:00
Eric Harney
7bb16f7501 Don't build two tox envs for pep8(-constraints)
Currently, "tox -e pep8" builds two tox envs:
a "pep8" env, and then a "venv" env for genopts.

This is rather wasteful -- reuse the pep8 env
instead.

Change-Id: I8ba3d4e541f279e5bc62dba2b1caaac447fee7e3
2015-11-23 14:06:24 +00:00
Eric Harney
f1f8f5e98e Add check_uptodate.sh --checkopts to "pep8"
2f9e416 Downstream Fix for Genconfig
  collided with
fb17ed8 Add -constraints for CI jobs

while merging and this only got added to
pep8-constraints.  It should be in "pep8" as well.

opts.py is updated here since it is out of sync,
due to this test not running.

Change-Id: I668df6be4301a052a4b47549c702a81b43abd8fc
2015-11-20 09:23:20 -05:00
Kendall Nelson
2f9e4163f4 Downstream Fix for Genconfig
This patch adds the opts.py file to the tree as a fix for deployers
that package cinder. The opts.py file is no longer being deleted right
away by generate_sample.sh after the cinder.sample.conf is being
generated.

This patch also introduces a pep8 check to make sure that the opts.py
file is up to date, so that it will catch when new opts get added to
Cinder without the opts.py being updated.

To support the ability to keep and check the opts file a number
of changes were needed in the check_uptodate.sh script as well as
the generate_sample.sh script:
    - check_uptodate now takes --checkopts instead of --checkonly
      When checkopts is used the opts.py file is generated using the
      current code and the generated file is compared to the existing
      file.  The check fails if there are differences.
    - generate_sample now has the --nosamplefile option.
      When this option is used, only the opts.py file is generated.
      The oslo-config-generator code is skipped so no sample file
      is created.
    - generate_sample also has some coding style consistency changes.
    - Added the 'genopts' option to tox so users can generate
      a fresh opts.py without a sample file when necessary.

Closes-Bug: 1501820
Co-Author: Jay Bryant <jsbryant@us.ibm.com>

Change-Id: I1f5494ebb19d5f4e8c651cbeef0acad07ad96829
2015-11-13 12:36:21 -06:00
Matt Riedemann
17cc592c68 Test for object version hash changes
Use the ObjectVersionChecker fixture from oslo.versionedobjects to check
when Cinder's registered objects have a change (new remotable method,
new field, etc) that require a version bump.

Fixes some registration problems along the way:

1. CinderObject doesn't need to be registered.
2. TestObject should be registered where it's used, not globally.

Removes deps=requirements.txt from tox.ini since it's not necessary and
prevents us from pip installing oslo.versionedobjects[fixtures].

Closes-Bug: #1514926

Change-Id: Id57c56a75ea11411e9e54104165bd44b577c1485
2015-11-11 13:09:22 -08:00
Michał Dulko
7da1684f3c Add reno for release notes management
Change-Id: Id4d6604d9775c34e9a8e911ec6e7afd02d56ef47
2015-11-05 14:58:17 +01:00
Sean McGinnis
f17c0bbd75 Update minimum tox version to 1.8
Other projects have updated to 1.8 or later for the minimum
tox version (nova, neutron), so we should probably do the
same.

Change-Id: Iddb9ab286519d555671ec39c0ce75c9351025730
2015-11-03 16:47:01 -06:00
Sachi King
fb17ed8653 Add -constraints for CI jobs
This adds the constraints factor to the base section and duplicates
the sections required for CI with -constraints as tox does not
currently support factors in sections.  Work towards enabling factor
support in sections is currently stalled, as such we will need to
duplicate sections adding -constraints to facilitate running sections
with the constrained install_command.

Implements Blueprint: Requirements-Management

Change-Id: I7afd52517750b82dcac94b8aae68dc0e94d00e54
2015-10-21 10:13:58 +11:00
Eric Harney
928926be80 Tox fast8: use pep8 env dir
This reuses the .tox/pep8 environment rather than
building a separate .tox/fast8 environment for
fast8.

This reduces duplicated install time and disk space.

Change-Id: I230465b28ba3d0f2c3a2ba38011ff9f7d472d719
2015-10-12 10:59:31 -04:00
Victor Stinner
a37618f55b py3: Run unit tests with ostestr on Python 3.4
"tox -e py34" now uses ostestr instead of testtools.run to run tests.
Tests are now executed in parallel.

Move the tests whitelist from tox.ini to a new file tests-py3.txt.

Fix NaElement.__str__() of netapp driver: return Unicode on Python 3,
not bytes.

Partial-Implements: blueprint cinder-python3
Change-Id: I4cda85fbe76ac853beaec3a9587726166daa063d
2015-10-07 15:35:27 +02:00
Jenkins
33666c6964 Merge "Port scheduler host manager to Python 3" 2015-10-07 06:50:20 +00:00
Jenkins
b85cbda2c4 Merge "Port test_quobyte to Python 3" 2015-10-06 21:55:34 +00:00
Jenkins
1f314662a8 Merge "Port test_netapp to Python 3" 2015-10-06 07:14:33 +00:00
Victor Stinner
76bcf57418 Port test_quobyte to Python 3
Modify assertRaisesAndMessageMatches() to tolerate subclasses. On
Python 3, a permission error now raises an exception PermissionError
which is a subclass of OSError, whereas the test expected exactly the
OSError type.

Partial-Implements: blueprint cinder-python3
Change-Id: Ia232c27de6a67eafad30b25301531e292fe349dc
2015-10-03 02:07:46 +02:00
Victor Stinner
f5f2b2ef78 Port netapp SSC Cmode to Python 3
Fix query_cluster_vols_for_ssc(): cast string (records) to int. On
Python 3, comparison between str and int now raise a TypeError.

tox.ini: add cinder.tests.unit.test_netapp_ssc to Python 3.4.

Partial-Implements: blueprint cinder-python3
Change-Id: I05127340029a5bb6c8444f51341be8008aab8dc8
2015-10-02 17:48:28 +02:00
Victor Stinner
28e9a94032 Port test_netapp to Python 3
* Add close() method to FakeHttplibSocket,
  FakeDirectCmodeHTTPConnection, FakeDirect7modeHTTPConnection
* Replace StringIO() with BytesIO() for socket buffers
* makefile(): ignore optional arguments after mode
* Fix bytes versus Unicode: use bytes for HTTP body. Encode HTTP body
  to UTF-8 and HTTP headers to Latin1.
* tox.ini: add cinder.tests.unit.test_netapp to Python 3.4

Partial-Implements: blueprint cinder-python3
Change-Id: I3dc8c36a84ffbb47be3e85eed34518c6adad48ec
2015-10-02 17:47:39 +02:00
Victor Stinner
3a7f1ffb95 Port scheduler host manager to Python 3
* Rewrite ReadOnlyDict using collections.Mapping: collections.Mapping
  has no method to modify the dictionary and is available on Python 2
  and Python 3. Add also a __repr__() method to ease debug. Remove
  ReadOnlyDict.update() method: a read-only dictionary must not be
  modifiable.
* test_host_manager.py: sort dictionaries using a key function to
  have a reliable order for the list of dictionaries. On Python 3,
  the hash function is now randomized.
* tox.ini. add cinder/tests/unit/scheduler/ tests to Python 3.4

Blueprint cinder-python3
Change-Id: I34d98d6a75fef907251719f0d46b025d8e8b2b65
2015-10-02 10:28:36 +02:00
Victor Stinner
2e20e70e14 Fix volume throttling to Python 3
BlkioCgroup._set_limits(): sort devices before iterating on them to
have a reliable behaviour.

The devs variable is a dictionary. On Python 3, the hash function is
now randomized, so iterating on a dictionary gives items in a random
order. Use sorted() to iterate on the list of sorted devices instead.

tox.ini: add cinder.tests.unit.test_volume_throttling to Python 3.

Blueprint cinder-python3
Partial-Bug: #1348818
Change-Id: Icf7141f772397c7ac08f0f1e21ad74cb86a06351
2015-10-02 07:01:03 +00:00
Jenkins
ae370d9ac9 Merge "Port test_volume to Python 3" 2015-10-01 02:56:29 +00:00
Victor Stinner
068db12d09 Port test_volume to Python 3
* don't use hasttr() to check if a lazy SQLAlchemy is loaded or not: use
  the obj_attr_is_set() method instead. On Python 3, hasattr() pass
  through SQLAlchemy exceptions which is unexpected.
* Replace a/b with a//b to use integer division, not float division.
* Replace filter() with list-comprehension using an if.
* Replace file() with open() and open /dev/null in binary mode (not text
  mode, so Unicode on Python 3).
* test_volume require "import cinder.volume.targets.tgt" on Python 3
* Use a key function to sort a list of dictionaries. Dictionaries are no
  more comparable on Python 3.
* tox.ini: add the following tests to Python 3.4.

  - cinder.tests.unit.keymgr.test_mock_key_mgr
  - cinder.tests.unit.test_volume

Note: test_list_availability_zones_enabled_service() of test_volume
was broken, the test checked that the result of .sort() is None...

Blueprint cinder-python3
Change-Id: If1a26acc0138db9bda7fde1cb1f40093d9b3c494
2015-09-29 14:22:40 +02:00
Jenkins
3d50faaad5 Merge "Add "fast8" tox env" 2015-09-29 10:15:51 +00:00
Victor Stinner
3a1c871927 py3: Port pure driver test to Python 3
* Replace func.func_name with func.__name__: the func_name attribute
  was removed in Python 3, whereas the __name__ attribute exists on
  Python 2 and Python 3
* Create INITIATOR_TARGET_MAP using list(set()) to get the same order
  than the tested code. The exact order is not reliable, it depends
  on the hash function which is now randomized by default on
  Python 3. The hash function is also different between Python 2.7
  and 3.4.
* tox.ini: add test_pure to Python 3.4

Blueprint cinder-python3
Change-Id: I78601278259f1d34ad6ac3458d2dd0a3aca9d77a
2015-09-28 17:56:37 +02:00
Jenkins
a61a2dbcf6 Merge "Port image_utils to Python 3" 2015-09-25 17:02:30 +00:00
Jenkins
89f8e9887c Merge "Port volume transfer to Python 3" 2015-09-25 16:38:33 +00:00
Eric Harney
7e5c074351 Add "fast8" tox env
This replicates the run_tests.sh -8 behavior, running
flake8 only on changes made in the last commit and
working tree.

Change-Id: Iebd5746c78e840225f0860843c77f4c638877c26
2015-09-22 15:43:51 -04:00
Kendall Nelson
111a056c0f Dynamically create cinder.conf.sample
As it stands, the opts.py file that is passed into
oslo-config-generator isn't being generated dynamically
and the old way of generating the cinder.conf.sample is
dependent on oslo-incubator which Cinder is trying to
move away from. oslo-config-generator works differently
than oslo-incubator so a number of changes had to be made
in order to make this switch.

This patch adds the config directory to Cinder and in it
are two files:

    -generate_cinder_opts.py that will take the
     results of a grep command to create the opts.py
     file to be passed into oslo-config-generator.

    -cinder.conf which is the new configuration for
     oslo-config-generator. The file is inside the config
     directory to be consistent with other projects.

Some changes were made to the generate_sample.sh file in
order to give the base directories and target directories
to the generate_cinder_opts.py program.

tox.ini was edited to remove the checkonly option because
all that needs to happen in check_uptodate.sh is a check to
ensure that the cinder.conf.sample is actually being
generated with no issues.

All options were removed from the check_uptodate.sh
because they were unnecessary given the new, more simple
way of generating the cinder.conf.sample.

setup.cfg was also edited in order to add information
oslo-config-generator needs to run.

Co-Authored By: Jay Bryant <jsbryant@us.ibm.com>
Co-Authored By: Jacob Gregor <jgregor@us.ibm.com>

Change-Id: I643dbe5675ae9280e204f691781e617266f570d5
Closes-Bug: 1473768
Closes-Bug: 1437904
Closes-Bug: 1381563
2015-09-18 17:27:27 +00:00
Victor Stinner
d95bd7c376 Port image_utils to Python 3
Port cinder.image_utils.check_qemu_img_version() to Python 3.

Add the following tests to Python 3.4 in tox.ini:

* cinder.tests.unit.test_image_utils
* cinder.tests.unit.test_migrations
* cinder.tests.unit.test_misc
* cinder.tests.unit.test_service

Partially implements: blueprint cinder-python3
Change-Id: I69feda94c90206adc8771fbd5ddc6aae5bb6b64b
2015-09-01 13:20:40 +02:00
Victor Stinner
d548495e9e Port volume transfer to Python 3
Encode Unicode to UTF-8 for salt and authentication key when computing
the crypt hash.

Partially implements: blueprint cinder-python3
Change-Id: I19d337cf0d40d91378e1c42061bc51b6009970a2
2015-09-01 13:11:13 +02:00
Eric Harney
b985bae7c4 Detect addition of executable files
When running pep8 checks, fail if executable
files have accidentally been added.

Change-Id: Ia0efa629acf794f8e0e9b78de24b5385776c1e66
2015-08-31 12:19:51 -04:00
Jenkins
3b40049cc8 Merge "Port test_nfs to Python 3" 2015-08-27 16:14:20 +00:00
Jenkins
a8d98325e3 Merge "Test whether sample config generation works" 2015-08-19 19:59:16 +00:00
Victor Stinner
125fec9495 Port test_nfs to Python 3
testtools.ExpectedException(exc_class, regex) expects the exact
exc_class class, wheras the
test_setup_should_throw_exception_if_mount_nfs_command_fails() test
raises a PermissionError on Python 3.3+ instead of OSError. Replace
testtools.ExpectedException() with self.assertRaisesRegex() to port the
test to Python 3, this method accepts subclasses of exc_class.

Replace also testtools.ExpectedException() with self.assertRaisesRegex()
in other tests.

Blueprint https://blueprints.launchpad.net/cinder/+spec/cinder-python3
Change-Id: If5a74529e5ac68a2118afa8ecfd86bf24307b0b7
2015-08-19 11:29:08 -07:00
Eric Harney
683dd65d41 Test whether sample config generation works
Adds --checkonly to tools/config/check_uptodate.sh.

This allows verifying that a configuration file was
generated.

We have had a number of issues introduced which
have caused config file generation to fail since we
removed the sample config file check in the gate.
This does not add back checks to ensure that the file
has been updated, it only checks to ensure that it is
still possible to update the sample file.

Change-Id: I2e0e376fbdec5a1bb584c6739231649e1d0f5d8e
2015-08-18 17:31:01 -04:00
Eric Harney
6da01c1b41 Revert "mark oslo.vmware as optional dependency"
This reverts commit e67adc385c.

Adding optional dependency tracking would be nice,
but Cinder is not yet in a place to do this.

- Cinder supports tox 1.6 (minversion in tox.ini)
- This functionality does not work on 1.6, requires 1.7.  Tox
  fails to build a test environment if using tox 1.6.
- We have decided we can't move to tox 1.7 for now.
  ( https://review.openstack.org/#/c/211614/ )

This reverts commit 478e8e970e.
- Need to remove oslo.vmware from test-requirements.txt now, as
well.

Closes-Bug: #1484035

Change-Id: Icff52508ff1cd67e8c17964c840d3a5cdcfbbce3
2015-08-18 16:28:36 -04:00
Victor Stinner
4055078083 Fix Python 3 issues in Windows tests
* Replace __builtin__ with six.moves.builtins
* tox.ini: add the following tests to Python 3.4

  - cinder.tests.unit.windows.test_smbfs
  - cinder.tests.unit.windows.test_vhdutils
  - cinder.tests.unit.windows.test_windows
  - cinder.tests.unit.windows.test_windows_remotefs
  - cinder.tests.unit.windows.test_windows_utils

This change requires the Python 3 fixes in os-brick.

Blueprint cinder-python3
Change-Id: I1ffa65745923459e993e81d8a95a39e19c1bd1e4
2015-08-13 08:43:40 -05:00
Matthew Edmonds
e67adc385c mark oslo.vmware as optional dependency
Driver-specific requirements are not hard requirements, since the
choice of drivers is up to the operator. The oslo.vmware module is
herein moved out of requirements.txt and instead listed as an optional
dependency using the extras functionality in setup.cfg.

A check is added to the vmdk driver to gracefully handle import errors
if the vmdk driver is used and oslo.vmware is not found.

Change-Id: I9b00edc38f0700304a1a164f0679a734f8701ebe
Closes-Bug: #1475739
2015-08-10 15:20:52 +00:00
Jenkins
1da350a3f1 Merge "Fix Python 3 issues in Hitachi HNAS tests" 2015-08-10 01:04:00 +00:00
Jenkins
dad3cf7a77 Merge "Port remotefs driver to Python 3" 2015-08-10 01:03:49 +00:00
Jenkins
cf3859bb87 Merge "Port IBM driver to Python 3" 2015-08-10 01:03:28 +00:00
Jenkins
690f1b24eb Merge "Add entry create and cast tasks to manage workflow" 2015-08-07 00:05:31 +00:00
Jenkins
b864ac106e Merge "Add bandit for security static analysis testing" 2015-08-06 01:19:39 +00:00
Cindy Pallares
cafb5d449f Fix multi-line docstrings to meet hacking rules
According to the PEP8(H405), multi-line docstring summaries
should be separated by an empty line.

Change-Id: I5cd8a9064dcefc504e85946ecdf1f56f10145d35
Closes-bug: #1407162
2015-07-30 11:34:16 -05:00
Anton Arefiev
a1e4ad9ff2 Add entry create and cast tasks to manage workflow
This change adds manage existing api task in flow. The task
is used in the volume api to provide full value task flow for
manage existing process. All errors occurred during manage
flow set volume to 'error' state.

Entry creating moved from volume api to EntryCreateTask. Also
added ManageCastTask to provide manage process to scheduler.

Related-Bug: #1364550
Change-Id: I12a4311953c1c86d584b5bf2fe2888e5b5127d43
2015-07-27 19:20:22 +03:00
Eric Brown
6cddec7db9 Add bandit for security static analysis testing
This change adds a basic bandit config for Cinder. It can be invoked
by running the tox environment for bandit;
    tox -e bandit

This is intended as a starting point for using bandit with Cinder
and it should be revisited to improve the testing as more is learned
about the specific needs of the Cinder code base.

Tox is configured to only show results for high and medium severity
results.

https://wiki.openstack.org/wiki/Security/Projects/Bandit

Change-Id: I0247e0ccaed6faacacb2b8d2f8b141a8edc704af
2015-07-25 21:37:31 -07:00
Victor Stinner
d24d075242 Fix Python 3 issues in Hitachi HNAS tests
* Replace __builtin__ with six.moves.builtins.
* Replace tempfile.mkstemp() with tempfile.NamedTemporaryFile() to get a
  text file instead of a binary file.
* tox.ini: add the following tests to Python 3.4

  - cinder.tests.unit.test_hitachi_hnas_backend
  - cinder.tests.unit.test_hitachi_hnas_iscsi
  - cinder.tests.unit.test_hitachi_hnas_nfs

Blueprint cinder-python3
Change-Id: I9a26b4e67033a443271e8f13bcaea5e122ec865a
2015-07-25 03:10:31 +02:00
Victor Stinner
ba1ae4ada0 Port remotefs driver to Python 3
* When parsing share configuration: replace "\040" with a space
  explicitly instead of using the Python unicode_escape encoding:
  see the bug #1180984.
* Encode Unicode to UTF-8 before hash it with MD5.
* tox.ini: add cinder.tests.unit.test_glusterfs to Python 3.4

Blueprint cinder-python3
Change-Id: Iea943a1f2dfb050fe43ff41411e258adb3f3b6d0
2015-07-25 03:10:07 +02:00
Victor Stinner
3c93229c0d Port IBM driver to Python 3
* Replace map(None, it1, it2) with zip(it1, it2)
* tox.ini: add the following tests to Python 3.4

  - cinder.tests.unit.test_ibm_flashsystem_iscsi
  - cinder.tests.unit.test_ibmnas

This patch requires Python 3 fixes in os-brick.

Blueprint cinder-python3
Change-Id: I735c39e08cf6e4edd06cf07243fe798b65e8c1a5
2015-07-25 03:01:39 +02:00
Jenkins
57c6d7ff6f Merge "Removing OpenvStorage for no CI" 2015-07-24 03:08:23 +00:00
Anton Arefiev
a7f264f3af Add drivers list generator
It is useful to have a maintained (in source control) list
of drivers that exist in Cinder. It could be used in docs
and unit tests to check method impl on backend drivers.

This change add tool for generate list of drivers based on
existing BaseVD class in class hierarchy of volume drivers.

Output example:
Drivers: ['cinder.volume.drivers.lvm.LVMVolumeDriver',
          'cinder.volume.drivers.rbd.RBDDriver',
         ...]

Implements: blueprint drivers-list-generator
Change-Id: I0e10906873e659e09a6e34531a0c932495d7c399
2015-07-21 18:04:29 +03:00
chenzongliang
c9bb99b52f Refactor Huawei Volume driver
This patch attempts to refactor Huawei volume driver in liberty.
We add a base driver to implement the basic functions.
The sub-class will inherit from the base driver according to different
storages.

The following changes were made in this refactor:
1. Abstract a base class named HuaweiBaseDriver to make Huawei driver more
universal. You can find it in the huawei_driver.py.
2. Put all static variables into the constants.py.
3. Rename rest_common.py to rest_client.py. rest_client.py stores the
relevant methods implemented for Huawei driver.
4. Migrate some public methods from rest_client.py to huawei_utils.py,
such as parse_xml_file(), _get_volume_type() and so on.
5. This refactor only involves structural adjustment and does not involve
functional changes.

Change-Id: I768889e2577a4d975397218eb31e89b42e08e04f
Implements: blueprint refactor-huawei-volume-driver
2015-07-18 16:49:05 +08:00
Mike Perez
f0ab819732 Removing OpenvStorage for no CI
It has been over a month since the CI maintainer was communicated via
third party mailing list, and the CI is still not reporting.

Change-Id: I25e125d76beb27da10a8ac617c70db357fcef57c
UpgradeImpact: OpenvStorage driver removed
2015-07-14 15:22:32 -07:00
Jenkins
88f65f0372 Merge "Port huawei driver to Python 3" 2015-07-08 02:06:53 +00:00