Port the legacy legacy-tempest-dsvm-multibackend-matrix job to the
native Zuul v3 syntax, and rename it following the guidelines
(cinder-multibackend-matrix-migration).
This job tests the migration between two different backends
specified through the volume.backend_names configuration
key in tempest.conf.
Now the job leverages the existing zuul code, namely the
run-tempest role, which is called multiple times with all
the possible combinations of the 3 tested backends
(lvm, ceph, nfs) where the source and the destination differ.
The final JUnitXML output summarizes the test results
for each of the tested combinations.
Change-Id: I34e7e48ee63c4c269f82ae178a7118ed402cad6d
The cinder-mypy job is failing because library stubs aren't installed
for requests [0]. Modify the mypywrap.sh to accept options specified
in an environment variable named OS_MYPY_OPTS to the mypy invocation,
and set this var in tox.ini.
The value is "--install-types --non-interactive" which is suitable for
CI purposes, and seems to make sense for local tox use as well [1].
The downside is it basically runs mypy twice, once to determine
whether there are any library stubs missing and then install them, and
againto do the actual check. If we don't want this setting in
tox.ini, we can move it to .zuul.yaml for the cinder-mypy job run.
Also, update the version of mypy in test-requirements to a version
that supports the above options.
And, run mypy in its own env (instead of reusing pep8) so that the
tox logs are preserved during CI runs.
[0] https://zuul.opendev.org/t/openstack/build/b66ee6c21e594940941585b0e9e5082a
[1] https://mypy.readthedocs.io/en/stable/running_mypy.html#library-stubs-not-installed
Change-Id: Id69cb519ee7300b33ff087de4e7d46cdad67d162
Tox issues a warning because bash is not
in whitelist_externals and indicates that this
will fail in tox 4+.
Just call mypywrap.sh directly to prevent this.
Change-Id: I3c1747b255338d189806b331ae6a3c1b37e6d744
[[ is a bash extension and therefore not
available on platforms where sh is not bash.
Use bash explicitly.
Change-Id: I3f31acd5fc3d7422270be53510962dcbc08db602
Add a "mypy" tox environment which runs mypy
type checking against Cinder code.
Taken from Stephen Finucane's Nova work at
https://review.opendev.org/#/c/676208/
Added "show_error_codes" and "pretty" options.
Generates an html report in ./mypy-report/
This adds stubs for oslo.i18n, so that _() calls
are annotated as intended. It may be possible to
do this with less .pyi files carried along here.
Change-Id: I2589d22c1f16f2e177d34730a520591743c0c1e3
Starting from mysql version 8 it's not possible to create a user
implictly when using GRANT.
This patch makes the behavior compatible with that.
Change-Id: Ia13402213bc82027e3d658918a22003be26576d6
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
There's some remainings of Python 2 in a few .py that this patch
intends to fix. This patch has been in Debian for the last 2
years, it's probably about time to get this in upstream Cinder too.
Change-Id: I9c08c3ff1177c323a1c8cfd234ad58bd607e3c1a
This adds usage of the flake8-import-order extension to our flake8
checks to enforce consistency on our import ordering to follow the
overall OpenStack code guidelines.
Since we have now dropped Python 2, this also cleans up a few cases for
things that were third party libs but became part of the standard
library such as mock, which is now a standard part of unittest.
Some questions, in order of importance:
Q: Are you insane?
A: Potentially.
Q: Why should we touch all of these files?
A: This adds consistency to our imports. The extension makes sure that
all imports follow our published guidelines of having imports ordered
by standard lib, third party, and local. This will be a one time
churn, then we can ensure consistency over time.
Q: Why bother. this doesn't really matter?
A: I agree - but...
We have the issue that we have less people actively involved and less
time to perform thorough code reviews. This will make it objective and
automated to catch these kinds of issues.
But part of this, even though it maybe seems a little annoying, is for
making it easier for contributors. Right now, we may or may not notice
if something is following the guidelines or not. And we may or may not
comment in a review to ask for a contributor to make adjustments to
follow the guidelines.
But then further along into the review process, someone decides to be
thorough, and after the contributor feels like they've had to deal with
other change requests and things are in really good shape, they get a -1
on something mostly meaningless as far as the functionality of their
code. It can be a frustrating and disheartening thing.
I believe this actually helps avoid that by making it an objective thing
that they find out right away up front - either the code is following
the guidelines and everything is happy, or it's not and running local
jobs or the pep8 CI job will let them know right away and they can fix
it. No guessing on whether or not someone is going to take a stand on
following the guidelines or not.
This will also make it easier on the code reviewers. The more we can
automate, the more time we can spend in code reviews making sure the
logic of the change is correct and less time looking at trivial coding
and style things.
Q: Should we use our hacking extensions for this?
A: Hacking has had to keep back linter requirements for a long time now.
Current versions of the linters actually don't work with the way
we've been hooking into them for our hacking checks. We will likely
need to do away with those at some point so we can move on to the
current linter releases. This will help ensure we have something in
place when that time comes to make sure some checks are automated.
Q: Didn't you spend more time on this than the benefit we'll get from
it?
A: Yeah, probably.
Change-Id: Ic13ba238a4a45c6219f4de131cfe0366219d722f
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
This allows "tox -e pep8" to succeed on
CentOS 8.
This is only called from the pep8 env which
already specifies a basepython of python3, so
it should be ok.
Change-Id: Id11e137bc8e1fb50e551639c2be067647c5bc9d4
Adds a doc about configuring and troubleshooting a service user
to send service tokens.
Also adds the config necessary for successful use of a service token
to the sample configuration file. Only adds keystone v3 options
because v3 is required for service tokens. Lists the service user
options before the session conf opts because the former are required
while the latter are less likely to be changed.
Closes-bug: #1837449
Change-Id: I4223624d00c713ab6b129ed278147f1c89995567
On el7 w/git 1.8.3, "tox -e pylint" just returns:
fatal: ambiguous argument '*.py': unknown revision
or path not in the working tree.
Before this change.
Change-Id: I63bf975f85dda88ab5e76d8df46ba9a176bb2a3b
"-j 0" runs on the number of CPUs available, just
pass that instead of counting CPUs ourselves.
(This removes a stealthy python2 dependency for us,
too.)
Change-Id: Ib7d90efdbc692fa8e1f42615edec58ba3cef926b
- colorizer.py: stestr (and ostestr) can be used instead,
as they include its code;
- enable-pre-commit-hook.sh: no references in the documentation,
it looks like a relic;
- with_venv.sh: its only usage was removed when run_tests.sh
was removed, and it's being phased out from other repositories
as well.
Change-Id: I566f893ae754687555fb3c4cb581e1b8a4eed95d
The upgrade checker is going to need to be able to load
configuration settings to check for things that will
interfere with an upgrade. This will cause genopts
to incorrectly think an entry needs to be added in
cinder/opts.py . This change add status.py to the
exception list.
Change-Id: Iab71497b5e9405429976b385dde2f6b7d585abae
This patch adds a new static method to drivers to expose their
configuration options. This also adds the same call to backup drivers
as well as the zone manager drivers.
Updated the generate_driver_list to expose the options as well.
This patch also orders the driver list alphabetically to make it easier
and consistent to find drivers. The driver list is now broken down
into supported and unsupported drivers as well.
Change-Id: I3e7db26ef3df24a12e3bfa219fe25bfb315335ec
We recently exposed the privsep opts for config generator use, so
projects that depend on oslo.privsep should include them in their
sample configs.
Change-Id: I59702c8b84bbcf21d32150ede0a3758d1a456bd6
Determines how many commits to check based
on the $FAST8_NUM_COMMITS env variable.
If set to "smart", it uses git to try to run
against all unsubmitted commits. This allows
fast8 to be more useful when actively developing
a series of patches.
Change-Id: I0adfba626ba2ea3479faf0f6ade712f14d3080d3
This commit does several things:
- Setup and run pylint directly rather than running
through a script. This allows the user to see what is happening
while the user is running through pylint.
- Allow the user to either run pylint on a particular changeset,
or the entire cinder tree.
- Allow the user to run on a particular changeset. Using like
HEAD~1, etc.
- Since the pylint gate check I disabled the tests that were
reported by pylint. The thought here would be go through
the failures and correct them.
- Update pylint to 2.1.1.
- Removed old pylintrc.
Change-Id: I708c93843d991e8c0ac114b0b2d6ede373b96d0a
Signed-off-by: Chuck Short <chucks@redhat.com>
The [service_user] group contains keystoneauth's session
options as well. This patch add these options back to list_opts.
Change-Id: I5a890c37b1bc23befb1f5792aa6129ca0c250917
The default process concurrency for pylint is 1. Since our gate
images used to run the job have 8 cores, this results in slower
job execution with many cores sitting idle. To speed things up,
this sets the process count to match the number of cores available.
Change-Id: If4ba3333a1cb3b8bb6b4797f9149f412d32047ef
Now cinder had some Versioned Objects which names do not
match their ORM counterparts. In method: get_model_for_versioned_object,
we handles those exceptions.
This patch fix this issue to keep the names match.
Change-Id: Icf709d87be99df95e5b52204032b730cd790096c
Closes-Bug: #1493112
This switches the pylint target to run under python3. In
order to work right, it also raises the version of pylint
used to a newer version.
Related-bug: #1761800
Change-Id: I0bc68a5aee24a53b9d3a08c20549b89d08989678
The long term goal for OpenStack is to support python3 for
all things except explicit python2 testing by the T release.
Part of preparing for that is ensuring all ancillary things
like docs jobs and other tooling work with python3.
This switches the default for our docs jobs to python3 so
anything that does not specify another version will end up
using it.
Further work is necessary for the pylint job due to changes
between the runtimes. That will be done in a follow up patch.
There was also a difference in behavior with the genopts job
where it ends up trying to recreate the venv under which it
is currently running under, resulting in a corrupted venv
and a failure. This cleans up that script and changes it so
rather than a tox job calling a tox job it just runs the
script.
Change-Id: Ie79528b33070e6c60f9f5e1f2219084a2f0cb039
The driver list generation takes driver docstrings and attempts
to parse that into rst formatted output for publication. With the
way this content is formatted within the docstring, this would
cause some slightly off formatting in the html output.
This attempts to better extract that information to make the
output cleaner and more readable.
Change-Id: I3b7e0817ade1c86b6d157c96872f00bb1b99c3d8
This patch adds documentation and sample
file for default policy in code feature.
Change-Id: I597971a29ec61a1bf8c991b2715ec7644b2e2692
Partial-Implements: blueprint policy-in-code
generate_cinder_opts.py doesn't really handle indirect variables
in group names, so add manual handlig of NOVA_GROUP.
Change-Id: I30a4666a058ebbf108fd42028fb0c39e5cb8bcae
Closes-Bug: #1717008
The location of the generate_cinder_opts script was moved in
d3abafdee4affe457e39705fd9b8e27b0f80f657. There is a check in
the script to exclude the script to its parsing, but this path
was not updated when it was moved. This updated the path to
the new location.
Change-Id: I369dec534c69f67fea72959bc9d626dddd8768ff
The sections in the cinder/opts.py file that are currently
upper case (I.E. BACKEND) cause a sphinx warning to be
generated. When we switch to treating warnings as errors
this will break our doc build.
This patch updates the code that generates the cinder/opts.py
file to make sure that all of the section names are lower-case
with the exception of the 'DEFAULT' section. With this
change in place we will not getting Sphinx warnings during
a doc build from the automatic config file generation.
Change-Id: Ibabbcaf7618623219690a2b27a2e548ea00be9d1
In the project, some of the terminology, like URL, URLs, API, APIs, OpenStack,
UUID, Cinder are neglectfully written as url, api, openstack, uuid, cinder.
This patch is to keep consistent of naming convention.
Change-Id: I98777fb4748cbc58b6e2fd1aca058d3e44069d07
The command to edit tempest.conf should be run as the same
user that owns that file. This is causing the scripts to run always
with the same backends[1].
[1] http://bit.ly/2juDims
Change-Id: Ie865042c052746b5001f6132c159bc32165021a3
This test was left behind as it wasn't passing the LVM -> NFS
migration setup. As it wasn't possible to reproduce and fix localy,
let's add this to the gate so we can reproduce the error scenario.
Additionaly to that, some bug fixes need this to be fully tested.
Change-Id: Icd9138c1bc1335e9465184749bf6dd5733c605a3
Remove unused variable.
Fix comment nit.
This addresses a review comment in https://review.openstack.org/430069.
Change-Id: I47e75a68c7e97340dab8b51b9fab9ed2983061ef
Add simple script to setup mysql and postgresql databases, this script
can be run by users during testing and will be run by CI systems for
specific setup before running unit tests. This is exactly what is
currently done by OpenStack CI in project-config.
This allows to change in project-config the python-db jobs to
python-jobs since python-jobs will call this script initially.
See also
http://lists.openstack.org/pipermail/openstack-dev/2016-November/107784.html
Update devref for this.
Needed-By: I628f825d9200f7951eae46b7be78b111b1a8141c
Change-Id: If549e6f108ec6184a432d5050da4386efe794a4e
There's been an intermittent issue where getting a list of all
drivers would fail to get all defined classes. With recent local
tests, once it got 103 drivers, but several times it got 69
drivers.
This appears to be from exceptions thrown while loading the
decorated classes with various cinder.objects classes not being
defined. To get around this, this adds an explicit load of all
cinder.objects before attempting to get the driver classes.
Change-Id: Ief4a93ec2d8256424166c7bf3b9d48a79a6e5e54
For lack of a better way, for drivers that have CI run by Jenkins we
denote this using the name 'Cinder_Jenkins'. CI links were added to
the generated driver list, but for these drivers this results in a
link to an invalid wiki page. For these drivers we should just skip
adding the CI info link.
Change-Id: I8a3087f8102392ca604a42fdf006e1a4e1a026d6
This patch is part of the effort to add migration tests to OpenStack
gate. It will rely in 2 tempest tests that are on the dependency list and that
the gate job will configure the following backends: ceph nfs glusterfs lvm.
Once running, the script will configure 2 tempest backends and run the
migration tests on all possible combinations of the backend listed above. The
output of the test run will be something like:
ceph nfs
ceph --- SUCCESS
nfs SUCCESS ---
If all tests in the matrix result in SUCCESS, then the test script returns.
Otherwise, other value is returned and the job is marked as failed.
Depends-on: I4ed9b1f30d4e4e595c44a0ce243c2463069833d7
Depends-on: Iff4d880a0f8e928cbc07b84ff0c357ad59ef929c
Change-Id: I17b5cd9dafb3ad17e8721376ae1e6790bd93108c
