191 Commits

Author SHA1 Message Date
Sean McGinnis
1c741636c7 Increase pylint processes
The default process concurrency for pylint is 1. Since our gate
images used to run the job have 8 cores, this results in slower
job execution with many cores sitting idle. To speed things up,
this sets the process count to match the number of cores available.

Change-Id: If4ba3333a1cb3b8bb6b4797f9149f412d32047ef
2018-06-20 10:35:26 -05:00
wanghao
e396560f33 Keep ORM names matching their VO counterparts
Now cinder had some Versioned Objects which names do not
match their ORM counterparts. In method: get_model_for_versioned_object,
we handles those exceptions.

This patch fix this issue to keep the names match.

Change-Id: Icf709d87be99df95e5b52204032b730cd790096c
Closes-Bug: #1493112
2018-06-19 17:50:47 +08:00
Zuul
f266406325 Merge "Clean output of driver list generation" 2018-06-14 22:03:37 +00:00
Sean McGinnis
d675de18de Run pylint job under py3
This switches the pylint target to run under python3. In
order to work right, it also raises the version of pylint
used to a newer version.

Related-bug: #1761800

Change-Id: I0bc68a5aee24a53b9d3a08c20549b89d08989678
2018-06-06 16:28:34 -05:00
Sean McGinnis
37510ac034 Default tox jobs to python3
The long term goal for OpenStack is to support python3 for
all things except explicit python2 testing by the T release.
Part of preparing for that is ensuring all ancillary things
like docs jobs and other tooling work with python3.

This switches the default for our docs jobs to python3 so
anything that does not specify another version will end up
using it.

Further work is necessary for the pylint job due to changes
between the runtimes. That will be done in a follow up patch.

There was also a difference in behavior with the genopts job
where it ends up trying to recreate the venv under which it
is currently running under, resulting in a corrupted venv
and a failure. This cleans up that script and changes it so
rather than a tox job calling a tox job it just runs the
script.

Change-Id: Ie79528b33070e6c60f9f5e1f2219084a2f0cb039
2018-06-05 14:16:34 -05:00
Sean McGinnis
113b383196 Clean output of driver list generation
The driver list generation takes driver docstrings and attempts
to parse that into rst formatted output for publication. With the
way this content is formatted within the docstring, this would
cause some slightly off formatting in the html output.

This attempts to better extract that information to make the
output cleaner and more readable.

Change-Id: I3b7e0817ade1c86b6d157c96872f00bb1b99c3d8
2018-04-25 17:15:23 -05:00
TommyLike
988f79479e Add policy documentation and sample file
This patch adds documentation and sample
file for default policy in code feature.

Change-Id: I597971a29ec61a1bf8c991b2715ec7644b2e2692
Partial-Implements: blueprint policy-in-code
2017-10-20 10:47:34 +08:00
Gyorgy Szombathelyi
db6564bb74 Add another hack from NOVA_GROUP to generate_cinder_opts.py
generate_cinder_opts.py doesn't really handle indirect variables
in group names, so add manual handlig of NOVA_GROUP.

Change-Id: I30a4666a058ebbf108fd42028fb0c39e5cb8bcae
Closes-Bug: #1717008
2017-09-14 10:19:17 +02:00
Sean McGinnis
b4485ec09f Update generate_cinder_opts path
The location of the generate_cinder_opts script was moved in
d3abafdee4affe457e39705fd9b8e27b0f80f657. There is a check in
the script to exclude the script to its parsing, but this path
was not updated when it was moved. This updated the path to
the new location.

Change-Id: I369dec534c69f67fea72959bc9d626dddd8768ff
2017-09-11 14:40:35 -06:00
Sorin Sbarnea
236f57e6ae Unhide errors from check_uptodate.sh
Displays execution errors instead of hiding them.

Change-Id: I8f723d17dffb96f2aa07e7b584fa124af1441592
Signed-off-by: Sorin Sbarnea <ssbarnea@redhat.com>
2017-08-30 16:36:03 +01:00
Jay S. Bryant
8db6335bb5 [DOC BLD FIX] Make opts sections lowercase
The sections in the cinder/opts.py file that are currently
upper case (I.E. BACKEND) cause a sphinx warning to be
generated.  When we switch to treating warnings as errors
this will break our doc build.

This patch updates the code that generates the cinder/opts.py
file to make sure that all of the section names are lower-case
with the exception of the 'DEFAULT' section.  With this
change in place we will not getting Sphinx warnings during
a doc build from the automatic config file generation.

Change-Id: Ibabbcaf7618623219690a2b27a2e548ea00be9d1
2017-08-29 10:53:48 -05:00
Eric Harney
d3abafdee4 Move config-generator to tools
Moving this out of cinder/ separates it from the
actual run-time Cinder code.

Change-Id: I056f40120fdef60a0dc1a5926729748fd2835938
2017-08-16 16:18:48 -04:00
Eric Harney
6460050c5c Add releasenotes to check_exec list
Prevent addition of executable files in
the releasenotes/notes directory.

Change-Id: I80ac0b7fce7b16a453fe9139f0dacf88325af8e7
2017-08-16 15:50:44 -04:00
junboli
db4327b6fe Keep consistent of naming convention
In the project, some of the terminology, like URL, URLs, API, APIs, OpenStack,
UUID, Cinder are neglectfully written as url, api, openstack, uuid, cinder.
This patch is to keep consistent of naming convention.

Change-Id: I98777fb4748cbc58b6e2fd1aca058d3e44069d07
2017-07-10 10:46:44 +08:00
Jenkins
a58cb0c693 Merge "Add driver supported status to dict output format" 2017-03-27 20:58:49 +00:00
Erlon R. Cruz
183dca0f91 Fix multibackend-matrix confs
The command to edit tempest.conf should be run as the same
user that owns that file. This is causing the scripts to run always
with the same backends[1].

[1] http://bit.ly/2juDims

Change-Id: Ie865042c052746b5001f6132c159bc32165021a3
2017-02-15 16:01:32 -02:00
Erlon R. Cruz
91229faf59 Adds attached migration to multibackend-matrix
This test was left behind as it wasn't passing the LVM -> NFS
migration setup. As it wasn't possible to reproduce and fix localy,
let's add this to the gate so we can reproduce the error scenario.
Additionaly to that, some bug fixes need this to be fully tested.

Change-Id: Icd9138c1bc1335e9465184749bf6dd5733c605a3
2017-02-15 16:01:32 -02:00
Walter A. Boring IV
76304ef091 Output the driver info as json
This patch fixes the output of the dict format to json, so it can be
consumed a bit easier.

Change-Id: I7574c6eefddb8f085422387f9d4892b8dbe9977d
2017-02-10 21:24:10 +00:00
Andreas Jaeger
afeb6f5aeb Minor cleanup on tools/test-setup.sh
Remove unused variable.
Fix comment nit.

This addresses a review comment in https://review.openstack.org/430069.

Change-Id: I47e75a68c7e97340dab8b51b9fab9ed2983061ef
2017-02-07 11:40:16 +01:00
Andreas Jaeger
1c91c0f033 Prepare for using standard python tests
Add simple script to setup mysql and postgresql databases, this script
can be run by users during testing and will be run by CI systems for
specific setup before running unit tests. This is exactly what is
currently done by OpenStack CI in project-config.

This allows to change in project-config the python-db jobs to
python-jobs since python-jobs will call this script initially.

See also
http://lists.openstack.org/pipermail/openstack-dev/2016-November/107784.html

Update devref for this.

Needed-By: I628f825d9200f7951eae46b7be78b111b1a8141c
Change-Id: If549e6f108ec6184a432d5050da4386efe794a4e
2017-02-03 07:34:35 +01:00
Jenkins
275be41863 Merge "Gate migration tests: Add Cinder tempest hook" 2017-01-24 16:04:23 +00:00
Vipin Balachandran
d34cd5cbac Add driver supported status to dict output format
Adding driver 'supported' status to the 'dict' output format
of driver listing.

Change-Id: I5ac1b151bd561991f12e4697b73fbbc3b8e5999b
2017-01-24 16:48:14 +05:30
Sean McGinnis
d1166474ab Prevent driver load failures from objects
There's been an intermittent issue where getting a list of all
drivers would fail to get all defined classes. With recent local
tests, once it got 103 drivers, but several times it got 69
drivers.

This appears to be from exceptions thrown while loading the
decorated classes with various cinder.objects classes not being
defined. To get around this, this adds an explicit load of all
cinder.objects before attempting to get the driver classes.

Change-Id: Ief4a93ec2d8256424166c7bf3b9d48a79a6e5e54
2017-01-16 17:26:54 -06:00
Sean McGinnis
cf6744db62 Skip CI link in list for infra supported drivers
For lack of a better way, for drivers that have CI run by Jenkins we
denote this using the name 'Cinder_Jenkins'. CI links were added to
the generated driver list, but for these drivers this results in a
link to an invalid wiki page. For these drivers we should just skip
adding the CI info link.

Change-Id: I8a3087f8102392ca604a42fdf006e1a4e1a026d6
2017-01-06 13:52:38 -06:00
Erlon R. Cruz
4f5a5962fa Gate migration tests: Add Cinder tempest hook
This patch is part of the effort to add migration tests to OpenStack
gate. It will rely in 2 tempest tests that are on the dependency list and that
the gate job will configure the following backends: ceph nfs glusterfs lvm.
Once running, the script will configure 2 tempest backends and run the
migration tests on all possible combinations of the backend listed above. The
output of the test run will be something like:

                    ceph             nfs
    ceph             ---         SUCCESS
     nfs         SUCCESS             ---

If all tests in the matrix result in SUCCESS, then the test script returns.
Otherwise, other value is returned and the job is marked as failed.

Depends-on: I4ed9b1f30d4e4e595c44a0ce243c2463069833d7
Depends-on: Iff4d880a0f8e928cbc07b84ff0c357ad59ef929c
Change-Id: I17b5cd9dafb3ad17e8721376ae1e6790bd93108c
2016-11-23 11:34:14 -02:00
Gorka Eguileor
0d2dceddd0 Add more operations to cluster
This patch adds cluster support to these methods:
 - update_consistencygroup
 - update_group
 - create_cgsnapshot
 - create_group_snapshot
 - create_snapshot
 - copy_volume_to_image
 - accept_transfer
 - get_backup_device
 - secure_file_operations_enabled
 - get_manageable_volumes
 - get_manageable_snapshots

APIImpact: Now listings of manageable volumes and snapshots accept
           cluster parameter.
Specs: https://review.openstack.org/327283
Implements: blueprint cinder-volume-active-active-support
Change-Id: I5d25fd2e442c0cc077149bec6a58f5a5380365be
2016-12-20 19:20:52 +01:00
Sean McGinnis
5d870b5621 Remove run_tests.sh wrapper
The run_tests.sh mechanism for performing operations
is no longer the preferred approach. It is now better
to just run tox. There are also issues with enforcing
upper constraints that may cause test runs to fail.

To get rid of confusion by having two different ways
to run unit tests, flake8 checks, etc., it's probably
better to just remove the run_tests.sh script completely
and direct folks to use tox instead. This way we are all
consistent and there is less concern that changes will
be made to one and not the other.

Change-Id: I03f4b49cffd9f64e4eb3bdc27995d543be9efa43
2016-12-09 14:15:54 -06:00
Vipin Balachandran
2a247e89e5 Add unsupported status to driver listing
Appending "unsupported" to the driver name in the driver listing
for unsupported drivers. The instance variable "_supported" in
BaseVD is changed to class variable "SUPPORTED" so that the supported
status of drivers can be read after loading the driver modules.

Change-Id: I21ca9137a847873f2ec026520daf8b97603d1888
2016-11-29 18:04:22 +05:30
Cady_Chen
aaa51695da Fix "wrap functions with 2 blank lines" pep8 check
Now gates are requires this pep8 rule as mandatory so until it'll be
fixed merge is blocked

Change-Id: Ic34859a0397a770db0a5203491f035cb81461bce
2016-11-17 02:50:35 +00:00
Michał Dulko
073d824deb Cleanup lintstack exceptions related to objects
Pylint doesn't like our cinder.objects namespace that is populated
dynamically and shows numerous errors for non-existing members of that
namespace. To fight this we've added multiple pylint exceptions like
"Module 'cinder.objects' has no 'Backup' member". New additions were
needed in case of new objects introduction or first usages of them.

This commit switches these exceptions to a regexp to make sure no new
additions will be needed in the future. Furthermore it moves some
non-object related messages into a general exception list.

Change-Id: I6252d3447f8793d4acd666265a59887e637ce614
2016-09-27 10:24:54 +02:00
xing-yang
325f99a64a Add group snapshots - db and objects
This is the third patch that implements the generic-volume-group
bluerpint. It adds database and object changes in order to support
group snapshots and create group from source. The API changes will
be added in the next patch.

This patch depends on the second patch which adds create/delete/update
groups support which was already merged:
    https://review.openstack.org/#/c/322459/

The next patch to add volume manager changes is here:
    https://review.openstack.org/#/c/361376/

Partial-Implements: blueprint generic-volume-group
Change-Id: I2d11efe38af80d2eb025afbbab1ce8e6a269f83f
2016-07-18 22:19:10 -04:00
Luong Anh Tuan
8df7241198 Remove self.__dict__ for formatting strings
Following OpenStack Style Guidelines:
http://docs.openstack.org/developer/hacking/#dictionaries-lists
It is not clear as using explicit dictionaries
and help avoid some errors during refactoring.

Change-Id: Iac53f4ae1fcfa8c0953bf635880f9b883ca9592b
2016-08-29 16:58:56 +07:00
xing-yang
8c74c74695 Add generic volume groups
This is the second patch that implements the generic-volume-group
bluerpint. It adds the groups table and introduces create/delete/
update/list/show APIs for groups.

It depends on the first patch which adds group types and group specs:
    https://review.openstack.org/#/c/320165/

Client side patch is here:
    https://review.openstack.org/#/c/322627/

Current microversion is 3.13. The following CLI's are supported:
cinder --os-volume-api-version 3.13 group-create --name my_group
    <group type uuid> <volume type uuid>
cinder --os-volume-api-version 3.13 group-list
cinder --os-volume-api-version 3.13 create --group-id <group uuid>
    --volume-type <volume type uuid> <size>
cinder --os-volume-api-version 3.13 group-update <group uuid>
    --name new_name  description new_description
    --add-volumes <uuid of volume to add>
    --remove-volumes <uuid of volume to remove>
cinder --os-volume-api-version 3.13 group-show <group uuid>
cinder --os-volume-api-version 3.13 group-delete
    --delete-volumes <group uuid>

APIImpact
DocImpact
Change-Id: I35157439071786872bc9976741c4ef75698f7cb7
Partial-Implements: blueprint generic-volume-group
2016-07-16 19:34:39 -04:00
Vipin Balachandran
2a8d7f858c Add CI wiki page to driver listing
The 'dict' output format prints the CI wiki name, but it is
missing in the 'str' output format. Adding CI wiki page to
the 'str' output format.

Change-Id: Ib62864221702d18a4f49118180b9af65f069ec4a
2016-08-12 13:12:11 +05:30
Walter A. Boring IV
1a5de5d4bd CI: Add CI_WIKI_NAME to all drivers
This patch adds a CI_WIKI_NAME to each driver object.  The value is the exact
name of the ThirdPartySystems wiki page.   This allows us to create an
automated tool to associated jobs to drivers and track their CI reporting
status correctly.

This patch also updates the generate_driver_list.py script to output the
driver list as a python list of dicts that can be directly consumed.

Change-Id: I0ec5f705e91f680a731648cf50738ea219565f70
2016-08-09 08:24:00 -07:00
Jenkins
16f6867ea0 Merge "Switch request_spec in create volume calls to ovo" 2016-08-03 13:55:12 +00:00
Walter A. Boring IV
c28cd113be Don't puke when exiting driver list
When you run the generate_driver_list.py file from inside the
tools dir, it pukes out an AttributeError for NoneType object
has no attribute 'close'.  This patch makes sure that exiting
doesn't cause a puke.

Change-Id: Ib6c4f451096cd07d0423941cab52ffa8cd8dc471
2016-08-02 07:03:33 -07:00
Jenkins
6713f1bb0c Merge "Replace OpenStack LLC with OpenStack Foundation" 2016-08-02 15:31:45 +00:00
Michał Dulko
294425d347 Switch request_spec in create volume calls to ovo
To be able to make changes in request_spec passed in create_volume calls
(in c-sch and c-vol RPC API) while supporting rolling upgrades we need
to convert it from arbitrary dict blob to a well-defined versioned
object. This commit does so while maintaining backwards compatibility
with Mitaka. When version is pinned to Mitaka we're serializing the
dict before sending and when receiving a dict we're converting it to the
object.

We actually have a lot of duplicated/unused data sent using request_spec
and filter_properties. We should start to clean that up.

The future patches should:
* Convert filter_properties to ovo.
* Make services use only one occurrence of the data.
* Remove the unused duplicates while maintaining backwards compatibility
  through object translations.

Change-Id: I7dae83667a1aca92c552fbfaa1e90c6558e293bf
Partial-Implements: blueprint cinder-objects
2016-08-01 18:40:48 +00:00
Jenkins
76dd0d4003 Merge "Fix mistakes introduced with QoSSpecs object" 2016-07-29 16:37:56 +00:00
Michał Dulko
e1b254dad1 Fix mistakes introduced with QoSSpecs object
If15ea8b628a6f88211a5d5cc7aadff44f7840138 introduced some mistakes that
I was able to identify when rebasing RequestSpec object patch:

* Missing loading volume_types relationship in QualityOfServiceSpecs
  _from_db_object.
* Wrong source of qos_specs relationship in VolumeType _from_db_object.
* Inefficient loading od QoSSpecs associations in db layer.
* Use of self in classmethod.

This commit fixes the issues.

Change-Id: I3b78127791a03e1a35a30d7f84f66e97d89c7cf9
Closes-Bug: 1603472
2016-07-28 13:34:24 +02:00
Sean McGinnis
8148038e92 Add driver list to doc build
We have tox -e gendriverlist that outputs an RST-ish report of all drivers
in the tree. This output can be used in the docs build to automatically
publish the list of drivers to make it easier to find officially supported
drivers.

This effectively removes the existing drivers.html that was generated prior
that did not actually contain any useful information.

Change-Id: I8de78723af76aabcc976733ac4b248db0b8ca16f
2016-07-26 11:09:05 -05:00
dineshbhor
1761d87faf Replace OpenStack LLC with OpenStack Foundation
Change-Id: I199fffd139a4d077985373231354343228d9d8b8
Closes-Bug: #1214176
2016-07-26 15:34:53 +05:30
Sean McGinnis
3c18c6f440 Add interface documentation to driver devref
Link the docstring generated documentation for the driver interfaces into
the drivers devref page for easy access to driver method documentation.

Change-Id: Ie22599fef99ebfac12f3b9bb251ae5bdfff4bff6
2016-07-24 17:23:29 -05:00
Jenkins
c1a206f80a Merge "Remove generate_driver_list dependency on cur dir" 2016-07-19 17:16:44 +00:00
Vipin Balachandran
71cf291cf2 Remove generate_driver_list dependency on cur dir
tools/generate_driver_list.py assumes that the current directory is
the Cinder root directory. If the script is run from a different
directory, it doesn't generate the driver info. This patch changes
the current directory to the root directory before calling the
utility functions to generate the driver list.

Closes-bug: #1604269
Change-Id: I35829003868531ccd76ee26e83e0b51067984a6a
2016-07-19 13:10:37 +05:30
Ryan McNair
e9c217fb5d Move QualityOfServiceSpecs to use VersionedObject
Change QualityOfServiceSpecs from a Python dict to using a
VersionedObject as part of moving Cinder to support rolling
upgrades.

These changes also include changes to volume/qos_specs.py:
  - Removing the unused method get_qos_specs_by_name

There will be follow-up patches to transition all instances of
qos['key'] to use qos.key and updating
volume_types.get_volume_type_extra_specs to use the new VO.

Fixing the invalid UUID issues and issues raised by new test cases.

Co-Authored-By: Xinli Guan <xinli@us.ibm.com>

Change-Id: If15ea8b628a6f88211a5d5cc7aadff44f7840138
Partial-Implements: blueprint cinder-objects
2016-07-06 16:37:08 +00:00
Sean McGinnis
e7b40242f8 Add driver interface checks
This is the start of an effort to both validate that drivers fully
implement the expected minimum requirements as well as to create a clear
place for driver developers to learn what needs to be implemented and get
documentation explaining what is expected for each method.

This also enables us to create tooling for documenting the available
drivers and their capabilities, to some degree. A follow up patch will
show some of what I'm thinking there, but it will make it possible to write
scripts for different needs.

This is somewhat a cleanup attempt to the ABC work that was started a
while back. This does not aim to replace that effort, but give a
mechanism for some of the things expected out of that effort that ended
up not being possible with how it evolved.

In most cases we do not really care if a driver is inherited from a
certain base class, just that it conforms to the given interface.

The interface/inheritance work really centers around two separate
things:

 * Ensuring drivers conform to an expected interface
 * Allowing code reuse and common implementation

This is really for the first item. Additional work is needed to complete
the ABC work we've done, but that really focuses on the second item, and
is out of scope for the intent of this patch.

Change-Id: I4168225126fe88c31712d94f0a130e9e7ede3446
2016-06-13 15:21:47 +00:00
Jenkins
b230cb743c Merge "Add sample config file to cinder docs" 2016-06-09 23:00:39 +00:00
Gorka Eguileor
253f9ea67b Remove API races from consistency groups
There is a potential window of opportunity where races can happen in the
API on consistency group related actions, this patch removes those
windows of opportunity using compare-and-swap for DB updates.

Races have been removed in following actions:

- create
- delete
- update
- create_cgsnapshot
- delete_cgsnapshot

Specs: https://review.openstack.org/232599/

Implements: blueprint cinder-volume-active-active-support
Change-Id: I67aec4cd8bcf2f7e09473a8d296aa383fe85ad23
2016-06-01 17:18:06 +02:00