This is the third patch that implements the generic-volume-group
bluerpint. It adds database and object changes in order to support
group snapshots and create group from source. The API changes will
be added in the next patch.
This patch depends on the second patch which adds create/delete/update
groups support which was already merged:
https://review.openstack.org/#/c/322459/
The next patch to add volume manager changes is here:
https://review.openstack.org/#/c/361376/
Partial-Implements: blueprint generic-volume-group
Change-Id: I2d11efe38af80d2eb025afbbab1ce8e6a269f83f
This is the second patch that implements the generic-volume-group
bluerpint. It adds the groups table and introduces create/delete/
update/list/show APIs for groups.
It depends on the first patch which adds group types and group specs:
https://review.openstack.org/#/c/320165/
Client side patch is here:
https://review.openstack.org/#/c/322627/
Current microversion is 3.13. The following CLI's are supported:
cinder --os-volume-api-version 3.13 group-create --name my_group
<group type uuid> <volume type uuid>
cinder --os-volume-api-version 3.13 group-list
cinder --os-volume-api-version 3.13 create --group-id <group uuid>
--volume-type <volume type uuid> <size>
cinder --os-volume-api-version 3.13 group-update <group uuid>
--name new_name description new_description
--add-volumes <uuid of volume to add>
--remove-volumes <uuid of volume to remove>
cinder --os-volume-api-version 3.13 group-show <group uuid>
cinder --os-volume-api-version 3.13 group-delete
--delete-volumes <group uuid>
APIImpact
DocImpact
Change-Id: I35157439071786872bc9976741c4ef75698f7cb7
Partial-Implements: blueprint generic-volume-group
The 'dict' output format prints the CI wiki name, but it is
missing in the 'str' output format. Adding CI wiki page to
the 'str' output format.
Change-Id: Ib62864221702d18a4f49118180b9af65f069ec4a
This patch adds a CI_WIKI_NAME to each driver object. The value is the exact
name of the ThirdPartySystems wiki page. This allows us to create an
automated tool to associated jobs to drivers and track their CI reporting
status correctly.
This patch also updates the generate_driver_list.py script to output the
driver list as a python list of dicts that can be directly consumed.
Change-Id: I0ec5f705e91f680a731648cf50738ea219565f70
When you run the generate_driver_list.py file from inside the
tools dir, it pukes out an AttributeError for NoneType object
has no attribute 'close'. This patch makes sure that exiting
doesn't cause a puke.
Change-Id: Ib6c4f451096cd07d0423941cab52ffa8cd8dc471
To be able to make changes in request_spec passed in create_volume calls
(in c-sch and c-vol RPC API) while supporting rolling upgrades we need
to convert it from arbitrary dict blob to a well-defined versioned
object. This commit does so while maintaining backwards compatibility
with Mitaka. When version is pinned to Mitaka we're serializing the
dict before sending and when receiving a dict we're converting it to the
object.
We actually have a lot of duplicated/unused data sent using request_spec
and filter_properties. We should start to clean that up.
The future patches should:
* Convert filter_properties to ovo.
* Make services use only one occurrence of the data.
* Remove the unused duplicates while maintaining backwards compatibility
through object translations.
Change-Id: I7dae83667a1aca92c552fbfaa1e90c6558e293bf
Partial-Implements: blueprint cinder-objects
If15ea8b628a6f88211a5d5cc7aadff44f7840138 introduced some mistakes that
I was able to identify when rebasing RequestSpec object patch:
* Missing loading volume_types relationship in QualityOfServiceSpecs
_from_db_object.
* Wrong source of qos_specs relationship in VolumeType _from_db_object.
* Inefficient loading od QoSSpecs associations in db layer.
* Use of self in classmethod.
This commit fixes the issues.
Change-Id: I3b78127791a03e1a35a30d7f84f66e97d89c7cf9
Closes-Bug: 1603472
We have tox -e gendriverlist that outputs an RST-ish report of all drivers
in the tree. This output can be used in the docs build to automatically
publish the list of drivers to make it easier to find officially supported
drivers.
This effectively removes the existing drivers.html that was generated prior
that did not actually contain any useful information.
Change-Id: I8de78723af76aabcc976733ac4b248db0b8ca16f
Link the docstring generated documentation for the driver interfaces into
the drivers devref page for easy access to driver method documentation.
Change-Id: Ie22599fef99ebfac12f3b9bb251ae5bdfff4bff6
tools/generate_driver_list.py assumes that the current directory is
the Cinder root directory. If the script is run from a different
directory, it doesn't generate the driver info. This patch changes
the current directory to the root directory before calling the
utility functions to generate the driver list.
Closes-bug: #1604269
Change-Id: I35829003868531ccd76ee26e83e0b51067984a6a
Change QualityOfServiceSpecs from a Python dict to using a
VersionedObject as part of moving Cinder to support rolling
upgrades.
These changes also include changes to volume/qos_specs.py:
- Removing the unused method get_qos_specs_by_name
There will be follow-up patches to transition all instances of
qos['key'] to use qos.key and updating
volume_types.get_volume_type_extra_specs to use the new VO.
Fixing the invalid UUID issues and issues raised by new test cases.
Co-Authored-By: Xinli Guan <xinli@us.ibm.com>
Change-Id: If15ea8b628a6f88211a5d5cc7aadff44f7840138
Partial-Implements: blueprint cinder-objects
This is the start of an effort to both validate that drivers fully
implement the expected minimum requirements as well as to create a clear
place for driver developers to learn what needs to be implemented and get
documentation explaining what is expected for each method.
This also enables us to create tooling for documenting the available
drivers and their capabilities, to some degree. A follow up patch will
show some of what I'm thinking there, but it will make it possible to write
scripts for different needs.
This is somewhat a cleanup attempt to the ABC work that was started a
while back. This does not aim to replace that effort, but give a
mechanism for some of the things expected out of that effort that ended
up not being possible with how it evolved.
In most cases we do not really care if a driver is inherited from a
certain base class, just that it conforms to the given interface.
The interface/inheritance work really centers around two separate
things:
* Ensuring drivers conform to an expected interface
* Allowing code reuse and common implementation
This is really for the first item. Additional work is needed to complete
the ABC work we've done, but that really focuses on the second item, and
is out of scope for the intent of this patch.
Change-Id: I4168225126fe88c31712d94f0a130e9e7ede3446
There is a potential window of opportunity where races can happen in the
API on consistency group related actions, this patch removes those
windows of opportunity using compare-and-swap for DB updates.
Races have been removed in following actions:
- create
- delete
- update
- create_cgsnapshot
- delete_cgsnapshot
Specs: https://review.openstack.org/232599/
Implements: blueprint cinder-volume-active-active-support
Change-Id: I67aec4cd8bcf2f7e09473a8d296aa383fe85ad23
The oslo sphinxconfiggen module was added to the oslo.config
2.3.0 release. This enables config file generation as part of
the sphinx doc generation.
The generated config file will pick up the current config
options from the code base. And as an added bonus, it will
now be published to the docs.openstack.org site for easy
reference or download.
This also puts us inline with what other projects like Nova
are doing for sample config files and is the recommended
method from the Oslo team.
Change-Id: I912a97eb2686d3dc56e50d8641d7bd930179bc18
The way these are organized makes it rather hard to
associate comments and their messages, and makes it
more work to modify this list in a meaningful way (or
re-use it for a project like os-brick).
Move things around a bit to make this easier to manage.
Change-Id: Ieebca9f97f6b15832693a37fa03f97c1074d64e8
The most recent version of bandit no longer requires a configuration
file (bandit.yaml) to run the default test suite. Removing the
configuration file and updating the bandit arguments will run all of
the currently available bandit tests in the default test suite.
Change-Id: If144c4017ce9bb58c3e83b62a5e31325353200b5
Most of the parts for oslo-incubator have moved to
oslo libraries. Cinder only had imageutils.py and
_i18n.py remaining in cinder/openstack/common. I
was to able remove both of these remaining incubator
components.
oslo_utils now has imageutils available. I moved
cinder/image/image_utils.py to use the new
imageutils from oslo_utils and updated all the other
places that were using the old imageutils import.
_i18n was only being used in scheduler/base_filter.py
which appears to have been accidentally left when the
scheduler was previously pulled out of incubator. I
fixed this oversight.
Finally, this patch deletes cinder/openstack/common
cinder/openstack and the openstack-common.conf file.
Change-Id: I8fe0b5d6c278715620f93f682289df7ae60dbeba
Closes-bug: 1517861
This currently doesn't report failures.
Also restrict files tested to HEAD~1..HEAD
instead of HEAD~2..HEAD.
Change-Id: I4983db5594fc2746eb28ec87a74d5d0ba27c9aaf
Previously in the generation of the opts.py file, the option groups
were hard coded. Now, they are pulled out as the lines are parsed.
If a group doesn't already exist in the dictionary it gets added and
then the options are added to the section. If the group already exists,
the options are added to the proper section as you would expect, and if
the options being registered don't have a group, they are added to the
default section.
This patch also moves the check for a non-zero exit code from
oslo-config-generator to the right location in the file. This
logic had been put too late in the file and was not catching
oslo-config-generator failures as it should have been.
Change-Id: I59c8067e8091d8bb212e3609f3a174ddc287707e
This patch adds the opts.py file to the tree as a fix for deployers
that package cinder. The opts.py file is no longer being deleted right
away by generate_sample.sh after the cinder.sample.conf is being
generated.
This patch also introduces a pep8 check to make sure that the opts.py
file is up to date, so that it will catch when new opts get added to
Cinder without the opts.py being updated.
To support the ability to keep and check the opts file a number
of changes were needed in the check_uptodate.sh script as well as
the generate_sample.sh script:
- check_uptodate now takes --checkopts instead of --checkonly
When checkopts is used the opts.py file is generated using the
current code and the generated file is compared to the existing
file. The check fails if there are differences.
- generate_sample now has the --nosamplefile option.
When this option is used, only the opts.py file is generated.
The oslo-config-generator code is skipped so no sample file
is created.
- generate_sample also has some coding style consistency changes.
- Added the 'genopts' option to tox so users can generate
a fresh opts.py without a sample file when necessary.
Closes-Bug: 1501820
Co-Author: Jay Bryant <jsbryant@us.ibm.com>
Change-Id: I1f5494ebb19d5f4e8c651cbeef0acad07ad96829
The following patch updates create_volume API to use
volume versionedobjects. Changes were made to be
backwards compatible with older RPC clients. It
only includes changes to the core cinder code.
Changes in the drivers are left to each driver
maintainer to update.
Note that this patch DOES NOT try to use
object dot notation everywhere, since it would
increase the size of the patch. Instead, it
will be done in subsequent patches.
Co-Authored-By: Michal Dulko <michal.dulko@intel.com>
Change-Id: Ic1b0f09132f8fc500b29650abbd57f18ea8bd9dd
Partial-Implements: blueprint cinder-objects
Change-Id: Ief9c63e8bddb2b40bdef4465b9099cff33d7c3bc
This replicates the run_tests.sh -8 behavior, running
flake8 only on changes made in the last commit and
working tree.
Change-Id: Iebd5746c78e840225f0860843c77f4c638877c26
As it stands, the opts.py file that is passed into
oslo-config-generator isn't being generated dynamically
and the old way of generating the cinder.conf.sample is
dependent on oslo-incubator which Cinder is trying to
move away from. oslo-config-generator works differently
than oslo-incubator so a number of changes had to be made
in order to make this switch.
This patch adds the config directory to Cinder and in it
are two files:
-generate_cinder_opts.py that will take the
results of a grep command to create the opts.py
file to be passed into oslo-config-generator.
-cinder.conf which is the new configuration for
oslo-config-generator. The file is inside the config
directory to be consistent with other projects.
Some changes were made to the generate_sample.sh file in
order to give the base directories and target directories
to the generate_cinder_opts.py program.
tox.ini was edited to remove the checkonly option because
all that needs to happen in check_uptodate.sh is a check to
ensure that the cinder.conf.sample is actually being
generated with no issues.
All options were removed from the check_uptodate.sh
because they were unnecessary given the new, more simple
way of generating the cinder.conf.sample.
setup.cfg was also edited in order to add information
oslo-config-generator needs to run.
Co-Authored By: Jay Bryant <jsbryant@us.ibm.com>
Co-Authored By: Jacob Gregor <jgregor@us.ibm.com>
Change-Id: I643dbe5675ae9280e204f691781e617266f570d5
Closes-Bug: 1473768
Closes-Bug: 1437904
Closes-Bug: 1381563
Change I3febb95bb2b3dd8f10da4f01ffa705e39cb20520 wasn't passing pylint
gate because of missing objects_ignore_messages entries in linstack.py.
This commits adds these entries.
Change-Id: I9e42a065d6584b445f2fed8995de32adbea7739b
Closes-Bug: 1493905
When importing backup metadata only required fields were being imported,
and all other fields were being ignored.
This led to some serious problems when importing backups back, since
you would lose not only the original Volume ID, but also relevant parent
id information needed to restore incremental backups.
This patch fixes this by importing everything back and creating backup
records with the right ID.
Closes-Bug: #1455043
Closes-Bug: #1476416
Depends-On: Id7ab6e174c1fe85772477f03059c4f457c5c8b17
Change-Id: Ia42ba73e9078d993c63f8e16308151ad11721ea9
Add versionedobjects abstraction layer to services. Distinguish time
zone aware DateTime fields. The object derives from
CinderObjectDictCompat, so it supports both object (obj.prop) and dict
(obj['prop']) syntax to access properties. Complete move to object
notation will be made in a follow up clean up patch.
Co-Authored-By: Michal Dulko <michal.dulko@intel.com>
Change-Id: I09f593f9f9aa8befa40d989b731159b78a429071
Partial-Implements: blueprint cinder-objects
Adds --checkonly to tools/config/check_uptodate.sh.
This allows verifying that a configuration file was
generated.
We have had a number of issues introduced which
have caused config file generation to fail since we
removed the sample config file check in the gate.
This does not add back checks to ensure that the file
has been updated, it only checks to ensure that it is
still possible to update the sample file.
Change-Id: I2e0e376fbdec5a1bb584c6739231649e1d0f5d8e
This change adds a basic bandit config for Cinder. It can be invoked
by running the tox environment for bandit;
tox -e bandit
This is intended as a starting point for using bandit with Cinder
and it should be revisited to improve the testing as more is learned
about the specific needs of the Cinder code base.
Tox is configured to only show results for high and medium severity
results.
https://wiki.openstack.org/wiki/Security/Projects/Bandit
Change-Id: I0247e0ccaed6faacacb2b8d2f8b141a8edc704af
It is useful to have a maintained (in source control) list
of drivers that exist in Cinder. It could be used in docs
and unit tests to check method impl on backend drivers.
This change add tool for generate list of drivers based on
existing BaseVD class in class hierarchy of volume drivers.
Output example:
Drivers: ['cinder.volume.drivers.lvm.LVMVolumeDriver',
'cinder.volume.drivers.rbd.RBDDriver',
...]
Implements: blueprint drivers-list-generator
Change-Id: I0e10906873e659e09a6e34531a0c932495d7c399
SQLite doesn't support 'drop constraint' statament. So it notifies:
SAWarning: Table 'encryption' specifies columns 'volume_type_id'
as primary_key=True, not matching locally specified columns
'encryption_id'; setting the current primary key columns to
'encryption_id'. This warning may become an exception in a future
release.
Add new strategy for sqlite upgrade: table is renamed to temporary
name, new table is created followed by INSERT statements, and
renamed to original one.
Change-Id: I723fdadb946f6944dde03954409899a9ddc3ee0e
Closes-Bug: #1431374
Use oslo.service.service and periodic_task entry points to generate
a sample configuration file. oslo.service library doesn't have
an oslo.service entry point.
Change-Id: Ib3c320d73494258e2ef973a62a54b6afb6b95988
Closes-Bug: 1468955
oslo.service has graduated, so cinder should consume it.
Change-Id: I80487f559302b0e88953835ffb4a3bfe2a347501
Closes-Bug: #1466851
Depends-On: I305cf53bad6213c151395e93d656b53a8a28e1db
Noticed today while working on fixing issues
with the config generator that oslo_policy
had been added in the wrong location for
pulling the config options in to the config
generator.
This fix moves the items that had been put in
oslo.config.generator.rc but were missing in generate_sample.sh
over to the appropriate location. I also removed the option for
importing additional libraries in oslo.config.generator.rc
given that it will not work.
I am still working on a longer term solution, but
we really should get this fixed so we can generate sample
config files again.
Change-Id: I2c14c1acc49d68305dca5a3dedc686ab07ed2088
This patch adds VersionedObjects abstraction layer to volume backups.
The object derives from CinderObjectDictCompat, so it supports both
object (obj.prop) and dict (obj['prop']) syntax to access properties.
Complete move to object notation will be made in a follow up clean up
patch.
Co-Authored-By: Grzegorz Grasza <grzegorz.grasza@intel.com>
Change-Id: Icff37261b367463b71a1268be16f9c97f595bf0c
Partial-Implements: blueprint cinder-objects
Use graduated oslo.policy instead of sync with oslo-incubator code.
Co-Authored-By: Ivan Kolodyazhny <e0ne@e0ne.info>
Change-Id: Ibb295830ac6abde9317c2549f75c1780ac454125
Closes-Bug: #1421863
Partial-Bug: #1458945
