Implement 'manage snapshot' feature in ZFSonLinux share driver.
Prerequisites:
- snapshot being managed should exist.
Details:
- snapshot being managed will be renamed by share driver based on
taken ID.
- size for managed snapshot will be taken either from
REST API (--driver_options size=5) or
from its "used" attribute.
Implements BP zfsonlinux-manage-snapshot
DocImpact
Change-Id: If240ecde5676c334d39faaccd5703e93544aaa06
Depends-On: Ifc762a882a78159adacfe168a4edbe824178301a
Added dummy driver for ease of testing and development of REST APIs
and internal manila interfaces.
It can be enabled as backend in the same way as all other share drivers.
But, it does not provide any resources.
Also, update tempest to be able to work with share-networks when
networks are defined via config and not API.
CI hooks are configured to run tempest twice using both available
driver modes. Where DHSS=True mode uses StandAloneNetworkPlugin as
network backend.
Change-Id: I9053dddfc643cb5a6602f15235b91bbaea727dbd
Implements BluePrint dummy-driver
Neutron is using tempest plugins too and it's needed newer changes
in tempest we need to update our commit id.
Change-Id: Ic6621fbad9e5ba932643e0c0fb717b71a1fc5808
Closes-bug: #1599429
Implement 'manage share' feature in ZFSonLinux share driver.
Prerequisites:
- share being managed should exist.
- share being managed should not be part of any replication relations.
- share being managed should not have redefined mountpoint.
Details:
- share being managed will be renamed by share driver based on taken ID.
- share being managed will get "quota" attribute set based on provided
value in API (--driver_options size=5) or it will set quota
to nearest bigger rounded integer of already
used space (2,4 -> 3, 3.6 -> 4).
Implements blueprint zfsonlinux-manage-share
DocImpact
Change-Id: I03eab5e02835fe660dde0020601c02c0136bddeb
Currently, Manila CI Tempest jobs use deprecated options of Tempest.
Recently, latest version of Tempest dropped them [1].
Manila CI jobs work ok, because they check out specific version of
Tempest. But it will work until first Devstack update for some one
more Tempest feature and then we will be forced to use latest Tempest
again. Switch to supported approach of setting up users in Tempest,
to avoid such sudden blockers.
[1] I8c24cd17f643083dde71ab2bd2a38417c54aeccb
Change-Id: I5b3ebab52ea1401f6f7a116d1260268eb10ebe0c
Closes-Bug: #1586129
Grenade is an OpenStack test harness to exercise the upgrade
process between releases. It uses DevStack to perform an initial
OpenStack install and as a reference for the final configuration.
Add support of it to Manila project, doing following changes:
- Add all required scripts for 'grenade' job, such as following:
-- settings
-- shutdown.sh
-- upgrade.sh
- Add optional 'resources.sh' script that performs functional
tests using bash and manilaclient. List of scenarios:
-- 1. create and delete share setting metadata and access rule
-- 2. attach security service to share network
-- 3. update quotas
-- 4. create private share type and add access for project
Note following:
- Tempest is not used by design.
- Manilaclient is used of stable version - version of 'base' devstack,
which is 'stable/mitaka'. It also means that used API microversion
is latest for that branch.
Change-Id: I7d7e0c50933776ca1a710bcf069abb0282b821ac
Depends-On: I4232382d3c944666bbf0916a3f285b84f6ff9d64
Implements blueprint grenade
With merge of commit [1] all calls of clients in Manila
post_test_hook.sh script fail, because used Devstack's file 'openrc'
does not export env var that contains domain name. That became
required after switch to Keystone v3 API in CI jobs [1].
Note: manilaclient calls should have been worked, as default values are
automatically picked up by Keystone server. But such clients as nova
requires set of domain name explicitly.
[1] Ic211a11308a3295409467efd88bff413482ee58d
Change-Id: Ia1976af058164da8ef9e33296006750aab7b3f50
Closes-Bug: #1559456
This reverts commit 466a19f18f12cb77b45aa7c7a272fae7c45c4c93.
Also this reverts commit 4f74b224fad7bf9ca665d70a66c79c98cf8d5674.
Manila has decided to remove LXD from the Mitaka release.
http://lists.openstack.org/pipermail/openstack-dev/2016-March/089522.html
Change-Id: I3c4b15e8c9d92c874be754205b0b981151809cc1
By default, these tests are disabled; 3rd party CI systems
should enable these tests through the
RUN_MANILA_REPLICATION_TESTS environment variable
if their driver supports share replication.
Partially implements: blueprint manila-share-replication
Co-Authored-By: Goutham Pacha Ravi <gouthamr@netapp.com>
Co-Authored-By: Andrew Kerr <andrew.kerr@netapp.com>
Change-Id: Idaf3c61600337cc30568b982dfb23948209c1d6e
Removed functionality of Share Migration relying on Manila Share
Service node, moved code to Data Service node for copy phase.
Added parameter 'notify' and share/api methods for future
implementation (see dependent patches).
Added new copy operation statuses, in order to implement future
API calls to obtain progress and cancel migration.
Added possibility of 2-phase migration for driver migration and
generic (fallback) migration.
Added admin export location support and removed approach of
replacing IP with config parameter.
Added Admin-only API entry points to:
- Migration Cancel (only during copying)
- Reset Task State field
- Migration Get Progress (only during copying)
- Migration Complete (2nd phase migration)
- Notify parameter on Migrate Share
APIImpact
DocImpact
Implements: blueprint data-service-migration
Change-Id: I1d65aac2f36942cd70eb214be561d59a15a4ba26
This driver exploits a container as a share server instead
of a virtual machine. The container acts as a NFS/CIFS server
and shares logical volume attached to it. The volume gets created
simultaneously to the container on a host in a volume group
specified by the user. Currently NFS is limited to v3 since the
only userspace NFS server that works in a container out of the box
seems to be unfs3. Later it will be replaced with nfs-ganesha.
Co-Authored-By: Alexey Ovchinnikov <aovchinnikov@mirantis.com>
Implements: blueprint lxc-driver
Depends-On: Ib1896997f2e7a505b5bf8ec0c9e5ee35942f79a6
Change-Id: Iea73bd34b94919d77ebe17cf054dde1f580384be
Add support for ZFS (on Linux) filesystem.
Following features are supported:
- Create/delete share
- Create/delete snapshot
- Create share from snapshot
- Extend/shrink share
- Update NFS IP-based access rules using new interface.
(SMB support planned for future).
- create/delete/update/promote share replica
ZFS-related notes:
- Any amount of ZFS zpools can be used by share driver.
- Allowed to configure default options for ZFS datasets that are used
for share creation.
- Any amount of nested datasets is allowed to be used.
- All share replicas are read-only.
- All share replicas are synched periodically, not continuously.
So, status 'in_sync' means latest sync was successful.
Time range between syncs equals to value of
config global opt 'replica_state_update_interval'.
Driver-related notes:
- Able to use remote ZFSonLinux storage as well as local.
Other made changes:
- updated driver private data DB methods removing filtering by host
as redundant operation. Replication requires some common metadata
storage and filtering by host breaks it. It is safe to do so, because
if driver gets some ID of entity then it is allowed to read its info
too.
Implements bp zfsonlinux-driver
DocImpact
Change-Id: I3ddd3767184e4843037de0ac75ff18dce709b6dc
Fix several issues in scenario test to get them stable again:
- The SSH timeout was set to 80 seconds what is way too low for an
Nova instance to come up in the gate.
- Scenario test access rule cleanup is not necessary, access rules
are deleted when share instance is deleted.
Co-Authored-By: Rodrigo Barbieri <rodrigo.barbieri@fit-tecnologia.org.br>
Closes-bug #1545715
Change-Id: I9f22f0416bb248666fe906e1ca653517054f21b5
Add APIs to support manage and unmanage share snapshots.
Also add support in the Generic driver.
This only supports for DHSS=False driver mode.
Add provider_location column to the share_snapshots table
to save data used to identify the snapshot on the backend.
Also need to bump microversion.
APIImpact
DocImpact
Change-Id: I87a066173c85d969607d132accd9f0e9bd49c235
Implements: blueprint manage-unmanage-snapshot
The post_test_hook.sh script was hardcoding the Tempest concurrency
setting to 8 for DHSS=False test runs. This patch has it honor the
MANILA_TEMPEST_CONCURRENCY environment variable, if it exists.
Change-Id: Ia2f0495560517484ba4d9290ced2c4dc3de6fb34
Closes-bug: 1546150
Reuse code of old LVM share driver.
LVM Driver is 1st party Manila driver with NFS and CIFS support
and no share server support.
LVM Driver doesn't rely on Neutron, Nova, or Cinder. As such, it
is ideal for testing purposes and, after a period of maturation,
it should be useful in production environments.
Move generic driver's helpers to manila/share/drivers/helpers.py
and reuse them in LVM driver.
Implement 'ro' access, user access, extend_share function for
LVM driver.
Implements bp lvm-driver
Change-Id: Ia46c51ed400dbb0f1d87a758fb165ca711ed3d7c
Use arguments of Devstack hook scripts to configure Manila and Tempest in
CI jobs instead of dependency on names of CI jobs.
Change-Id: I96d8bca54320f4476789c00e1061d4e96b697d05
Depends-On: I521f0419e65e27f22acbc03aaa1f0f997492e8f8
After merge of change [1] to devstack project, preconfigured creds for
Tempest cannot be used anymore. But Manila requires such.
For the moment, latest Tempest interfaces cannot allow us to use
preconfigured creds because of bug #1524717.
So, until this bug fixed, we should workaround Devstack changes and
set legacy opts explicitly.
Also, redefine new Nutron option 'dnsmasq_local_resolv' to 'False' value,
that is 'True' by default and was added in change [2].
Default value causes Nova operations initiated by Manila be very slow.
That leads to breakage of CI.
[1] I65b56ff681d6c27094380693c953fbc3664eceb0
[2] I17a884f467d307432a06f67a9dd93ed2fa6081a3
Change-Id: I516a6c9ab4396b14e3984d5f810210e4fcf7ec85
Related-Bug: #1531049
Closes-Bug: #1530847
Manila CI jobs fail very often exceeding timeout.
So, do several changes to speed up Tempest jobs:
1) Separate NFS and CIFS tests between jobs.
Run NFS jobs on MySQL ones and CIFS on PostgreSQL.
2) Refactor couple of tests removing creation of redundant shares
testing the same things.
3) Increase amount of threads, to minimize probability of
running heavy test suites in one thread.
Change-Id: I6b60db039a96b47386f8ff538d1cb74559691ccc
Manila contains CI hooks located in "contrib/ci" dir.
Split common logic to separate file to be able to just "source" it and
then use env vars and bash functions in different CI jobs and projects.
Current change allows to do two things:
- Reuse env var "MANILA_TEMPEST_COMMIT" in third-party CIs
- Wait for drivers init in other projects like OpenStack Rally. Where,
for the moment, one of jobs fails because of absense of "waiting".
Usage:
$ source %manila%/contrib/ci/common.sh
$ echo $MANILA_TEMPEST_COMMIT
$ manila_wait_for_drivers_init "/etc/manila/manila.conf"
Change-Id: I94514405628ab8337d1f4153e431f064614e3596
Our Tempest job "*no-share-servers*" fails after merge of change [1]
The root cause for it is in changed behaviour:
After merge of change [1] share services with driver that use Nova
started verify availability of a Nova VM by its net interface and
do not report its capabilities until it is available.
It takes about 2,5 minutes. During this time scheduler has no available
hosts and we get "no hosts available" errors from it in case of CI that
runs Tempest tests much faster than 2,5 minutes after VM creation.
So, add waiter in post_test_hook.sh script.
[1] I325d6461c4b12f481769bb01f792ef850dba789c
Change-Id: I58d617dd76cf079e5e26bf6a4ddc6b4fba4e20e6
Closes-Bug: #1507605
wating -> waiting
acknowledgement -> acknowledgment
occured -> occurred
Should never get here. If get, then error occurred.
->
Should never get here unless an error occurred.
Change-Id: Ibe89d652e7d76b3d8e6d7549140c50c88144036e
Signed-off-by: Zhao Lei <zhaolei@cn.fujitsu.com>
This patch addresses issues commented on
change I64b0a3ee77b27278cc294f72702408a27888e0e9
after it was merged, according to bug below.
Closes-bug: #1494000
Change-Id: I9079ea16e0edc359b380705bebba2a7e98446d5c
Use novaclient for creation of keypair instead of openstack client in
devstack plugin, because the latter is not compatible with
nova microversion 2.1
Also, specify explicitly Nova API version in manila wrapper over
novaclient that is used by Generic and Windows drivers.
Also, revert back amount of tempest threads for Ci jobs to avoid job
timeout failures.
Also, fix raise of SnapshotBuildErrorException in Tempest plugin.
Tempest tries to raise exception SnapshotBuildErrorException from wrong
place. It should use "share_exceptions" instead of "exceptions".
Change-Id: Ie9ea9f6d25a13f8cd8f0ccbc73f6fa75cca35e0b
Closes-Bug: #1493008
Closes-Bug: #1493072
This commit adds tempest functional tests for Manila consistency
groups and cgsnapshot objects and actions. By default these tests are
enabled; 3rd party CI systems should disabled these tests through the
RUN_MANILA_CG_TESTS environment variable if their driver does not
support consistency groups.
Partially implements bp manila-consistency-groups
Change-Id: I3297e02ad53c328f0bfe5245fefdb6af80552b4a
The Generic Driver in Manila has no concept of consistency
groups. This patch proposes to implement a loose
consistency group support mechanism for the Generic Driver,
using existing logic to snapshot shares and create shares
from snapshots.
The actions concerning creating and deleting a consistency
group are handled by the API and the Generic Driver does not
need to remember any relevant information persistently.
The actions concerning consistency group snapshots are not
simultaneous, but are rather let to run in succession.
DocImpact
Partially implements bp manila-consistency-groups
Change-Id: Ie5e51205a1154786f9057691aa82bffcafe04de7
Make use of the Tempest plugin interface instead of copying Manila
files into Tempest. This will remove the burden to port Manila
tests onto Tempest master recurrently.
This ports all existing Manila Tempest test to the new structure.
It uses manila_tempest_tests as new top folder for all Tempest
tests. It follow the model of Heat (see [1]).
[1]: https://github.com/openstack/heat/tree/master/heat_integrationtests
Change-Id: Ie5ed64a6777ed1acf8dd56522c26705ae897596d
Partly-implements: bp tempest-plugin-interface
Depends-On: I26dd32b1de8cceeaa6dc674092efec683df71889
Using approach without handling of share servers we have bigger load for
volume creation in Cinder using Generic driver. So, reduce amount of threads
to avoid errors for Cinder volume creations that appear because of lack of
free space.
Change-Id: I738dd9837eb6f951899aef091a6aafe8a4a0945b
To make the script more flexible to include more or less tests,
Allow the test regex to be defined prior to calling to post_test_hook.sh.
This will also allow the various jenkins jobs to define the regex directly
instead of relying on their name.
Change-Id: Id64d761e7ad035a2e6d6351b04b9f224aa3b8e86
This patch allows the following Manila tempest options to be set,
rather than hardcoded, in Jenkins CI jobs:
- suppress_errors_in_cleanup
- run_manage_unmanage_tests
- multitenancy_enabled
Change-Id: If566501cd5c97be37c6e018080e604fd76b6605d
Adapt manager and add first initial test for Manila. A ping check will
be performed to check the connectivity from the VM to the share
server/service. The test can be triggered using the experimental test
job introduced by [1].
Open todo's that can be addressed later:
- Support of flat network
- More flexible configuration options
[1]: https://review.openstack.org/#/c/164102/
Change-Id: Ie72e877a88c337866e9cb591f8099218d84ee5dd
Partially-implements: blueprint scenario-tests
Wikipedia's list of common misspellings [1] has a machine-readable
version. This patch fixes those misspellings mentioned in the list
which don't have multiple right variants (as e.g. "accension", which can
be both "accession" and "ascension"), such misspellings are left
untouched. The list of changes was manually re-checked for false
positives.
[1] https://en.wikipedia.org/wiki/Wikipedia:Lists_of_common_misspellings/For_machines
Change-Id: I2446995aa8b7c89c7f78c056f9ddb6d8e7899e4e
Signed-off-by: Martin Kletzander <mkletzan@redhat.com>
Add implementation of manage_existing() to Generic driver.
Generic driver accepts only one driver_option 'volume_id'.
If an administrator provides this option, then this Cinder
volume will be managed by Manila as well.
Partially implements bp manage-shares
Change-Id: I8767a7af75d575c57e8708563fa4a6bf2f4aa8ad
bashate checks shell script syntax (similar to pep8 for python). Fix
currently available scripts and enable the check for the scripts.
Change-Id: I33dc8c44a58f40e8bca9e29cd717caf46e5cd586
Since CLI tests were implemented in manilaclient itself we do not need it
anymore in manila. So, remove it as duplication.
Change-Id: Ie6ff3e7e822276c247f73ab6495ca2a2acee91ce
Currently TEMPEST_CONCURRENCY is hardcoded to 12. This change
uses a different variable name to control concurrency, called
MANILA_TEMPEST_CONCURRENCY. The value still defaults to 12, but
can be overridden by the environment by 3rd party CI systems. A
different variable name is used to ensure that in upstream CI
it will not conflict with any existing variables.
Change-Id: Ia7c76a3ef18a6d114336417bfda08321f8d8971d
After appearence of second mode for Generic driver we need to run additional
job with functional tests using this second mode.
Changes:
- Add possibility to create service share server for each generic driver with
mode that does not allow share servers handling.
- Add autodetection of driver mode for post_test_hook.
Implements bp generic-driver-single-svm-ci-job
Change-Id: I77127115a89caaaa8701c2089cd5db4b609e6c7f
For the moment bash function 'iniset' is imported from devstack only for
multi backend Tempest job. It causes absence of set options for single backend
installation, because absent function 'iniset' is attempted to be used.
Due to described problem single backend installation does not have enabled
error suppressing in resource cleanup and retry mechanism is disabled.
Change-Id: Ib79be4c6b781e7c9e000dcea617cc8b03e5b4e7a
For the moment config option for suppression of error within Tempest cleanup
is set only for multibackend job. Make it be set for all jobs.
Change-Id: Ic61f164e33ac1fff513947cf7f3e8cef115208ed
Add possibility to retry share creation to be able to avoid failures
caused by unstable environment.
To enable it just update config opt "share_creation_retry_number" in config
group "share". This opt is integer and if it equals to or less than zero (0),
then no recreations will be performed.
All shares created with recreation logic will be cleaned up in the same way
as it was before.
Enable 2 retries for CI Tempest jobs.
Implements blueprint retry-for-functional-tests
Change-Id: I2b72b3908b8c87e4ce67b0aa277c19a2339681b1
Now if something can not be cleaned up we get error and in case of CI job it is
'fail' of job. We should be able to suppress such errors.
Added config option 'suppress_errors_in_cleanup' that allows us to raise errors
when it is 'False' and log errors when it is 'True'.
Disable it by default.
Enable it for CI Tempest jobs.
Implements blueprint tempest-clean-up-suppress-errors
Change-Id: I03a39cc020f52d54211369c8f5cbfc2e14651a21
Increase quotas for Cinder (Generic driver case) and Manila.
Increase number of threads for tempest run to simulate concurrency and
have test run in shorter time slot.
Change-Id: Ic7473c94eeb2b45533c028352b56e4027cd7eead
Now Tempest tests are not running with our Tempest jobs.
Tempest commit https://review.openstack.org/#/c/123589/ has changed Tempest
tox jobs, so we need to use another tox job to be able to run our Tempest
tests.
Change-Id: I166637916347e03010781e4ca97fd901a6181b48
There are cases when Cinder volume quota is exceeded with
high load in gate tests with generic driver, that uses Cinder.
Until we have proper way to increase Cinder volume quota, amount
of threads is decreased for better stability of gate tests.
Change-Id: I2c74aee51aa6c1c3cb4fb4ccd9f28d5a05290b52
Changes:
Fix multibackend job be multibackend indeed.
Increase amount of threads to run tempest with, after increase of
volume group capacity for Cinder, that is used by generic driver.
Partially-implements blueprint multibackend-installation-tempest-job
Change-Id: Ia7aada6c29d4c2a4929384e9e03057b3cccc470d
