38 Commits

Author SHA1 Message Date
Valeriy Ponomaryov
7145891ef8 Add dummy driver
Added dummy driver for ease of testing and development of REST APIs
and internal manila interfaces.
It can be enabled as backend in the same way as all other share drivers.
But, it does not provide any resources.

Also, update tempest to be able to work with share-networks when
networks are defined via config and not API.

CI hooks are configured to run tempest twice using both available
driver modes. Where DHSS=True mode uses StandAloneNetworkPlugin as
network backend.

Change-Id: I9053dddfc643cb5a6602f15235b91bbaea727dbd
Implements BluePrint dummy-driver
2016-07-22 17:57:59 +03:00
Valeriy Ponomaryov
e2aecad547 [Tempest] Bump tempest version
Fix small incompatibility that left to be done and bump Tempest version
to unblock our CI jobs.

Also, remove temporary fix [1] that was done for unblocking CI.

[1] I44655c83cc5588c5b94d0fad81162dd0c238b5b4

Change-Id: I12e2652638ba26833f9365f06fa84a98ca24765c
2016-06-08 11:53:29 +03:00
Valeriy Ponomaryov
82a571c486 [Tempest] Stop using deprecated Tempest opts
Currently, Manila CI Tempest jobs use deprecated options of Tempest.
Recently, latest version of Tempest dropped them [1].
Manila CI jobs work ok, because they check out specific version of
Tempest. But it will work until first Devstack update for some one
more Tempest feature and then we will be forced to use latest Tempest
again. Switch to supported approach of setting up users in Tempest,
to avoid such sudden blockers.

[1] I8c24cd17f643083dde71ab2bd2a38417c54aeccb

Change-Id: I5b3ebab52ea1401f6f7a116d1260268eb10ebe0c
Closes-Bug: #1586129
2016-06-08 11:44:08 +03:00
Valeriy Ponomaryov
e96e7cb299 [Tempest] HotFix for broken CI jobs
Recently merged commit to "neutron-fwaas" project [1] started using
one of Tempest's modules that appeared after version our plugin uses.
So, implement hotfix-workaround until we make all compatibility changes
to latest Tempest in our plugin.

For the moment, we cannot switch to latest Tempest as we already
have incompatibility for it, see bug [2].

[1] https://review.openstack.org/#/c/326150/
[2] https://bugs.launchpad.net/manila/+bug/1586129

Change-Id: I44655c83cc5588c5b94d0fad81162dd0c238b5b4
Closes-Bug: #1589964
2016-06-07 15:55:58 +03:00
Valeriy Ponomaryov
91ebe5bf65 Increase Cinder oversubscription ratio in CI
Generic driver fails from time to time on Cinder not being able
to schedule shares because of exceeded size limit.
Since we do not use size in API tests at all increase
Cinder oversubscription ratio from 20.0 to 100.0
to make it enough for sure.

Such error appears from time to time and
more often with addon of new Tempest tests that create shares.

TrivialFix

Change-Id: I8a14f806c0235f5a12930eee362a838404cd5c06
2016-04-01 18:10:44 +03:00
Yogesh
7bf4f0262e Add tests to ensure snapshots across replicas
Related-Bug: #1546303
Depends-On: Ia4cd2a36e31418e7a3d1c218080caa632755fe16
Depends-On: Id318a4adc0faf64a4bef57252aa2f0d9083b82b1
Change-Id: I269225b976efe13b2cbb9e0648d541a063df70e5
2016-03-19 11:53:02 +00:00
Marc Koderer
7fc492ea79 Fix keystone v3 issues for all clients
This fix basically ports nova fix https://review.openstack.org/#/c/136931/
plus additional bug fixing to Manila. It creates a common class to get
used for all clients (neutron, nova, cinder).

Idea is to create an auth session and pass this to the client object
instead of let the client do that. The auth session will be created by
a config group which means the configuration for the clients needed to be
heavily reworked. Patch is also backward compatible with old options but
flag them as deprecated.

DocImpact

Change-Id: Ic211a11308a3295409467efd88bff413482ee58d
Closes-bug: #1555093
2016-03-18 14:42:33 +01:00
Alexey Ovchinnikov
5dfbef4f95 Revert "LXC/LXD driver"
This reverts commit 466a19f18f12cb77b45aa7c7a272fae7c45c4c93.
Also this reverts commit 4f74b224fad7bf9ca665d70a66c79c98cf8d5674.

Manila has decided to remove LXD from the Mitaka release.
http://lists.openstack.org/pipermail/openstack-dev/2016-March/089522.html

Change-Id: I3c4b15e8c9d92c874be754205b0b981151809cc1
2016-03-16 18:48:17 +04:00
Ben Swartzlander
7d3fc0ed7f Run ZfsOnLinux gate tests with SSH enabled
SSH is required for replication to work in the ZfsOnLinux
driver, but the gate defaults to disabling it. Make it
configurable and make sure to preinstall host key.

Change-Id: Iece2ad2e966fafe3ab4d63c82d062830ac9f3170
Closes-Bug: 1552244
2016-03-08 10:50:05 -05:00
vponomaryov
b0c510e8ae Download service image only when needed
Download of Generic/Windows image for Nova should be disabled for
all other share drivers in CI.

Change-Id: Ib2c952e85dd0a6eb36fe945e0a33950c38dfdfae
Closes-Bug: #1553343
2016-03-04 21:48:19 +02:00
Igor Malinovskiy
4f74b224fa LXC/LXD driver
This driver exploits a container as a share server instead
of a virtual machine. The container acts as a NFS/CIFS server
and shares logical volume attached to it. The volume gets created
simultaneously to the container  on a host in a volume group
specified by the user. Currently NFS is limited to v3 since the
only userspace NFS server that works in a container out of the box
seems to be unfs3. Later it will be replaced with nfs-ganesha.

Co-Authored-By: Alexey Ovchinnikov <aovchinnikov@mirantis.com>

Implements: blueprint lxc-driver

Depends-On: Ib1896997f2e7a505b5bf8ec0c9e5ee35942f79a6

Change-Id: Iea73bd34b94919d77ebe17cf054dde1f580384be
2016-03-02 18:55:22 +04:00
Valeriy Ponomaryov
312cd3321a Add ZFSonLinux share driver
Add support for ZFS (on Linux) filesystem.

Following features are supported:
- Create/delete share
- Create/delete snapshot
- Create share from snapshot
- Extend/shrink share
- Update NFS IP-based access rules using new interface.
(SMB support planned for future).
- create/delete/update/promote share replica

ZFS-related notes:
- Any amount of ZFS zpools can be used by share driver.
- Allowed to configure default options for ZFS datasets that are used
for share creation.
- Any amount of nested datasets is allowed to be used.
- All share replicas are read-only.
- All share replicas are synched periodically, not continuously.
So, status 'in_sync' means latest sync was successful.
Time range between syncs equals to value of
config global opt 'replica_state_update_interval'.

Driver-related notes:
- Able to use remote ZFSonLinux storage as well as local.

Other made changes:
- updated driver private data DB methods removing filtering by host
  as redundant operation. Replication requires some common metadata
  storage and filtering by host breaks it. It is safe to do so, because
  if driver gets some ID of entity then it is allowed to read its info
  too.

Implements bp zfsonlinux-driver
DocImpact
Change-Id: I3ddd3767184e4843037de0ac75ff18dce709b6dc
2016-03-01 13:34:18 +02:00
vponomaryov
8c3c73ada6 Fix Cinder's NoValidHostFound errors
Manila CI jobs create lots of shares and for Generic driver it means
lots of Cinder volumes. After merge of [1] to Cinder project, we started
having NoValidHostFound errors in Cinder logs, it changed default
oversubscription ratio to 1.0 from 20.0. But Manila CI requires it
be bigger than 1.0, so set it explicitly to Cinder's default value 20.0
that is used by all other driver but not LVM.

[1] I5a214462b573d00d26086512821b4f8cc1db4fc3

Change-Id: If0a6e535b92b1471572e1de98e3b44ab6a2c65c9
Closes-Bug: #1540947
2016-02-02 22:21:54 +02:00
Julia Varlamova
401c8d982e Add LVM driver
Reuse code of old LVM share driver.

LVM Driver is 1st party Manila driver with NFS and CIFS support
and no share server support.

LVM Driver doesn't rely on Neutron, Nova, or Cinder. As such, it
is ideal for testing purposes and, after a period of maturation,
it should be useful in production environments.

Move generic driver's helpers to manila/share/drivers/helpers.py
and reuse them in LVM driver.

Implement 'ro' access, user access, extend_share function for
LVM driver.

Implements bp lvm-driver

Change-Id: Ia46c51ed400dbb0f1d87a758fb165ca711ed3d7c
2016-01-20 17:15:30 -05:00
vponomaryov
09e7125583 Use new approach for setting up CI jobs
Use arguments of Devstack hook scripts to configure Manila and Tempest in
CI jobs instead of dependency on names of CI jobs.

Change-Id: I96d8bca54320f4476789c00e1061d4e96b697d05
Depends-On: I521f0419e65e27f22acbc03aaa1f0f997492e8f8
2016-01-08 18:02:59 +02:00
vponomaryov
6b8319e32a Split common logic of CI hooks to separate file
Manila contains CI hooks located in "contrib/ci" dir.
Split common logic to separate file to be able to just "source" it and
then use env vars and bash functions in different CI jobs and projects.

Current change allows to do two things:
 - Reuse env var "MANILA_TEMPEST_COMMIT" in third-party CIs
 - Wait for drivers init in other projects like OpenStack Rally. Where,
   for the moment, one of jobs fails because of absense of "waiting".

Usage:
 $ source %manila%/contrib/ci/common.sh
 $ echo $MANILA_TEMPEST_COMMIT
 $ manila_wait_for_drivers_init "/etc/manila/manila.conf"

Change-Id: I94514405628ab8337d1f4153e431f064614e3596
2015-11-10 11:24:51 +00:00
Valeriy Ponomaryov
0af26bbba2 Sync Manila Tempest plugin with latest Tempest
Two reasons to do it:
1) Broken compatibility of old Tempest we use with latest dependencies.
2) Manila Tempest plugin is incompatible with latest Tempest.

Change-Id: Iae5a656b72e774d9d62b52d7ffb63edecbb23ca6
Closes-Bug: #1513105
2015-11-06 12:15:54 +02:00
Andrew Kerr
333474047c Add functional tests for Manila consistency groups
This commit adds tempest functional tests for Manila consistency
groups and cgsnapshot objects and actions. By default these tests are
enabled; 3rd party CI systems should disabled these tests through the
RUN_MANILA_CG_TESTS environment variable if their driver does not
support consistency groups.

Partially implements bp manila-consistency-groups
Change-Id: I3297e02ad53c328f0bfe5245fefdb6af80552b4a
2015-09-07 13:59:22 -04:00
Marc Koderer
09798caae1 Use Tempest plugin interface
Make use of the Tempest plugin interface instead of copying Manila
files into Tempest. This will remove the burden to port Manila
tests onto Tempest master recurrently.

This ports all existing Manila Tempest test to the new structure.

It uses manila_tempest_tests as new top folder for all Tempest
tests. It follow the model of Heat (see [1]).

[1]: https://github.com/openstack/heat/tree/master/heat_integrationtests

Change-Id: Ie5ed64a6777ed1acf8dd56522c26705ae897596d
Partly-implements: bp tempest-plugin-interface
Depends-On: I26dd32b1de8cceeaa6dc674092efec683df71889
2015-09-05 00:36:33 +00:00
Valeriy Ponomaryov
6c44bff516 Add possibility to setup password for generic driver
Make it possible to setup password for generic driver service image and
scenario tests. Also allow to just disable its usage not unsetting the password

It is useful for testing purpose when we have some problems with SSH keys
injection to Nova VMs.

Change-Id: I26dd32b1de8cceeaa6dc674092efec683df71889
Related-Bug: #1480870
2015-09-04 18:56:57 +03:00
vponomaryov
d1c1c009f4 Fix Neutron config setting in pre_test_hook
Change-Id: I843416f091c6804f3a14581a1992ee2f119cbd4c
Closes-Bug: #1486501
2015-08-19 13:35:07 +03:00
Valeriy Ponomaryov
35300fc4e6 Use all types of migrations in devstack installation
For the moment we install Manila database using only 'upgrade' migrations.
In that case we do not test Manila after using 'downgrade' migrations.
So, allow enabling of both types of migrations to verify that our downgrade
migrations do not break the world of data having functional tests running
after it.

Change-Id: If55a38692abca5e008095eb99b55adb465cad2b6
2015-07-14 09:46:46 +00:00
Igor Malinovskiy
d3d5d77ab6 Use new manila-service-image with public-key auth
- Update devstack plugin
- Update tempest scenario tests

Implements bp reproducible-server-image

Change-Id: I7c71ee2f20074e04021b5c95d8c18096d21b0404
Closes-Bug: #1421104
2015-06-22 16:03:13 +03:00
Valeriy Ponomaryov
5246c0e67f Bump supported tempest version
Change-Id: I38ddd6f4a7424cc61cdd6e4e1dd250a63bb8b7ff
2015-06-15 13:20:29 +03:00
Valeriy Ponomaryov
2335afa915 Sync tempest plugin with latest tempest
Sync our Tempest plugin to latest state of tempest repo because it has
incompatibilities.

Change-Id: I0d62f8a6f11f4bdba25807123e4cc4396a75799d
2015-05-08 14:51:24 +03:00
Marc Koderer
0d44022800 Sync contrib/tempest to newer state
An update is needed to support multiple identity versions.

Change-Id: I5c1417642c2bfe476647dcf5549dfed0bc2905c0
2015-04-23 09:05:06 +02:00
vponomaryov
8376ca41fc Sync tempest plugin with latest tempest
We are forced to sync our tempest plugin to latest tempest due to changes
done to CI system, that are  based on recent tempest project changes.
Such as reusage oslo libs instead of built-in incubator code, that leads to
changes of config opts that are set/read within tempest config used in CI.

Change-Id: I3e64dc373f5d81034afc4c5d8dfd5d7f4abaa603
Closes-Bug: #1434958
2015-03-22 13:23:46 +02:00
Valeriy Ponomaryov
98a6c6621b Use devstack plugin in CI hooks
We have devstack plugin stored in dir 'devstack' and now we do not need
extras.d hooks anymore, stored in dir 'contrib/devstack'.

Switch CI scripts for usage of devstack plugin and remove extras.d hooks.

Change-Id: I49008d169758710df81bf8bbc6ec9d455f162ba1
2015-02-25 23:45:52 +02:00
vponomaryov
46857b6ffc Sync tempest plugin
Our plugin for Tempest project is not compatible anymore.
Sync it and update commit ID that should be checked out.

Change-Id: I4db33a120da29f2417f2f3f2153f473626ae6b5f
2015-02-23 18:27:07 +02:00
Valeriy Ponomaryov
4dd9a33c83 Add possibility to define driver mode within pre_test_hook for CI
Make it possible to provide argument with boolean value to script
pre_test_hook.sh that will be considered for driver mode setting.
'False' means "no share servers handling" and
'True' means "handling of share servers exist".

It is useful for all drivers that do not handle share servers and are going
to have CI job with such setup.

Also, fix Nova service VM creation func within Devstack.

Change-Id: I4145986e224bb934503f79085ea6359d90d55755
2015-02-13 06:05:21 +00:00
vponomaryov
0ccc3ac55c Remove workaround for Nova VM boot bug
Boot of VM with provided Neutron port id works again after merge of following:
I01b561ac8b9b29cc4721d88567678a42a4fd1f6b
that is revert for
d8cafb9a6efb95b78609aca8fcb3532c33a1788f

So, we do not need workaround anymore and can test Manila against latest Nova.

Change-Id: I3d1c06e14380efaaad0fe0b06d9d4435ebb1a863
2015-01-31 13:26:36 +02:00
Valeriy Ponomaryov
7393fb03b9 Workaround Nova VM boot bug
Generic driver is unable to boot VMs anymore. The cause is in appeared bug in
Nova, that was introduced by commit d8cafb9a6efb95b78609aca8fcb3532c33a1788f,
where Nova is unable to boot VM if port from Neutron is provided.

Checkout Nova working commit until this bug is fixed in Nova.

Change-Id: I6dd2892392b5d08a7cede9c750d6328bee7dea9b
Closes-Bug: #1415770
2015-01-29 15:02:45 +02:00
vponomaryov
d36a343608 Sync tempest plugin with latest Tempest
Tempest has been changed and our plugin no more compatible with it.
Update files that are broken and bump Tempest commit that is used as HEAD.

Change-Id: I5b13d060b552154345ecfc5c1c6679496f3f3ea6
2015-01-10 11:44:11 +00:00
Ponomaryov Valeriy
be716c383a Make Tempest repo stable for Manila
Manila has Tempest plugin for mainstream Tempest.
Now, our jobs run over latest Tempest.
It causes incompatibilities from time to time.
Fix it by checking out Tempest repo HEAD to some specific commit that is
compatible with Manila plugin.

Implements blueprint stable-base-for-functional-tests

Change-Id: I5760461c1ebf8cf53ccfae55606fe6036c915e02
2014-12-23 14:33:46 +02:00
Valeriy Ponomaryov
738c7780b7 Update ci scripts
Changes:
Fix multibackend job be multibackend indeed.
Increase amount of threads to run tempest with, after increase of
volume group capacity for Cinder, that is used by generic driver.

Partially-implements blueprint multibackend-installation-tempest-job

Change-Id: Ia7aada6c29d4c2a4929384e9e03057b3cccc470d
2014-06-26 16:33:47 +00:00
vponomaryov
6abf3b9160 Update contrib/ci bash scripts
Actions from setup.sh files are supposed
to be moved to job itself. All changes needed only for tempest
are stored in post_test_hook.

So, no need to keep these files.
When files from multi_backend become not used, they
should be removed too.

Change-Id: I3da41e2843de3e2dfaf4952aea5f7e00a6ff9940
2014-06-05 11:07:09 +03:00
Valeriy Ponomaryov
f915c5430f Fixed ci bash scripts
1) Exec permissions were disabled for all. Set it as 755
2) Was syntax error in post_test_hook.sh
3) Added flags -xe for bash
4) Removed redundant exports from setup.sh

Change-Id: I0fbe1680b9903c8be6a1b7519775d68843df289b
2014-05-31 21:50:53 +03:00
vponomaryov
fa3dd801a8 Added bash scripts for ci jobs
Partially implements: blueprint multibackend-installation-tempest-job

Change-Id: I919bbdd6264ab1c6ad30b6edd210a70f84d32c68
2014-05-23 17:23:24 +03:00