122 Commits

Author SHA1 Message Date
Zuul
d290514706 Merge "Some clean up" 2019-09-20 02:02:39 +00:00
Zuul
a67a6ed2ef Merge "Fix issue with pip2 command and update pip3 for mongodb" 2019-09-19 23:49:04 +00:00
Lingxian Kong
757a873963 Some clean up
- Remove the unused variables in trovestack.rc which leads to
  module not found error.
- Remove non-voting gate job

Change-Id: I583945addb6cb0aa442db1c9d9c6ce3cb57793f7
2019-09-20 08:51:21 +12:00
Lingxian Kong
0cdf8b20b6 Add image build experimental CI job
Story: 2006553
Task: 36637
Change-Id: I3d909082fb887b24ed25c93e11792031d5053b1e
2019-09-17 22:23:43 +12:00
Nguyen Van Trung
38efc97251 Fix issue with pip2 command and update pip3 for mongodb
This patch will fix following issue logs:

   2019-09-02 15:11:13.121 | dib-run-parts Running /tmp/in_target.d/install.d/25-trove-mongo-dep
   2019-09-02 15:11:13.124 | + pip2 install pymongo
   2019-09-02 15:11:13.125 | /tmp/in_target.d/install.d/25-trove-mongo-dep: line 9: pip2: command not found

And move on pip3.

Change-Id: I7cf16820091a0f977b41e3222abb866a9ae66166
2019-09-12 10:11:30 +00:00
Zuul
1b159c7d6a Merge "Improve image build" 2019-09-04 05:03:24 +00:00
Lingxian Kong
89386032ae Fix MariaDB image build
The error in MariaDB installation during the guest image building:

```
apt-get -y install socat percona-xtrabackup
percona-xtrabackup : Depends: libcurl4 (>= 7.16.3) but it is not installable
                     Depends: libgcrypt20 (>= 1.8.0) but 1.6.5-2ubuntu0.5 is to be installed
                     Depends: libssl1.1 (>= 1.1.0) but it is not installable
E: Unable to correct problems, you have held broken packages.
```

This patch fixes the issue based on the process described in
https://mariadb.com/kb/en/library/installing-mariadb-deb-files

Change-Id: I368bddabffcc0f11f7181b3a33be1f213f0bdaba
2019-09-04 09:38:37 +12:00
Lingxian Kong
c23c1b9d0c Improve image build
- Remove network-scripts package from guest agent, otherwise image
  creation will fail with network-scripts not found.
- Improve the way to install percona-xtrabackup package on Xenial to
  avoid the failure: "write error: No space left on device"

Change-Id: I6d8de1ad0f675db9635f4762cefc26169e738bbc
2019-09-02 22:58:28 +12:00
Lingxian Kong
a1aa15282e Support management security group
Allow the cloud admin to control the security groups on the management
port of Trove instance, a new config option `management_security_groups`
is introduced for that purpose.

Change-Id: I4b22b87d37792be700d4ec7f78a7ea479ddb5814
Story: 2006466
Task: 36395
2019-09-02 10:06:49 +12:00
Lingxian Kong
0ef474a70c Support keypair in devstack
Since Trove already supports to specify a Nova keypair when creating
instance for management convenience, devstack needs to be changed to
create the management keypair and add to Trove config file.

One extra change in this patch is to use a single config file for Trove
API, task-manager and conductor.

Change-Id: I1e6c4f4305104815bdf89b31776a4955de61bc89
Story: 2005429
Task: 30463
2019-08-29 15:41:58 +12:00
Lingxian Kong
7b9a752eb0 Support python3 in guest agent
Story: #2006455
Task: #36376
Change-Id: I62323ac1c1edfbf0378d0c89e156fdb3a879aa88
2019-08-27 15:03:45 +12:00
Lingxian Kong
dfa5ce93d5 Improve devmode=flase when building the image
During debugging, the following changes are also included:

- Support to specify an image ID to run the integration test.
- Fix the reboot function bug.
- Remove the unsuccessful restart test.

How to run integration test with dev_mode=false:

    ADMIN_PASSWORD=password \
    SERVICE_PASSWORD=password \
    DEV_MODE=false \
    /opt/stack/trove/integration/scripts/trovestack gate-tests mysql mysql

Change-Id: I31d4ee579a554f4c98f9facb9fd4b7779665a3dd
2019-08-25 23:11:54 +12:00
Mark Kirkwood
7d2e986778 Fix 31-fix-init-script for Postgresql
Ironically the fix is never applied as the systemd file uses
/run not /var/run.

Change-Id: Ie6022a13c181f5c27e19b54bd49d9a77c90dc1ac
2019-08-14 22:09:56 +00:00
Andreas Jaeger
7b3483723a Fix Trove CI jobs
Changes to get jobs working

1) After [1] devstack no longer changes the ownership of the whole
   /opt/stack tree to the stack user unconditionally. Switch to the
   stack user when running integration test.

2) Add bindep.txt file[2]. The default fallback file is not installed
   anymore and therefore a bindep.txt file is needed to add install
   additional packages.

3) Use trovestack script rather than devstack to build image so many
   global variables could be used for consistency. By default, devstack
   won't build image.

4) Remove the tools/test-setup.sh as it is not used any more.

5) Instance upgrade test keeps failing in CI for some reason, although
   it's always passed on my local environment. In order not to block
   other patches, skip the instance upgrade tests temporarily.

[1] https://review.opendev.org/203698
[2] http://lists.openstack.org/pipermail/openstack-discuss/2019-June/006888.html

Change-Id: I35e17afb9e827b1fead9d28dbf32f11ce4034a9b
2019-08-14 21:20:34 +12:00
Lingxian Kong
49a11dd9c7 trovestack guide
Also fixed the migration script issue after upgrading SQLAlchemy
version from 1.2.19 to 1.3.4

Change-Id: Ia0d954237dd3c9c2b2ebedf440fa3c656da30777
2019-06-11 00:09:42 +12:00
Lingxian Kong
2e052b0262 Enable service tenant deployment model by default in DevStack
The service tenant deployment model means Trove creates most of the
resources(vm, volume, security group, etc.) relating to a database
instance in the Trove service tenant rather than the end user.

With this deployment model, most of the related resources behind the
scenes are invisible to the user, which is a more secure deployment
model for either private or public cloud provider.

DevStack should follow this model as it will be recommended for Trove
deployment in production.

Changes included in this patch that are necessary in order to make that
happen:

- Add 'admin' role to Trove service user(username: trove, project:
  service) in DevStack.
- Create Trove management network resources for Trove service user in
  DevStack.
- Enable Trove remote client configuration by default in DevStack.
- Mainly use alt_demo user in alt_demo project for integration tests,
  config trove user as the admin role user in integration tests.
- Disable the module related tests(module_groups) for now because of no
  use cases but need effort to fix all the failed tests in the service
  tenant model.

Story: #2005445
Task: #30489
Change-Id: I2efb69d3d50344914a875b773f62a227dba2ccaf
2019-06-08 00:46:21 +12:00
Lingxian Kong
37e8dedac2 Remove the trove-tox-apiexamples CI job
- The job relies on python-troveclient, sometimes python-troveclient
  only changes the URL sending to Trove but neither request nor
  response.
- The job is hard to maintain, especially when there are changes
  relating the API. The recommended way is to manually update the API
  doc and add tests in trove-tox-fakemodetests job.
- Remove that trovestack subcommand as well.

Change-Id: I108203915ef79ef93bbfbb15acdb7480854878e0
2019-06-06 15:57:26 +12:00
ZhongShengping
6a5afea40a Move to opendev
1.Use opendev.org instead of git.openstack.org.
2.Use review.opendev.org instead of review.openstack.org.

Change-Id: I8e9da8fbb04bdec42a0b2f012f6a38b8ee413b27
2019-04-24 16:08:08 +08:00
Lingxian Kong
eafd303e0a Use opendev.org instead of git.openstack.org
After moving to opendev.org, the following command fails when building
the Trove guest image:

curl -o /opt/stack/devstack/upper-constraints.txt \
  'https://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt?h=master'

https://opendev.org/openstack/requirements/raw/branch/master/upper-constraints.txt
should be used instead.

Other places using git.openstack.org are also changed.

Change-Id: Id357dd61e229e3336219723c1c1ae6b50e983f26
2019-04-22 00:05:41 +12:00
Kasper Hasior
7f7ae90ee9 Fix cloudinit mariadb scenario test error
For now mariadb scenario single test case fails because the script
cannot find cloudinit file.

This change adds cloudinit file creation if the file doesn't exist.

Before this patch we were getting the following message:
    tee: /etc/trove/cloudinit/mysql.cloudinit: No such file or
    directory
    #!/bin/sh
    ERROR: InvocationError for command /opt/stack/trove/integration/
    scripts/trovestack gate-tests mariadb mariadb-supported-single
    (exited with code 1)

Change-Id: Iea74dc2c58b1c4376c698f0084454cfcbb3f3b7d
Signed-off-by: Kasper Hasior <k.hasior@samsung.com>
2019-04-17 11:45:58 +02:00
Zuul
ee0ed4de0d Merge "Improve trove guest agent image building" 2019-04-11 00:04:21 +00:00
Ian Wienand
4367dd5e7a Merge "Replace openstack.org git:// URLs with https://" 2019-04-10 00:52:15 +00:00
Lingxian Kong
798af4d02a Improve trove guest agent image building
- Add a new element 'guest-agent' for image building. This element is
  used when dev_mode=false, so that the trove code is downloaded into
  the image during the building phase rather than during the guest
  agent initialization.
- Improve trovestack sub-command 'build-image'.

  ./trovestack build-image ${datastore_type} \
     ${guest_os} \
     ${guest_release} \
     ${dev_mode}

- Improve documentation.

Story: #2005387
Task: #30375

Change-Id: I9d7acbd6a97f8c01b48b0f2cf94398d549d89124
2019-04-10 09:29:40 +12:00
Lingxian Kong
6ea816312e Fix tests for Ubuntu Bionic migration of CI jobs
Change-Id: I2fee668c3774f7e87ce3478ca998f8b366c6cd41
2019-03-28 10:32:26 +00:00
Ian Wienand
007821e830 Replace openstack.org git:// URLs with https://
This is a mechanically generated change to replace openstack.org
git:// URLs with https:// equivalents.

This is in aid of a planned future move of the git hosting
infrastructure to a self-hosted instance of gitea (https://gitea.io),
which does not support the git wire protocol at this stage.

This update should result in no functional change.

For more information see the thread at

 http://lists.openstack.org/pipermail/openstack-discuss/2019-March/003825.html

Change-Id: I7f2db9189611a1e2121d9e7bacd35c05382b5915
2019-03-24 20:36:20 +00:00
lijunjie
c531ab3c55 Fix the misspelling of "directory"
Change-Id: If9cca7b453daf63ce4bc2ecff4a90c3a701858e4
2019-01-23 17:24:46 +08:00
Dariusz Krol
68a4819dd4 Add image setup to trove devstack plugin
Main goal is to have a working vm image registered in
trove after devstack installation.

It is a first step towards trovestack-redesign as described
in https://etherpad.openstack.org/p/trovestack-redesign

It is enabled by default during devstack installation with
trove enabled plugin, but it can be skipped by setting
DISABLE_TROVE_IMAGE_SETUP=TRUE

Implementation details:
* export environmental variables for diskimage-builder
* build an image with disk-image-create script
* register the newly created image in trove
* add tripleo-elements information to default settings

Change-Id: I6e57890c5b6fb65e9b8c63363f92e5a7c70e523e
Signed-off-by: Dariusz Krol <d.krol@samsung.com>
2018-12-14 21:26:01 +01:00
Marcin Piwowarczyk
be17146943 Fix home direcroty path in trovestack script
This simple fix solves "no such file or directory" error when user
home directory is not prefixed by /home.

This is the case, for example in devstack, where stack user home is
set to /opt/stack/.

Lets use $HOME variable to determine user home location.

Change-Id: I07432d27e1a2035296bc62793fecbe13d0265e7b
Signed-off-by: Marcin Piwowarczyk <m.piwowarczy@samsung.com>
2018-10-26 20:33:32 +02:00
Zhangfei Gao
2d7ecd9385 Fix build ubuntu-geust issue on arm64
To build ubuntu-geust on arm64, libffi-dev & libssl-dev are required

Closes-Bug: #1797014

Change-Id: Ia215deac11976fc9fc67e6d823ae11d181bf8083
Signed-off-by: Zhangfei Gao <zhangfei.gao@linaro.org>
2018-10-16 11:38:22 +00:00
Marcin Piwowarczyk
dc7303a135 Be compilance with latest oslo.messaging
Support for the [oslo_messaging_rabbit] section has been removed in
oslo.messaging==9.0.0 package [1].
That's why integration tests failures on build devstack step.

This is the moment where we need to use the transport_url directive
in the [DEFAULT] section instead.

Moreover rpc_backend property (which was used in trove to enable fake
RPC backend) has been removed from DEFAULT section and API tests
fails on TIME_OUT as they starts on rabbit RPC backend.

Fake RPC can be now configured as described here [2]:
transport_url = 'fake:/'

[1] https://docs.openstack.org/releasenotes/oslo.messaging/unreleased.html
[2] https://docs.openstack.org/oslo.messaging/latest/configuration/conffixture.html

Change-Id: Id6c5a9198d5a213cb085407a1d8b534e7c755f69
Signed-off-by: Marcin Piwowarczyk <m.piwowarczy@samsung.com>
2018-10-13 20:00:16 +02:00
Zuul
56050fce95 Merge "More reliable gpg keys importing in DIB elements" 2018-07-26 11:49:02 +00:00
Zhao Chao
d2868a59cb Migrate to Zuul v3 native job definitions
This patch migrates the legacy gate jobs to Zuul v3 native ones(a new
trove-devstack role is created for all those legecy jobs, and the
previous way of running gate jobs is still kept as we still cannot
migrate the genade job to a Zuul v3 native one), following the
documents below:
https://docs.openstack.org/infra/manual/zuulv3.html
https://docs.openstack.org/devstack/latest/zuul_ci_jobs_migration.html

Change-Id: I10a1adae3a79b7322118fce180119f0ee5ff2e1b
Signed-off-by: Zhao Chao <zhaochao1984@gmail.com>
2018-07-17 10:29:55 +08:00
Zhao Chao
8057b2d926 More reliable gpg keys importing in DIB elements
We're still seeing apt-key failed to import gpg keys these days during
the images building in the gate jobs, the problem is keys.gnupg.net and
keyserver.ubuntu.com are both not stable according to [1] and [2], it's
better to adopt pool.sks-keyservers.net instead and with simple retries.

To reduce code duplication, this common apt-key importing function is
also moved to ubuntu-guest as an environment snippet.

[1] https://www.gnupg.org/faq/gnupg-faq.html#new_user_default_keyserver
[2] https://sks-keyservers.net/overview-of-pools.php

Closes-Bug: #1579094

Change-Id: I0fe200d140f6f9c4d423dd498797a225e3295a71
Signed-off-by: Zhao Chao <zhaochao1984@gmail.com>
2018-07-16 12:03:16 +08:00
Fan Zhang
4279fd253b Remove pycrypto from requirements
Remove pycrypto completely from requirements and fix the
related tests.

Closes-Bug: #1749574
Depends-On: I5c0c1a238023c116af5a84d899e629f1c7c3513f
Change-Id: Ibfedd9e2ab0a5e78959108112f57103a089f02d1
Signed-off-by: Fan Zhang <zh.f@outlook.com>
2018-06-14 12:56:05 +08:00
Zuul
9cdb08cb7b Merge "Switch to cryptography from pycrypto" 2018-06-13 06:05:19 +00:00
Zhao Chao
43b5807f2d Add a hook for restore process to check if successful
1.This is the same solution with the backup process. Previously we only
check the stderr of the restore command, but this method is not
reliable, and the changes during percona-xtrabackup 2.4.11:
https://jira.percona.com/browse/PXB-1542
causes the gate jobs failed in creating a slave MySQL instance.

The new hook currently is only used by InnoBackupEx restore runner with
checking the exit status and stderr output of xbstream.

2.with[1] merged,this makes DIB_CLOUD_IMAGES more flexible, but it
break trovestack build image, now we need to specify a more
detailed path to DIB_CLOUD_IMAGES to get the appropriate content.

[1]:https://review.openstack.org/#/c/568697/

Co-Authored-By: Zhao Chao <zhaochao1984@gmail.com>
Co-Authored-By: zhanggang <zhanggang@cmss.chinamobile.com>
Co-Authored-By: jiansong <jian.song@easystack.cn>

Closes-Bug: #1771990

Change-Id: Ibb54d6b5953a775be339fb991a0771961d27eba2
2018-06-07 23:37:44 -07:00
Zhao Chao
46a031e765 Switch to cryptography from pycrypto
PyCrypto isn't active developed for quite a while, cryptography is
recommended instead. This patch does this migration, but still keeps
pycrytpo as a fallback solution.

Random generation is also migrated to os.urandom as the cryptography
document suggests:
https://cryptography.io/en/latest/random-numbers/

Closes-Bug: #1749574

Change-Id: I5c0c1a238023c116af5a84d899e629f1c7c3513f
Co-Authored-By: Fan Zhang <zh.f@outlook.com>
Signed-off-by: Zhao Chao <zhaochao1984@gmail.com>
2018-06-08 10:55:44 +08:00
wangqi
de26c5ec79 drop extra word to fix typo
Change-Id: I6c55ce58bae79da6d2d18a5fd017d4932a637e17
2018-02-24 04:27:57 +00:00
Gaetan Trellu
e15fe71733 Remove security.authorization option from mongos
The "security.authorization" option is only usable for
mongod and not for mongos service.

When this option is applied to /etc/mongos.conf configuration
file, mongos service doesn't start.

Change-Id: Id798d7269ee80706002bbebe2ff666fc2fecdd1e
Closes-Bug: #1748322
2018-02-08 18:54:06 -05:00
Zuul
692100db17 Merge "Accept the IBM DB2 license during the DIB process" 2018-02-02 23:04:18 +00:00
Zuul
4b95da49cb Merge "Improve Vertica 9.x support" 2018-02-02 18:13:38 +00:00
Gaetan Trellu
c334c5e46d Accept the IBM DB2 license during the DIB process
During the DB2 installation, the db2_install
process asks a question about the license. Disk
Image Builder will hang until "yes" or "no" is
manually entered.

db2_install script provides the "-y" option to
accept automatically the license.

From db2_install help:
  Specifies that you have read and agreed to the
  license agreement file in the db2/license
  directory on the CD. This parameter is mandatory
  when -n is specified.

Change-Id: I8a8330adc41cbed60c8472f925b10cf816aed44e
Closes-Bug: #1747031
2018-02-02 11:12:02 -05:00
Zuul
278c100240 Merge "Remove hardcoded version in DB2 install path" 2018-02-02 04:51:20 +00:00
Gaetan Trellu
770e311075 Remove hardcoded version in DB2 install path
The DB2 path needs to be changed to something
basic and not using a version number because we
don't want to update the code every time than a
new DB2 version is released.

Updated DB2 version from 10.5 to 11.1, version
10.5 is no more available on IBM website.

Closes-Bug: #1746351
Change-Id: I4fd7c66c3cd5ff8d540e994ada979b59c7def35e
2018-02-01 12:41:48 -05:00
Gaetan Trellu
3bc8a63c33 Improve Vertica 9.x support
During the Vertica installation, the install script try to get AWS
metadata even if --ignore-aws-instance-type is specified.

The only way to avoid the _get_macs error is to define
--ignore-install-config option.

During the cluster creation, if the process is too long, Vertica
ask a question:
  Do you want to continue waiting? (yes/no) [yes]

From the adminTools documentation there is no non-interactive or
timeout options. The only way to have this question answered is
to run "echo yes | " before the "adminTools -t create_db" command.

The only available version from Vertica website is 9.0.1.

Closes-Bug: #1745493
Change-Id: I94472082fac606f4ff14f26c2ca2b620ef0a3bbb
2018-02-01 12:32:38 -05:00
Pierre Blanc
c1b82ffa9c Revert Cassandra version to 2 on ubuntu element
The element is configured to use version 3 of cassandra, unfortunatly
trove is actually not compatible with version 3.

Closes-Bug: #1745056
Change-Id: Ic22202eeae6f404433a84def8cf5da0e5dc28c7b
2018-01-26 11:47:15 -05:00
Zuul
d5186ca06f Merge "Fix Cassandra element" 2018-01-26 06:10:47 +00:00
Luke Browning
2ae9da7623 Fix Cassandra element
The pip command was not installed.  Added a couple of
prerequisites (snappy, Cython) and db management package.
The cassandra-tools package provides commands for enabling
and disabling incremental backup, snapshots, and many other
features.  It is provided by the cassandra community.

Change-Id: If0f807878d6f3da17e33bc9e40ecb9563de61a25
2018-01-25 18:18:07 -06:00
Luke Browning
264f8be347 Change file permissions on element script
Needs to be executable to run

Change-Id: I1302d8ea805f3dc023ef7224ed4aa9b4d06ff371
2018-01-25 16:13:18 -06:00
Zuul
808182b56c Merge "Missing element-deps files for xenial" 2018-01-25 01:42:17 +00:00