61 Commits

Author SHA1 Message Date
Lingxian Kong
aa1d4d2246 Datastore containerization
Significant changes:

* Using docker image to install datastore.
* Datastore image is common to different datastores.
* Using backup docker image to do backup and restore.
* Support MariaDB replication
* Set most of the functional jobs as non-voting as nested
  virtualization is not supported in CI.

Change-Id: Ia9c97a63a961eebc336b70d28dc77638144c1834
2020-05-27 10:31:50 +12:00
Lingxian Kong
523d66e8fd Fix CI job
* Fix pep8: ambiguous variable name
* Install Python 3.7 for trove guest image. oslo.concurrency requires
  python 3.6 or newer. See
  https://bugs.launchpad.net/tripleo/+bug/1861803
* Mark tempest job non-voting temporarily because of some tempest bugs.

Change-Id: I6d316779cc7220a855ce187437056b667bbe1f75
2020-05-25 17:37:38 +12:00
Lingxian Kong
7d78c74158 Add python-troveclient in requirements.txt
Change-Id: I2bd0d81171df85c05423e19ab2e4afa82fcf880e
2020-03-15 20:57:39 +13:00
Lingxian Kong
602c4d42de Improve the function tests
- Execute test groups in serial to make sure no more than 2 database
  instance are created at the same time.
- Remove some unneccesary tests
- Remove unneeded datastore, e.g. 'Test_Datastore_1'
- Remove unsupported trovestack subcommands
- Move unsupported DIB elements to the 'deprecated-elements' folder
- Decrease default value of 'agent_call_high_timeout' to 5min
- Add initial_deplay for pooling task
- Use socket file to connect with database instead of using localhost
  IP

Change-Id: Ie5030a671fbeb453eafa6cbe04e08da7b52e33c9
2020-01-28 14:27:52 +13:00
Lingxian Kong
72524f9d85 Support PostgreSQL 12
Change-Id: Ic1d4982a479c2154d7ddcc807d863e2a37fea506
2019-12-18 22:13:32 +00:00
Zuul
94331620a3 Merge "Support pip3 and run on guest-agent service for redis" 2019-12-06 09:57:17 +00:00
Nguyen Van Trung
d5549f30cf Support pip3 and run on guest-agent service for redis
This patch will solve some following items:
- Support python3/ pip3
- Permit redis is running with guest-agent service

Tested with standalone, replicate and cluster successfully.

Change-Id: I81b72fae1c31ce2995232b52460d0ad0d6f894be
2019-12-05 09:35:12 +00:00
Nguyen Van Trung
1d314fdf91 Support pip3-virtualenv during image creation
This patch will auto-process with `pip-and-virtualenv`
element when use `guest-agent` in python3. We also can add
this element in disk-image-create cli which instead of
this patch to fix following issue:

   2019-09-12 09:19:58.674 | + python3 -m virtualenv /opt/guest-agent-venv
   2019-09-12 09:19:58.752 | /usr/bin/python3: No module named virtualenv

Change-Id: I21090320ba7396979cee31431de7198150dd763e
2019-12-05 09:34:52 +00:00
Lingxian Kong
11b0b8d6f2 Fix mariadb CI - trove-scenario-mariadb-single
- Trove supports MariaDB 10.4
- From MariaDB 10.3, Mariabackup is used instead of Percona XtraBackup
  for backup functionality
- Some log improvements

Change-Id: Ibaa6fd7273b98451097b32fb6b881008a236be9f
2019-12-05 07:48:26 +13:00
Lingxian Kong
8fc0b7695d Build reusable Trove guest image for dev
When building the guest image for dev_mode=true, the controller IP
address is injected to the image in order for the guest-agent to
download Trove code during initialization. As a result, we have to build
guest image each time in the Trove CI because the image build relies on
the devstack host IP address.

If we could remove the dependency, we can build the image(in
devmode=true) for a specific datastore once and use that image for all
related CI jobs, which could save some time for Trove CI.

Fix the current CI issue as well.

Change-Id: If23f4f179a6ab72cfb35e4c45d55142fedb76498
2019-12-05 00:45:56 +13:00
Lingxian Kong
d244c9d666 Support to specify branch when building the image
- Explicity specify branch name for building non-dev image
- Use branch name for building dev image in devstack
- Update the doc

Candidate backport for stable/train

Change-Id: I4d2fe38fac34c75b42234fa7a20aba51e6dc1c4e
2019-10-10 21:55:53 +13:00
Nguyen Van Trung
38efc97251 Fix issue with pip2 command and update pip3 for mongodb
This patch will fix following issue logs:

   2019-09-02 15:11:13.121 | dib-run-parts Running /tmp/in_target.d/install.d/25-trove-mongo-dep
   2019-09-02 15:11:13.124 | + pip2 install pymongo
   2019-09-02 15:11:13.125 | /tmp/in_target.d/install.d/25-trove-mongo-dep: line 9: pip2: command not found

And move on pip3.

Change-Id: I7cf16820091a0f977b41e3222abb866a9ae66166
2019-09-12 10:11:30 +00:00
Zuul
1b159c7d6a Merge "Improve image build" 2019-09-04 05:03:24 +00:00
Lingxian Kong
89386032ae Fix MariaDB image build
The error in MariaDB installation during the guest image building:

```
apt-get -y install socat percona-xtrabackup
percona-xtrabackup : Depends: libcurl4 (>= 7.16.3) but it is not installable
                     Depends: libgcrypt20 (>= 1.8.0) but 1.6.5-2ubuntu0.5 is to be installed
                     Depends: libssl1.1 (>= 1.1.0) but it is not installable
E: Unable to correct problems, you have held broken packages.
```

This patch fixes the issue based on the process described in
https://mariadb.com/kb/en/library/installing-mariadb-deb-files

Change-Id: I368bddabffcc0f11f7181b3a33be1f213f0bdaba
2019-09-04 09:38:37 +12:00
Lingxian Kong
c23c1b9d0c Improve image build
- Remove network-scripts package from guest agent, otherwise image
  creation will fail with network-scripts not found.
- Improve the way to install percona-xtrabackup package on Xenial to
  avoid the failure: "write error: No space left on device"

Change-Id: I6d8de1ad0f675db9635f4762cefc26169e738bbc
2019-09-02 22:58:28 +12:00
Lingxian Kong
0ef474a70c Support keypair in devstack
Since Trove already supports to specify a Nova keypair when creating
instance for management convenience, devstack needs to be changed to
create the management keypair and add to Trove config file.

One extra change in this patch is to use a single config file for Trove
API, task-manager and conductor.

Change-Id: I1e6c4f4305104815bdf89b31776a4955de61bc89
Story: 2005429
Task: 30463
2019-08-29 15:41:58 +12:00
Lingxian Kong
7b9a752eb0 Support python3 in guest agent
Story: #2006455
Task: #36376
Change-Id: I62323ac1c1edfbf0378d0c89e156fdb3a879aa88
2019-08-27 15:03:45 +12:00
Lingxian Kong
dfa5ce93d5 Improve devmode=flase when building the image
During debugging, the following changes are also included:

- Support to specify an image ID to run the integration test.
- Fix the reboot function bug.
- Remove the unsuccessful restart test.

How to run integration test with dev_mode=false:

    ADMIN_PASSWORD=password \
    SERVICE_PASSWORD=password \
    DEV_MODE=false \
    /opt/stack/trove/integration/scripts/trovestack gate-tests mysql mysql

Change-Id: I31d4ee579a554f4c98f9facb9fd4b7779665a3dd
2019-08-25 23:11:54 +12:00
Mark Kirkwood
7d2e986778 Fix 31-fix-init-script for Postgresql
Ironically the fix is never applied as the systemd file uses
/run not /var/run.

Change-Id: Ie6022a13c181f5c27e19b54bd49d9a77c90dc1ac
2019-08-14 22:09:56 +00:00
Lingxian Kong
eafd303e0a Use opendev.org instead of git.openstack.org
After moving to opendev.org, the following command fails when building
the Trove guest image:

curl -o /opt/stack/devstack/upper-constraints.txt \
  'https://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt?h=master'

https://opendev.org/openstack/requirements/raw/branch/master/upper-constraints.txt
should be used instead.

Other places using git.openstack.org are also changed.

Change-Id: Id357dd61e229e3336219723c1c1ae6b50e983f26
2019-04-22 00:05:41 +12:00
Lingxian Kong
798af4d02a Improve trove guest agent image building
- Add a new element 'guest-agent' for image building. This element is
  used when dev_mode=false, so that the trove code is downloaded into
  the image during the building phase rather than during the guest
  agent initialization.
- Improve trovestack sub-command 'build-image'.

  ./trovestack build-image ${datastore_type} \
     ${guest_os} \
     ${guest_release} \
     ${dev_mode}

- Improve documentation.

Story: #2005387
Task: #30375

Change-Id: I9d7acbd6a97f8c01b48b0f2cf94398d549d89124
2019-04-10 09:29:40 +12:00
Zhangfei Gao
2d7ecd9385 Fix build ubuntu-geust issue on arm64
To build ubuntu-geust on arm64, libffi-dev & libssl-dev are required

Closes-Bug: #1797014

Change-Id: Ia215deac11976fc9fc67e6d823ae11d181bf8083
Signed-off-by: Zhangfei Gao <zhangfei.gao@linaro.org>
2018-10-16 11:38:22 +00:00
Zhao Chao
8057b2d926 More reliable gpg keys importing in DIB elements
We're still seeing apt-key failed to import gpg keys these days during
the images building in the gate jobs, the problem is keys.gnupg.net and
keyserver.ubuntu.com are both not stable according to [1] and [2], it's
better to adopt pool.sks-keyservers.net instead and with simple retries.

To reduce code duplication, this common apt-key importing function is
also moved to ubuntu-guest as an environment snippet.

[1] https://www.gnupg.org/faq/gnupg-faq.html#new_user_default_keyserver
[2] https://sks-keyservers.net/overview-of-pools.php

Closes-Bug: #1579094

Change-Id: I0fe200d140f6f9c4d423dd498797a225e3295a71
Signed-off-by: Zhao Chao <zhaochao1984@gmail.com>
2018-07-16 12:03:16 +08:00
wangqi
de26c5ec79 drop extra word to fix typo
Change-Id: I6c55ce58bae79da6d2d18a5fd017d4932a637e17
2018-02-24 04:27:57 +00:00
Gaetan Trellu
e15fe71733 Remove security.authorization option from mongos
The "security.authorization" option is only usable for
mongod and not for mongos service.

When this option is applied to /etc/mongos.conf configuration
file, mongos service doesn't start.

Change-Id: Id798d7269ee80706002bbebe2ff666fc2fecdd1e
Closes-Bug: #1748322
2018-02-08 18:54:06 -05:00
Gaetan Trellu
c334c5e46d Accept the IBM DB2 license during the DIB process
During the DB2 installation, the db2_install
process asks a question about the license. Disk
Image Builder will hang until "yes" or "no" is
manually entered.

db2_install script provides the "-y" option to
accept automatically the license.

From db2_install help:
  Specifies that you have read and agreed to the
  license agreement file in the db2/license
  directory on the CD. This parameter is mandatory
  when -n is specified.

Change-Id: I8a8330adc41cbed60c8472f925b10cf816aed44e
Closes-Bug: #1747031
2018-02-02 11:12:02 -05:00
Zuul
278c100240 Merge "Remove hardcoded version in DB2 install path" 2018-02-02 04:51:20 +00:00
Gaetan Trellu
770e311075 Remove hardcoded version in DB2 install path
The DB2 path needs to be changed to something
basic and not using a version number because we
don't want to update the code every time than a
new DB2 version is released.

Updated DB2 version from 10.5 to 11.1, version
10.5 is no more available on IBM website.

Closes-Bug: #1746351
Change-Id: I4fd7c66c3cd5ff8d540e994ada979b59c7def35e
2018-02-01 12:41:48 -05:00
Pierre Blanc
c1b82ffa9c Revert Cassandra version to 2 on ubuntu element
The element is configured to use version 3 of cassandra, unfortunatly
trove is actually not compatible with version 3.

Closes-Bug: #1745056
Change-Id: Ic22202eeae6f404433a84def8cf5da0e5dc28c7b
2018-01-26 11:47:15 -05:00
Zuul
d5186ca06f Merge "Fix Cassandra element" 2018-01-26 06:10:47 +00:00
Luke Browning
2ae9da7623 Fix Cassandra element
The pip command was not installed.  Added a couple of
prerequisites (snappy, Cython) and db management package.
The cassandra-tools package provides commands for enabling
and disabling incremental backup, snapshots, and many other
features.  It is provided by the cassandra community.

Change-Id: If0f807878d6f3da17e33bc9e40ecb9563de61a25
2018-01-25 18:18:07 -06:00
Luke Browning
264f8be347 Change file permissions on element script
Needs to be executable to run

Change-Id: I1302d8ea805f3dc023ef7224ed4aa9b4d06ff371
2018-01-25 16:13:18 -06:00
Zuul
808182b56c Merge "Missing element-deps files for xenial" 2018-01-25 01:42:17 +00:00
Manoj Kumar
3934e301ac Missing element-deps files for xenial
The dependency files for cassandra, couchbase and couchdb
were missing. Added them.

Change-Id: Ib7684dabe41f505c34a598678037cebb4526cae0
2018-01-23 22:38:25 +00:00
Manoj Kumar
61832ea5e0 Adding missing dependencay
For percona on xenial, the element-deps file was missing.

Change-Id: I1e44986a5caaab8ddf0a2891a3d692d6f4684153
2018-01-23 20:08:50 +00:00
Zuul
de547c2e71 Merge "Unable to build cassandra images" 2018-01-23 10:02:22 +00:00
Zuul
19253e2c89 Merge "Add missing permission on 10-fix-mycnf for Percona" 2018-01-22 21:25:45 +00:00
Gaëtan Trellu
32830483b5 Add missing permission on 10-fix-mycnf for Percona
Related to Luke's comment on this review:
https://review.openstack.org/#/c/535387/

Set executable permission on 10-fix-mycnf file.

Change-Id: I15bebe2ea5c4b9ade53147cb7859052895804181
2018-01-22 11:02:23 -05:00
Pierre Blanc
8e68d163a8 Unable to build cassandra images
A wrong comparison option is used to test the existence of variables.

This issue block the build for cassandra image.

Closes-bug: #1744501
Change-Id: I8f801c78a260bb82b78a9ce3275b264b7acd1c11
2018-01-20 17:10:03 -05:00
Zuul
91cb668adb Merge "Upgrade Postgresql support to v9.6" 2018-01-20 05:09:53 +00:00
zhanggang
75bc5a48df Upgrade Postgresql support to v9.6
Modified existing Postgresql elements to support v9.6.
This change will no longer support pg<9.6, so "wal_log_hints"
is supported by default. Remove "checkpoint_segments" from
config.template because the parameter is no longer supported
from 9.5. By defult, "trovestack kick-start postgresql" will
create a pg-9.6 image. The steps in "30-postgresql" has been
tested in ubuntu trusty. Since yum.postgresql.org is no longer support
fedora-22[1], trovestack use fedora-27 by default, and steps in
"10-postgresql" has been tested too.
Releated commits: https://review.openstack.org/#/c/409524/
[1] https://yum.postgresql.org/9.6/fedora/
Closes-bug: 1649084

Change-Id: I65d93635ad45838522abed014825d1f7e567c30c
2018-01-19 17:00:01 -06:00
Gaetan Trellu
fe49104288 Guest agent won't start on Xenial Percona 5.7
Trove guest agent on Ubuntu Xenial Percona 5.7 image won't start
because of a missing header/section in "/etc/mysql/my.cnf" file.

To fix that, we only have to check if a section exists, if not
a [mysql] section is automatically created.

Change-Id: If3b1166fc04985763ae522f5eac022d53f2a366f
Closes-Bug: #1744107
2018-01-18 11:01:35 -05:00
Pierre Blanc
396619eae8 Fix systemd service mongodb on xenial element
We have several issues on mongodb element for xenial.

This change propose these actions:
  - fix a typo on check-numa script.
  - fix variable issue on the systemd parameter ExecStart.
  - create a dependency and an order between numa-check and mongo service.

Closes-bug: #1743612
Change-Id: I50123dad58b9a368f528d086fa28f6abc86464a6
2018-01-16 12:15:08 -05:00
Zuul
7061c37854 Merge "Update ubuntu cassandra to supported repo and version" 2018-01-12 04:25:36 +00:00
Zuul
5ec6236bae Merge "Adding mongodb support to xenial" 2018-01-11 01:05:21 +00:00
Luke Browning
36926f39e6 Add support for MySQL 5.7 on Ubuntu Xenial
Ubuntu Trusty continues to get MySQL 5.6.

The selection of the MySQL version is based on the version
of Ubuntu that is installed.  If the script trovestack is
invoked on Xenial, the version that is chosen is 5.7.  If
it is invoked on Trusty, then version 5.6 is chosen.

The only thing that is eliminated is the dubious combination
of MySQL 5.6 on Ubuntu Xenial.  This combination is probably
not supported by Ubuntu as it is down level from their
standard offering is probably not tested anywhere.

Note Xenial does not provide the 5.6 version of mysql.  This
was accomplished by initializing the Apt repo with the
Trusty software repository.

The size of the root file system is increased from 3 to 4 GBs
as the Mysql Xenial image does not fit in 3 GBs. This has an
impact on the flavors that are used by Trove for testing as the
name of the flavor includes the size of the root file system.

This is turn caused a change to each of the db specific test
config files as the trove falvors are referenced in them.

Change-Id: I4f4e497208b8f4728580e48239a8ae208e0a96dd
2018-01-08 14:57:56 -06:00
Zuul
ccb6752f69 Merge "Configure guestagent on Ubuntu guest images to use CA certificates" 2017-12-03 18:29:20 +00:00
Luke Browning
2ff5c83009 Add #!/bin/bash to /etc/rc.local
Lack of shell designation in script results in a degraded systemctl
state on Xenial.  Make this change to all element instances for
code maintenance purposes.

Change-Id: I7bcbd4bdd1623a18d1cb01a83aa7bd5a76883f7f
2017-11-20 11:33:38 -06:00
Luke Browning
62a0b47525 Utilize Ubuntu's hardware enablement stack for trovestack
This brings in the latest fixes which is often important for
nested virtualization.  It is required for ppc64el.  This stack
is installed via a new post-install.d script in the trove
integration element ubuntu-xenial-guest.

A user that is manually creating datastore images may install
this stack by setting the following environment variable prior
to running the disk-image-create command.

export DIB_USE_HWE_KERNEL=true

Change-Id: Iaf01c543ea210bd4725ab909aff6884df451b198
2017-11-16 16:45:47 -06:00
Tomasz Nowak
043ad08237 Configure guestagent on Ubuntu guest images to use CA certificates
Currently there is no way to provide SSL configuration for Trove, so it
fails e.g. when uploading backups to a secured Swift endpoint. This
patch sets an environment variable (REQUESTS_CA_BUNDLE [1]) understood
by Requests library for Python, so all HTTPS calls done by trove-guest
service will trust the provided CAs.

For Ubuntu Xenial and Fedora a systemd drop-in sets this environment
variable for trove-guest service, so it uses Ubuntu's/Fedora's system
certificate store to validate server certificates.

For Ubuntu Trusty the upstart script is modified to build and use a
bundle file from certificates in /usr/local/share/ca-certificates,
because Requests library doesn't support CA directories in such old
Python versions.

On Ubuntu systems the custom certificates are taken from
/usr/local/share/ca-certificates; please use PEM format, .crt extension
and call update-ca-certificates.

On Fedora systems custom certificates can be put in
/usr/share/pki/ca-trust-source/anchors; please use PEM format, .pem
extension and call update-ca-trust.

[1] http://docs.python-requests.org/en/master/user/advanced/#ssl-cert-verification

Change-Id: I0025e7c72fa2d863ae9540941956b1ab63bcc636
2017-11-09 09:29:16 +00:00