Significant changes:
* Using docker image to install datastore.
* Datastore image is common to different datastores.
* Using backup docker image to do backup and restore.
* Support MariaDB replication
* Set most of the functional jobs as non-voting as nested
virtualization is not supported in CI.
Change-Id: Ia9c97a63a961eebc336b70d28dc77638144c1834
* Fix pep8: ambiguous variable name
* Install Python 3.7 for trove guest image. oslo.concurrency requires
python 3.6 or newer. See
https://bugs.launchpad.net/tripleo/+bug/1861803
* Mark tempest job non-voting temporarily because of some tempest bugs.
Change-Id: I6d316779cc7220a855ce187437056b667bbe1f75
- Execute test groups in serial to make sure no more than 2 database
instance are created at the same time.
- Remove some unneccesary tests
- Remove unneeded datastore, e.g. 'Test_Datastore_1'
- Remove unsupported trovestack subcommands
- Move unsupported DIB elements to the 'deprecated-elements' folder
- Decrease default value of 'agent_call_high_timeout' to 5min
- Add initial_deplay for pooling task
- Use socket file to connect with database instead of using localhost
IP
Change-Id: Ie5030a671fbeb453eafa6cbe04e08da7b52e33c9
This patch will solve some following items:
- Support python3/ pip3
- Permit redis is running with guest-agent service
Tested with standalone, replicate and cluster successfully.
Change-Id: I81b72fae1c31ce2995232b52460d0ad0d6f894be
This patch will auto-process with `pip-and-virtualenv`
element when use `guest-agent` in python3. We also can add
this element in disk-image-create cli which instead of
this patch to fix following issue:
2019-09-12 09:19:58.674 | + python3 -m virtualenv /opt/guest-agent-venv
2019-09-12 09:19:58.752 | /usr/bin/python3: No module named virtualenv
Change-Id: I21090320ba7396979cee31431de7198150dd763e
- Trove supports MariaDB 10.4
- From MariaDB 10.3, Mariabackup is used instead of Percona XtraBackup
for backup functionality
- Some log improvements
Change-Id: Ibaa6fd7273b98451097b32fb6b881008a236be9f
When building the guest image for dev_mode=true, the controller IP
address is injected to the image in order for the guest-agent to
download Trove code during initialization. As a result, we have to build
guest image each time in the Trove CI because the image build relies on
the devstack host IP address.
If we could remove the dependency, we can build the image(in
devmode=true) for a specific datastore once and use that image for all
related CI jobs, which could save some time for Trove CI.
Fix the current CI issue as well.
Change-Id: If23f4f179a6ab72cfb35e4c45d55142fedb76498
- Explicity specify branch name for building non-dev image
- Use branch name for building dev image in devstack
- Update the doc
Candidate backport for stable/train
Change-Id: I4d2fe38fac34c75b42234fa7a20aba51e6dc1c4e
This patch will fix following issue logs:
2019-09-02 15:11:13.121 | dib-run-parts Running /tmp/in_target.d/install.d/25-trove-mongo-dep
2019-09-02 15:11:13.124 | + pip2 install pymongo
2019-09-02 15:11:13.125 | /tmp/in_target.d/install.d/25-trove-mongo-dep: line 9: pip2: command not found
And move on pip3.
Change-Id: I7cf16820091a0f977b41e3222abb866a9ae66166
The error in MariaDB installation during the guest image building:
```
apt-get -y install socat percona-xtrabackup
percona-xtrabackup : Depends: libcurl4 (>= 7.16.3) but it is not installable
Depends: libgcrypt20 (>= 1.8.0) but 1.6.5-2ubuntu0.5 is to be installed
Depends: libssl1.1 (>= 1.1.0) but it is not installable
E: Unable to correct problems, you have held broken packages.
```
This patch fixes the issue based on the process described in
https://mariadb.com/kb/en/library/installing-mariadb-deb-files
Change-Id: I368bddabffcc0f11f7181b3a33be1f213f0bdaba
- Remove network-scripts package from guest agent, otherwise image
creation will fail with network-scripts not found.
- Improve the way to install percona-xtrabackup package on Xenial to
avoid the failure: "write error: No space left on device"
Change-Id: I6d8de1ad0f675db9635f4762cefc26169e738bbc
Since Trove already supports to specify a Nova keypair when creating
instance for management convenience, devstack needs to be changed to
create the management keypair and add to Trove config file.
One extra change in this patch is to use a single config file for Trove
API, task-manager and conductor.
Change-Id: I1e6c4f4305104815bdf89b31776a4955de61bc89
Story: 2005429
Task: 30463
During debugging, the following changes are also included:
- Support to specify an image ID to run the integration test.
- Fix the reboot function bug.
- Remove the unsuccessful restart test.
How to run integration test with dev_mode=false:
ADMIN_PASSWORD=password \
SERVICE_PASSWORD=password \
DEV_MODE=false \
/opt/stack/trove/integration/scripts/trovestack gate-tests mysql mysql
Change-Id: I31d4ee579a554f4c98f9facb9fd4b7779665a3dd
- Add a new element 'guest-agent' for image building. This element is
used when dev_mode=false, so that the trove code is downloaded into
the image during the building phase rather than during the guest
agent initialization.
- Improve trovestack sub-command 'build-image'.
./trovestack build-image ${datastore_type} \
${guest_os} \
${guest_release} \
${dev_mode}
- Improve documentation.
Story: #2005387
Task: #30375
Change-Id: I9d7acbd6a97f8c01b48b0f2cf94398d549d89124
We're still seeing apt-key failed to import gpg keys these days during
the images building in the gate jobs, the problem is keys.gnupg.net and
keyserver.ubuntu.com are both not stable according to [1] and [2], it's
better to adopt pool.sks-keyservers.net instead and with simple retries.
To reduce code duplication, this common apt-key importing function is
also moved to ubuntu-guest as an environment snippet.
[1] https://www.gnupg.org/faq/gnupg-faq.html#new_user_default_keyserver
[2] https://sks-keyservers.net/overview-of-pools.php
Closes-Bug: #1579094
Change-Id: I0fe200d140f6f9c4d423dd498797a225e3295a71
Signed-off-by: Zhao Chao <zhaochao1984@gmail.com>
The "security.authorization" option is only usable for
mongod and not for mongos service.
When this option is applied to /etc/mongos.conf configuration
file, mongos service doesn't start.
Change-Id: Id798d7269ee80706002bbebe2ff666fc2fecdd1e
Closes-Bug: #1748322
During the DB2 installation, the db2_install
process asks a question about the license. Disk
Image Builder will hang until "yes" or "no" is
manually entered.
db2_install script provides the "-y" option to
accept automatically the license.
From db2_install help:
Specifies that you have read and agreed to the
license agreement file in the db2/license
directory on the CD. This parameter is mandatory
when -n is specified.
Change-Id: I8a8330adc41cbed60c8472f925b10cf816aed44e
Closes-Bug: #1747031
The DB2 path needs to be changed to something
basic and not using a version number because we
don't want to update the code every time than a
new DB2 version is released.
Updated DB2 version from 10.5 to 11.1, version
10.5 is no more available on IBM website.
Closes-Bug: #1746351
Change-Id: I4fd7c66c3cd5ff8d540e994ada979b59c7def35e
The element is configured to use version 3 of cassandra, unfortunatly
trove is actually not compatible with version 3.
Closes-Bug: #1745056
Change-Id: Ic22202eeae6f404433a84def8cf5da0e5dc28c7b
The pip command was not installed. Added a couple of
prerequisites (snappy, Cython) and db management package.
The cassandra-tools package provides commands for enabling
and disabling incremental backup, snapshots, and many other
features. It is provided by the cassandra community.
Change-Id: If0f807878d6f3da17e33bc9e40ecb9563de61a25
Related to Luke's comment on this review:
https://review.openstack.org/#/c/535387/
Set executable permission on 10-fix-mycnf file.
Change-Id: I15bebe2ea5c4b9ade53147cb7859052895804181
A wrong comparison option is used to test the existence of variables.
This issue block the build for cassandra image.
Closes-bug: #1744501
Change-Id: I8f801c78a260bb82b78a9ce3275b264b7acd1c11
Modified existing Postgresql elements to support v9.6.
This change will no longer support pg<9.6, so "wal_log_hints"
is supported by default. Remove "checkpoint_segments" from
config.template because the parameter is no longer supported
from 9.5. By defult, "trovestack kick-start postgresql" will
create a pg-9.6 image. The steps in "30-postgresql" has been
tested in ubuntu trusty. Since yum.postgresql.org is no longer support
fedora-22[1], trovestack use fedora-27 by default, and steps in
"10-postgresql" has been tested too.
Releated commits: https://review.openstack.org/#/c/409524/
[1] https://yum.postgresql.org/9.6/fedora/
Closes-bug: 1649084
Change-Id: I65d93635ad45838522abed014825d1f7e567c30c
Trove guest agent on Ubuntu Xenial Percona 5.7 image won't start
because of a missing header/section in "/etc/mysql/my.cnf" file.
To fix that, we only have to check if a section exists, if not
a [mysql] section is automatically created.
Change-Id: If3b1166fc04985763ae522f5eac022d53f2a366f
Closes-Bug: #1744107
We have several issues on mongodb element for xenial.
This change propose these actions:
- fix a typo on check-numa script.
- fix variable issue on the systemd parameter ExecStart.
- create a dependency and an order between numa-check and mongo service.
Closes-bug: #1743612
Change-Id: I50123dad58b9a368f528d086fa28f6abc86464a6
Ubuntu Trusty continues to get MySQL 5.6.
The selection of the MySQL version is based on the version
of Ubuntu that is installed. If the script trovestack is
invoked on Xenial, the version that is chosen is 5.7. If
it is invoked on Trusty, then version 5.6 is chosen.
The only thing that is eliminated is the dubious combination
of MySQL 5.6 on Ubuntu Xenial. This combination is probably
not supported by Ubuntu as it is down level from their
standard offering is probably not tested anywhere.
Note Xenial does not provide the 5.6 version of mysql. This
was accomplished by initializing the Apt repo with the
Trusty software repository.
The size of the root file system is increased from 3 to 4 GBs
as the Mysql Xenial image does not fit in 3 GBs. This has an
impact on the flavors that are used by Trove for testing as the
name of the flavor includes the size of the root file system.
This is turn caused a change to each of the db specific test
config files as the trove falvors are referenced in them.
Change-Id: I4f4e497208b8f4728580e48239a8ae208e0a96dd
Lack of shell designation in script results in a degraded systemctl
state on Xenial. Make this change to all element instances for
code maintenance purposes.
Change-Id: I7bcbd4bdd1623a18d1cb01a83aa7bd5a76883f7f
This brings in the latest fixes which is often important for
nested virtualization. It is required for ppc64el. This stack
is installed via a new post-install.d script in the trove
integration element ubuntu-xenial-guest.
A user that is manually creating datastore images may install
this stack by setting the following environment variable prior
to running the disk-image-create command.
export DIB_USE_HWE_KERNEL=true
Change-Id: Iaf01c543ea210bd4725ab909aff6884df451b198
Currently there is no way to provide SSL configuration for Trove, so it
fails e.g. when uploading backups to a secured Swift endpoint. This
patch sets an environment variable (REQUESTS_CA_BUNDLE [1]) understood
by Requests library for Python, so all HTTPS calls done by trove-guest
service will trust the provided CAs.
For Ubuntu Xenial and Fedora a systemd drop-in sets this environment
variable for trove-guest service, so it uses Ubuntu's/Fedora's system
certificate store to validate server certificates.
For Ubuntu Trusty the upstart script is modified to build and use a
bundle file from certificates in /usr/local/share/ca-certificates,
because Requests library doesn't support CA directories in such old
Python versions.
On Ubuntu systems the custom certificates are taken from
/usr/local/share/ca-certificates; please use PEM format, .crt extension
and call update-ca-certificates.
On Fedora systems custom certificates can be put in
/usr/share/pki/ca-trust-source/anchors; please use PEM format, .pem
extension and call update-ca-trust.
[1] http://docs.python-requests.org/en/master/user/advanced/#ssl-cert-verification
Change-Id: I0025e7c72fa2d863ae9540941956b1ab63bcc636