116 Commits

Author SHA1 Message Date
wu.chunyang
09fd606163 synchronize guest logs to controller node
use rsync to synchronize guest logs to the controller node so that we can
find the logs even after the instance is deleted

Add a guest-log-collection timer, this timer triggers
guest-log-collection service every 10 seconds

Change-Id: I264235ce7b7df585358654c46bdac4dcd743fb3d
2023-06-13 10:39:25 +08:00
wu.chunyang
06c8ed94ab Revert "Add debug log for CI tests"
This reverts commit 49c16b889b68039b068eb420f345efdfaafcc1cc.

Reason for revert: this patch doesn't work as expected

Change-Id: I398dd7ac8c628d19b75693dcac0521346e8f5d64
2023-06-07 06:25:10 +00:00
wu.chunyang
49c16b889b Add debug log for CI tests
When the CI fails, specifically related to the Trove guest agent,
we are unable to find a way to debug it.
this patch synchronizes the guest agent logs to controller nodes.

Change-Id: I24695238075f9b1354ee4c54673d021de481a868
2023-06-06 15:43:44 +08:00
Takashi Kajinami
f62d3197c4 Do not configure removed [DEFAULT] rpc_backend
This option was removed in oslo.messaging 9.0.0[1].

[1] https://docs.openstack.org/releasenotes/oslo.messaging/stein.html#relnotes-9-0-0-stable-stein-upgrade-notes

Change-Id: I11660751532dde117bbf509302fbde594e3911b1
2023-03-20 03:29:35 +00:00
wu.chunyang
1f1e2becfe Using local registry in devstack.
This patch runs a registry container during the installation of
devstack

Change-Id: I5e5a89cd9c65d1ea5b47fece985dbc044bfdec5e
2023-02-26 17:27:17 +08:00
Hirotaka Wakabayashi
6e4dd51c02 Rename api_wsgi.py to app_wsgi.py
This PR changes wsgi script filename to app_wsgi.py.

Original problem is that mod_wsgi script not found after devstack
installation. This bug is created by #854356[1].

--
[1]: https://review.opendev.org/c/openstack/trove/+/854356

Story: 2010264
Task: 46163
Change-Id: I22883c8a8bb7e2818679d667b6868587a040a602
2022-09-02 11:43:00 +09:00
wu.chunyang
44c3cdef5c Rename app.wsgi to app_wsgi.py
1. app.wsgi is a python script and should be end with py extension.
2. Generate trove-wsgi script which is needed by kolla to support
https and ipv6.
3. Ignore build directory in git
4. update docs

Change-Id: Ibbd6ecb8db25aa3a443d3f526f4cf6d418eb26f3
2022-08-25 11:10:33 +08:00
wu.chunyang
0e44c623ea Set os release to focal in devstack
trove built image failed in devstack due to the bionic
os release. in 'Z' cycle, openstack doesn't support
bionic any more.

Change-Id: I1138c4dc346d4b4bbe4d72e72ff8ae59f2fce606
2022-08-11 20:47:42 +08:00
wu.chunyang
a8d9a82df1 Add a check before executing ip replace
sometimes, ROUTER_GW_IP or IPV4_ADDRS_SAFE_TO_USE
may not defined in localrc, we add a check before
executing ip replace command.

Change-Id: I48ca40cc2259e21b8a8f0a9fa763746c30f0e8ab
2022-07-19 22:33:21 +08:00
Dylan McCulloch
f6327f0668 Use os-cloud trove creds
Task: 44414
Story: 2009826
Change-Id: I382b86c341de2d5c4687fbb4cc9cc48b17a03081
2022-02-03 20:00:23 +11:00
Lingxian Kong
5590ecdce0 Show user network ID for getting instance
Change-Id: Ia1e9112ae69e04f8c3e9e9d1b4a0189c743d7448
2021-08-02 14:25:54 +12:00
Lingxian Kong
02971d850b Add periodic task to remove postgres archived wal files
* Added a periodic task for postgresql datastore to clean up the
  archived WAL files.
* Added a check when creating incremental backups for postgresql.
* A new container image ``openstacktrove/db-backup-postgresql:1.1.2``
  is uploaded to docker hub.

Story: 2009066
Task: 42871
Change-Id: I235e2abf8c0405e143ded6fb48017d596b8b41a1
2021-07-23 22:16:20 +12:00
Lingxian Kong
03e5512695 Support customized database container images in DevStack
Story: 2008990
Task: 42643
Change-Id: Ie62e9aa1863d705a60b244981c9dee160d50d2dc
2021-06-19 16:47:31 +12:00
Ghanshyam Mann
bcb8992f99 [goal] Deprecate the JSON formatted policy file
As per the community goal of migrating the policy file
the format from JSON to YAML[1], we need to do two things:

1. Change the default value of '[oslo_policy] policy_file''
config option from 'policy.json' to 'policy.yaml' with
upgrade checks.

2. Deprecate the JSON formatted policy file on the project side
via warning in doc and releasenotes.

Also replace policy.json to policy.yaml ref from doc and tests.

[1]https://governance.openstack.org/tc/goals/selected/wallaby/migrate-policy-format-from-json-to-yaml.html

Change-Id: I790409da69df8479ad2fe152b15c32ba45067c23
2020-12-26 05:30:18 +00:00
Lingxian Kong
cc57e32c48 Use nested virt node for trove functional test
Enable the jobs trove-functional-mysql and trove-tempest.

Change-Id: I930a1468fec5922916af0819698fceb94bddb2f3
2020-11-30 09:48:09 +13:00
Zuul
d38defed9b Merge "Remove use of removed tail_log command" 2020-11-26 01:58:52 +00:00
Lingxian Kong
da2da6969a Support --image-tags for trove-manager datastore_version_update command
image-tags was introduced in commit 1d24b65 but the implementation was
backward incompatible for the CLI `trove-manager datastore_version_update`.

This patch changes that by supporting an optional parameter `--image-tags`.

Change-Id: I583c296f15a453bdfadf10de5c678ac483cbd4dd
2020-11-26 11:06:55 +13:00
Sam Morrison
c615f7ead9 Remove use of removed tail_log command
This was deprecatd in queens and removed recently from devstack

Change-Id: Iac9f6e4cff6780a3983cfb814d3745d9ac43acbb
2020-11-25 10:14:00 +11:00
Lingxian Kong
33c0b64051 Add 'version' to datastore version
This patch is part of story 2008358 implementation.

1. DB schema change
2. trove-manage datastore_version_update subcommand change
3. API change for creating and listing datastore version.

Story: 2008358
Task: 41264

Change-Id: I8069e6f4b972497f2b8be540ea35047d5fb2f9a5
2020-11-17 22:10:56 +13:00
Lingxian Kong
d1af33f17b Support mysql 8.0
* MySQL 5.7 and MySQL 8.0 need different percona-xtrabackup package version.
  Added Percona XtraBackup 8 support for MySQL 8.x backup and restore.
* Construct different backup container image names for MySQL 5.7 and MySQL 8.0
  based on the default option value.
* Two docker images are uploaded for backup/restore:
  openstacktrove/db-backup-mysql5.7:1.0.0 and
  openstacktrove/db-backup-mysql8.0:1.0.0. Trove guest agent can automatically
  choose the approriate one based on the datastore version.
* Added option "secure-file-priv=NULL" in MySQL config template to fix
  https://github.com/docker-library/mysql/issues/541.
* Stop using IDENTIFIED BY in GRANT clause (also REVOKE). Starting with MySQL 8
  creating a user implicitly using the GRANT command is not supported.

Story: #2008275
Task: #41143

Change-Id: Ibdec63324b1b39ba9b8a38dbe529da17bbb06767
2020-10-23 23:50:39 +13:00
Lingxian Kong
1d24b65052 Image tags support in datastore version
Change-Id: I0e51d08515c121e3a7b0e82b6e4c4161bb4fbc4a
2020-10-09 11:43:48 +13:00
Lingxian Kong
cf3e9a6e74 Support to rebuild instance by admin
Change-Id: I48f8c6f997daeb6d82ff62b3325277d86fee2147
2020-08-06 20:38:11 +12:00
Lingxian Kong
aa1d4d2246 Datastore containerization
Significant changes:

* Using docker image to install datastore.
* Datastore image is common to different datastores.
* Using backup docker image to do backup and restore.
* Support MariaDB replication
* Set most of the functional jobs as non-voting as nested
  virtualization is not supported in CI.

Change-Id: Ia9c97a63a961eebc336b70d28dc77638144c1834
2020-05-27 10:31:50 +12:00
Lingxian Kong
429c39890e Support ip address type for instances
Include address type in getting instance response.

* Deprecate confip option network_label_regex as we don't reply on Nova
  to get addresses, network names don't make any sense.
* Add 'addresses' in instance API response, keep 'ip' as is but mark
  it deprecated in API doc, python-troveclient shouldn't break.

Story: 2007562
Task: 39445

Change-Id: Ia0458b5ddae8959ce29c17e444e1a51a026283cd
2020-04-26 09:56:50 +12:00
Lingxian Kong
c952838a9b Support to test non-dev guest image in CI
Make the job non-voting because it may fail when the guest agent code
changes.

Change-Id: Ibd595b5b5c02cf72f1e5f05c79469cd4cb15a3ca
2020-03-16 11:10:05 +13:00
Lingxian Kong
602c4d42de Improve the function tests
- Execute test groups in serial to make sure no more than 2 database
  instance are created at the same time.
- Remove some unneccesary tests
- Remove unneeded datastore, e.g. 'Test_Datastore_1'
- Remove unsupported trovestack subcommands
- Move unsupported DIB elements to the 'deprecated-elements' folder
- Decrease default value of 'agent_call_high_timeout' to 5min
- Add initial_deplay for pooling task
- Use socket file to connect with database instead of using localhost
  IP

Change-Id: Ie5030a671fbeb453eafa6cbe04e08da7b52e33c9
2020-01-28 14:27:52 +13:00
Lingxian Kong
dfe826402b Config admin clients as default
Change-Id: Ib4dc349ab7ed4193189ed6bc2bae699d51064998
2020-01-19 16:11:41 +13:00
Lingxian Kong
72524f9d85 Support PostgreSQL 12
Change-Id: Ic1d4982a479c2154d7ddcc807d863e2a37fea506
2019-12-18 22:13:32 +00:00
Lingxian Kong
a0a10f0b94 Support HEALTHY status for db instance
- 'HEALTHY' means the db service is responsive, 'ACTIVE' means the db
  service is alive.
- Remove the CI job fakemodetests, but will add similar testing task in
  the future.
- Fix the periodic CI job
- Remove MongoDB and  related jobs

Change-Id: I5abe9091ba203297dc87db5fba139179166321f7
2019-12-14 12:55:56 +13:00
Lingxian Kong
d93a410943 Fix delete instance
Deal with the situation when Trove fails to look for the Nova server
when waiting for the db instance ACTIVE

Change-Id: I484d45f24176c89d999864d3eb1c48860b3038bd
2019-12-10 12:18:32 +13:00
Zuul
b2a31c5ac5 Merge "Support incremental backup for MariaDB" 2019-12-08 23:17:08 +00:00
Lingxian Kong
605ff34608 Support incremental backup for MariaDB
Enable the tests in CI as well.

Change-Id: Ie9706d26355bd325baf50ec874f05e6904768a1a
2019-12-08 22:27:36 +13:00
Lingxian Kong
4c551947c9 Move the iptable change from trovestack to devstack
Change-Id: I6e10744f86f19f3c29babc07fda77936c95022ff
2019-12-08 18:50:42 +13:00
Lingxian Kong
8fc0b7695d Build reusable Trove guest image for dev
When building the guest image for dev_mode=true, the controller IP
address is injected to the image in order for the guest-agent to
download Trove code during initialization. As a result, we have to build
guest image each time in the Trove CI because the image build relies on
the devstack host IP address.

If we could remove the dependency, we can build the image(in
devmode=true) for a specific datastore once and use that image for all
related CI jobs, which could save some time for Trove CI.

Fix the current CI issue as well.

Change-Id: If23f4f179a6ab72cfb35e4c45d55142fedb76498
2019-12-05 00:45:56 +13:00
Lingxian Kong
fc0c4fff4c Rename devstack variable TROVE_DISABLE_IMAGE_SETUP
Rename TROVE_DISABLE_IMAGE_SETUP to TROVE_ENABLE_IMAGE_BUILD to avoid
double negative definition.

Change-Id: Iede80c39bb2a7aaf3c0453bd67f85375fa5ee76c
2019-10-11 14:35:12 +13:00
Zuul
a643c5ecff Merge "Use dedicated service credential config" 2019-10-10 23:41:47 +00:00
Lingxian Kong
d244c9d666 Support to specify branch when building the image
- Explicity specify branch name for building non-dev image
- Use branch name for building dev image in devstack
- Update the doc

Candidate backport for stable/train

Change-Id: I4d2fe38fac34c75b42234fa7a20aba51e6dc1c4e
2019-10-10 21:55:53 +13:00
Lingxian Kong
f3bb46542b Use dedicated service credential config
The current admin credential config options are confusing

Change-Id: I92c15c065edf364b955f2243b3599e0b8cae8b4f
2019-10-10 11:49:04 +13:00
Lingxian Kong
32de39fa4d Use correct Swift credential for instance backup
DevStack is deploying Trove in service tenant model, for swift backed
instance backup, the swift container should be transparent to the end
users.

Story: 2006647
Task: 36883

Change-Id: I5859f4c9911fc2c129c8f23611c6607044fbc145
2019-10-02 22:15:48 +13:00
Lingxian Kong
0cdf8b20b6 Add image build experimental CI job
Story: 2006553
Task: 36637
Change-Id: I3d909082fb887b24ed25c93e11792031d5053b1e
2019-09-17 22:23:43 +12:00
Lingxian Kong
f771e939eb Fix Trove CI failure
- Remove Designate v1 support. python-designateclient removed v1 support
  in
  093d8d7170
- Fix get port IP address error in devstack script.

Change-Id: I08916ddfba85c0218aeab5bfbad58ce3222521f3
2019-09-17 13:52:21 +12:00
Lingxian Kong
c33fa67066 Support to create public trove instance
- The users need to specify the network to create Trove instance, but
  trove-taskmanager will create port in that network for Nova instance
  creation. Using port gives Trove more capabilities to define how the
  database service is exposed.
- Deprecate ICMP protocol for the instance.
- Restrict 'nics' parameter for creating instance.
- Add 'access' parameter for creating instance.
- Add 'public_network_id' option in order to create floating IP for the
  instance.
- Do not create records for security groups, but Trove can still delete
  existing instances for backward compatibility.
- Delete unreasonable Host, Account, Storage API.

Story: 2006500
Task: 36468
Task: 36466
Change-Id: I80827e1ad5e6b130cbf94c2bb7a909c44d5cf1e5
2019-09-11 11:28:00 +12:00
Lingxian Kong
3c09e6178a Improve guest image creation in devstack
Change-Id: I0ef8cbc4babc5c6b9f2936923ffe2837ecc3da74
2019-09-07 14:12:58 +12:00
Lingxian Kong
a1aa15282e Support management security group
Allow the cloud admin to control the security groups on the management
port of Trove instance, a new config option `management_security_groups`
is introduced for that purpose.

Change-Id: I4b22b87d37792be700d4ec7f78a7ea479ddb5814
Story: 2006466
Task: 36395
2019-09-02 10:06:49 +12:00
Lingxian Kong
0ef474a70c Support keypair in devstack
Since Trove already supports to specify a Nova keypair when creating
instance for management convenience, devstack needs to be changed to
create the management keypair and add to Trove config file.

One extra change in this patch is to use a single config file for Trove
API, task-manager and conductor.

Change-Id: I1e6c4f4305104815bdf89b31776a4955de61bc89
Story: 2005429
Task: 30463
2019-08-29 15:41:58 +12:00
Lingxian Kong
dfa5ce93d5 Improve devmode=flase when building the image
During debugging, the following changes are also included:

- Support to specify an image ID to run the integration test.
- Fix the reboot function bug.
- Remove the unsuccessful restart test.

How to run integration test with dev_mode=false:

    ADMIN_PASSWORD=password \
    SERVICE_PASSWORD=password \
    DEV_MODE=false \
    /opt/stack/trove/integration/scripts/trovestack gate-tests mysql mysql

Change-Id: I31d4ee579a554f4c98f9facb9fd4b7779665a3dd
2019-08-25 23:11:54 +12:00
Mark Kirkwood
e85c843445 Refactor variables in plugin.sh to simplify setting a datastore
Remove several variables local to plugin.sh so it is possible to
set a non default datastore by changing only the 3 most obvious ones:

TROVE_DATASTORE_TYPE
TROVE_DATASTORE_VERSION
TROVE_DATASTORE_PACKAGE

Change-Id: I782757647985bdfb859043fe68e2b4cd27569ab3
2019-08-14 22:09:43 +00:00
Andreas Jaeger
7b3483723a Fix Trove CI jobs
Changes to get jobs working

1) After [1] devstack no longer changes the ownership of the whole
   /opt/stack tree to the stack user unconditionally. Switch to the
   stack user when running integration test.

2) Add bindep.txt file[2]. The default fallback file is not installed
   anymore and therefore a bindep.txt file is needed to add install
   additional packages.

3) Use trovestack script rather than devstack to build image so many
   global variables could be used for consistency. By default, devstack
   won't build image.

4) Remove the tools/test-setup.sh as it is not used any more.

5) Instance upgrade test keeps failing in CI for some reason, although
   it's always passed on my local environment. In order not to block
   other patches, skip the instance upgrade tests temporarily.

[1] https://review.opendev.org/203698
[2] http://lists.openstack.org/pipermail/openstack-discuss/2019-June/006888.html

Change-Id: I35e17afb9e827b1fead9d28dbf32f11ce4034a9b
2019-08-14 21:20:34 +12:00
Lingxian Kong
2e052b0262 Enable service tenant deployment model by default in DevStack
The service tenant deployment model means Trove creates most of the
resources(vm, volume, security group, etc.) relating to a database
instance in the Trove service tenant rather than the end user.

With this deployment model, most of the related resources behind the
scenes are invisible to the user, which is a more secure deployment
model for either private or public cloud provider.

DevStack should follow this model as it will be recommended for Trove
deployment in production.

Changes included in this patch that are necessary in order to make that
happen:

- Add 'admin' role to Trove service user(username: trove, project:
  service) in DevStack.
- Create Trove management network resources for Trove service user in
  DevStack.
- Enable Trove remote client configuration by default in DevStack.
- Mainly use alt_demo user in alt_demo project for integration tests,
  config trove user as the admin role user in integration tests.
- Disable the module related tests(module_groups) for now because of no
  use cases but need effort to fix all the failed tests in the service
  tenant model.

Story: #2005445
Task: #30489
Change-Id: I2efb69d3d50344914a875b773f62a227dba2ccaf
2019-06-08 00:46:21 +12:00
ZhongShengping
c704af6e32 Fix syntax error
README of Devstack shows a problem[1].

[1]https://opendev.org/openstack/trove/src/branch/master/devstack

Change-Id: I00eec0b5edfedbea954ce9e093e7a6cb5a84ddb5
2019-05-06 18:32:53 +08:00