4125 Commits

Author SHA1 Message Date
Przemysław Godek
4823dd2f55 Add releasenotes for Redis upgrade
Change-Id: I3a878b7cc1f1e50d8860a5e9bcbc9469d101b232
Signed-off-by: Kasper Hasior <k.hasior@samsung.com>
2019-05-23 09:57:33 +02:00
Zuul
a038b356fd Merge "Add Redis datastore upgrade" 2019-05-20 09:20:56 +00:00
Przemysław Godek
2fd324357b Add Redis datastore upgrade
This patch adds support for upgrading Redis datastore - both single
instance and cluster.

It is achievied in a similar way to MySQL/Mariadb. Steps are:
    + For each node do:
        + stop database
        + preserve configuration files,
        + unmount data volume,
        + create new instance with new datastore,
        + enter restarting mode
        + mount data volume,
        + restore configuration files,
        + reload datastore

Note that due to the Redis Cluster design and Trove approach (all
nodes are master nodes) we do not perform real rolling upgrade - some
Redis slots may be unavailable while upgrading node associated with
them.

This patch adds also a user documentation about upgrading datastores
using Redis as an example.

Story: #2005421
Task: #30443
Co-Authored-By: Kasper Hasior <k.hasior@samsung.com>
Change-Id: I60cddb3a41d2cc7024cbec3d2fd2038d79446507
Signed-off-by: Kasper Hasior <k.hasior@samsung.com>
2019-05-16 13:07:00 +00:00
Zuul
8da960b8eb Merge "Fix tox debug mode" 2019-05-14 06:53:34 +00:00
Zuul
b1529cd4f2 Merge "Extend cluster events" 2019-05-13 17:45:52 +00:00
Minmin Ren
85d2b17c49 Fix tox debug mode
- add oslotest in test-requirement.txt for
fixing tox debug mode failed.
- Update jsonschema/sphinx following the global-requirments.txt
to pass zuul requirements-check job.
(See: https://review.openstack.org/649789
https://review.opendev.org/#/c/657890/)

Change-Id: I5f2cabc35415cac3c4e81afa9084cbd8677ccd22
Story: #2005615
Task: #30862
2019-05-13 10:27:00 +00:00
Bartosz Zurkowski
59e8cb7e75 Extend cluster events
This patch extends adds a new field 'instance_ids' in payloads of two
cluster events:
     - DBaaSClusterShrink (during start and end notification),
     - DBaaSClusterGrow (during end notification).

Moreover, additional end notifications after growing and shrinking
cluster have been added.

The purpose of this change if to enable better integration with
tools for monitoring resources usage.

Change-Id: I2c39b2c3bff65f88e46944eda22209bdc92803bc
Signed-off-by: Kasper Hasior <k.hasior@samsung.com>
Co-Authored-By: Kasper Hasior <k.hasior@samsung.com>
Story: #2005520
Task: #30639
2019-05-12 02:06:42 +00:00
Zuul
e1a027067c Merge "Pass kwargs to exception to get better format of error message" 2019-05-08 14:57:50 +00:00
Zuul
f56329a4aa Merge "Skip image building in DevStack for functional tests" 2019-05-08 14:57:44 +00:00
Zuul
3c17a88543 Merge "Fix error URL" 2019-05-08 14:31:44 +00:00
Zuul
731ce0834e Merge "Update min tox version to 2.0" 2019-05-08 14:31:42 +00:00
Zuul
81c31ca606 Merge "Instead of deprecated keystone cli in docs" 2019-05-08 12:19:16 +00:00
ZhongShengping
c704af6e32 Fix syntax error
README of Devstack shows a problem[1].

[1]https://opendev.org/openstack/trove/src/branch/master/devstack

Change-Id: I00eec0b5edfedbea954ce9e093e7a6cb5a84ddb5
2019-05-06 18:32:53 +08:00
Minmin Ren
27cf32564e Instead of deprecated keystone cli in docs
The keystone CLI is deprecated in favor of
python-openstackclient, update docs using
openstack CLI instead of keystone CLI.

Change-Id: Ifddb5d0d38e4353e17e14ecd65a0d086a6886763
2019-05-06 09:29:35 +00:00
ZhongShengping
687cd4be4a Fix error URL
The CONTRIBUTING.rst url and HACKING.rst URLs throwing 404 not found.
So fix them.

Change-Id: Ifc0a2aec565c00d29bb3d29c55fb16fef259c66e
2019-04-29 10:45:37 +08:00
ZhongShengping
d9a63f7cb0 Update min tox version to 2.0
The commands used by constraints need at least tox 2.0.  Update to
reflect reality, which should help with local running of constraints
targets.

Change-Id: Ifc2c2ad88d77fb2590b5bb598458bdfd4a423a9e
2019-04-29 08:39:54 +08:00
ZhongShengping
4e63540245 Dropping the py35 testing
All the integration testing has been moved to
Bionic now[1] and py3.5 is not tested runtime for
Train or stable/stein[2].

As per below ML thread, we are good to drop the py35
testing now:
http://lists.openstack.org/pipermail/openstack-discuss/2019-April/005097.html

[1] http://lists.openstack.org/pipermail/openstack-discuss/2019-April/004647.html
[2]
https://governance.openstack.org/tc/reference/runtimes/stein.html
https://governance.openstack.org/tc/reference/runtimes/train.html

Change-Id: I0d0a3f62d652853963dc4fe204a413eec2bc1830
2019-04-28 15:36:48 +08:00
Zuul
b53872099d Merge "Move to opendev" 2019-04-28 07:28:23 +00:00
Zuul
d5d84653cf Merge "Changing file owner when upgrading mariadb" 2019-04-26 06:35:28 +00:00
Lingxian Kong
aa836df310 Skip image building in DevStack for functional tests
DevStack builds a mysql guest image by default, but trovestack is used
heavily in Trove functional test and image building is also part of the
script.

This patch disables the image building in devstack for all functional
CI jobs except the mysql ones so we could save some time for those jobs.

Change-Id: Iefef03066c1a054bf760f83a66ce7795ad92c44a
Story: #2005490
Task: #30590
2019-04-26 10:30:36 +12:00
Kasper Hasior
a1df0dbb03 Changing file owner when upgrading mariadb
When we upgrade mariadb version we mount volumes with data and
config files.

As we use new images we cannot make assumption that UIDs are the same
as on the old image so we set a new files and directories owner.

Two new functions have been added: _restore_directory() and
_restore_home_directory() in class Manager which are responsible to
copy files from the volume to the instance file system and to change
owner and group of the files.

The new function is located in base Manager class as it very likely
that other datastores may use it as well.

Co-Authored-By: Przemysław Godek <p.godek@partner.samsung.com>
Change-Id: I3c39f51b471081eeabed55070dc91807544a2dda
Story: #2005398
Task: #30391
Signed-off-by: Kasper Hasior <k.hasior@samsung.com>
2019-04-24 16:28:05 +02:00
ZhongShengping
6a5afea40a Move to opendev
1.Use opendev.org instead of git.openstack.org.
2.Use review.opendev.org instead of review.openstack.org.

Change-Id: I8e9da8fbb04bdec42a0b2f012f6a38b8ee413b27
2019-04-24 16:08:08 +08:00
Lingxian Kong
eafd303e0a Use opendev.org instead of git.openstack.org
After moving to opendev.org, the following command fails when building
the Trove guest image:

curl -o /opt/stack/devstack/upper-constraints.txt \
  'https://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt?h=master'

https://opendev.org/openstack/requirements/raw/branch/master/upper-constraints.txt
should be used instead.

Other places using git.openstack.org are also changed.

Change-Id: Id357dd61e229e3336219723c1c1ae6b50e983f26
2019-04-22 00:05:41 +12:00
OpenDev Sysadmins
13711bd589 OpenDev Migration Patch
This commit was bulk generated and pushed by the OpenDev sysadmins
as a part of the Git hosting and code review systems migration
detailed in these mailing list posts:

http://lists.openstack.org/pipermail/openstack-discuss/2019-March/003603.html
http://lists.openstack.org/pipermail/openstack-discuss/2019-April/004920.html

Attempts have been made to correct repository namespaces and
hostnames based on simple pattern matching, but it's possible some
were updated incorrectly or missed entirely. Please reach out to us
via the contact information listed at https://opendev.org/ with any
questions you may have.
2019-04-19 19:26:15 +00:00
Zuul
a44ad54d79 Merge "Nova keypair support" 2019-04-18 14:23:51 +00:00
Kasper Hasior
7f7ae90ee9 Fix cloudinit mariadb scenario test error
For now mariadb scenario single test case fails because the script
cannot find cloudinit file.

This change adds cloudinit file creation if the file doesn't exist.

Before this patch we were getting the following message:
    tee: /etc/trove/cloudinit/mysql.cloudinit: No such file or
    directory
    #!/bin/sh
    ERROR: InvocationError for command /opt/stack/trove/integration/
    scripts/trovestack gate-tests mariadb mariadb-supported-single
    (exited with code 1)

Change-Id: Iea74dc2c58b1c4376c698f0084454cfcbb3f3b7d
Signed-off-by: Kasper Hasior <k.hasior@samsung.com>
2019-04-17 11:45:58 +02:00
zhufl
74370e8c18 Pass kwargs to exception to get better format of error message
If we do not pass kwargs to exception, the parameter will be deemed
as message and will be displayed directly. This is to pass kwargs to
some exceptions, to get better format of error message.

Change-Id: Ia7d0ec0152de9ee4bdf10c8de80e8d32bbf4a42b
2019-04-17 16:19:19 +08:00
Lingxian Kong
a3de34dbc7 Nova keypair support
Now Trove doesn't support to specify keypair when creating the db
instance, the ssh key is injected into the guest agent image at the
build time, which makes it very hard to manage.

This patch adds a config option `nova_keypair` that is used as keypair
name when creating db instance. The old way of the image building will
be changed in the subsequent patches.

Change-Id: I41d4e41fc4bc413cdd48b8d761429b0204481932
Story: #2005429
Task: #30462
2019-04-17 20:04:58 +12:00
Zuul
f4e17a34b9 Merge "Update master for stable/stein" 2019-04-11 00:12:43 +00:00
Zuul
e7ef1dd722 Merge "Disable devstack image building for trove-scenario-mariadb-single CI job" 2019-04-11 00:12:37 +00:00
Zuul
ee0ed4de0d Merge "Improve trove guest agent image building" 2019-04-11 00:04:21 +00:00
Zuul
eab1b7f236 Merge "Add error handling when Swift is not installed" 2019-04-10 02:00:58 +00:00
Ian Wienand
4367dd5e7a Merge "Replace openstack.org git:// URLs with https://" 2019-04-10 00:52:15 +00:00
Zuul
6a022e8bea Merge "Fix mariadb status after upgrade" 2019-04-09 23:43:01 +00:00
Zuul
ddb839b4e4 Merge "Remove SecurityGroup API extension" 2019-04-09 23:36:02 +00:00
Lingxian Kong
798af4d02a Improve trove guest agent image building
- Add a new element 'guest-agent' for image building. This element is
  used when dev_mode=false, so that the trove code is downloaded into
  the image during the building phase rather than during the guest
  agent initialization.
- Improve trovestack sub-command 'build-image'.

  ./trovestack build-image ${datastore_type} \
     ${guest_os} \
     ${guest_release} \
     ${dev_mode}

- Improve documentation.

Story: #2005387
Task: #30375

Change-Id: I9d7acbd6a97f8c01b48b0f2cf94398d549d89124
2019-04-10 09:29:40 +12:00
Marcin Piwowarczyk
99b30c3737 Add error handling when Swift is not installed
So far, when Swift wasn't installed, there was no error handlig
in trove API.

This change adds LogsNotAvailable exception which is raised when
swift endpoints are not available.

Change-Id: I9a8a75b40bde90163560cdd758f17f9a98a41f0a
Story: #2005378
Task: #30360
Signed-off-by: Kasper Hasior <k.hasior@samsung.com>
2019-04-09 21:19:01 +00:00
Lingxian Kong
4fe95fbaf3 Disable devstack image building for trove-scenario-mariadb-single CI job
Trove mainly uses trovestack script to trigger the CI tests, including
the guest agent image building, so the image building in devstack could
be skipped to decrease the testing duration.

Change-Id: I26fa0adc1a5e39612be15b75d37d22ea1cca3be1
Story: #2005391
Task: #30380
2019-04-09 09:20:22 +12:00
Kasper Hasior
603c5114eb Fix mariadb status after upgrade
For now, after upgrading mariadb in guestagent logs we were constantly
getting message:
    "DB server is not installed or is in restart mode, so for now we'll
     skip determining the status of DB on this instance."

It is caused by the fact that we do not exit restart mode.

This patch adds exiting restart mode after successful upgrade.

Change-Id: Iabf2466f3f68838a54ca15714e832968b2803572
Story: #2005396
Task: #30389
Signed-off-by: Kasper Hasior <k.hasior@samsung.com>
2019-04-08 15:36:23 +02:00
Lingxian Kong
fe4c72807c Remove SecurityGroup API extension
SecurityGroup API extension is not mentioned in Trove API doc. Security
group information could be retrieved from Neutron.

Change-Id: Ifb134eaada09ca4dc739eddb5772681b486cad93
Story: #2005366
Task: #30341
2019-04-06 23:46:07 +13:00
Zuul
6aa86cfaf4 Merge "Add new Galera Cluster bootstraping method" 2019-04-05 23:09:37 +00:00
Zuul
ecf8a78ef9 Merge "Additional logs for Mariadb restore strategy" 2019-04-05 22:59:29 +00:00
Zuul
fc1c246516 Merge "Fix redis expected parameter type" 2019-04-04 12:18:07 +00:00
Kasper Hasior
97d9c34baf Add new Galera Cluster bootstraping method
For systems with systemd there is a wrapper program to bootstrap
Galera Cluster - galera_new_cluster.

This patch introduces a simple method to find a location of a given
executable.

Using this method we can check if a command 'galera_new_cluster' exists
- if it is present it is used to bootstrap Galera Cluster.

Co-Authored-By: Jacek Kaniuk <j.kaniuk@samsung.com>

Change-Id: Iab8f6dd7f908f1e031135e2136abeb228890108a
Signed-off-by: Kasper Hasior <k.hasior@samsung.com>
2019-03-29 10:53:34 +01:00
ghanshyam
d98edf6798 Migrate legacy jobs to Ubuntu Bionic
We have migrated the zuulv3 job to Bionic during Dec/Jan month.
 - http://lists.openstack.org/pipermail/openstack-discuss/2018-December/000837.html
 - https://etherpad.openstack.org/p/devstack-bionic
But that effort does not move all gate job to Bionic as there are
large amount of jobs are still legacy jobs. All the legacy jobs still
use Xenial as nodeset.

As per the decided runtime for Stein, we need to test everything on openstack
CI/CD on Bionic - https://governance.openstack.org/tc/reference/runtimes/stein.html

Below patch move the legacy base jobs to bionic which will move the derived jobs
automatically to bionic. These jobs are modified with branch variant so that they will use
Bionic node from stein onwards and xenial for all other stable branches
until stable/rocky.
- https://review.openstack.org/#/c/639096

This commit mofiy the parent job from devstack-xenial to devstack so that
all trove jobs will start running on bionic.

More Details: 
- https://etherpad.openstack.org/p/legacy-job-bionic
- http://lists.openstack.org/pipermail/openstack-discuss/2019-March/003614.html

Change-Id: I8c618cccc9f62665f0735ac5bd513e991e66d602
2019-03-29 10:55:00 +13:00
Zuul
ac9193d59c Merge "Fix tests for Ubuntu Bionic migration of CI jobs" 2019-03-28 20:02:58 +00:00
Lingxian Kong
6ea816312e Fix tests for Ubuntu Bionic migration of CI jobs
Change-Id: I2fee668c3774f7e87ce3478ca998f8b366c6cd41
2019-03-28 10:32:26 +00:00
Zuul
d6aabcb7c4 Merge "add python 3.7 unit test job" 2019-03-28 07:30:19 +00:00
Zuul
1f5940e283 Merge "Deprecate the config option default_neutron_networks" 2019-03-28 06:56:39 +00:00
Zuul
e590560a73 Merge "Fix poll_until exception type" 2019-03-27 20:50:03 +00:00