51 Commits

Author SHA1 Message Date
jiansong
6f6293f90d Display more flavor information in trove flavor-list command
This commit adds disk_space colmn to flavor-list

Change-Id: I71ca7ddbb6b835b5250bd2c4a0cb7ef4d72c9826
Closes-Bug:#1617987
2016-08-29 04:43:06 -07:00
Morgan Jones
8311ac938c Extend guest agent call timeouts
The gate is horribly overloaded and as a result our calls to the
guest are timing out during scenario tests.  I've been told that
the community won't support fixing the tests, so the only option
is to break production code.  This change makes the high timeout
10 minutes, which is now the amount of time the user will need
to wait to learn of an error in an asyncronous call to the guest.

Change-Id: I66d97dc4a60a1d2afe12d27d167ca426b51074ca
2016-08-22 14:46:02 -04:00
Petr Malik
e586638991 Add log retrieval to Cassandra
Add system.log retrieval.

The Cassandra's system/general log is located in the cassandra
log directory and is by default called system.log.

Cassandra 2.1 and higher uses 'SLF4J' with 'logback' backend.

Logging properties can be configured at runtime via the nodetool utility
(setlogginglevel) and/or persisted in 'logback.xml' configuration file.

Logging can be enabled on per-module/class basis with configurable
verbosity levels.

The logging in Trove is configured globally (on the root level)
- i.e. for all application modules.

A new configuration property 'system_log_level' was added to control the
log verbosity. It allows the operator define the log level that will be
set when the logging is enabled on an instance.
All logging will be turned off when disabled.

Note: The log level is not changed during prepare. The instances will be
provisioned with the default logging setting (INFO level).

Note: An XML codec was added to allow parsing XML files into/from Python
dicts.

Change-Id: Ia57587bf557771ea6d7f00e2fe9d674789b98559
Depends-On: I4a088b7a541c35a6c32d6985727bad65ff491815
Depends-On: I254b81b45e44aeda3049865bee76cd9075312761
Closes-Bug: 1550557
2016-08-14 17:45:02 +00:00
Jenkins
5510241c31 Merge "Use proper queries to update user properties" 2016-08-14 04:20:29 +00:00
Masaki Matsushita
1584f198a5 Introduce "icmp" option for security group rule
This change introduces new datastore option "icmp" to
configure whether to permit ICMP. It helps users to
check DB instance health in different way from access
DB ports.

DocImpact
Closes-Bug: #1485884
Change-Id: I61edeb38ded5543b7976a01363108a7b5b4fc5b5
2016-08-04 13:58:56 +09:00
Jenkins
5a0ba5c8b6 Merge "Fix xtrabackup-binlog file GTID parsing" 2016-07-29 21:38:43 +00:00
Jenkins
1dd56b91fb Merge "Add vCPUs to flavor-list" 2016-07-29 20:45:56 +00:00
Doug Shelley
8538ff6e11 Fix xtrabackup-binlog file GTID parsing
The code responsible for pulling the last GTID reference from
snapshot backup restored to the replica only handled the case
where the GTID reference contained one gtid. There are common
replication use cases where the GTID reference in the file is
comma separated list of GTID references. Change the code from
using the CSV reader to just read the entire file and split it
on tabs. This will guarantee that we get the full list of GTID
references in the file.

Change-Id: Ibbde5a4daa9741e1b997a618288bf6d469356bfe
Closes-bug: 1604914
2016-07-28 19:36:04 +00:00
Petr Malik
dc7ccce006 Use proper queries to update user properties
Bug 1380880 occurs when a user gets renamed
and another one gets created wit its original name.

It turns out that the new user wrongly inherits
access rights of the original user.

MySQL guest currently updates user properties by
changing fields in the internal 'mysql' database.

This patch set replaces the UPDATE with more generic
SET PASSWORD and RENAME statements that should work on
all MySQL versions.

We no longer need to transfer privileges on a user name/host
change since the RENAME statement does it for us under the covers.

As an additional benefit this solution is also fully compatible
with MySQL 5.7 which has changed the definition of the
internal 'user' table and the current code no longer works.

Closes-Bug: 1380880
Change-Id: I37fdec77184715ed889d8ea6d446282c98903258
2016-07-18 17:31:26 -04:00
Petr Malik
df509b7252 Preserve data type when parsing MySQL configs
The data type information was lost in conversions.

IniCodec should deserialize Python objects (like other codecs).
guestagent_utils.to_bytes should return byte values as ints.

* The IniCodec is also used by Cassandra.
  Tested with both MySQL and Cassandra scenario tests.

Change-Id: Ibb703b3db6814fc0c9ea4c6d96399f6c881cea03
Closes-Bug: 1599656
2016-07-18 11:07:00 -04:00
Jenkins
6e2922bc49 Merge "Add New Relic License module driver" 2016-07-17 03:54:48 +00:00
Masaki Matsushita
1a1deafcbf Add vCPUs to flavor-list
This commit adds vCPU colmn to flavor-list

Closes-Bug: #1261876
Change-Id: I2309d00fb8b1c428a8708d8dd934ccc462fb9366
2016-07-08 16:59:27 +09:00
Peter Stachowski
2a9fa44364 Persist error messages and display on 'show'
When an error occurs in Trove, it is very difficult
to determine the cause without access to the
server logs. To make these errors available to
the end user, they are now persisted in the database
and can be viewed using the standard 'show' command.

Also fixed TESTS_USE_INSTANCE_ID test path, as it
somehow got broken over time.

Change-Id: I84ed28ee73a24a2dd6bdbf895662d26e406e9fae
Depends-On: I5d3339e9cbfd6aeb0c3ff6936fefa8dbe9e841f8
Implements: blueprint persist-error-message
2016-07-01 22:27:30 +00:00
Jenkins
3196d347f0 Merge "Locality support for replication" 2016-06-24 21:19:16 +00:00
Jenkins
c902d250c3 Merge "Update 'myisam-recover-options' config template" 2016-06-23 19:53:39 +00:00
Peter Stachowski
187725fafb Locality support for replication
In order to allow replication sets to be all on the same hypervisor
(affinity) or all on different hypervisors (anti-affinity) a new
argument (locality) needed to be added to the Trove create API.

This changeset addresses the Trove server part of this feature.
'locality' can now be added to the ReST payload for a create command
and it is passed along as a scheduler hint to Nova.

The replication scenario tests were enhanced to test that 'affinity'
works and 'anti-affinity' fails (since devstack sets up a single
hypervisor by default). A check for the existance (and
lack of) server-groups was added. This is to ensure that not only is
the server-group created properly, but also that it has been deleted
after all the related instances are gone.

DocImpact: New functionality

Partially implements: blueprint replication-cluster-locality
Depends-On: I18f242983775526a7f1e2644302ebdc0dac025cf
Change-Id: I7d924c25d832f9ff4386e9497bfd214f1b2b3503
2016-06-20 19:38:48 +00:00
Alex Tomic
72dd3d1f5c Postgresql Incremental Backup and Restore
Full and incremental backup and restore strategy for postgres
based on pg_basebackup and WAL shipping.

Full backups are effectively data directory filesystem snapshots
aided by the use of the pg_basebackup tool. Incremental backups
are performed by creating recovery points and backing up the
appropriate WAL files since the previous backup. Restore is
done by recreating the data directory based on a pg_basebackup
snapshot and copying the needed WAL files for
incremental recovery.

A new scenario test group for incremental backups was also added.
(Note that parts of the new test have already been refactored in:
https://review.openstack.org/#/c/324060 )

Implements: bp/pgsql-incremental-backup
Change-Id: I60d52d8cdf7744976c2cae0ea1aca2fd8d07e902
2016-06-14 19:39:48 +00:00
Peter Stachowski
31b0fe39b6 Add New Relic License module driver
The recent addition of module support in Trove (see
https://blueprints.launchpad.net/trove/+spec/module-management ) does
not include the New Relic license module driver. This has been added.

A decorator to streamline writing drivers (by handling common errors)
was also added, and the ping driver modified to use it as well.

Since this code is dependent on having an image with New Relic
installed, no changes were made to the scenario tests with
respect to this new driver.

An addition flag was added to the 'apply' interface that passes in
whether a module was created with 'admin options.'  This allows
some rudimentary access control to be implemented.

Depends-On: I6fb23b3dbbec98de9ee1e2731bcfc56ab3c0ca42
Change-Id: I282cf533c99e351d23f3b86aae727ae4bf279b64
Closes-Bug: #1571711
2016-06-07 17:44:41 +00:00
Jenkins
2f0a8610ca Merge "Backup to swift using wrong large object type" 2016-06-01 11:36:04 +00:00
Jenkins
80bcf8b227 Merge "Unsupported module driver not logged correctly" 2016-05-25 19:14:09 +00:00
xhzhf
b06f64128c spelling mistake in release description for cassandra
In releasenotes description of cassandra backup and restore, 
"resture" should be "restore".
Closes-Bug: #1584684

Change-Id: I0004b4998efa426f2a1707ebb057de52420a450f
2016-05-24 06:24:14 +00:00
Jenkins
654a1a228e Merge "Fixed module-apply on removed module" 2016-05-18 00:09:07 +00:00
Jenkins
a713478cce Merge "Make 'default_password_length' datastore-specific" 2016-05-17 20:28:37 +00:00
Jenkins
29c30e522b Merge "Fix MariaDB clusters failing with TypeError" 2016-05-17 19:01:37 +00:00
Jenkins
946bd0cdb5 Merge "Reuse Cassandra connections" 2016-05-17 07:23:57 +00:00
Craig Vyvial
15ea555624 Fix MariaDB clusters failing with TypeError
This would apply to any type of cluster that uses the galera strategy
while setting the nics on a create call. When we called cast to set()
the object was a list of lists. The set method can not has a list so
this was causesing a unhashable error.

The change is to make the instance_nics a list of strings (what we
originaly expected) to resolve this issue.

Change-Id: I6b04f8b580720e5791e0977a9347d031cdbf9710
Closes-Bug: #1570602
2016-05-14 12:38:23 +00:00
Jenkins
b8004d477d Merge "Add db-models and RootController for Postgres" 2016-05-14 07:08:57 +00:00
Jenkins
c008650ced Merge "Fix MariaDB config groups" 2016-05-13 21:26:52 +00:00
Peter Stachowski
0c506cb404 Fixed module-apply on removed module
If you apply a module, remove it and then apply it again it would
no longer show up in the module-query command. This has been fixed
and corresponding tests added to the module scenario run.

Also added missing 'object' to ModuleManager class

Change-Id: I3302a2547cf88f1da1ba4abca6617981572782d4
Closes-Bug: #1571799
2016-05-12 17:19:53 -04:00
Petr Malik
25ab8e2924 Add db-models and RootController for Postgres
Add Postgres guestagent db models.

Implement RootController extension for the Postgres datastore.
The extension is still using MySQL User models which
will likely cause troubles later (e.g. host parameter).

Change-Id: I1a2eb72fe192943e6843c0ea020d7747740e0714
Partial-Bug: 1543739
2016-05-12 17:13:26 -04:00
Petr Malik
728df1640f Reuse Cassandra connections
Cache and reuse the connections in the Admin and
Status objects to avoid leaking resources.
Initialize the App and Admin objects themselves lazily
at the first access.

The Connection object has been changed to act as both
an instance object and context manager.

Change-Id: I4263b870a1099d345ab1940c61ccdf4680a706ca
Closes-Bug: 1566946
2016-05-12 19:56:20 +00:00
Jenkins
e8388b93ff Merge "Remove unused 'override.config.template'" 2016-05-12 16:54:38 +00:00
Petr Malik
e07262d711 Update 'myisam-recover-options' config template
Note that the option name has changed.
Also removed an unnecessary commented line.

Change-Id: I963724e553baf0870c95aecd1cabc8cbba3da422
Closes-Bug: 1580315
2016-05-12 16:25:00 +00:00
Jenkins
40d72aea7d Merge "Mysql GTID replication fails when data inserted" 2016-05-12 15:29:21 +00:00
Peter Stachowski
970d23ff2a Unsupported module driver not logged correctly
If an invalid module driver is found, the method is erroneously
printed out instead of the actual 'type.'  This is due
to missing parenthesis in the logging call.

This has been fixed.

Change-Id: I6fb23b3dbbec98de9ee1e2731bcfc56ab3c0ca42
Closes-Bug: #1579900
2016-05-09 21:22:39 +00:00
Peter Stachowski
9f7be1a10b Notification exceptions not sent back correctly
With the new notification feature added recently, exceptions
that occur on the guest are sent back to the conductor.
Due to a mis-match in the code however, the wrong call is being
issued. The conductor api makes a call to notify_exception
instead of notify_exc_info.

This has been fixed.

Change-Id: Ib2e04bb903c92b4d570913aba2e985db63c45096
Closes-Bug: #1577848
2016-05-06 02:05:52 -04:00
Matt Van Dijk
6672fe868a Backup to swift using wrong large object type
Trove backups are stored in Swift as Dynamic Large Objects. This is not
ideal, and could cause restore to fail as DLOs may not return all of
their parts. Static Large Objects should be used instead. SLOs perform
checksum and object size validation and can reduce object download times
significantly.

Changes are needed to the Swift storage strategy and taskmanager's
backup delete model to perform actions on SLOs. The corresponding
unittests and fakes are also updated.

A special case occurs when the backup object fits in a single Swift
object segment, in which case it will not be made a SLO.

Change-Id: Ide30a1e7f909a3b7fecb16268367b56468306d42
Closes-bug: 1489997
2016-05-02 15:11:15 -04:00
Petr Malik
7859d49f14 Remove unused 'override.config.template'
Override templates have been ignored since Liberty.
Remove the unused files from the sourcetree and
last stale references from the code.

Change-Id: I17939dbafa229cccb2548aecc1f8f82ead2e70ef
Closes-Bug: 1575852
2016-04-29 15:17:09 -04:00
Petr Malik
f4cbaf6ccd Make 'default_password_length' datastore-specific
Make 'default_password_length' per-datastore-property.
Set max length to 24 in Couchbase config.

Change-Id: I391bc68b44028412ed25490264cbb1e396d540d2
Closes-Bug: 1572230
2016-04-22 19:34:04 +00:00
Jenkins
c3169966ba Merge "Mysql replicas need to set binlog_format" 2016-04-21 16:53:03 +00:00
Doug Shelley
09a312ae3a Mysql GTID replication fails when data inserted
If you have a master and a slave configured and you insert
new data into the master, it will cause subsequent replica
create to fail.

The problem is that we weren't setting the gtid_purged variable
using the metadata in the xtrabackup_binlog_info file. The
Mysql GTID replication strategy was adjusted to account for this.

A release note has been added.

The replication scenario tests were enhanced to validate this
issue.

Note: this issue doesn't occur with MariaDB GTID replication
because it mechanism is different.

Scenario tests were run succesfully on Mysql, Percona and
MariaDB with this change in place.

Change-Id: I66c8b6278afa50ba14e4bb7888e3a25dc657a9e4
Closes-bug: 1563574
2016-04-21 16:05:24 +00:00
Doug Shelley
9b03fec1e1 Mysql replicas need to set binlog_format
For GTID replication we set the replica source (i.e. master)
to have a binlog_format of MIXED. We aren't setting this on
the replicas so they default to STATEMENT. This causes a problem
with certain "non-deterministic" functions (e.g. RAND()). This
changes replica config templates for Mysql and Percona to use
MIXED mode.

A new scenario test was introduced to validate that the
replicas have this set.

MariaDB doesn't appear to have this issue so it wasn't changed.

Scenario tests for Mysql, Percona and MariaDB were run to
validate this change.

Change-Id: I936cd9bc53a812af19653e9b5b472103fab2b6c1
Closes-bug: 1563541
2016-04-19 13:07:09 +00:00
OpenStack Proposal Bot
c09d8f3e8c Imported Translations from Zanata
For more information about this automatic import see:
https://wiki.openstack.org/wiki/Translations/Infrastructure

Change-Id: I6fb050b46ea7f34a1a17d17ba5c34b98f41d8a58
2016-04-19 06:12:05 +00:00
Jenkins
1d2ff0304a Merge "Do not remove root user on disable" 2016-04-18 12:09:16 +00:00
OpenStack Proposal Bot
143991ea5d Imported Translations from Zanata
For more information about this automatic import see:
https://wiki.openstack.org/wiki/Translations/Infrastructure

Change-Id: I64f1781889347ea69844a56774b54e99cf258f8b
2016-04-17 06:11:01 +00:00
Doug Shelley
ef27d82bec Fix MariaDB config groups
MariaDB historically leveraged the mysql manager for guest
support including the configuration groups implementation.
With MariaDB now having it's own manager class that inherits
from Mysql, it needs to have validation_rules and a
ConfigParser setup.

This commit adds those items and turns back on the configuration
scenario tests (which were run with MariaDB to test this change)

Change-Id: Iaae1bb3916f0d05f7d1566dbeb39903763447657
Closes-bug: 1532256
2016-04-16 14:39:35 +00:00
Petr Malik
52bc1ab9c8 Do not remove root user on disable
The existence of the root user (with remote access)
is used to determine whether root was ever enabled on a restored
instance.
Do not remove it, just generate a new random password for it.

Change-Id: I8a4321ac062b1ec565945b49dbb7c619b6da867f
Closes-Bug: 1549600
2016-04-15 07:54:12 -04:00
Thierry Carrez
ea7e5934ad Update reno for stable/mitaka
Change-Id: Ie98d9732f758b0988c9a6943e0e54857a45040cf
2016-03-18 08:49:15 +00:00
Craig Vyvial
a5dd762cf0 updating the release notes from mitaka commits
Change-Id: I360c32c686161cf7bdd8afa1971a06c9ae2aa664
Author: Craig Vyvial <cp16net@gmail.com>
Co-Authored-By: Amrith Kumar <amrith@tesora.com>
2016-03-17 23:14:16 -04:00
Craig Vyvial
6c11ae108f updating with other reno changes
We need a release notes page to show notes for unreleased versions by
scanning the "current" branch. This ensures that notes files cannot be
merged if they break the releasenotes job.

also running job to treat warnings as errors.

Change-Id: I14a993f2830d3c0f9fd8d1bec7b2b71b315a13d2
2015-12-02 13:55:48 -06:00