The grow and shrink operations in Galera_common were eating exceptions.
This change passes the exceptions back and adds unit tests to
ensure such.
Change-Id: I3202d6d4daa77a8a185a427b31b2187eab4d82e9
Closes-bug: 1634627
<long rant deleted>
This change enables the hacking extensions ability in trove. That
includes bumping the version of hacking that we use, and also adding
the enable-extensions like in the [flake8] section. I've enabled two
of the hacking extensions here.
There are some limitations to the hacking rules that I've enabled so
I'm also going to enter two bugs that others can fix (if they so
desire).
Change-Id: I6d8fc4bc03f948b84ad823e5248d78a4caa5c042
Related-Bug: 1642554
Related-Bug: 1642552
this is a pylint wrapper for Trove's tox tests. This commit includes a
basic infrastructure for running pylint through tox.
It also fixes some very obvious import errors that are flagged by the
tool. One is to handle missing imports for _ and _LE.
There is one instance where an exception is being raised but
trove.common.exception isn't imported, and another where an exception
is being incorrectly thrown.
A short readme is also provided.
Change-Id: I0a38f5efde3cb491f1f6c27f6c6500ab29987968
Partial-Bug: #1621636
Since this repo does not publish DocBook XML content anymore, we can
remove openstack-doc-tools and the tox.ini targets for it.
Also, remove the config file doc-test.conf, it's not used anymore.
Depends-On: Ie90eb7fa6f1b1d7b99869083099201d2bfe10018
Change-Id: Ia92429f120fd18553e2785f63f6a12ad0fc9f12f
With this email[0], you must migrate API reference docs into RST. The
conf.py and the tox environment are also cribbed from nova.
Still need to retain the install_command in tox.ini, otherwise the
api-ref job fails.
[0] http://lists.openstack.org/pipermail/openstack-dev/2016-May/093765.html
Co-Authored-By: Anne Gentle <agentle@cisco.com>
Co-Authored-By: Amrith Kumar <amrith@tesora.com>
Change-Id: I3315261aa18729fa7a6aa79d4a1d6c24de1e2c6b
Replace testtools.run test runner with os-testr for Python 3 to run
tests in parallel.
Replace also the whitelist with a blacklist to quickly identify tests
which still must be ported to Python 3.
Blacklist backup.test_backup_models: even if it worked when run with
testtools.run, it fails when run using os-testr. We can investigate
this issue later, it's more important to reduce the duration of the
py34 job to reduce the risk of random timeout.
Add a dependency to os-testr.
Partially implements: blueprint trove-python3
Change-Id: I5ea7ebf7eec15b75a5fed07aab7bd15e5e6cd6d6
Bandit is an OpenStack security linter that uses python AST modules
to perform static code analysis of a project. It can help identify
low, medium, and high security "hotspots" in a project codebase.
This change provides an initial Bandit config based on the existing
one used by Sahara.
Change-Id: Idcfb5641007bb27de8df262b236b9a46d7e11d5d
Closes-Bug: 1547730
The recommended method for doing full backups in CouchDB
has been a simple filesystem copy of the data files. This
is because CouchDB stores data in wholly contained append
only files. For example, when a user creates a database,
a corresponding <database-name>.couch file is created in
the database directory.
The backup functionality has been implemented by compressing
the database directory and then encrypting it and sending it
over to store in Swift. Similarly, the restore functionality
has been implemented by fetching the files from Swift and
uncompressing them into the database directory. After this,
the ownership of the directory needs to be updated.
To test the changes, follow the steps:
- Create a CouchDB instance
- Access the admin console called Futon using the following
url: http://10.0.0.5:5984/_utils/
- Create a database from there and create one or more documents
- Create a backup of this CouchDB instance
- Create another CouchDB instance from the backup created above
- Access the admin console for this new instance and verify that
the database created above is there
couchdb client library for the integration tests has been added
to global-requirements: https://review.openstack.org/#/c/285191/
Change-Id: Iad6d69bf60ace73825819081964a43ad53d6c6fc
Implements: blueprint couchdb-backup-restore
pymongo version 3.1 produces segmentation faults when running unit
tests and has broken the trove gate, see bug 1512870. This is one part
of the solution, the other part of the solution is to push the change
to global-requirements.txt. That's in the review
https://review.openstack.org/#/c/290233/2
Change-Id: I0ac00931cc696e53f1845b60c23f84c9846a6a5e
Depends-On: I94dd0d031bb2c6ecae45ec9482cc2f481e77ce03
Closes-Bug: 1554838
Related-Bug: 1512870
This changeset handles the details of creating,
updating, listing and deleting Trove 'modules.'
Two new tables have been added to the Trove database:
modules
instance_modules
although the instance_modules table is at present unused.
Scenario tests have been written as well, to exercise the
new functionality. These tests can be run by:
./redstack int-tests --group=module_create
In the follow-up changeset, all module tests can be run
by:
./redstack int-tests --group=module
Since module support is available for all datastores
(controlled by a CONF option) the module test has been
added to the common modules group.
Note: Trying to do admin tasks with none admin
credentials results in an Unauthorized exception
being thrown, instead of Forbidden. This
is due to the fact that Forbidden is in the
HTTPUnauthorized section of wsgi.py instead of
the HTTPForbidden section. Moving the exception
caused too many failures, so I created a 'Module'
Forbidden exception and put it in the right section.
Change-Id: I755b0431b33b870ae02d903527f071fd8e23130d
Depends-On: I54d37025275dee4731ad49ebbd21612c4464e4c4
Depends-On: I779c24472d3d96a7b2fe4ed0284fd5869cdef93b
Partially-Implements: blueprint module-maintenance
This patch set implements the following functionality for Cassandra
datastore.
create/delete/get user
list users
change password
grant/revoke/list access
update attributes
create/delete database
list databases
Notes on Cassandra users:
In Cassandra only SUPERUSERS can create other users and
grant permissions to database resources.
Trove uses the 'os_admin' superuser to perform its administrative
tasks. It proactively removes the built-in 'cassandra' superuser
on prepare.
The users it creates are all 'normal' (NOSUPERUSER) accounts.
The permissions it can grant are also limited to non-superuser
operations. This is to prevent anybody from creating a new superuser via
the Trove API.
Updatable attributes include username and password.
The configuration template had to be updated to enable authentication
and authorization support (original configuration allowed anonymous
connections). Default implementations used are:
authenticator: org.apache.cassandra.auth.PasswordAuthenticator
authorizer: org.apache.cassandra.auth.CassandraAuthorizer
The superuser password is set to a random Trove password which is then
stored in a Trove-read-only file in '~/.cassandra/cqlshrc' which is
also the default location for client settings.
Notes on Cassandra keyspaces:
Cassandra stores replicas on multiple nodes to ensure reliability and
fault tolerance. All replicas are equally important;
there is no primary or master.
A replication strategy determines the nodes where
replicas are placed.
The total number of replicas across the cluster is referred to as the
replication factor.
The above 'create database' implementation uses 'SimpleStrategy'
with just a single replica on the guest machine.
This is a very simplistic configuration only good for the most basic
applications and demonstration purposes. SimpleStrategy is for a single
data center only.
The following system keyspaces have been included in the default
'ignore_dbs' configuration list and therefore excluded from all database
operations: 'system', 'system_auth', 'system_traces'
Notes on user rename:
Cassandra does not have a native way for renaming users.
The reason why Cassandra itself does not implement rename is apparently just
lack of demand for that feature.
We implement it by creating a new user, transferring permissions and
dropping the old one (which also removes its existing permissions).
I asked about the sanity of this rename approach on the Cassandra mailing
list and IRC channel and there should not be anything inherently wrong
with the proposed procedure.
This method, however, requires the user to always provide a password.
Additional notes:
Trove uses the official open-source Python driver for Cassandra
to connect to the database and execute queries.
The connection is implemented in CassandraConnection. It is now also
used to obtain the current database status as opposed to the original
method of parsing output of the client tool.
The 'common/operating_system' module was extended with two new functions
for reading/writing ini-style and YAML configuration files to/from Python
dicts. Unit tests were added to 'guestagent/test_operating_system'.
The existing Manager unit tests were extended to include the added
functionality.
Also includes some minor improvements to comments and log messages.
Used the existing operating_system interface to update file
ownership.
The system module was removed and its contents moved to the Application
class. This is to reduce the number of files and help facilitate
overriding.
Implements: blueprint cassandra-database-user-functions
Depends-On: I0faa3a4b9c7302064fb2413b572e2fc515efff0d
Change-Id: I7021f6a0e9a3a933f00cfb7a5d987dc6fe2f95a6
When you run devstack, the trove plugin installs python-troveclient by
forcing upper-constraints.txt in requirements to be changed as below.
-python-troveclient===2.0.0
+-e file:///opt/stack/python-troveclient#egg=python-troveclient
As a result, pinning test-requirements.txt to use a file for
python-troveclient to come from tarballs.openstack.org causes a
failure.
The error you will see is "Could not satisfy constraints for
'python-troveclient': installation from path or url cannot be
constrained to a version".
This change modifies test-requirements.txt and matches
global-requirements.txt, and devstack makes the change (above) to
upper-constraints.txt which accomplishes the same thing.
Change-Id: I3bb748a575d67f9f419f51837555d18ed5d7ca32
Closes-Bug: 1539818
Implements log file retrieval from the guest agent. The contents
of the log file are pushed up to a swift container as a series of
objects that represent a subset of the lines in the log.
The following trove CLI commands are now supported:
trove log-list <instance> : lists log files available on guest
trove log-enable <instance> <log> : enables writing to log file
trove log-disable <instance> <log>: disables writing to log file
trove log-publish <instance> <log>: publishes updates to swift container
trove log-discard <instance> <log>: discards published logs
trove log-tail <instance> <log> : displays last lines of log
trove log-save <instance> <log> : saves the entire log to a file
Log declarations and scenario tests were added for MySQL and
PostgreSQL.
Co-Authored-By: Morgan Jones <morgan@tesora.com>
Co-Authored-By: Alex Tomic <atomic@tesora.com>
Co-Authored-By: Peter Stachowski <peter@tesora.com>
Implements: blueprint datastore-log-operations
Change-Id: I16c3bba4a3183d05af2971be6ba56110105797a6
The default PostgreSQL administration account is 'postgres'.
This account always exists and cannot be removed.
Its attributes and access can however be altered.
Clients can connect from the localhost or remotely via TCP/IP:
Local clients (e.g. psql) can connect from a preset *system* account
called 'postgres'.
This system account has no password and is *locked* by default,
so that it can be used by *local* users only.
It should *never* be enabled (or it's password set)!!!
That would just open up a new attack vector on the system account.
Remote clients should use a build-in *database* account of the same
name. It's password can be changed using the "ALTER USER" statement.
Access to this account is disabled by Trove exposed only once the
superuser access is requested.
In the current implementation Trove uses the 'postgres' account and
return a new superuser called 'root' when the root access is requested.
The user 'root' has however no special meaning in PostgreSQL and the
existing applications may rely on the default superuser name 'postgres'.
This patch includes the following fixes:
- Make Trove create and use its own administrative account (os_admin).
- Disable (disable logins) the built-in 'postgres' account by default.
- Enable and return the 'postgres' user when the root access is
requested.
- Fix failure on repeated 'root-enable' calls. The new implementation
just regenerates the user's password.
- Prevent users from hijacking the 'os_admin' account by allowing only
local access.
- Use the existing framework to generate secure passwords.
- Stop using the system 'postgres' user to execute command-line client
calls. Communicate with database via a native Python interface
(like other datastores) instead.
- Remove unnecessary complex result-set caching on the guest.
- Simplify the status-checking code using the native exceptions.
Avoid performing complex checks with uncertain benefits
(i.e. attempt to guess the state of the process).
Notes:
The current implementation is broken for variaous reasons:
- It uses UUIDs in place of 'secure' password.
- It creates a 'root' user, but no database for it.
The clients won't be able to authenticate without explicitly
providing an existing database name.
- The created 'root' user has no 'SUPERUSER' attribute and
hence is not a real superuser (cannot perform certain tasks)...
- The implementation suffers a defect that allows a non-root user
gain root access to an instance without marking is as 'root-enabled'
A similar defect exists in other datastores (MySQL) too:
1. Create an instance.
2. Enable root.
3. Use your root access to change the password of the built-in
'postgres' account (Trove will still work because it uses the
'peer' authentication method - the UNIX account).
4. Login as 'postgres' using the changed password and drop the
created 'root' account.
5. Backup & restore the instance.
6. Trove reports the root has never been enabled (it checks for
existence of superuser accounts other than the built-in
'postgres').
7. You enjoy the root access of the 'postgres' user
(the password is not reset on restore).
Depends-On: I9d6b3df5bebc3c499cea8306f5a1a6bab18adef6
Change-Id: Ie56ac0850bf4de742ce7841195ce29ca3b30c9ef
Closes-Bug: 1514569
Also includes:
Provide a reusable interface for the OS service 'start', 'stop',
'enable_on_boot' and 'disable_on_boot' commands using the existing
service discovery functionality.
Implement a Trove Redis client based on the native Python 'redis-py'
https://pypi.python.org/pypi/redis
API reference: https://kushal.fedorapeople.org/redis-py/html/api.html
The client can connect via TCP/IP or a Unix socket (currently used
for Linux OSs) and accepts an optional password retrieved from the
configuration file.
The driver already exists in OpenStack global requirements.
It does not have to be included in the 'requirements.txt' file, but
it will need to be added to 'test-requirements.txt' file to enable unit
tests.
The image provider will be required to install it ('redis')
in the Redis images.
Change-Id: Iad8f16acaf18588733162b235925d711b97a366f
Implements blueprint: redis-configuration-groups
NOTE: The code to handle MongoDB authentication is being added, but the auth
switch will NOT be turned on in this commit.
Implement client authentication in the MongoDB guest agent. This involves
creating an os_admin user, authenticating the client before all requests,
and passing a keyfile to new cluster instances.
A MongoDB client driver is being added to improve request handling.
The os_admin password and the keyfile key will be randomly generated and
stored on the guests.
This work should be transparent to the user. The result of Trove instance
and cluster creation should not be effected.
New classes:
* MongoDBSchema - represents Mongo databases in Trove
* MongoDBUser - represents Mongo users in Trove
* MongoDBAdmin - wrapper for running administrative commands
* MongoDBClient - wrapper for the PyMongo client driver
* MongoDBCredentials - utility class for handling username/password
Test:
* Added 'pymongo' to test-requirements.txt
* Refactored the test_mongodb_manager unit test to make it easier to add
new tests.
* Cleaned up dangling mocks.
* Added test for the 'secure' procedure during 'prepare'.
RELATED CHANGES:
Upcoming patch sets that enable Trove management of MongoDB databases and
users require these changes. See the dependancy tree below.
--\ authentication
--\ users create/list/show/delete
--- users root enable/show
--- users access grant/show/revoke
--- database
Closes-Bug: #1467513
Partially Implements: blueprint mongodb-database
Change-Id: I90793f95984b360a2c3668421cb5351aac89e46f
The gate-trove-pep8 job fails on Jenkins with the following error:
pkg_resources.VersionConflict:
(pbr 1.0.1 (/home/jenkins/workspace/gate-trove-pep8/
.tox/pep8/lib/python2.7/site-packages),
Requirement.parse('pbr<1.0,>=0.5.21'))
This is caused by the fact that pbr has now released 1.0.1 and
although most of the dependencies in Trove have switched to:
pbr>=0.11,<2.0
we are still using the hacking module from last year
(hacking>=0.8.0,<0.9), which has pbr pegged to <1.0
The hacking module needs to be updated and the new rules ignored.
The following rules are now ignored:
E111,E122,E123,E128,E251,E265,E713,F821,H105,H237,H238,H301,H305,H306,
H307,H402,H404,H405,H407,H501,H904
Change-Id: Ief129369bbffbaec7694e3f0d36668dde1772f05
Closes-Bug: #1458985