93 Commits

Author SHA1 Message Date
Andreas Jaeger
59f522aefb Remove openstack-doc-tools
Since this repo does not publish DocBook XML content anymore, we can
remove openstack-doc-tools and the tox.ini targets for it.

Also, remove the config file doc-test.conf, it's not used anymore.

Depends-On: Ie90eb7fa6f1b1d7b99869083099201d2bfe10018
Change-Id: Ia92429f120fd18553e2785f63f6a12ad0fc9f12f
2016-08-23 16:56:37 +02:00
Amrith Kumar
3b0d1ea25d Adds the api-ref migrated RST + YAML files
With this email[0], you must migrate API reference docs into RST. The
conf.py and the tox environment are also cribbed from nova.

Still need to retain the install_command in tox.ini, otherwise the
api-ref job fails.

[0] http://lists.openstack.org/pipermail/openstack-dev/2016-May/093765.html

Co-Authored-By: Anne Gentle <agentle@cisco.com>
Co-Authored-By: Amrith Kumar <amrith@tesora.com>

Change-Id: I3315261aa18729fa7a6aa79d4a1d6c24de1e2c6b
2016-08-17 17:46:41 -04:00
Amrith Kumar
0d3ca9c29a Constraints are ready to be used for tox.ini
Per email[1] from Andreas, we don't need to hack at install_command
any longer.

[1] http://openstack.markmail.org/thread/a4l7tokbotwqvuoh

Change-Id: Ibb642f76d92cb56e1764e0460822a57f68931263
2016-08-11 14:46:54 -04:00
Peter Stachowski
dda9aa730b Remove times.dbm file for each tox run
If you run the py27 tox tests then attempt to run the py34 tests,
the following error will occur:

    py34 runtests: commands[0] | ostestr
    --blacklist_file=blacklist-py3.txt
    db type could not be determined
    running=${PYTHON:-python} -m subunit.run discover
    ./trove/tests/unittests --list
    The test run didn't actually run any tests

This happens because the apparently the times.dbm file generated
by py27 is not compatible with py34. The converse is not true though -
if you run py34 first, py27 will work.

The work-around is to delete the file, which is what is now done
before running every tox test. Once the bug has been fixed, the
work-around can be removed.

This was originally noticed in python-troveclient, however it
also manifests in the Trove project.

coverage.xml was also added to the .gitignore file as it
was appearing in the git list of modified files.

Change-Id: I312b3879c764b7b803d35e0bb62f0f47291f6693
Closes-Bug: #1565928
2016-07-20 19:43:08 +00:00
Peter Stachowski
798d9354f6 Serialize py34 tests
It turns out that the way the python34 tests are run in parallel is
to spin up several processes (not threads).  This means that putting
locks around creating the database in a single 'run' would need an
inter-process lock not a thread lock.

While the feasibility of this is investigated, change the behaviour
of the tests to run in series to increase stability.

On a 4 cpu test machine this only increased the run time by 70s.

Partially implements: blueprint trove-python3

Change-Id: I530fcf22b697148a3d0a286015c5a79481528d8e
2016-07-19 18:43:16 +00:00
Andreas Jaeger
4ad0dfe88c Use constraints for coverage job
OpenStack CI supports now constraints in the coverage job - as a first
job to check that the recent changes for zuul-cloner work correctly.

Use constraints for cover environment.

Change-Id: I148a50e3f17fcbf2187898a59b28ea4ed0a827a2
2016-07-17 15:57:37 +02:00
Victor Stinner
71284ddc5f Run Python 3 unit tests in parallel
Replace testtools.run test runner with os-testr for Python 3 to run
tests in parallel.

Replace also the whitelist with a blacklist to quickly identify tests
which still must be ported to Python 3.

Blacklist backup.test_backup_models: even if it worked when run with
testtools.run, it fails when run using os-testr. We can investigate
this issue later, it's more important to reduce the duration of the
py34 job to reduce the risk of random timeout.

Add a dependency to os-testr.

Partially implements: blueprint trove-python3
Change-Id: I5ea7ebf7eec15b75a5fed07aab7bd15e5e6cd6d6
2016-06-22 09:23:47 +02:00
Victor Stinner
b938999f7d Port more guestagent unit tests to Python 3
* mysql: open temporary file in text mode, not in binary mode
* operating_system: open files in text mode, not in binary mode, and
  flush explicitly the file rather than using unbuffered file. On
  Python 3, it's no more possible to open a text file in unbuffered
  mode (it would be very inefficient).
* CouchbaseRootAccess.write_password_to_file(): encode password to
  UTF-8 if it's an unicode string.
* Replace ConfigParser import with six.moves.configparser
* Replace exc.message with exception_to_unicode(exc)
* tox: run the following guestagent unit tests on Python 3.4

  - test_backups.py
  - test_configuration.py
  - test_couchbase_manager.py
  - test_couchdb_manager.py
  - test_db2_manager.py
  - test_dbmodels.py
  - test_galera_manager.py
  - test_manager.py
  - test_mariadb_manager.py
  - test_mongodb_cluster_manager.py
  - test_mysql_manager.py
  - test_redis_manager.py
  - test_vertica_manager.py

Partially implements: blueprint trove-python3
Change-Id: I7bf477b328dfe15c5339cda1b80e44358019bfff
2016-06-22 09:23:32 +02:00
Victor Stinner
9a0ff4e126 Port backup test_storage to Python 3
* Swift: use byte strings
* Encode Unicode (to UTF-8) before hashing
* tox.ini: run backup/test_storage.py on Python 3.4

Partially implements: blueprint trove-python3
Change-Id: Ia5201d2f15db4a66ce9c51d3b9404e3c82fe7c13
2016-06-21 19:08:00 +02:00
Jenkins
f865fa7dc3 Merge "Add Install Guide" 2016-06-17 17:56:13 +00:00
Andreas Jaeger
696d80d4a8 Add Install Guide
This adds trove-specific contents of the OpenStack Installation Guide
in the trove repo per [1]. A separate change will remove the heat
contents from the OpenStack Installation Guide for Newton per [2].

This is a verbatim import from the content in openstack-manuals with
minimal changes to get content building.

Also adds tox.ini environment for install-guide and adds
openstackdocs-theme to test-requirements.txt.

From lmichaels -- added a statement to the next-steps doc.

[1]
http://specs.openstack.org/openstack/docs-specs/specs/newton/project-specific-installguides.html
[2]
http://specs.openstack.org/openstack/docs-specs/specs/newton/installguide.html
[3] https://review.openstack.org/#/c/314229/

Partially-Implements: blueprint projectspecificinstallguides
Change-Id: Idf8cb90536fd4b4d9a2d0695ef30f498b1283410
2016-06-16 15:48:29 -07:00
Victor Stinner
4f888ddfa3 pkg: replace commands module with subprocess
The commands is deprecated since Python 2.6 and was removed from
Python 3. This change replaces it with subprocess. By the way, it
also changes the code to avoid a shell and so indirectly makes the
code safer (shell injections are no more possibles).

tox: run test_backupagent and test_pkg on Python 3.4.

Partially implements: blueprint trove-python3
Change-Id: I3818fa1498819cc77a3da42c12245c50b96cad59
2016-06-16 02:26:49 +02:00
Victor Stinner
61614653d6 Port more unit tests to Python 3
* Replace string.find(a, b) with a.find(b)
* NamedTemporaryFile: open file in text mode rather than opening it
  in binary mode.
* Module.process_contents(): if contents is a Unicode string, first
  encode it to UTF-8 before hashing the content to MD5.
* ClusterController: replace dict.items()[0] with next(iter(dict.items()))
* tox: run the following tests on Python 3.4

  - mysql/test_common.py
  - taskmanager/test_models.py
  - module/test_module_models.py
  - test_cluster_controller.py

Partially implements: blueprint trove-python3
Change-Id: I5372520f9717b4f0279c4b436ca88c160ddf894b
2016-05-30 10:47:26 +02:00
Jenkins
081e41fded Merge "Trove's tox tests should respect upper-constraints" 2016-05-17 20:54:52 +00:00
Jenkins
ebb3bfa79d Merge "Add bandit security config" 2016-05-13 23:16:21 +00:00
Peter MacKinnon
64fc24516b Add bandit security config
Bandit is an OpenStack security linter that uses python AST modules
to perform static code analysis of a project. It can help identify
low, medium, and high security "hotspots" in a project codebase.
This change provides an initial Bandit config based on the existing
one used by Sahara.

Change-Id: Idcfb5641007bb27de8df262b236b9a46d7e11d5d
Closes-Bug: 1547730
2016-05-13 13:41:12 +00:00
Petr Malik
2db8983c74 Trove's tox tests should respect upper-constraints
This was missed by Ie1133e4c020106e8098685ba9125f84b356d8e0c
and breaks 'checkbuild' (runs when API goldfiles change).

Change-Id: I26fd96c0a6906ab03dec7516e71e1c8e70abaa4a
Closes-Bug: #1563038
2016-05-12 12:24:22 -04:00
Victor Stinner
410eb932e9 Enable more unit tests on Python 3
tox.ini: add more unit tests to Python 3.4, tests already pass thanks
to other changes.

Partially implements: blueprint trove-python3
Change-Id: Iebbf2f413153e0f91f47c3dc5671f6f20415dda9
2016-05-03 12:16:37 +02:00
Victor Stinner
ced1438566 Port instance unit tests to Python 3
* Replace long() with int()
* tox.init: add instance unit tests to Python 3

Partially implements: blueprint trove-python3
Change-Id: Ic8825f05e4f648c959147d1b25d158541c50a3eb
2016-05-02 22:31:06 +02:00
Victor Stinner
a007c948ae Port more API unit tests to Python 3
* HTTP body type must be bytes on Python 3:

  * WsgiLimiterProxy, JSONDictSerializer: encode JSON as bytes
  * Use byte strings for literal HTTP bodies
  * Encode explicitly HTTP body to UTF-8 on Python 3

* Fix the prototype of FakeHttplibSocket.makefile(): only the first
  mode parameter is mandatory
* Replace dict.keys().sort() with sorted(dict.keys())
* tox.ini: add the following unit tests to Python 3.4:

  - api/common/test_extensions.py
  - api/test_versions.py

Partially implements: blueprint trove-python3
Change-Id: I175dc55424e9b5786819f08a584ac967d083db11
2016-05-02 22:31:06 +02:00
Victor Stinner
bdf664bbdf Port designate code to Python 3
* Replace map() with list-comprehension to get a list on Python 3.
* Unicode dance: encode text to UTF-8 for MD5, decode Base32 from
  ASCII to get text.
* tox.ini: test_designate_driver and test_datastore_versions to
  Python 3.4

Partially implements: blueprint trove-python3
Change-Id: I1aa5387908b3f108487ed731cee5f7d9bbfd7d69
2016-05-02 22:31:05 +02:00
Jenkins
386434d267 Merge "Port crypto_utils to Python 3" 2016-05-02 16:42:40 +00:00
Victor Stinner
93f448b577 pep8: exclude .git/ subdirectory
When a git branch is called trove-xxx, "tox -e pep8" fails on
checking .git/logs/refs/heads/trove-xxx. It doesn't make sense to
validate Git branches using flake8.

Simply exclude .git from flake8.

Note: [testenv:pep8] of tox.ini looks for "trove-*" to validate
contrib/trove-guestagent.

Change-Id: I7c5582282a63da8cc1b3bbb6b76a267ef894a5d7
2016-04-20 15:36:43 +02:00
Victor Stinner
f9ed13a576 Port crypto_utils to Python 3
This change is backward compatible on Python 2:

* Data encrypted with the old code can be decrypted with the new code
* Data encrypted with the new code can be decrypted with the old code

Effect on Python 2: it now is possible to encode Unicode strings
which contain non-ASCII characters. Before encode_data() failed with
UnicodeEncodeError.

Changes:

* pad_for_encryption(): replace chr() with six.int2byte() to get
  bytes on Python 3.
* use byte strings in tests
* encode_data() now encodes data to UTF-8 (if it is Unicode)
* encrypt() and decrypt() now encode the encryption key to UTF-8 (if
  it's Unicode)
* encrypt() now encodes data to UTF-8 (if it's Unicode), it's only
  needed on Python 3 since encode_data() returns Unicode on Python 3.
* test_encode_decode_string(): test also Unicode
* Add new unit tests on encrypt() and decrypt() with known values and
  hardcoded IV.
* tox.ini: add test_common_extensions and test_crypto_utils
  to Python 3.4 (test_common_extensions already passed before).

Note: UTF-8 is the defacto standard encoding in OpenStack. It is
already used widely in all parts of OpenStack.

Partially implements: blueprint trove-python3
Change-Id: I8e991994698d558177580595cf2c7da93ed1e1c8
2016-04-20 14:45:26 +02:00
Victor Stinner
fe52d5549e Port more common unit tests to Python 3
* fakes/swift.py: replace httplib import with six.moves.http_client
* replace sorted(dict.iterkeys()) with sorted(dict)
* test_stream_codecs: use byte strings
* test_stream_codecs: add test on non-ASCII byte string
* Replace types.StringTypes with (bytes, six.text_type)
* tox.ini: add test_notification, test_remote and test_stream_codecs
  to Python 3

Partially implements: blueprint trove-python3
Change-Id: I9498ff6a8f0a7bae2e9292eb74fc9c44a400754a
2016-04-11 12:09:56 +02:00
Victor Stinner
df9e3c571a Port test_template unit test to Python 3
* Fix test_template: replace server_id > 1 with len(server_id) > 1,
  server_id is a string. On Python 3, str cannot be compared to int,
  a TypeError is raised.
* Replace ConfigParser import with six.moves.configparser
* Replace basestring with six.string_types
* Replace dict.iteritems() with dict.items(). The iteritems() was
  removed in Python 3.
* Replace a/b with a//b to get integer on Python 3.
* tox.ini: add common/test_template.py to Python 3.4

Partially implements: blueprint trove-python3
Change-Id: Ibe2ccdcba7d55edcc014ea00c3b927d8201d6c1b
2016-04-11 12:09:42 +02:00
Jenkins
1d5e743da3 Merge "Add a minimal py34 test environment" 2016-04-01 16:15:29 +00:00
Amrith Kumar
aca90168f5 Trove's tox tests should respect upper-constraints.txt
This will force pip install to use the upper-constraints.txt specified
version of pip modules. When you don't do this, you are out on the
bleeding edge and become unstable everytime some python library in the
wide world changes in a way that you don't expect.

The releasenotes job has no constraints file, so override the install
command there.

Change-Id: Ie1133e4c020106e8098685ba9125f84b356d8e0c
Closes-Bug: #1563038
2016-03-30 11:51:25 -04:00
Victor Stinner
9906bc9a6b Add a minimal py34 test environment
The first step to port Trove to Python 3 is to have a working
"tox -e py34" test to avoid regressions.

This change creates such minimum test environment just by running 3
very simple unit tests which have no dependency. With the change, it
becomes possible to make the py34 gate voting to avoid regressions.

Partially implements: blueprint trove-python3
Change-Id: Ie08b45db885639541804c3bcc486e4e451833347
2016-03-25 18:45:38 +01:00
Amrith Kumar
f7cda9912d Block pymongo version 3.1
pymongo version 3.1 produces segmentation faults when running unit
tests and has broken the trove gate, see bug 1512870. This is one part
of the solution, the other part of the solution is to push the change
to global-requirements.txt. That's in the review
https://review.openstack.org/#/c/290233/2

Change-Id: I0ac00931cc696e53f1845b60c23f84c9846a6a5e
Depends-On: I94dd0d031bb2c6ecae45ec9482cc2f481e77ce03
Closes-Bug: 1554838
Related-Bug: 1512870
2016-03-08 21:12:11 -05:00
Craig Vyvial
9e96241d12 pxc grow/shrink cluster implementation
* The adds support for pxc to grow a cluster.
* api and taskmanager support for shrinking a cluster
* validate that the networks given are the same for each instance in
  the cluster.
* make sure to add the existing networks on an instance in the
  cluster.
* add new Error task for grow and shrink.
* nova client version configuration changed to a string option rather
  than an int option because the nova microversions change nova api
  output. This was needed for the network interfaces on existing
  instances.
* testing for grow and shrink cluster

Change-Id: I8602cad1a98970fcd99ee9793a89e1abdceb0982
Partially-Implements-Blueprint: pxc-grow-shrink-cluster
2016-02-05 11:43:36 -06:00
ting.wang
6457dc3d3b Add debug testenv in tox
Once we add debug testenv, we can use "tox debug -- --debug"
to debug test cases when tox is running.

Change-Id: I8613c96bc1bd696ffa6a55e53441e9dda3e25711
2016-01-09 17:26:33 +08:00
Ondřej Nový
c010e52c72 Deprecated tox -downloadcache option removed
Caching is enabled by default from pip version 6.0

More info:
https://testrun.org/tox/latest/config.html#confval-downloadcache=path
https://pip.pypa.io/en/stable/reference/pip_install/#caching

Change-Id: I81fbb20af8394ec6bb834cf099a5cdbaf52c69da
2015-12-11 21:29:31 +01:00
Craig Vyvial
6c11ae108f updating with other reno changes
We need a release notes page to show notes for unreleased versions by
scanning the "current" branch. This ensures that notes files cannot be
merged if they break the releasenotes job.

also running job to treat warnings as errors.

Change-Id: I14a993f2830d3c0f9fd8d1bec7b2b71b315a13d2
2015-12-02 13:55:48 -06:00
Jenkins
d2a4f3e6cb Merge "Port run_tests.py to Python 3" 2015-12-02 00:22:12 +00:00
Jenkins
e5c5ac70f0 Merge "Delete python bytecode before every test run" 2015-11-20 18:08:14 +00:00
Victor Stinner
41f2606435 Port run_tests.py to Python 3
* Replace urllib import with six.moves.urllib
* gettext.install() doesn't require the unicode parameter on Python 3
* tox.ini: add py34 testenv

Tests don't pass. It's the very first step to start the working on
porting Trove to Python 3.

Partially implements: blueprint trove-python3
Change-Id: Ide9ec1d2aee84905c03cacdb3ea35d0dcbf01596
2015-11-19 10:25:01 +00:00
OTSUKA, Yuanying
c57f205a2a Delete python bytecode before every test run
Because python creates pyc files during tox runs, certain
changes in the tree, like deletes of files, or switching
branches, can create spurious errors.

Change-Id: Id8798992dd71d9a7b16cba9e822bab236f0fcad7
Closes-Bug: #1368661
2015-11-17 10:09:04 +09:00
Craig Vyvial
ab29656bc5 Add reno for release notes management
Depends-On: I2a445cdea20e70399e331ea4e97042ecd0137c2e
Change-Id: Ia0520aff6d96ce3fcf20e94e2a100dae0f90aa76
2015-11-05 13:12:35 -06:00
Craig Vyvial
7a72321cd3 this should fix the failing gate
This reverts the oslo.util pin we had now that its in our requirments.txt
and also avoids pymonogo 3.1 causing seg faults.

Change-Id: I4f4daf7a458520799bfbf4368851434c67029324
Closes-Bug: #1512870
2015-11-04 14:53:18 -06:00
Craig Vyvial
03674b10d9 oslo.utils 2.6.0 causing tox test to hang
Forcing the change in requirements.txt is not feasible as all
entries there must match the global-requirements.txt file.
This forces a specific version of oslo.utils *before* the
requirements are loaded, thus ensuring that 2.6.0 is *not* used.

This should be considered a temporary fix only.

Change-Id: I4363c4b8b86efb15fd95d07ce6fff374024b3616
Partial-Bug: #1506402
Co-Authored-By: Amrith Kumar <amrith@tesora.com>
2015-10-15 14:51:54 +00:00
Craig Vyvial
a94987f8ba Percona Xtradb Cluster implementation
Created the clustering implementation for Percona XtraDB Cluster
Server. Used the existing framework for adding a new cluster strategy
for percona. Created guestagent, api, and taskmanager calls to setup
the cluster.

Ran into a weird issue where creating the admin user was not being
saved even though there was no error logged. I believe this was
happening because mysql was technically up from the mysqladmin ping
cmd but was not processing the grant call yet. I've heard that to be
sure you should run a 'select 1;' call to verify that mysql is really
ready after starting it. I added a method to verify this.

PXC needs a few more ports opened up for the group comm, state
transfer, and incremental state transfer. Added the percona
cluster.cnf template file for the setting needed in PXC. When a
joining instance first joins the cluster it syncs all the data using
xtrabackup, this will cause the joining instance to lose its admin
user os_admin. So we reset the admin to a new password across the
board.

Make Percona Xtradb Cluster a different datastore because there are
muliple version of PXC. This has been tested with pxc 5.5 from the
ubuntu repository.

Adding coverage xml output in tox.ini

DocImpact
Adding a new datastore percona-xtradb-cluster(pxc) that should not
have any special cases for create and delete cluster.

Implements: blueprint support-pxc-56
Change-Id: I239433f0491cea15b214c41ceecc3a5e72b5bbeb
2015-09-04 03:25:06 -05:00
Zhenguo Niu
a8a2060de7 Remove H305,H307,H402,H407,H904
These rules were removed from hacking.

Change-Id: Ice36d79a17ba38bb52a79c039535e7cb61150804
2015-07-16 11:14:33 +08:00
Jared Rohe
8ad786e9be Remove rsdns directory
The rsdns service is no longer needed since trove now supports designate.
It is not being tested in the gate and is currently unsupported.

Remove other rsdns related files and references.

Change-Id: I44009dace44afb5467c51def33c794641ffa33c0
Closes-Bug: #1454028
2015-06-03 12:15:01 -07:00
Sushil Kumar
843bb0cd53 Fixes hacking rules
This patchset helps reducing the ignored pep8 errors.

Fixed some ignored hacking rules, as listed follows:
- E111 indentation is not a multiple of four
- E122 continuation line missing indentation or outdented
- E123 closing bracket does not match indentation of opening bracket's line
- E128 continuation line under-indented for visual indent
- E251 unexpected spaces around keyword / parameter equals
- E265 block comment should start with '# '
- E713 test for membership should be 'not in'
- H105  Don't use author tags
- H306  imports not in alphabetical order

Change-Id: Iadf2af4f6ec90420153ad63b5a41650392ef2cbd
2015-05-29 07:27:59 +00:00
Peter Stachowski
4e3e0e83c6 Fix gate failure on gate-trove-pep8
The gate-trove-pep8 job fails on Jenkins with the following error:

pkg_resources.VersionConflict:
(pbr 1.0.1 (/home/jenkins/workspace/gate-trove-pep8/
.tox/pep8/lib/python2.7/site-packages),
Requirement.parse('pbr<1.0,>=0.5.21'))

This is caused by the fact that pbr has now released 1.0.1 and
although most of the dependencies in Trove have switched to:

pbr>=0.11,<2.0

we are still using the hacking module from last year
(hacking>=0.8.0,<0.9), which has pbr pegged to <1.0

The hacking module needs to be updated and the new rules ignored.

The following rules are now ignored:

E111,E122,E123,E128,E251,E265,E713,F821,H105,H237,H238,H301,H305,H306,
H307,H402,H404,H405,H407,H501,H904

Change-Id: Ief129369bbffbaec7694e3f0d36668dde1772f05
Closes-Bug: #1458985
2015-05-26 19:37:17 +00:00
Andreas Jaeger
40bfbd8f16 Remove now obsolete tox targets
With https://review.openstack.org/156328 merged, some of the tox targets
are not needed anymore, remove them.

Change-Id: I818ca7c2ff9323af04acc59f2254dc50c0a940a6
2015-02-25 19:52:51 +01:00
Andreas Jaeger
0917c1dcb3 Create docs test target
Consolidate the docs job to have:
* non-voting checklinks
* single docs job that includes all the voting jobs (checkbuild)
* (keep) publish docs job to publish documents as post job
* marked all other check* jobs as true, they can be removed later.

Note that checkbuild and the gate-trove-docs jobs are separate since the
project has both XML and sphinx docs that have different ways of
publishing.

Change-Id: I3432ddaee4c79fbe395a5c76f2b192cfc2f01e24
2015-02-16 19:44:00 +01:00
Julien Danjou
414507227e Remove Python 2.6 classifier
Trove does not support Python 2.6 anymore starting with Kilo and might
not work correctly with it, so remove the classifier.

Change-Id: Ida17ce3ce76258442222b591846e1e1717119eed
2014-12-02 09:57:52 +01:00
Tim Simpson
52fc67e51e Create example generator
This code adds a feature to the tests where all of the example
snippets are generated and then validated. Tests fail if the new
examples don't match the old ones, meaning a dev changing the API
must update the snippets, triggering a conversation about the changes
during the pull request.

Implements: blueprint example-snippet-generator
Change-Id: I5f1bfd47558a646a56e519614ae76a55759a4422
2014-11-17 13:51:24 -06:00