Since this repo does not publish DocBook XML content anymore, we can
remove openstack-doc-tools and the tox.ini targets for it.
Also, remove the config file doc-test.conf, it's not used anymore.
Depends-On: Ie90eb7fa6f1b1d7b99869083099201d2bfe10018
Change-Id: Ia92429f120fd18553e2785f63f6a12ad0fc9f12f
With this email[0], you must migrate API reference docs into RST. The
conf.py and the tox environment are also cribbed from nova.
Still need to retain the install_command in tox.ini, otherwise the
api-ref job fails.
[0] http://lists.openstack.org/pipermail/openstack-dev/2016-May/093765.html
Co-Authored-By: Anne Gentle <agentle@cisco.com>
Co-Authored-By: Amrith Kumar <amrith@tesora.com>
Change-Id: I3315261aa18729fa7a6aa79d4a1d6c24de1e2c6b
If you run the py27 tox tests then attempt to run the py34 tests,
the following error will occur:
py34 runtests: commands[0] | ostestr
--blacklist_file=blacklist-py3.txt
db type could not be determined
running=${PYTHON:-python} -m subunit.run discover
./trove/tests/unittests --list
The test run didn't actually run any tests
This happens because the apparently the times.dbm file generated
by py27 is not compatible with py34. The converse is not true though -
if you run py34 first, py27 will work.
The work-around is to delete the file, which is what is now done
before running every tox test. Once the bug has been fixed, the
work-around can be removed.
This was originally noticed in python-troveclient, however it
also manifests in the Trove project.
coverage.xml was also added to the .gitignore file as it
was appearing in the git list of modified files.
Change-Id: I312b3879c764b7b803d35e0bb62f0f47291f6693
Closes-Bug: #1565928
It turns out that the way the python34 tests are run in parallel is
to spin up several processes (not threads). This means that putting
locks around creating the database in a single 'run' would need an
inter-process lock not a thread lock.
While the feasibility of this is investigated, change the behaviour
of the tests to run in series to increase stability.
On a 4 cpu test machine this only increased the run time by 70s.
Partially implements: blueprint trove-python3
Change-Id: I530fcf22b697148a3d0a286015c5a79481528d8e
OpenStack CI supports now constraints in the coverage job - as a first
job to check that the recent changes for zuul-cloner work correctly.
Use constraints for cover environment.
Change-Id: I148a50e3f17fcbf2187898a59b28ea4ed0a827a2
Replace testtools.run test runner with os-testr for Python 3 to run
tests in parallel.
Replace also the whitelist with a blacklist to quickly identify tests
which still must be ported to Python 3.
Blacklist backup.test_backup_models: even if it worked when run with
testtools.run, it fails when run using os-testr. We can investigate
this issue later, it's more important to reduce the duration of the
py34 job to reduce the risk of random timeout.
Add a dependency to os-testr.
Partially implements: blueprint trove-python3
Change-Id: I5ea7ebf7eec15b75a5fed07aab7bd15e5e6cd6d6
* mysql: open temporary file in text mode, not in binary mode
* operating_system: open files in text mode, not in binary mode, and
flush explicitly the file rather than using unbuffered file. On
Python 3, it's no more possible to open a text file in unbuffered
mode (it would be very inefficient).
* CouchbaseRootAccess.write_password_to_file(): encode password to
UTF-8 if it's an unicode string.
* Replace ConfigParser import with six.moves.configparser
* Replace exc.message with exception_to_unicode(exc)
* tox: run the following guestagent unit tests on Python 3.4
- test_backups.py
- test_configuration.py
- test_couchbase_manager.py
- test_couchdb_manager.py
- test_db2_manager.py
- test_dbmodels.py
- test_galera_manager.py
- test_manager.py
- test_mariadb_manager.py
- test_mongodb_cluster_manager.py
- test_mysql_manager.py
- test_redis_manager.py
- test_vertica_manager.py
Partially implements: blueprint trove-python3
Change-Id: I7bf477b328dfe15c5339cda1b80e44358019bfff
The commands is deprecated since Python 2.6 and was removed from
Python 3. This change replaces it with subprocess. By the way, it
also changes the code to avoid a shell and so indirectly makes the
code safer (shell injections are no more possibles).
tox: run test_backupagent and test_pkg on Python 3.4.
Partially implements: blueprint trove-python3
Change-Id: I3818fa1498819cc77a3da42c12245c50b96cad59
* Replace string.find(a, b) with a.find(b)
* NamedTemporaryFile: open file in text mode rather than opening it
in binary mode.
* Module.process_contents(): if contents is a Unicode string, first
encode it to UTF-8 before hashing the content to MD5.
* ClusterController: replace dict.items()[0] with next(iter(dict.items()))
* tox: run the following tests on Python 3.4
- mysql/test_common.py
- taskmanager/test_models.py
- module/test_module_models.py
- test_cluster_controller.py
Partially implements: blueprint trove-python3
Change-Id: I5372520f9717b4f0279c4b436ca88c160ddf894b
Bandit is an OpenStack security linter that uses python AST modules
to perform static code analysis of a project. It can help identify
low, medium, and high security "hotspots" in a project codebase.
This change provides an initial Bandit config based on the existing
one used by Sahara.
Change-Id: Idcfb5641007bb27de8df262b236b9a46d7e11d5d
Closes-Bug: 1547730
This was missed by Ie1133e4c020106e8098685ba9125f84b356d8e0c
and breaks 'checkbuild' (runs when API goldfiles change).
Change-Id: I26fd96c0a6906ab03dec7516e71e1c8e70abaa4a
Closes-Bug: #1563038
tox.ini: add more unit tests to Python 3.4, tests already pass thanks
to other changes.
Partially implements: blueprint trove-python3
Change-Id: Iebbf2f413153e0f91f47c3dc5671f6f20415dda9
* HTTP body type must be bytes on Python 3:
* WsgiLimiterProxy, JSONDictSerializer: encode JSON as bytes
* Use byte strings for literal HTTP bodies
* Encode explicitly HTTP body to UTF-8 on Python 3
* Fix the prototype of FakeHttplibSocket.makefile(): only the first
mode parameter is mandatory
* Replace dict.keys().sort() with sorted(dict.keys())
* tox.ini: add the following unit tests to Python 3.4:
- api/common/test_extensions.py
- api/test_versions.py
Partially implements: blueprint trove-python3
Change-Id: I175dc55424e9b5786819f08a584ac967d083db11
* Replace map() with list-comprehension to get a list on Python 3.
* Unicode dance: encode text to UTF-8 for MD5, decode Base32 from
ASCII to get text.
* tox.ini: test_designate_driver and test_datastore_versions to
Python 3.4
Partially implements: blueprint trove-python3
Change-Id: I1aa5387908b3f108487ed731cee5f7d9bbfd7d69
When a git branch is called trove-xxx, "tox -e pep8" fails on
checking .git/logs/refs/heads/trove-xxx. It doesn't make sense to
validate Git branches using flake8.
Simply exclude .git from flake8.
Note: [testenv:pep8] of tox.ini looks for "trove-*" to validate
contrib/trove-guestagent.
Change-Id: I7c5582282a63da8cc1b3bbb6b76a267ef894a5d7
This change is backward compatible on Python 2:
* Data encrypted with the old code can be decrypted with the new code
* Data encrypted with the new code can be decrypted with the old code
Effect on Python 2: it now is possible to encode Unicode strings
which contain non-ASCII characters. Before encode_data() failed with
UnicodeEncodeError.
Changes:
* pad_for_encryption(): replace chr() with six.int2byte() to get
bytes on Python 3.
* use byte strings in tests
* encode_data() now encodes data to UTF-8 (if it is Unicode)
* encrypt() and decrypt() now encode the encryption key to UTF-8 (if
it's Unicode)
* encrypt() now encodes data to UTF-8 (if it's Unicode), it's only
needed on Python 3 since encode_data() returns Unicode on Python 3.
* test_encode_decode_string(): test also Unicode
* Add new unit tests on encrypt() and decrypt() with known values and
hardcoded IV.
* tox.ini: add test_common_extensions and test_crypto_utils
to Python 3.4 (test_common_extensions already passed before).
Note: UTF-8 is the defacto standard encoding in OpenStack. It is
already used widely in all parts of OpenStack.
Partially implements: blueprint trove-python3
Change-Id: I8e991994698d558177580595cf2c7da93ed1e1c8
* Fix test_template: replace server_id > 1 with len(server_id) > 1,
server_id is a string. On Python 3, str cannot be compared to int,
a TypeError is raised.
* Replace ConfigParser import with six.moves.configparser
* Replace basestring with six.string_types
* Replace dict.iteritems() with dict.items(). The iteritems() was
removed in Python 3.
* Replace a/b with a//b to get integer on Python 3.
* tox.ini: add common/test_template.py to Python 3.4
Partially implements: blueprint trove-python3
Change-Id: Ibe2ccdcba7d55edcc014ea00c3b927d8201d6c1b
This will force pip install to use the upper-constraints.txt specified
version of pip modules. When you don't do this, you are out on the
bleeding edge and become unstable everytime some python library in the
wide world changes in a way that you don't expect.
The releasenotes job has no constraints file, so override the install
command there.
Change-Id: Ie1133e4c020106e8098685ba9125f84b356d8e0c
Closes-Bug: #1563038
The first step to port Trove to Python 3 is to have a working
"tox -e py34" test to avoid regressions.
This change creates such minimum test environment just by running 3
very simple unit tests which have no dependency. With the change, it
becomes possible to make the py34 gate voting to avoid regressions.
Partially implements: blueprint trove-python3
Change-Id: Ie08b45db885639541804c3bcc486e4e451833347
pymongo version 3.1 produces segmentation faults when running unit
tests and has broken the trove gate, see bug 1512870. This is one part
of the solution, the other part of the solution is to push the change
to global-requirements.txt. That's in the review
https://review.openstack.org/#/c/290233/2
Change-Id: I0ac00931cc696e53f1845b60c23f84c9846a6a5e
Depends-On: I94dd0d031bb2c6ecae45ec9482cc2f481e77ce03
Closes-Bug: 1554838
Related-Bug: 1512870
* The adds support for pxc to grow a cluster.
* api and taskmanager support for shrinking a cluster
* validate that the networks given are the same for each instance in
the cluster.
* make sure to add the existing networks on an instance in the
cluster.
* add new Error task for grow and shrink.
* nova client version configuration changed to a string option rather
than an int option because the nova microversions change nova api
output. This was needed for the network interfaces on existing
instances.
* testing for grow and shrink cluster
Change-Id: I8602cad1a98970fcd99ee9793a89e1abdceb0982
Partially-Implements-Blueprint: pxc-grow-shrink-cluster
Once we add debug testenv, we can use "tox debug -- --debug"
to debug test cases when tox is running.
Change-Id: I8613c96bc1bd696ffa6a55e53441e9dda3e25711
We need a release notes page to show notes for unreleased versions by
scanning the "current" branch. This ensures that notes files cannot be
merged if they break the releasenotes job.
also running job to treat warnings as errors.
Change-Id: I14a993f2830d3c0f9fd8d1bec7b2b71b315a13d2
* Replace urllib import with six.moves.urllib
* gettext.install() doesn't require the unicode parameter on Python 3
* tox.ini: add py34 testenv
Tests don't pass. It's the very first step to start the working on
porting Trove to Python 3.
Partially implements: blueprint trove-python3
Change-Id: Ide9ec1d2aee84905c03cacdb3ea35d0dcbf01596
Because python creates pyc files during tox runs, certain
changes in the tree, like deletes of files, or switching
branches, can create spurious errors.
Change-Id: Id8798992dd71d9a7b16cba9e822bab236f0fcad7
Closes-Bug: #1368661
This reverts the oslo.util pin we had now that its in our requirments.txt
and also avoids pymonogo 3.1 causing seg faults.
Change-Id: I4f4daf7a458520799bfbf4368851434c67029324
Closes-Bug: #1512870
Forcing the change in requirements.txt is not feasible as all
entries there must match the global-requirements.txt file.
This forces a specific version of oslo.utils *before* the
requirements are loaded, thus ensuring that 2.6.0 is *not* used.
This should be considered a temporary fix only.
Change-Id: I4363c4b8b86efb15fd95d07ce6fff374024b3616
Partial-Bug: #1506402
Co-Authored-By: Amrith Kumar <amrith@tesora.com>
Created the clustering implementation for Percona XtraDB Cluster
Server. Used the existing framework for adding a new cluster strategy
for percona. Created guestagent, api, and taskmanager calls to setup
the cluster.
Ran into a weird issue where creating the admin user was not being
saved even though there was no error logged. I believe this was
happening because mysql was technically up from the mysqladmin ping
cmd but was not processing the grant call yet. I've heard that to be
sure you should run a 'select 1;' call to verify that mysql is really
ready after starting it. I added a method to verify this.
PXC needs a few more ports opened up for the group comm, state
transfer, and incremental state transfer. Added the percona
cluster.cnf template file for the setting needed in PXC. When a
joining instance first joins the cluster it syncs all the data using
xtrabackup, this will cause the joining instance to lose its admin
user os_admin. So we reset the admin to a new password across the
board.
Make Percona Xtradb Cluster a different datastore because there are
muliple version of PXC. This has been tested with pxc 5.5 from the
ubuntu repository.
Adding coverage xml output in tox.ini
DocImpact
Adding a new datastore percona-xtradb-cluster(pxc) that should not
have any special cases for create and delete cluster.
Implements: blueprint support-pxc-56
Change-Id: I239433f0491cea15b214c41ceecc3a5e72b5bbeb
The rsdns service is no longer needed since trove now supports designate.
It is not being tested in the gate and is currently unsupported.
Remove other rsdns related files and references.
Change-Id: I44009dace44afb5467c51def33c794641ffa33c0
Closes-Bug: #1454028
This patchset helps reducing the ignored pep8 errors.
Fixed some ignored hacking rules, as listed follows:
- E111 indentation is not a multiple of four
- E122 continuation line missing indentation or outdented
- E123 closing bracket does not match indentation of opening bracket's line
- E128 continuation line under-indented for visual indent
- E251 unexpected spaces around keyword / parameter equals
- E265 block comment should start with '# '
- E713 test for membership should be 'not in'
- H105 Don't use author tags
- H306 imports not in alphabetical order
Change-Id: Iadf2af4f6ec90420153ad63b5a41650392ef2cbd
The gate-trove-pep8 job fails on Jenkins with the following error:
pkg_resources.VersionConflict:
(pbr 1.0.1 (/home/jenkins/workspace/gate-trove-pep8/
.tox/pep8/lib/python2.7/site-packages),
Requirement.parse('pbr<1.0,>=0.5.21'))
This is caused by the fact that pbr has now released 1.0.1 and
although most of the dependencies in Trove have switched to:
pbr>=0.11,<2.0
we are still using the hacking module from last year
(hacking>=0.8.0,<0.9), which has pbr pegged to <1.0
The hacking module needs to be updated and the new rules ignored.
The following rules are now ignored:
E111,E122,E123,E128,E251,E265,E713,F821,H105,H237,H238,H301,H305,H306,
H307,H402,H404,H405,H407,H501,H904
Change-Id: Ief129369bbffbaec7694e3f0d36668dde1772f05
Closes-Bug: #1458985
With https://review.openstack.org/156328 merged, some of the tox targets
are not needed anymore, remove them.
Change-Id: I818ca7c2ff9323af04acc59f2254dc50c0a940a6
Consolidate the docs job to have:
* non-voting checklinks
* single docs job that includes all the voting jobs (checkbuild)
* (keep) publish docs job to publish documents as post job
* marked all other check* jobs as true, they can be removed later.
Note that checkbuild and the gate-trove-docs jobs are separate since the
project has both XML and sphinx docs that have different ways of
publishing.
Change-Id: I3432ddaee4c79fbe395a5c76f2b192cfc2f01e24
Trove does not support Python 2.6 anymore starting with Kilo and might
not work correctly with it, so remove the classifier.
Change-Id: Ida17ce3ce76258442222b591846e1e1717119eed
This code adds a feature to the tests where all of the example
snippets are generated and then validated. Tests fail if the new
examples don't match the old ones, meaning a dev changing the API
must update the snippets, triggering a conversation about the changes
during the pull request.
Implements: blueprint example-snippet-generator
Change-Id: I5f1bfd47558a646a56e519614ae76a55759a4422