50 Commits

Author SHA1 Message Date
Davanum Srinivas
47fdbc1391 Deal with underscores in blacklist during generate-constraints
In change id  I60adf0dca4aa32f4ef6bca61250b375c8a3703c6, we added
flake8_docstrings to the black list. However as seen in
I8bfff1b2e65541959e7f2ae0fa6ca6c17889eb54, the generate-constraints
job ends up proposing adding "flake8-docstrings===0.2.1.post1" to
the upper-constraints.txt. This is because of the '-' vs '_'
difference. So just like in Ibaa22657a2cf2c0ad96dbd0b9bc43cdafe6a1d56,
we need to convert the items in black list to the correct safe name
before trying to check if the item is in the black list.

Change-Id: Ieb1cd44e908b80d58a20bf0d0e1e6b10f17ae95e
2015-11-07 23:00:12 -05:00
Davanum Srinivas
33636c1d81 Expose extras information in requirements
Starting with oslo.db+Nova, we would like to specify say
oslo.db[fixtures] in Nova's test-requirements. If you see
the proposed review I4131e534d4cb12e4888d398fa4fb7c922e369210
the gate-nova-requirements fails because code in
project-config/jenkins/scripts/project-requirements-change.py
does not understand extras at all and treats both oslo.db
and oslo.db[fixtures] the same. Since that module
uses requirement.py, we should expose the information here
first and then fix project-config after that.

Change-Id: I146baa3ef94cc8bbf29af371786f3ea95a42cb9f
2015-10-22 15:26:48 -04:00
Jenkins
ef82c13bf7 Merge "Deal with underscores in names in edit-constraints" 2015-09-17 16:09:16 +00:00
Robert Collins
b95d747cfa Update our handling of pbr.
*) We're removing our upper caps - they are just time bombs. Make pbr
   be consistent here too.

*) pbr 1.8 handles comments in marker lines properly. We don't need to
   update the run-time minimum, but we do need a build-time minimum of
   1.8 to ensure OpenStack packages can build properly.

Change-Id: I9fe0c62e774c00c2e521e1e69c5f2f5d87865ec1
2015-09-17 10:20:17 +12:00
Davanum Srinivas
3ebedf84f0 Deal with underscores in names in edit-constraints
Problem showed up with django_openstack_auth job that installs from
source. The change to devstack to verify LIBS_FROM_GIT kicked in
(Iffef2007f99a0e932b68c4c897ebbfb748cac2b4) and starting failing the
job. The problem was that the edit_constraint was not fixing the
django-openstack-auth line in upperconstraints.txt as the setup.cfg
had name = django_openstack_auth. Note that when things finally get
installed setup_tools' safe_name kicks in to standardize names and for
example glance_store will show up as glance-store in pip freeze. So we
should be able to treat this situation better and allow constraints to
have the safe names (using dashes).

Co-Authored-By: Robert Collins <rbtcollins@hp.com>
Change-Id: Ibaa22657a2cf2c0ad96dbd0b9bc43cdafe6a1d56
2015-09-17 10:12:11 +12:00
Victor Sergeyev
e45b270ad8 Fix url regexp to proceed oslo-specific url's properly
The current url_re regexp ignores dots in egg section, so it proceeds
url with "#egg=oslo.messaging" as "#egg=oslo". This patch fixes it.

Change-Id: Ifcea3e825218578f799c0a951baac8f8e8ef5de5
2015-08-06 17:14:39 +03:00
Doug Hellmann
2dbf39295c Move data validation out of the unit tests and strengthen
Having the data validation happening inside the unit tests makes
understanding requirement update failures confusing because the output
is hard to read and does not explain what the error actually is. Move
the checks to their own command and set up a tox env to run it. A
separate patch in project-config will add a new job for the repository.

Address the dedent comment from
https://review.openstack.org/#/c/204181/3/openstack_requirements/tests/test_requirement.py,cm

Add a test to ensure that all items in global-requirements.txt are also
listed in either upper-constraints.txt or blacklist.txt. Ignore a few
items that we don't know how to constrain yet.

Add a test to ensure that items in blacklist.txt are not in
upper-constraints.txt.

Change-Id: Icb717b0f36afb6ea29f50bc6935917dddf47fd4c
2015-08-03 20:45:37 +00:00
Doug Hellmann
43a876b678 Fix constraint integration test
The test to validate the constraints file against the global requirements
list was looking for any spec from any package to match. Fix it to only
look at the specs for the package from the constraint being checked.

Add some more unit tests to demonstrate the old conditions that were
failing, and to verify the new logic passes.

Add stdout capture to the tests so we can provide more helpful error
messages to folks contributing patches to the different requirement
lists about why the integration test might be failing.

Add a test to enforce the format of upper-constraints.txt using 3 = to
indicate an absolute constraint. Fix 2 constraints which were previously
specified with only 2 =.

Change-Id: I46fd11031ba8f775fe62406380bf6417684f1f06
2015-07-23 16:55:13 +00:00
Jenkins
5174071897 Merge "Add checks for setup.cfg existence in project dictionary" 2015-07-17 11:02:59 +00:00
Gal Sagie
97ddc985ab Add checks for setup.cfg existence in project dictionary
This check is needed by new projects at their first commit
Without this fix the following initial commit for new project
failed:
https://review.openstack.org/#/c/201701/

The exception that it failed was originated in this method
with KeyError on the project['setup.cfg'] (see previous
jenkins logs on the above commit)

Log: http://logs.openstack.org/01/201701/6/check/gate-kuryr-requirements/e922983/console.html

Change-Id: I39db2918ea277b939f67d3c856f52bd53c226f27
2015-07-15 11:08:48 +03:00
Jenkins
8ddf1fcffd Merge "Place an accurate lower bound on pbr" 2015-07-15 00:08:33 +00:00
Jenkins
d859b7dcb3 Merge "Stop adding extra newline after unparseable lines." 2015-07-15 00:08:30 +00:00
Robert Collins
0937f0d3a2 Place an accurate lower bound on pbr
Without this version, markers will fail to install in a growing number
of projects.

Change global-requirements to prevent
pip -r requirements.txt; pip install .
failing with an already installed and too old for setup_requires pbr.

Change update.py to signal to setuptools the minimum version required,
giving clearer error messages in the case of downgrades during
devstack.

Change-Id: I074274b470a7a4f2c95211677b135038ab2c942c
2015-07-12 09:54:33 +12:00
Russell Bryant
931defba21 Stop adding extra newline after unparseable lines.
I have a project that includes a requirement line this:

    -e git://git.openstack.org/openstack/neutron.git#egg=neutron

Every time update.py runs against the file, it adds an extra newline
after this.  For example, if I run update.py a few times, I'll have a
few new lines in the file.  This patch prevents the extra newlines
from getting added.

Change-Id: I92d02a45c437025b16df067d7b58a3727d42c0e0
Signed-off-by: Russell Bryant <rbryant@redhat.com>
2015-07-08 10:37:02 -04:00
Robert Collins
6b06c5d0fd Don't permit embedded #'s in URLs.
This is a fix for a minor issue caught in review of 198186

Change-Id: Ic27acf79b4a189f9348edcd37523403202e6fc0b
2015-07-08 10:38:44 +12:00
Robert Collins
63e5747001 Fix editing constraints files with urls in them.
constraints files, unlike global_requirements, need to be able to
contain urls (specifically file:/// paths) so that we can constrain
libraries to be run from their git trees, rather than having them flap
all over the place as different servers specify different bounds. The
first edit-constraints implementation failed to parse such lines,
throwing errors instead.

This patch makes the support for urls optional, preventing changing
the semantics of requirements update.py logic.

Change-Id: I33bd2d9dff9cb7dc1a50177db7286b7317966784
Depends-On: I0f07858e96ea3baf46f8a453e253b9ed29c7f7e2
2015-07-04 09:54:39 +12:00
Robert Collins
b838ab1e43 Add location to requirement.Requirement
This will provide space to store the result of parsing requirements
like -e file:///opt/stack/new/oslo.THING, which we need to do for
edit-constraint to work in devstack when more than one library is
being installed from git. This is a separate patch to make reviewing
the actual new functionality easier - it is entirely mechanical,
adding the field and dealing with test fallout.

Change-Id: Ied4e92717b0dab871dac42ae639d1f2a6c236414
2015-07-03 12:23:09 +12:00
Robert Collins
893490c958 Add edit-constraints command.
This allows shell editing of a constraints file, which we need to do
in devstack when installing a constrained library from git.

Change-Id: Ibdbfc786b5d2f56d2e41d329c8766bef7b2e6854
2015-06-29 15:44:17 +12:00
Robert Collins
3aa37f7ea4 Move commands into cmds subdir.
This is a useful hint to readers about whether a module is library or UI.

Change-Id: I177449fc6820e82fed938f2c216b6b9f9192957f
2015-06-29 15:27:25 +12:00
Robert Collins
50eee6fc57 Split project and requirement types out.
These don't need to be hidden within update.py anymore, and we need
them for lint and edit.

Change-Id: I92f4ff51f98f6343cc94a74a69660306b742dbf3
2015-06-29 15:27:25 +12:00
Robert Collins
24fe404d97 Prevent incompatible version bumps.
We shouldn't be able to bump global requirement versions if that
would be incompatible with the current test set of versions. To assess
'works' we need to run them in tests which means that we need to
update upper-constraints.txt at the same time. We don't test all
versions however, so just checking that the constraints version is
compatible with the updated global requirements is sufficient.

This adds a dependency on the pypa packaging library. The library is
maintained by the authors of setuptools and pip, is Apache V2, in good
condition and Python 3 ready. Its a test-only dependency so I don't
think the distro packaging status matters. This is *the* reference
library for packaging facilities on Python.

Change-Id: I032dd9a384453dcfa4911ea905c8f89b0c974211
2015-06-25 11:20:56 +12:00
Robert Collins
5e4ca1b8b8 Test that changes to requirements are freezable.
If we can't generate a new constraints file, don't let changes in.

Change-Id: If5aa15bb9ca75db0b8ef89a13c9b4bbf8d45cae2
2015-06-25 11:20:56 +12:00
Robert Collins
8bfdcc55ee Show not-in-global errors again.
This regressed when I half-split it into actions and IO (exception
raising). Moving it entirely into actions lets us see all the errors
very easily.

Change-Id: I12e6d910bf983841b4ad81aa41a95bf341456fe4
2015-06-24 09:47:24 +12:00
Robert Collins
5cd10eb1f6 Teach the constraints generator about excludes.
Some things like flake8 can't be constrained, so we need an opt-out
clause.

Change-Id: I11eb0d762869ad8920795fb710f1b2eeb9354f12
2015-06-24 09:42:52 +12:00
Doug Hellmann
a85286e3b2 Report bad package when found
When a non-standard package is found in the requirements list, we had an
error saying that but not saying what the bad package name was.

Change-Id: I860c548dce0c8f988f8fd29cead582bf9b9b6850
2015-06-22 19:24:38 +00:00
Robert Collins
4273910b78 Add upper-constraints.txt generator.
This script when run creates a constraints file with exact pins for
the transitive dependencies of a requirements file.

Change-Id: I1282f8e4010c0ec46c473495bacddf044d06c0af
2015-06-20 19:19:21 +12:00
Robert Collins
9436d30a7d Remove the shebang from generated setup.py's.
We don't expect users to run setup.py directly, and the world didn't
end when the x bit was removed accidentally, so this just completes
the statement.

Change-Id: Ib3934e55949ed6bc72408ddf020b4300a1fccc80
2015-06-20 19:19:21 +12:00
Robert Collins
7fb4566746 Remove spurious output in test suite.
This was a debugging thing, but we don't need this verbosity for the
smoke test.

Change-Id: I3a088fae7048074a5f9ef6a2ca94dbd90067f0b2
2015-06-20 19:19:21 +12:00
Robert Collins
4b22b94752 Also update extras in setup.cfg.
This required some care. There doesn't seem to be a sane Python3 ready
comment-preserving ini parser around, so I wrote a
minimal-for-our-case one in Parsley. Parsley is already in use in
infra in bindep, but I need to add it to global-requirements as this
is the first use in a managed project of it.

Change-Id: I48de3a2f36e945f75b534f689e3af802bbdc5be9
Depends-On: I7d7e91694c9145fac0ddab8a9de5f789d723c641
Depends-On: I16e967356d5c56f1474ee661b954b3db11a608cb
2015-06-20 19:19:21 +12:00
Robert Collins
a848fdc879 Handle : as well as ; for markers.
setup.cfg uses foo:MARKER rather than foo;MARKER as requirements.txt
files do. To be able to process setup.cfg sections we need to support
':' as a marker separator.

Change-Id: I3d78b3df8ea9fad1fd7b3c634eb0fb0f7e7e9124
2015-06-19 09:15:25 +12:00
Robert Collins
91f24223ea Pull serialisation out of _sync_requirements_file.
This means we can use _sync_requirements for setup.cfg stanzas, which
have a different format to requirements.txt files.

Change-Id: Ifdb612017677fa2cf3e91de09e4b152dd11ca8ed
2015-06-19 09:15:25 +12:00
Robert Collins
937d378926 Pull parsing out of sync_requirements_file.
We're going to be parsing different sorts of files soon, and
sync_requirements shouldn't need to figure that out.

Change-Id: I98677e083cdf55a8626334bee03786aa021045e3
2015-06-17 13:54:46 +12:00
Robert Collins
a3a3f7f57a Support multiple marked reqs for the same package
This is needed when a single specifier set is insufficient to handle
both Python2.7 and 3.x. We're not currently dealing with this, but I
have observed it in other projects and we can be forearmed quite
easily.

Change-Id: If1493625d0d2ebf750159ccb8ed0fd22699766a1
2015-06-17 01:32:23 +12:00
Robert Collins
32fed9acdb Support environment markers in requirements.
This is needed to allow universal wheels when requirements differ
between Python versions (or platforms).

Change-Id: I3a6e88dc3988c9ddf0d4923e5ee6c1aa81e7a4d9
2015-06-17 01:32:23 +12:00
Robert Collins
75408531ba Move to a single unified parser for requirements.
This gets rid of potential inconsistency points between global
handling and per-file handling.

Change-Id: Ia24fd7d213fa6363f16d70c8ef798b7b67e4de4c
2015-06-17 01:32:23 +12:00
Robert Collins
a26f9e1323 Simplify requirement comparisons.
The _functionally_equal thing was needed in the past but isn't anymore.

Change-Id: Ie88814a71ddbc5bf72c22970f728d4ffd99caa62
2015-06-17 01:32:23 +12:00
Robert Collins
74cdbd94e4 Make requirements parsing testable.
Change-Id: I3e5f4568a9cd0e6ba4da1d2193f613426a701fd6
2015-06-17 01:32:23 +12:00
Robert Collins
81c6d1a736 Reduce dataloss window through interrupted writes.
If we're interrupted mid-file-update we shouldn't leave the
requirements file half written. Instead write to a temp file and
rename into place.

Change-Id: I7c7c2553b90f756d74df1967369732b21b8e8c0d
2015-06-17 01:32:23 +12:00
Robert Collins
4c5bd8e6be Move test_update to be unit tests.
We preserve a single full end to end test to ensure all the dots are
in fact joined.

Change-Id: I0d99ae378f3bf0f2cc27a3b5eddf7730606c649e
2015-06-17 01:32:23 +12:00
Robert Collins
81522c16c2 Convert test_pbr to unit tests.
Change-Id: I07cd6e2b0158cd8787d829ac2004d6f5e3b2d18d
2015-06-17 01:32:23 +12:00
Robert Collins
8441666452 Convert test_update_suffix to unit tests.
Change-Id: I818929eac8d9bc9f186a78fb192eeddbe620ceb0
2015-06-17 01:32:23 +12:00
Robert Collins
380eb67075 Add basic testing of options -> parameters.
Change-Id: Id8594de19085e34730019bb5c7e59248e471e756
2015-06-16 09:40:14 +12:00
Robert Collins
3b4e5228c1 Split main out a little for testing.
Change-Id: I9040db8dd7f829e8a50c738710933435cc8ddbd0
2015-06-16 09:40:14 +12:00
Robert Collins
847df92dbb Make _copy_requires pure.
Easier testing and better separation of logic.

Change-Id: I70189487bdf33ce8061b8abc21ae610aed2b031d
2015-06-16 09:40:14 +12:00
Robert Collins
7cf29aa22e Remove VERBOSE global state.
Instead pass a verbose closure down, which while one more parameter
will prevent any skew in future unit tests.

Change-Id: I310a8fa185b2381a43b28c92ddc3f08e3c6ba035
2015-06-16 09:40:14 +12:00
Robert Collins
1d80fea293 Don't write partial requirements files.
If we detected nonstandard requirements we were previously rewriting
half the requirements file, making this non-idempotent: all the
subsequent contents was being lost.

Change-Id: If550d98fcab33a915bd44347bac1b538386d691f
2015-06-16 09:40:14 +12:00
Robert Collins
fb956e40b0 Make _write_setup_py pure.
Factoring out the IO lets us look at the function more clearly, as
well as increasing testability.

Change-Id: I2366ff0ef7cba16863da9d10890a3fc60098e4d0
2015-06-16 09:40:14 +12:00
Robert Collins
f0cf095a5b Group pure and IO functions in update.py.
Starting to make this more testable.

Change-Id: I18b764c6b58ab409c7beac73314b4e52ad093742
2015-06-13 03:00:10 +12:00
Robert Collins
21fa543926 Move update.py into the python package.
A symlink is kept to update.py to keep this working for infra until we
update the propose scripts to try the new location and fall back.

Until that is done and the symlink removed, update.py cannot refer to
other modules (though they can refer to it).

Change-Id: If878bc2bb426c0e5478ed031f1144e26f05e85a2
2015-06-13 03:00:10 +12:00
Robert Collins
41abe54de4 Move tests into openstack_requirements.
There's no reason to have this project be a special snowflake, lets
use the convention we do elsewhere in OpenStack.

Change-Id: I6ee4cbaa958dcc9c1018dbfdfda083dcb07eea00
2015-06-08 22:07:01 +12:00