In change id I60adf0dca4aa32f4ef6bca61250b375c8a3703c6, we added
flake8_docstrings to the black list. However as seen in
I8bfff1b2e65541959e7f2ae0fa6ca6c17889eb54, the generate-constraints
job ends up proposing adding "flake8-docstrings===0.2.1.post1" to
the upper-constraints.txt. This is because of the '-' vs '_'
difference. So just like in Ibaa22657a2cf2c0ad96dbd0b9bc43cdafe6a1d56,
we need to convert the items in black list to the correct safe name
before trying to check if the item is in the black list.
Change-Id: Ieb1cd44e908b80d58a20bf0d0e1e6b10f17ae95e
Starting with oslo.db+Nova, we would like to specify say
oslo.db[fixtures] in Nova's test-requirements. If you see
the proposed review I4131e534d4cb12e4888d398fa4fb7c922e369210
the gate-nova-requirements fails because code in
project-config/jenkins/scripts/project-requirements-change.py
does not understand extras at all and treats both oslo.db
and oslo.db[fixtures] the same. Since that module
uses requirement.py, we should expose the information here
first and then fix project-config after that.
Change-Id: I146baa3ef94cc8bbf29af371786f3ea95a42cb9f
*) We're removing our upper caps - they are just time bombs. Make pbr
be consistent here too.
*) pbr 1.8 handles comments in marker lines properly. We don't need to
update the run-time minimum, but we do need a build-time minimum of
1.8 to ensure OpenStack packages can build properly.
Change-Id: I9fe0c62e774c00c2e521e1e69c5f2f5d87865ec1
Problem showed up with django_openstack_auth job that installs from
source. The change to devstack to verify LIBS_FROM_GIT kicked in
(Iffef2007f99a0e932b68c4c897ebbfb748cac2b4) and starting failing the
job. The problem was that the edit_constraint was not fixing the
django-openstack-auth line in upperconstraints.txt as the setup.cfg
had name = django_openstack_auth. Note that when things finally get
installed setup_tools' safe_name kicks in to standardize names and for
example glance_store will show up as glance-store in pip freeze. So we
should be able to treat this situation better and allow constraints to
have the safe names (using dashes).
Co-Authored-By: Robert Collins <rbtcollins@hp.com>
Change-Id: Ibaa22657a2cf2c0ad96dbd0b9bc43cdafe6a1d56
The current url_re regexp ignores dots in egg section, so it proceeds
url with "#egg=oslo.messaging" as "#egg=oslo". This patch fixes it.
Change-Id: Ifcea3e825218578f799c0a951baac8f8e8ef5de5
Having the data validation happening inside the unit tests makes
understanding requirement update failures confusing because the output
is hard to read and does not explain what the error actually is. Move
the checks to their own command and set up a tox env to run it. A
separate patch in project-config will add a new job for the repository.
Address the dedent comment from
https://review.openstack.org/#/c/204181/3/openstack_requirements/tests/test_requirement.py,cm
Add a test to ensure that all items in global-requirements.txt are also
listed in either upper-constraints.txt or blacklist.txt. Ignore a few
items that we don't know how to constrain yet.
Add a test to ensure that items in blacklist.txt are not in
upper-constraints.txt.
Change-Id: Icb717b0f36afb6ea29f50bc6935917dddf47fd4c
The test to validate the constraints file against the global requirements
list was looking for any spec from any package to match. Fix it to only
look at the specs for the package from the constraint being checked.
Add some more unit tests to demonstrate the old conditions that were
failing, and to verify the new logic passes.
Add stdout capture to the tests so we can provide more helpful error
messages to folks contributing patches to the different requirement
lists about why the integration test might be failing.
Add a test to enforce the format of upper-constraints.txt using 3 = to
indicate an absolute constraint. Fix 2 constraints which were previously
specified with only 2 =.
Change-Id: I46fd11031ba8f775fe62406380bf6417684f1f06
Without this version, markers will fail to install in a growing number
of projects.
Change global-requirements to prevent
pip -r requirements.txt; pip install .
failing with an already installed and too old for setup_requires pbr.
Change update.py to signal to setuptools the minimum version required,
giving clearer error messages in the case of downgrades during
devstack.
Change-Id: I074274b470a7a4f2c95211677b135038ab2c942c
I have a project that includes a requirement line this:
-e git://git.openstack.org/openstack/neutron.git#egg=neutron
Every time update.py runs against the file, it adds an extra newline
after this. For example, if I run update.py a few times, I'll have a
few new lines in the file. This patch prevents the extra newlines
from getting added.
Change-Id: I92d02a45c437025b16df067d7b58a3727d42c0e0
Signed-off-by: Russell Bryant <rbryant@redhat.com>
constraints files, unlike global_requirements, need to be able to
contain urls (specifically file:/// paths) so that we can constrain
libraries to be run from their git trees, rather than having them flap
all over the place as different servers specify different bounds. The
first edit-constraints implementation failed to parse such lines,
throwing errors instead.
This patch makes the support for urls optional, preventing changing
the semantics of requirements update.py logic.
Change-Id: I33bd2d9dff9cb7dc1a50177db7286b7317966784
Depends-On: I0f07858e96ea3baf46f8a453e253b9ed29c7f7e2
This will provide space to store the result of parsing requirements
like -e file:///opt/stack/new/oslo.THING, which we need to do for
edit-constraint to work in devstack when more than one library is
being installed from git. This is a separate patch to make reviewing
the actual new functionality easier - it is entirely mechanical,
adding the field and dealing with test fallout.
Change-Id: Ied4e92717b0dab871dac42ae639d1f2a6c236414
This allows shell editing of a constraints file, which we need to do
in devstack when installing a constrained library from git.
Change-Id: Ibdbfc786b5d2f56d2e41d329c8766bef7b2e6854
We shouldn't be able to bump global requirement versions if that
would be incompatible with the current test set of versions. To assess
'works' we need to run them in tests which means that we need to
update upper-constraints.txt at the same time. We don't test all
versions however, so just checking that the constraints version is
compatible with the updated global requirements is sufficient.
This adds a dependency on the pypa packaging library. The library is
maintained by the authors of setuptools and pip, is Apache V2, in good
condition and Python 3 ready. Its a test-only dependency so I don't
think the distro packaging status matters. This is *the* reference
library for packaging facilities on Python.
Change-Id: I032dd9a384453dcfa4911ea905c8f89b0c974211
This regressed when I half-split it into actions and IO (exception
raising). Moving it entirely into actions lets us see all the errors
very easily.
Change-Id: I12e6d910bf983841b4ad81aa41a95bf341456fe4
When a non-standard package is found in the requirements list, we had an
error saying that but not saying what the bad package name was.
Change-Id: I860c548dce0c8f988f8fd29cead582bf9b9b6850
This script when run creates a constraints file with exact pins for
the transitive dependencies of a requirements file.
Change-Id: I1282f8e4010c0ec46c473495bacddf044d06c0af
We don't expect users to run setup.py directly, and the world didn't
end when the x bit was removed accidentally, so this just completes
the statement.
Change-Id: Ib3934e55949ed6bc72408ddf020b4300a1fccc80
This required some care. There doesn't seem to be a sane Python3 ready
comment-preserving ini parser around, so I wrote a
minimal-for-our-case one in Parsley. Parsley is already in use in
infra in bindep, but I need to add it to global-requirements as this
is the first use in a managed project of it.
Change-Id: I48de3a2f36e945f75b534f689e3af802bbdc5be9
Depends-On: I7d7e91694c9145fac0ddab8a9de5f789d723c641
Depends-On: I16e967356d5c56f1474ee661b954b3db11a608cb
setup.cfg uses foo:MARKER rather than foo;MARKER as requirements.txt
files do. To be able to process setup.cfg sections we need to support
':' as a marker separator.
Change-Id: I3d78b3df8ea9fad1fd7b3c634eb0fb0f7e7e9124
This means we can use _sync_requirements for setup.cfg stanzas, which
have a different format to requirements.txt files.
Change-Id: Ifdb612017677fa2cf3e91de09e4b152dd11ca8ed
We're going to be parsing different sorts of files soon, and
sync_requirements shouldn't need to figure that out.
Change-Id: I98677e083cdf55a8626334bee03786aa021045e3
This is needed when a single specifier set is insufficient to handle
both Python2.7 and 3.x. We're not currently dealing with this, but I
have observed it in other projects and we can be forearmed quite
easily.
Change-Id: If1493625d0d2ebf750159ccb8ed0fd22699766a1
This is needed to allow universal wheels when requirements differ
between Python versions (or platforms).
Change-Id: I3a6e88dc3988c9ddf0d4923e5ee6c1aa81e7a4d9
If we're interrupted mid-file-update we shouldn't leave the
requirements file half written. Instead write to a temp file and
rename into place.
Change-Id: I7c7c2553b90f756d74df1967369732b21b8e8c0d
Instead pass a verbose closure down, which while one more parameter
will prevent any skew in future unit tests.
Change-Id: I310a8fa185b2381a43b28c92ddc3f08e3c6ba035
If we detected nonstandard requirements we were previously rewriting
half the requirements file, making this non-idempotent: all the
subsequent contents was being lost.
Change-Id: If550d98fcab33a915bd44347bac1b538386d691f
Factoring out the IO lets us look at the function more clearly, as
well as increasing testability.
Change-Id: I2366ff0ef7cba16863da9d10890a3fc60098e4d0
A symlink is kept to update.py to keep this working for infra until we
update the propose scripts to try the new location and fall back.
Until that is done and the symlink removed, update.py cannot refer to
other modules (though they can refer to it).
Change-Id: If878bc2bb426c0e5478ed031f1144e26f05e85a2
There's no reason to have this project be a special snowflake, lets
use the convention we do elsewhere in OpenStack.
Change-Id: I6ee4cbaa958dcc9c1018dbfdfda083dcb07eea00