62 Commits

Author SHA1 Message Date
Matthew Thode
58d3af1960
remove the renderspec from requirements sync
remove pymod2pkg as well as it was only used in renderspec

Change-Id: If00a0efed80f8ec821397805f59ee4da483361e4
Signed-off-by: Matthew Thode <mthode@mthode.org>
2018-02-07 13:07:06 -06:00
Zuul
ea96aaa3d7 Merge "Add pycodestyle" 2018-01-17 18:15:42 +00:00
Swapnil Kulkarni
2f301abf8c Remove MySQL-python
The overwhelming majority is converted to PyMySQL, and we don't
want to add more users.
See https://wiki.openstack.org/wiki/PyMySQL_evaluation and
http://codesearch.openstack.org/?q=MySQL-python&i=nope&files=.*.txt&repos=

Depends-On: Ie17f4543fa3d72b507d88d2c4023a9a5a430419e
Change-Id: I1796dd30cb7adb30c7d51139288109c528e9e15e
2017-12-20 13:09:26 +05:30
Zuul
4fdf99df5e Merge "Remove django-openstack-auth" 2017-12-05 10:24:39 +00:00
Dirk Mueller
4d787dc4fa Remove django-openstack-auth
This was merged into horizon under the new name "openstack_auth"
see https://review.openstack.org/#/c/482561/

Depends-On: I93f6365ccf661568b37ea9c737f8ce6115873739
Change-Id: Iec8a64b25aa4ab188a940cf919545275d6967391
2017-12-01 08:22:42 +01:00
Dirk Mueller
1a9a31d361 Add pycodestyle
pep8 was renamed to pycodestyle, so allow that one in order to
be able to transition to it.

The depends-on remove pyflake mentions as most projects adopted
flake8/hacking, we don't need to allow explicit dependencies
on the tools integrated by flake8 (mccabe, pycodestyle/pep8, pyflakes).

Cleanup those from the tests/ files to avoid grep'ing finding them.

Depends-On: I12b5e114c5c6783e9da5cca2818ac9266a00fc87
Depends-On: Ib145e2afa97f441c07fada9c30f0f0e2410870ae
Depends-On: Id44c90ddd4a8a2204fd7ce600097bafbc9469993

Change-Id: If9f30bf39143d5e6fefca794385cafc4456d2197
2017-11-14 23:55:27 +01:00
Dirk Mueller
33aba8e192 Remove cliff-tablib
This is merged with cliff and shouldn't be used separately anymore

Depends-On: I8221542e666502986626b9fa53956721da1a3220
Depends-On: I24d0c348b040f601f9f36b08972c27d0289540de
Change-Id: I0607e905e4f2bb33d5a329c50d379b59250394a9
2017-11-14 10:18:18 +05:30
Dirk Mueller
4153903939 Add policy check on global-requirements.txt entries
We want to ensure that every entry in global-requirements has
at least a minimum version specified, and if there is an exclude,
the exclude is not outside the lower bounds.

Change-Id: Ie12d99d3317bd4c81de9a47a43fa99345b2b9664
2017-09-15 20:49:33 +00:00
howardlee
4b01754380 Update and replace http with https for doc links
1) Update doc links according to OpenStack document migration
2) Use https instead of http for docs links

Change-Id: I2d558e68b060ae893bafccf51333d1fd5d061322
2017-07-22 11:14:40 +08:00
Dougal Matthews
bb1a083fea Remove mistral from global requirements
Historically, to write Mistral actions you would need to depend on
Mistral itself. However, with the introduction of mistral-lib this is no
longer required. The only consumer of mistral-as-a-library within
OpenStack was tripleo-common. It has been fully migrated to mistral-lib
(see depends-on) and the requirement can thus be removed.

Change-Id: I0a99c19d3cbcfbb396b09c008aff7a379a273d4c
Depends-On: I7ae2e0e4dbca6c02ed8ceedc066ca35eeb13c322
2017-06-28 20:44:02 +00:00
Doug Hellmann
c671d3b0c1 do not sync the blacklist to projects
Depends-On: I90362ea9968f423f0e6aeb638adb0e38bdf097d0
Change-Id: I14dab0936e3bae196f29e867f4987f6b38c728aa
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
2017-06-15 16:26:53 -04:00
Doug Hellmann
d500f3c24c fix typo that made updating the tests confusing
Change-Id: I11cc7964ef3b977b2245a63d7d0582c80724b2d4
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
2017-06-13 23:14:55 +02:00
Jenkins
f2370c9c2c Merge "Create check_constraints command" 2017-05-22 23:38:48 +00:00
Zhongcheng Lao
f562496f7d Fixed wrong regular expression for vcs requirements
The regular expression to parse requirement line is also used for
constrains files. However, it will not match those lines which
point to VCS, such as git+http or git+https which pypi accept.

This commit fixes the issue.

Despite that we only need to change the RE to
^(?P<url>\s*(?:-e\s)?\s*(?:(?:git\+)?(?:https|http)|file)://[^#]*)#
egg=(?P<name>[-\.\w]+)
to fix the issue for git+http and git+https, pypi accepts more VCS
URI schemes other than git. This patch also looses the match for
these cases to be accepted.

Change-Id: I202078f5ca4ba6da4ab0ed52d97d41997674c1b4
2017-05-21 04:54:50 -07:00
Matthew Oliver
302ba18d81
Create check_constraints command
The check_constraints utility wraps a new check_exists cmd.
This command looks at a project's requirements and checks:

  - That the requirement name alone exists in both g-r and u-c
  - If it exists in both, then check that the version in u-c
    is covered.
  - Added an extra g-r check as a flag, but not quite happy with it.
    It needs more work.. but would be a _little_ more correct
    if/when g-r stops defining the minimum.

This patch is currently WIP as tests are still required. But is
the starting point for the fuzzy checking for projects managing
their own lower-constraints.

Change-Id: I36690fddcfc24d49aab0d28d7d46dcd8d9128b2a
2017-05-19 15:03:15 -05:00
Tony Breeds
ffe658de93 Add the concept of a version map for constraints generation
We need to generate constraints for all python versions we use.  This is
python 2.7 (trusty and xenial), 3.4 (trusty) and 3.5(xenial).  There
isn't a supportable way to install all 3 versions on a single node.

Add a version-map that allows us to clone the output of a freeze between
python versions.  This assumes that the python version in question are
compatible during the transition window.

It's possible to run with:
    ... --version-map 3.4:3.5 --version-map 3.5:3.4 ...

This will mean that we generate constraints consistently between versions
regardless of which version /usr/bin/python3 is.

Change-Id: I55a3597ea2f2bf48e30dc564a4bf4b2ef7f178b3
2016-10-06 16:22:18 +11:00
Tony Breeds
4188b2900d Quick wins for python3 support
Change-Id: I9b04543bd2fe9a3f52232040586041cb340a1133
2016-09-29 16:48:46 +10:00
Jenkins
5b3477502c Merge "Check that the argument is in fact a directory" 2016-05-30 06:00:57 +00:00
Swapnil Kulkarni (coolsvap)
c21062a307 Drop hgtools
Actually unreferenced in any project subject to g-r

Co-Authored-By: Swapnil Kulkarni <me@coolsvap.net>
Change-Id: Idb76cd4f450cf09853982b905c41473b20c51735
2016-05-27 09:19:17 +00:00
Tony Breeds
2ee3cbb721 Check that the argument is in fact a directory
If you mistakenly pass a path that isn't a directory, or doens't exist,
you get a traceback that ends with:
 ---
  File "/Users/tony8129/projects/openstack/openstack/requirements/openstack_requirements/cmds/update.py", line 229, in _copy_requires
    dest_path, project.merge_setup_cfg(proj['setup.cfg'], output_extras)))
KeyError: 'setup.cfg'
ERROR: InvocationError: '.../openstack/requirements/.tox/update/bin/update-requirements -H -s ./nova'
---

This is a little confusing.  This patch checks that the arg is a
directory so a more explicit error can be generated.

Change-Id: I9f3f43d5c9b90746bbf2d651ee4cb37afade1528
2016-05-23 14:09:52 +10:00
Jamie Lennox
5033a0f716 Allow maintaining extras from project
A project needs to be able to specify that it depends on a project
including extras for that project and still have it's version maintained
by global-requirements.

Currently if the requirements are seen to mismatch then the string from
global requirements is dropped in place of the project string, but this
drops any extras specified by project.

In the case that extras are different and the version needs updating add
a new combined requirement to the project with the original extras and
the new version.

Closes-Bug: #1567809
Change-Id: Ife48b7963a5e6706289f1b9a47cb95fae7f0bc22
2016-04-14 15:01:13 +10:00
Jamie Lennox
6410265574 Update requirements ordering
When using generated version output strings these produce stable but
unusually ordered outputs. Fudge the tests to create the expected
version strings to ease the update process.

Change-Id: I021f2befe2422525f528e27b041f60ec684e1eae
2016-04-14 11:41:30 +10:00
Davanum Srinivas (dims)
f247d93677 Revert "Tolerate both new and old setuptools"
This reverts commit 0cf3c06ad98fa86378ffd561b65c0154989b179c.

setuptools 20.3 was released today which fixes the
vendorized exception problem.

Change-Id: I24edaf6cb2f0b354e2ebb33451f1f15bfd888183
2016-03-15 21:05:40 -04:00
Davanum Srinivas
0cf3c06ad9 Tolerate both new and old setuptools
New setuptools 20.2.2 seems to be throwing a new exception
as it uses a vendorized version of packaging now. Let's make
sure our tests works with both the older and newer versions.

Change-Id: Iab9247e2026896bb7d25b125788ec3072f819dad
2016-02-28 14:57:24 -05:00
Davanum Srinivas
47fdbc1391 Deal with underscores in blacklist during generate-constraints
In change id  I60adf0dca4aa32f4ef6bca61250b375c8a3703c6, we added
flake8_docstrings to the black list. However as seen in
I8bfff1b2e65541959e7f2ae0fa6ca6c17889eb54, the generate-constraints
job ends up proposing adding "flake8-docstrings===0.2.1.post1" to
the upper-constraints.txt. This is because of the '-' vs '_'
difference. So just like in Ibaa22657a2cf2c0ad96dbd0b9bc43cdafe6a1d56,
we need to convert the items in black list to the correct safe name
before trying to check if the item is in the black list.

Change-Id: Ieb1cd44e908b80d58a20bf0d0e1e6b10f17ae95e
2015-11-07 23:00:12 -05:00
Davanum Srinivas
33636c1d81 Expose extras information in requirements
Starting with oslo.db+Nova, we would like to specify say
oslo.db[fixtures] in Nova's test-requirements. If you see
the proposed review I4131e534d4cb12e4888d398fa4fb7c922e369210
the gate-nova-requirements fails because code in
project-config/jenkins/scripts/project-requirements-change.py
does not understand extras at all and treats both oslo.db
and oslo.db[fixtures] the same. Since that module
uses requirement.py, we should expose the information here
first and then fix project-config after that.

Change-Id: I146baa3ef94cc8bbf29af371786f3ea95a42cb9f
2015-10-22 15:26:48 -04:00
Davanum Srinivas
3ebedf84f0 Deal with underscores in names in edit-constraints
Problem showed up with django_openstack_auth job that installs from
source. The change to devstack to verify LIBS_FROM_GIT kicked in
(Iffef2007f99a0e932b68c4c897ebbfb748cac2b4) and starting failing the
job. The problem was that the edit_constraint was not fixing the
django-openstack-auth line in upperconstraints.txt as the setup.cfg
had name = django_openstack_auth. Note that when things finally get
installed setup_tools' safe_name kicks in to standardize names and for
example glance_store will show up as glance-store in pip freeze. So we
should be able to treat this situation better and allow constraints to
have the safe names (using dashes).

Co-Authored-By: Robert Collins <rbtcollins@hp.com>
Change-Id: Ibaa22657a2cf2c0ad96dbd0b9bc43cdafe6a1d56
2015-09-17 10:12:11 +12:00
Victor Sergeyev
e45b270ad8 Fix url regexp to proceed oslo-specific url's properly
The current url_re regexp ignores dots in egg section, so it proceeds
url with "#egg=oslo.messaging" as "#egg=oslo". This patch fixes it.

Change-Id: Ifcea3e825218578f799c0a951baac8f8e8ef5de5
2015-08-06 17:14:39 +03:00
Doug Hellmann
2dbf39295c Move data validation out of the unit tests and strengthen
Having the data validation happening inside the unit tests makes
understanding requirement update failures confusing because the output
is hard to read and does not explain what the error actually is. Move
the checks to their own command and set up a tox env to run it. A
separate patch in project-config will add a new job for the repository.

Address the dedent comment from
https://review.openstack.org/#/c/204181/3/openstack_requirements/tests/test_requirement.py,cm

Add a test to ensure that all items in global-requirements.txt are also
listed in either upper-constraints.txt or blacklist.txt. Ignore a few
items that we don't know how to constrain yet.

Add a test to ensure that items in blacklist.txt are not in
upper-constraints.txt.

Change-Id: Icb717b0f36afb6ea29f50bc6935917dddf47fd4c
2015-08-03 20:45:37 +00:00
Doug Hellmann
43a876b678 Fix constraint integration test
The test to validate the constraints file against the global requirements
list was looking for any spec from any package to match. Fix it to only
look at the specs for the package from the constraint being checked.

Add some more unit tests to demonstrate the old conditions that were
failing, and to verify the new logic passes.

Add stdout capture to the tests so we can provide more helpful error
messages to folks contributing patches to the different requirement
lists about why the integration test might be failing.

Add a test to enforce the format of upper-constraints.txt using 3 = to
indicate an absolute constraint. Fix 2 constraints which were previously
specified with only 2 =.

Change-Id: I46fd11031ba8f775fe62406380bf6417684f1f06
2015-07-23 16:55:13 +00:00
Jenkins
5174071897 Merge "Add checks for setup.cfg existence in project dictionary" 2015-07-17 11:02:59 +00:00
Gal Sagie
97ddc985ab Add checks for setup.cfg existence in project dictionary
This check is needed by new projects at their first commit
Without this fix the following initial commit for new project
failed:
https://review.openstack.org/#/c/201701/

The exception that it failed was originated in this method
with KeyError on the project['setup.cfg'] (see previous
jenkins logs on the above commit)

Log: http://logs.openstack.org/01/201701/6/check/gate-kuryr-requirements/e922983/console.html

Change-Id: I39db2918ea277b939f67d3c856f52bd53c226f27
2015-07-15 11:08:48 +03:00
Jenkins
d859b7dcb3 Merge "Stop adding extra newline after unparseable lines." 2015-07-15 00:08:30 +00:00
Russell Bryant
931defba21 Stop adding extra newline after unparseable lines.
I have a project that includes a requirement line this:

    -e git://git.openstack.org/openstack/neutron.git#egg=neutron

Every time update.py runs against the file, it adds an extra newline
after this.  For example, if I run update.py a few times, I'll have a
few new lines in the file.  This patch prevents the extra newlines
from getting added.

Change-Id: I92d02a45c437025b16df067d7b58a3727d42c0e0
Signed-off-by: Russell Bryant <rbryant@redhat.com>
2015-07-08 10:37:02 -04:00
Robert Collins
6b06c5d0fd Don't permit embedded #'s in URLs.
This is a fix for a minor issue caught in review of 198186

Change-Id: Ic27acf79b4a189f9348edcd37523403202e6fc0b
2015-07-08 10:38:44 +12:00
Robert Collins
63e5747001 Fix editing constraints files with urls in them.
constraints files, unlike global_requirements, need to be able to
contain urls (specifically file:/// paths) so that we can constrain
libraries to be run from their git trees, rather than having them flap
all over the place as different servers specify different bounds. The
first edit-constraints implementation failed to parse such lines,
throwing errors instead.

This patch makes the support for urls optional, preventing changing
the semantics of requirements update.py logic.

Change-Id: I33bd2d9dff9cb7dc1a50177db7286b7317966784
Depends-On: I0f07858e96ea3baf46f8a453e253b9ed29c7f7e2
2015-07-04 09:54:39 +12:00
Robert Collins
b838ab1e43 Add location to requirement.Requirement
This will provide space to store the result of parsing requirements
like -e file:///opt/stack/new/oslo.THING, which we need to do for
edit-constraint to work in devstack when more than one library is
being installed from git. This is a separate patch to make reviewing
the actual new functionality easier - it is entirely mechanical,
adding the field and dealing with test fallout.

Change-Id: Ied4e92717b0dab871dac42ae639d1f2a6c236414
2015-07-03 12:23:09 +12:00
Robert Collins
893490c958 Add edit-constraints command.
This allows shell editing of a constraints file, which we need to do
in devstack when installing a constrained library from git.

Change-Id: Ibdbfc786b5d2f56d2e41d329c8766bef7b2e6854
2015-06-29 15:44:17 +12:00
Robert Collins
3aa37f7ea4 Move commands into cmds subdir.
This is a useful hint to readers about whether a module is library or UI.

Change-Id: I177449fc6820e82fed938f2c216b6b9f9192957f
2015-06-29 15:27:25 +12:00
Robert Collins
50eee6fc57 Split project and requirement types out.
These don't need to be hidden within update.py anymore, and we need
them for lint and edit.

Change-Id: I92f4ff51f98f6343cc94a74a69660306b742dbf3
2015-06-29 15:27:25 +12:00
Robert Collins
24fe404d97 Prevent incompatible version bumps.
We shouldn't be able to bump global requirement versions if that
would be incompatible with the current test set of versions. To assess
'works' we need to run them in tests which means that we need to
update upper-constraints.txt at the same time. We don't test all
versions however, so just checking that the constraints version is
compatible with the updated global requirements is sufficient.

This adds a dependency on the pypa packaging library. The library is
maintained by the authors of setuptools and pip, is Apache V2, in good
condition and Python 3 ready. Its a test-only dependency so I don't
think the distro packaging status matters. This is *the* reference
library for packaging facilities on Python.

Change-Id: I032dd9a384453dcfa4911ea905c8f89b0c974211
2015-06-25 11:20:56 +12:00
Robert Collins
8bfdcc55ee Show not-in-global errors again.
This regressed when I half-split it into actions and IO (exception
raising). Moving it entirely into actions lets us see all the errors
very easily.

Change-Id: I12e6d910bf983841b4ad81aa41a95bf341456fe4
2015-06-24 09:47:24 +12:00
Robert Collins
5cd10eb1f6 Teach the constraints generator about excludes.
Some things like flake8 can't be constrained, so we need an opt-out
clause.

Change-Id: I11eb0d762869ad8920795fb710f1b2eeb9354f12
2015-06-24 09:42:52 +12:00
Robert Collins
4273910b78 Add upper-constraints.txt generator.
This script when run creates a constraints file with exact pins for
the transitive dependencies of a requirements file.

Change-Id: I1282f8e4010c0ec46c473495bacddf044d06c0af
2015-06-20 19:19:21 +12:00
Robert Collins
7fb4566746 Remove spurious output in test suite.
This was a debugging thing, but we don't need this verbosity for the
smoke test.

Change-Id: I3a088fae7048074a5f9ef6a2ca94dbd90067f0b2
2015-06-20 19:19:21 +12:00
Robert Collins
4b22b94752 Also update extras in setup.cfg.
This required some care. There doesn't seem to be a sane Python3 ready
comment-preserving ini parser around, so I wrote a
minimal-for-our-case one in Parsley. Parsley is already in use in
infra in bindep, but I need to add it to global-requirements as this
is the first use in a managed project of it.

Change-Id: I48de3a2f36e945f75b534f689e3af802bbdc5be9
Depends-On: I7d7e91694c9145fac0ddab8a9de5f789d723c641
Depends-On: I16e967356d5c56f1474ee661b954b3db11a608cb
2015-06-20 19:19:21 +12:00
Robert Collins
a848fdc879 Handle : as well as ; for markers.
setup.cfg uses foo:MARKER rather than foo;MARKER as requirements.txt
files do. To be able to process setup.cfg sections we need to support
':' as a marker separator.

Change-Id: I3d78b3df8ea9fad1fd7b3c634eb0fb0f7e7e9124
2015-06-19 09:15:25 +12:00
Robert Collins
91f24223ea Pull serialisation out of _sync_requirements_file.
This means we can use _sync_requirements for setup.cfg stanzas, which
have a different format to requirements.txt files.

Change-Id: Ifdb612017677fa2cf3e91de09e4b152dd11ca8ed
2015-06-19 09:15:25 +12:00
Robert Collins
937d378926 Pull parsing out of sync_requirements_file.
We're going to be parsing different sorts of files soon, and
sync_requirements shouldn't need to figure that out.

Change-Id: I98677e083cdf55a8626334bee03786aa021045e3
2015-06-17 13:54:46 +12:00
Robert Collins
a3a3f7f57a Support multiple marked reqs for the same package
This is needed when a single specifier set is insufficient to handle
both Python2.7 and 3.x. We're not currently dealing with this, but I
have observed it in other projects and we can be forearmed quite
easily.

Change-Id: If1493625d0d2ebf750159ccb8ed0fd22699766a1
2015-06-17 01:32:23 +12:00