92 Commits

Author SHA1 Message Date
Dirk Mueller
4153903939 Add policy check on global-requirements.txt entries
We want to ensure that every entry in global-requirements has
at least a minimum version specified, and if there is an exclude,
the exclude is not outside the lower bounds.

Change-Id: Ie12d99d3317bd4c81de9a47a43fa99345b2b9664
2017-09-15 20:49:33 +00:00
Dirk Mueller
ee1a3433f1 Xfail the mitmproxy / pyOpenSSL dependency conflict
This is already fixed in master:
https://github.com/mitmproxy/mitmproxy/pull/2435

So it'll hopefully in the next release.

Because the pyOpenSSL package name and requirement name do not match we
also add code to lower the name for comparison.

Also the check-uc only complains about a single issue but there are 3.

Change-Id: I6657802908ac4052eeb6c73fb03098b0c3557eb2
2017-09-15 14:46:26 -06:00
howardlee
4b01754380 Update and replace http with https for doc links
1) Update doc links according to OpenStack document migration
2) Use https instead of http for docs links

Change-Id: I2d558e68b060ae893bafccf51333d1fd5d061322
2017-07-22 11:14:40 +08:00
Dougal Matthews
bb1a083fea Remove mistral from global requirements
Historically, to write Mistral actions you would need to depend on
Mistral itself. However, with the introduction of mistral-lib this is no
longer required. The only consumer of mistral-as-a-library within
OpenStack was tripleo-common. It has been fully migrated to mistral-lib
(see depends-on) and the requirement can thus be removed.

Change-Id: I0a99c19d3cbcfbb396b09c008aff7a379a273d4c
Depends-On: I7ae2e0e4dbca6c02ed8ceedc066ca35eeb13c322
2017-06-28 20:44:02 +00:00
Doug Hellmann
c671d3b0c1 do not sync the blacklist to projects
Depends-On: I90362ea9968f423f0e6aeb638adb0e38bdf097d0
Change-Id: I14dab0936e3bae196f29e867f4987f6b38c728aa
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
2017-06-15 16:26:53 -04:00
Doug Hellmann
d500f3c24c fix typo that made updating the tests confusing
Change-Id: I11cc7964ef3b977b2245a63d7d0582c80724b2d4
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
2017-06-13 23:14:55 +02:00
Jenkins
f2370c9c2c Merge "Create check_constraints command" 2017-05-22 23:38:48 +00:00
Zhongcheng Lao
f562496f7d Fixed wrong regular expression for vcs requirements
The regular expression to parse requirement line is also used for
constrains files. However, it will not match those lines which
point to VCS, such as git+http or git+https which pypi accept.

This commit fixes the issue.

Despite that we only need to change the RE to
^(?P<url>\s*(?:-e\s)?\s*(?:(?:git\+)?(?:https|http)|file)://[^#]*)#
egg=(?P<name>[-\.\w]+)
to fix the issue for git+http and git+https, pypi accepts more VCS
URI schemes other than git. This patch also looses the match for
these cases to be accepted.

Change-Id: I202078f5ca4ba6da4ab0ed52d97d41997674c1b4
2017-05-21 04:54:50 -07:00
Matthew Oliver
302ba18d81
Create check_constraints command
The check_constraints utility wraps a new check_exists cmd.
This command looks at a project's requirements and checks:

  - That the requirement name alone exists in both g-r and u-c
  - If it exists in both, then check that the version in u-c
    is covered.
  - Added an extra g-r check as a flag, but not quite happy with it.
    It needs more work.. but would be a _little_ more correct
    if/when g-r stops defining the minimum.

This patch is currently WIP as tests are still required. But is
the starting point for the fuzzy checking for projects managing
their own lower-constraints.

Change-Id: I36690fddcfc24d49aab0d28d7d46dcd8d9128b2a
2017-05-19 15:03:15 -05:00
Tony Breeds
bc10840484 Match the minimum from global-requirements in setup.py
It's somewhat strange to say you can build a package with pbr >= 1.8 but
to install it you need 2.0.0.  Synchronise these values.

Change-Id: I47640a0226be2e82f2fbbf866d2c235f5e7ed415
2017-03-02 14:24:58 +11:00
Dirk Mueller
e04ec4896c remove comment header generation
The validation command is not able to handle that and I'm not sure
if it is really worth fixing that.

Change-Id: I8abc66143333522b77ab4ff6b2fef66bc707a36a
2017-02-22 16:36:33 +01:00
Jenkins
d32dc2e1f5 Merge "use sha1 hashes to sort constraints" 2017-02-21 21:18:19 +00:00
Doug Hellmann
2c43fb140b add check-python2-support command
Change-Id: I3406acd2b1fea353fd1419cbe1aacc2b09588a3c
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
2017-01-24 10:34:40 -05:00
Doug Hellmann
26b6c243fc use sha1 hashes to sort constraints
We want to avoid packages with common names sorting close together in
the constraints list, because when we batch-release from projects like
Oslo we have a lot of merge conflicts trying to land those individual
patches. This changes the constraints list to be sorted based on the
sha1 hash of the dependency name (ignoring version information so we
have some consistency in the ordering). This gives us the appearance of
randomness while maintaining a stable sort.

Change-Id: I761967f8281c4d5df02e1dc88895b76d9ac89e6e
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
2017-01-18 16:23:50 -05:00
Cady_Chen
c443870c32 Fix typo in openstack_requirements/cmds/generate.py
Change-Id: Ic316c428b1704c8685e3d9533d2776f825c50af0
2016-12-07 10:11:59 +08:00
Tony Breeds
ffe658de93 Add the concept of a version map for constraints generation
We need to generate constraints for all python versions we use.  This is
python 2.7 (trusty and xenial), 3.4 (trusty) and 3.5(xenial).  There
isn't a supportable way to install all 3 versions on a single node.

Add a version-map that allows us to clone the output of a freeze between
python versions.  This assumes that the python version in question are
compatible during the transition window.

It's possible to run with:
    ... --version-map 3.4:3.5 --version-map 3.5:3.4 ...

This will mean that we generate constraints consistently between versions
regardless of which version /usr/bin/python3 is.

Change-Id: I55a3597ea2f2bf48e30dc564a4bf4b2ef7f178b3
2016-10-06 16:22:18 +11:00
Jenkins
53f820a155 Merge "Quick wins for python3 support" 2016-10-04 13:35:34 +00:00
Jenkins
3cc50f2a57 Merge "Fixes for check-uc jenkins job" 2016-10-04 04:59:10 +00:00
Dirk Mueller
d59c8ece29 Fixes for check-uc jenkins job
Implement XFAIL handling for ignoring known problems so that
we can make the job voting until the remaining issues are sorted
out.

Change-Id: Ia2cf2c440a72e6365a9ff08086704dc535bbc072
2016-09-29 20:33:34 +02:00
gengchc2
9d890b2856 Use ConfigParser instead of SafeConfigParser
in Python 3 The SafeConfigParser class has been renamed
to ConfigParser in Python 3.2 [1]. The alias SafeConfigParser
maybe removed in future versions of Python 3. Use ConfigParser
instead of SafeConfigParser in Python 3.We're switching from
SafeConfigParser to ConfigParser because we're we're not explicitly
using the 'Safe' functionality.So it make the code potentially neater
if SafeConfigParser is even de-aliased in python3.

[1] http://bugs.python.org/issue10627

Change-Id: I932ced300a93040144e89c7f4acc95480b32749d
2016-09-29 08:57:54 +00:00
Tony Breeds
4188b2900d Quick wins for python3 support
Change-Id: I9b04543bd2fe9a3f52232040586041cb340a1133
2016-09-29 16:48:46 +10:00
Ihar Hrachyshka
b97f008059 Use python3 to calculate freezes
We need the integration gate to work with both trusty (python3.4) and
xenial (python3.5).  Rather than hard code the full python binary just
use python 3 and let the OS mange this for us. This is less than ideal
but it's the best we can do right now to unblock the gate.

We also address a short term issue with out upper-constraints.txt files
and manually copy the constraints from 3.4 to 3.5

Related-Bug: 1620436
Change-Id: I6acabf86933b7a7bce7baf44e6b512e7d0d2f6db
2016-09-06 03:34:23 +00:00
Haikel Guemar
4ede8784c9 Add --save option to normalize-requirements command
Change-Id: Id6fe76e76fa87939475cb56f376aa084e61a9db0
2016-06-23 22:27:22 +02:00
Tony Breeds
4bb92e041c Integrate normalize as part of the requirements tools
Rather than have this as an external tool that open codes some of the logic
we have in the existing codebase we should integrate it.

Change-Id: I979dcefeec4ae28705088d4fac1e41b4d33e40e0
2016-06-23 10:16:23 +02:00
Davanum Srinivas
9e0ff3a550 Check conflicts script
Use pkg_resources.require to report conflicts to ensure
co-installability.

Co-Authored-By: Igor Yozhikov <iyozhikov@mirantis.com>
Change-Id: I7443496b6a06d02bb9cc9457b729dea71451f44a
2016-06-02 16:24:00 -04:00
Jenkins
5b3477502c Merge "Check that the argument is in fact a directory" 2016-05-30 06:00:57 +00:00
Swapnil Kulkarni (coolsvap)
c21062a307 Drop hgtools
Actually unreferenced in any project subject to g-r

Co-Authored-By: Swapnil Kulkarni <me@coolsvap.net>
Change-Id: Idb76cd4f450cf09853982b905c41473b20c51735
2016-05-27 09:19:17 +00:00
Tony Breeds
2ee3cbb721 Check that the argument is in fact a directory
If you mistakenly pass a path that isn't a directory, or doens't exist,
you get a traceback that ends with:
 ---
  File "/Users/tony8129/projects/openstack/openstack/requirements/openstack_requirements/cmds/update.py", line 229, in _copy_requires
    dest_path, project.merge_setup_cfg(proj['setup.cfg'], output_extras)))
KeyError: 'setup.cfg'
ERROR: InvocationError: '.../openstack/requirements/.tox/update/bin/update-requirements -H -s ./nova'
---

This is a little confusing.  This patch checks that the arg is a
directory so a more explicit error can be generated.

Change-Id: I9f3f43d5c9b90746bbf2d651ee4cb37afade1528
2016-05-23 14:09:52 +10:00
Jenkins
2e7df9c0ac Merge "add a validation command to ensure projects use check-requirements" 2016-05-17 10:24:01 +00:00
Jenkins
af1eccf965 Merge "Fix use of a renamed itertools function in python3" 2016-05-16 20:58:20 +00:00
Doug Hellmann
bcc1779def add a validation command to ensure projects use check-requirements
Add a command and tox environment to let us define a job to ensure that
all repos in projects.txt have the check-requirements job defined. Two
projects are missing the job now, so this patch depends on the patch to
add those jobs.

Change-Id: I61658a7265a77e72401978dda558f41b28a81d92
Depends-On: I81e1c70257e28583e776171580d75341e620bf8d
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
2016-05-12 16:15:46 -04:00
Jamie Lennox
5033a0f716 Allow maintaining extras from project
A project needs to be able to specify that it depends on a project
including extras for that project and still have it's version maintained
by global-requirements.

Currently if the requirements are seen to mismatch then the string from
global requirements is dropped in place of the project string, but this
drops any extras specified by project.

In the case that extras are different and the version needs updating add
a new combined requirement to the project with the original extras and
the new version.

Closes-Bug: #1567809
Change-Id: Ife48b7963a5e6706289f1b9a47cb95fae7f0bc22
2016-04-14 15:01:13 +10:00
Jamie Lennox
6410265574 Update requirements ordering
When using generated version output strings these produce stable but
unusually ordered outputs. Fudge the tests to create the expected
version strings to ease the update process.

Change-Id: I021f2befe2422525f528e27b041f60ec684e1eae
2016-04-14 11:41:30 +10:00
Jenkins
511a413e75 Merge "Add wheel to UNCONSTRAINABLE" 2016-04-06 12:54:53 +00:00
Jenkins
54961eb3b4 Merge "Add Windows as supported operating system" 2016-04-05 18:37:42 +00:00
Davanum Srinivas
5dcd3d3fe3 Add wheel to UNCONSTRAINABLE
pip freeze no longer outputs wheel which breaks
jobs in the review that the bot/script proposes.
So let's mark it as UNCONSTRAINABLE.

Change-Id: Ieb33a54117b4579e28dfdb9a9ec9629f29c42d8b
2016-04-05 06:39:29 -04:00
Stéphane Albert
62818c3069 Fix use of a renamed itertools function in python3
In python3 izip functions got renamed to zip in itertools module.
This was breaking support for python3.

Change-Id: I6fda87b78e40be134d7c94698cca5d85a51eb989
2016-03-30 11:40:14 +02:00
Davanum Srinivas (dims)
f247d93677 Revert "Tolerate both new and old setuptools"
This reverts commit 0cf3c06ad98fa86378ffd561b65c0154989b179c.

setuptools 20.3 was released today which fixes the
vendorized exception problem.

Change-Id: I24edaf6cb2f0b354e2ebb33451f1f15bfd888183
2016-03-15 21:05:40 -04:00
Davanum Srinivas
0cf3c06ad9 Tolerate both new and old setuptools
New setuptools 20.2.2 seems to be throwing a new exception
as it uses a vendorized version of packaging now. Let's make
sure our tests works with both the older and newer versions.

Change-Id: Iab9247e2026896bb7d25b125788ec3072f819dad
2016-02-28 14:57:24 -05:00
Davanum Srinivas
d83f02b08e Add pymi to UNCONSTRAINABLE list
the validate script has trouble with win32 in the
sys_platform. We have a special list that skips
such packages. Since pywin32 is just like pymi and
is already in the list, added pymi there as well.

Change-Id: Ib811aa88ba3b78915041590a2766cf6097c70061
2016-02-26 08:11:00 -08:00
Costin Galan
7bca319451 Add Windows as supported operating system
This change is necessary in order to use the project's
update-requirements.py script when preparing the environment on the
Hyper-V compute nodes used in the Hyper-V CI. Using the mentioned script
will set the latest requirements in nova, neutron, manila, etc.,
reducing the CI's amount of failures caused by outdated requirements.

Including Microsoft Windows as a supported operating system on this
project. Also, we update the edit-constraints.py and project.py
files to support os.rename() of Windows.

On Windows, os.rename() will create a file with the source
name, without deleting the initial file first.

Example of a traceback:

 Traceback (most recent call last):
   File "C:\Python27\Scripts\edit-constraints-script.py", line 9, in
 <module>
     load_entry_point('openstack.requirements==0.0.1.dev2497',
 'console_scripts', 'edit-constraints')()
   File
 "C:\Python27\lib\site-packages\openstack_requirements\cmds\edit_constraint.py",
 line 74, in main
     os.rename(args[0] + '.tmp', args[0])
 WindowsError: [Error 183] Cannot create a file when that file already exists

This could be easily solved by removing the file before the rename.

Proposed fix:
Add before "os.rename(args[0] + '.tmp', args[0])" the line "os.remove(args[0])"
in edit_constraint.py
Add before "os.rename(tmpname, fullname)" the line "os.remove(fullname)"
in project.py
Add "Operating System :: Microsoft :: Windows" in setup.cfg

DocImpact: Added Windows as a supported OS in setup.cfg
Change-Id: If123a65fd8d49d5c67a1db16827a9617ce520dba
Signed-off-by: Costin Galan <cgalan@cloudbasesolutions.com>
2016-02-26 14:25:50 +02:00
Zhao Lei
3abda1e284 Fix a little spelling typo in README and error message
s/occured/occurred/
s/occuring/occurring/

Change-Id: Iaec8aeb0ec5c2c3638146f525868b65ab943f71f
Signed-off-by: Zhao Lei <zhaolei@cn.fujitsu.com>
2015-12-11 09:26:28 +00:00
Davanum Srinivas
47fdbc1391 Deal with underscores in blacklist during generate-constraints
In change id  I60adf0dca4aa32f4ef6bca61250b375c8a3703c6, we added
flake8_docstrings to the black list. However as seen in
I8bfff1b2e65541959e7f2ae0fa6ca6c17889eb54, the generate-constraints
job ends up proposing adding "flake8-docstrings===0.2.1.post1" to
the upper-constraints.txt. This is because of the '-' vs '_'
difference. So just like in Ibaa22657a2cf2c0ad96dbd0b9bc43cdafe6a1d56,
we need to convert the items in black list to the correct safe name
before trying to check if the item is in the black list.

Change-Id: Ieb1cd44e908b80d58a20bf0d0e1e6b10f17ae95e
2015-11-07 23:00:12 -05:00
Davanum Srinivas
33636c1d81 Expose extras information in requirements
Starting with oslo.db+Nova, we would like to specify say
oslo.db[fixtures] in Nova's test-requirements. If you see
the proposed review I4131e534d4cb12e4888d398fa4fb7c922e369210
the gate-nova-requirements fails because code in
project-config/jenkins/scripts/project-requirements-change.py
does not understand extras at all and treats both oslo.db
and oslo.db[fixtures] the same. Since that module
uses requirement.py, we should expose the information here
first and then fix project-config after that.

Change-Id: I146baa3ef94cc8bbf29af371786f3ea95a42cb9f
2015-10-22 15:26:48 -04:00
Jenkins
ef82c13bf7 Merge "Deal with underscores in names in edit-constraints" 2015-09-17 16:09:16 +00:00
Robert Collins
b95d747cfa Update our handling of pbr.
*) We're removing our upper caps - they are just time bombs. Make pbr
   be consistent here too.

*) pbr 1.8 handles comments in marker lines properly. We don't need to
   update the run-time minimum, but we do need a build-time minimum of
   1.8 to ensure OpenStack packages can build properly.

Change-Id: I9fe0c62e774c00c2e521e1e69c5f2f5d87865ec1
2015-09-17 10:20:17 +12:00
Davanum Srinivas
3ebedf84f0 Deal with underscores in names in edit-constraints
Problem showed up with django_openstack_auth job that installs from
source. The change to devstack to verify LIBS_FROM_GIT kicked in
(Iffef2007f99a0e932b68c4c897ebbfb748cac2b4) and starting failing the
job. The problem was that the edit_constraint was not fixing the
django-openstack-auth line in upperconstraints.txt as the setup.cfg
had name = django_openstack_auth. Note that when things finally get
installed setup_tools' safe_name kicks in to standardize names and for
example glance_store will show up as glance-store in pip freeze. So we
should be able to treat this situation better and allow constraints to
have the safe names (using dashes).

Co-Authored-By: Robert Collins <rbtcollins@hp.com>
Change-Id: Ibaa22657a2cf2c0ad96dbd0b9bc43cdafe6a1d56
2015-09-17 10:12:11 +12:00
Victor Sergeyev
e45b270ad8 Fix url regexp to proceed oslo-specific url's properly
The current url_re regexp ignores dots in egg section, so it proceeds
url with "#egg=oslo.messaging" as "#egg=oslo". This patch fixes it.

Change-Id: Ifcea3e825218578f799c0a951baac8f8e8ef5de5
2015-08-06 17:14:39 +03:00
Doug Hellmann
2dbf39295c Move data validation out of the unit tests and strengthen
Having the data validation happening inside the unit tests makes
understanding requirement update failures confusing because the output
is hard to read and does not explain what the error actually is. Move
the checks to their own command and set up a tox env to run it. A
separate patch in project-config will add a new job for the repository.

Address the dedent comment from
https://review.openstack.org/#/c/204181/3/openstack_requirements/tests/test_requirement.py,cm

Add a test to ensure that all items in global-requirements.txt are also
listed in either upper-constraints.txt or blacklist.txt. Ignore a few
items that we don't know how to constrain yet.

Add a test to ensure that items in blacklist.txt are not in
upper-constraints.txt.

Change-Id: Icb717b0f36afb6ea29f50bc6935917dddf47fd4c
2015-08-03 20:45:37 +00:00
Doug Hellmann
43a876b678 Fix constraint integration test
The test to validate the constraints file against the global requirements
list was looking for any spec from any package to match. Fix it to only
look at the specs for the package from the constraint being checked.

Add some more unit tests to demonstrate the old conditions that were
failing, and to verify the new logic passes.

Add stdout capture to the tests so we can provide more helpful error
messages to folks contributing patches to the different requirement
lists about why the integration test might be failing.

Add a test to enforce the format of upper-constraints.txt using 3 = to
indicate an absolute constraint. Fix 2 constraints which were previously
specified with only 2 =.

Change-Id: I46fd11031ba8f775fe62406380bf6417684f1f06
2015-07-23 16:55:13 +00:00