165 Commits

Author SHA1 Message Date
Tony Breeds
5a16177116 Add a tool to generate the upper-constraints.txt file for publishing
Change-Id: I25a381e55362df0e7b31f18ff45c95ce3a28dc6a
2017-09-27 09:14:04 +10:00
Tony Breeds
6c75ce543e [tools] Add a README.txt
Change-Id: I1daba8a6852255daca62c0c772a44b822360a84e
2017-09-20 16:11:20 -04:00
Tony Breeds
5b32f5237b [tools] Add pike and remove mitaka from grep-all
We now have a stable/pike branch and mitaka is EOLd (though we can't yet
remove the branch because there are still projects that are using it :(

Update the grep-all script to look in releases we care about.

Change-Id: Ia3eeda9db6b5715a3bfff5fe4f0ccd3d50ebf653
2017-08-15 23:49:00 +00:00
Tony Breeds
2b1d8f9152 Remove bogus comment
When I added the license to grep-all I copied it from another shell file
clearly no one read it as it describes tools/cruft.sh

Change-Id: Ia48d5000a360519be596c09273ba653d10e3daa6
2017-08-03 15:18:27 +10:00
Dirk Mueller
958b700317 Use git grep -h for skipping the filenames
That seems to be more portable accross different environments
(aka make it work for dirk)

Change-Id: I4eee201ed0114cf7aa9f9be39d1335326691bf99
2017-02-21 16:19:54 +01:00
Tony Breeds
a4edd9d11a Add Ocata branch remove Liberty
Change the branches looked at to match the current tracked releases.

Change-Id: Ie406b3743c296905c8db196797b5c632d4d91dca
2017-02-16 11:52:03 +11:00
Tony Breeds
b4e3ac5714 Narrow down the results/matches
If you run grep-all.sh requests, you get all the item that match that
prefix.  It is a little messy.  Make the match patten more specific so
you only get the library you pass on the command line.

Change-Id: I16544372f7efff28933da728d11f85dc5e5f9aea
2017-02-16 11:51:56 +11:00
zhangyanxian
7148c8eeb2 Fix typo in what-broke.py
"programatically" should be "programmatically"

Change-Id: I4ab60ddc688e48a76862d739fddfcfe711ad19eb
2016-12-19 07:13:35 +00:00
Jenkins
5a4f776506 Merge "Fix "wrap functions with 2 blank lines" pep8 check" 2016-11-16 15:28:57 +00:00
Cady_Chen
6898f5d20d Fix "wrap functions with 2 blank lines" pep8 check
Now gates are requires this pep8 rule as mandatory so until it'll be
fixed merge is blocked.

Change-Id: I5b1ad7ec820911b5e765e184223f786c733836a5
2016-11-16 18:52:03 +08:00
Tony Breeds
e699ece94c Add a tool do dump requirements and constraints for all branches
Often it's helpful to quickly compare the requirements and constraints
for a given library on all branches.  This tool does that.

Change-Id: I2458c5929d8812cd74f657050eea5f4f3fa11df0
2016-11-16 11:59:00 +11:00
Ihar Hrachyshka
b97f008059 Use python3 to calculate freezes
We need the integration gate to work with both trusty (python3.4) and
xenial (python3.5).  Rather than hard code the full python binary just
use python 3 and let the OS mange this for us. This is less than ideal
but it's the best we can do right now to unblock the gate.

We also address a short term issue with out upper-constraints.txt files
and manually copy the constraints from 3.4 to 3.5

Related-Bug: 1620436
Change-Id: I6acabf86933b7a7bce7baf44e6b512e7d0d2f6db
2016-09-06 03:34:23 +00:00
Andreas Jaeger
1f3975247a Move other-requirements.txt to bindep.txt
The default filename for documenting binary dependencies has been
changed from "other-requirements.txt" to "bindep.txt" with the release
of bindep 2.1.0. While the previous name is still supported, it will
be deprecated.

Move the file around to follow this change.

Note that this change is self-testing, the OpenStack CI infrastructure
will use a "bindep.txt" file to setup nodes for testing.

For more information about bindep, see also:
http://docs.openstack.org/infra/manual/drivers.html#package-requirements
http://docs.openstack.org/infra/bindep/

As well as this announcement:
http://lists.openstack.org/pipermail/openstack-dev/2016-August/101590.html

Change-Id: If4db21748c4cf2b457e02efdd92b33eee44e2319
2016-08-13 16:45:13 +10:00
Tony Breeds
9a428f3ddb Check for missing run-time requirements
This change creates a new tox environment that *only* installs
openstack_requirements and then verifies that each of the console
scripts has all of the modules that it imports.

This will need to be added to our gate RSN

Change-Id: Ibc37593afcc4d9f820cb88168e1aa15e773b2087
2016-07-26 19:46:49 -05:00
Davanum Srinivas
836a390b91 Add semi-colon as separator for parsing package names
If we don't do this then the following line:
weakrefmethod;python_version=='2.7' or python_version=='2.6' # PSF

ends up as:
weakrefmethod;python_version

in the cruft output. So we need to add ';' to get the output
as 'weakrefmethod'

Change-Id: I7b32bfe70df3bb24ca301db9952d76451674a0db
2016-05-06 16:05:17 -04:00
Andreas Jaeger
e3305bf513 Babel Integration tests
With Babel 2.2 and then with Babel 2.3.2 and 2.3.3 releases, the
OpenStack infra scripts broke and uncovered bugs in the upstream
Babel release. The infra scripts run in post and periodic queues, so
failures here are not directly visible. With Babel 2.3.x invalid strings
were send to the translation server, and we had to recover old content
to fix this.

Let's add some Babel integration tests that can be run whenever the
upper-constraints file changes so that any new Babel release will be at
least minimally tested.

The shell script babel-test.sh runs a simple extraction of strings to
translate from a simple input file. It uses the same commands that the
infra scripts run and handles the two different cases of extraction of
the normal strings and extraction of the non-standard log-level strings.

The script then compares the new content with previously created good
content.

Once this change is in, we will run this test whenever
upper-constraints.txt file changes.

Needed-By: I15fa6a706323615d6072d8acaa859fea9bf35883
Change-Id: Iab7844ba748ecd84ec07ad4280b2317bd626fe9a
2016-04-24 17:21:18 -05:00
Swapnil Kulkarni (coolsvap)
533afe3671 Keep py3.X compatibility for urllib
Change-Id: Ib243de3c6dfef9fdc3e8d73629dc94cf45ee648b
Partial-Bug:#1280105
2016-01-05 08:39:43 +05:30
Jenkins
eeb5a1d827 Merge "Stop special casing jeepyb and pypi-mirror in test" 2015-10-23 18:29:02 +00:00
Doug Hellmann
cd88f6c3f6 do not commit requirements syncs
We need to test without committing the requirements changes because pbr
throws an error if the pre and post versioning do not match. The first
change on a stable branch must be the one to remove the pre-versioning
configuration.

Change-Id: I812af614b7a6c7f499c67310f078ac4835b9e91a
2015-10-16 14:27:08 +00:00
Clark Boylan
7fe60b6f33 Stop special casing jeepyb and pypi-mirror in test
We have removed these projects from the PROJECTS list so no longer need
to special case them here.

Change-Id: Ic3116a646389438ded03f285819529477bf849d8
Depends-On: I61c2f3f8a724dc789479278e7ba493e095c52305
2015-10-15 10:44:43 -07:00
Jenkins
96c547c1ed Merge "Move integration test dependencies into bindep" 2015-07-29 04:07:22 +00:00
Robert Collins
4c028dba0a Move integration test dependencies into bindep
bindep makes this a lot easier to work with as well as making it
possible to support Fedora, Suse etc as needed.

Change-Id: I4d19df8bbcd08e06edcf71ee51c30bb9c3d57fd0
2015-07-19 08:37:11 +12:00
Jenkins
b2788ceb19 Merge "Revert "Move integration test dependencies into bindep"" 2015-07-18 16:03:11 +00:00
Matt Riedemann
f236298949 Revert "Move integration test dependencies into bindep"
This reverts commit ce6ef0fe230ff939ae4d583a2262a16d598c32b9

We're only installing bindep on diskimage-builder built images, and
the devstack-trusty images in rax regions are snapshot-based. We'll
need to temporarily revert this until devstack-trusty is switched to
DIB everywhere, or DevStack-based jobs move to the ubuntu-trusty DIB
images we have available in rax currently.

Change-Id: I69c923297c3202fd35b8892ddfa4d8dbf272bf16
Closes-Bug: #1475880
2015-07-18 15:04:34 +00:00
Jenkins
ac231a5f6d Merge "Move integration test dependencies into bindep" 2015-07-18 01:51:06 +00:00
Robert Collins
ce6ef0fe23 Move integration test dependencies into bindep
bindep makes this a lot easier to work with as well as making it
possible to support Fedora, Suse etc as needed.

Change-Id: I29ffd7376c6f77ca104bf7d930fef178e5cab20f
2015-07-16 07:28:59 +12:00
Sean Dague
c7136bc2cd fix what-broke to strip environment markers
The addition of environment markers broke what-broke, remove them
before processing it.

Change-Id: I4bad24431a6e037e149401f5e4e33130f6ecb07f
2015-07-10 09:35:36 -04:00
Robert Collins
5e4ca1b8b8 Test that changes to requirements are freezable.
If we can't generate a new constraints file, don't let changes in.

Change-Id: If5aa15bb9ca75db0b8ef89a13c9b4bbf8d45cae2
2015-06-25 11:20:56 +12:00
Jenkins
37dc266677 Merge "Fix the empty string issue for pip" 2015-06-22 19:48:51 +00:00
Robert Collins
4b22b94752 Also update extras in setup.cfg.
This required some care. There doesn't seem to be a sane Python3 ready
comment-preserving ini parser around, so I wrote a
minimal-for-our-case one in Parsley. Parsley is already in use in
infra in bindep, but I need to add it to global-requirements as this
is the first use in a managed project of it.

Change-Id: I48de3a2f36e945f75b534f689e3af802bbdc5be9
Depends-On: I7d7e91694c9145fac0ddab8a9de5f789d723c641
Depends-On: I16e967356d5c56f1474ee661b954b3db11a608cb
2015-06-20 19:19:21 +12:00
Flavio Percoco
7428cd9e30 Use pyngus>=1.3.1
Pyngus 1.3.1 has been released and it fixes the issues that blocked the
gate last time. This version of the library is required for the amqp
driver in oslo.messaging.

Change-Id: I4ba583e062674b27ad873ca2228274f383f2bb22
2015-06-16 13:32:24 +02:00
Kai Qiang Wu(Kennan)
7578f91069 Fix the empty string issue for pip
When read from requirement file, the grep not deal with
blank line, which cause pip wheel have '' issue.

Check logs in below for details:
http://logs.openstack.org/21/181821/1/check/check-requirements-integration-dsvm/3ce99ff/console.html#_2015-05-13_05_27_33_105

Change-Id: I558a9488f576738a44dd746589c04bd28d7cd332
2015-05-13 20:51:01 +08:00
Robert Collins
020073faee Allow testing with specific pips
We want to use the new resolver branch of pip today, and perhaps
develop in future too. This introduces the same basic change we did to
pbr's tests to test with a custom pip.

Change-Id: Ia2b1a9c7ee7a7bf32f404932fe7fe872ac402b0e
Depends-On: I7600fa20391bb83491d630b277bf01cc3a20727c
2015-05-11 12:42:26 +12:00
Jenkins
5cd5005af6 Merge "add pycparser tracking to what-broke" 2015-05-06 03:15:46 +00:00
Jenkins
ab00508aed Merge "also install a new pbr into the venvs" 2015-05-04 18:42:04 +00:00
Sean Dague
8fae8b1e29 add pycparser tracking to what-broke
It broke us today, it will probably do so in the future. Let's track
it.

Depends-On: If48a8444b02ee1e105bc1d9ce78a0489ea0c405b

Change-Id: I203738fad804d3cd41808109a1671e4baabf48d0
2015-04-22 11:02:03 +00:00
Sean Dague
0c2eef0a4f add what-broke.py analayzer
This runs through all of global-requirements.txt and looks at the
release history of all components over the last N days, returning the
released list. This is helpful in unexplained gate breaks to at least
look for candidate libraries.

Fleshed out some docs to explain intent for future consumers /
modifiers of this script.

Change-Id: I5880a1d7f0a6063a45f7533319fed1ff56057144
2015-04-07 08:41:15 -04:00
Jenkins
03f906318f Merge "Add a script to find cruft global requirements" 2015-03-20 14:28:19 +00:00
Jenkins
558d1dd4af Merge "Import cap.py tool to cap explicit dependencies" 2015-03-20 13:41:52 +00:00
Jenkins
ce750be64f Merge "Do not sync requirements to possibly capped library's master branch" 2015-02-24 15:12:14 +00:00
Adam Gandelman
8c2ad8c2a8 Do not sync requirements to possibly capped library's master branch
We've hit a situation where a project (pycadf) is listed and capped
in stable/icehouse requirements.txt.  pycadf is also listed in
projects.txt. As part of the requirements integration test, we are
attempting to update pycadf's master branch with stable/icehouse
requirements, and failing because of new dependencies in pycadf that
are not managed by stable/icehouse's GR.

This updates integration.sh so that when we're testing a stable branch,
we are only syncing requirements to other projects have stable branches.
This avoids trying to sync stable requirements into library's master
branches.

Change-Id: I3b8845ad8e013cf6747995df5a62cc4f2ee85e01
2015-02-24 11:57:57 +01:00
Jenkins
c01252639a Merge "Don't use git diff --quiet to detect local change" 2015-02-23 13:27:18 +00:00
Ihar Hrachyshka
e1b3c1f099 Don't use git diff --quiet to detect local change
It seems that there is a bug in (old?) versions of git, particularily in
the one that is shipped with Ubuntu Precise (1.7.9.5) that (sometimes?)
makes the exit code of git diff incorrect.

Change-Id: Id2286147529f1072fe4ceff3fc5200aeb30267a1
Related-Bug: #1264422
Related-Bug: #1410434
2015-02-23 12:01:38 +01:00
Joe Gordon
ed868f3ab9 Import cap.py tool to cap explicit dependencies
This is a forward port of the tool we used to cap stable/juno
requirements. Save a copy of it in master so its available when we are
ready to cut future stable branches.

pin all global-requirements with the version found in pip-freeze
along with a tool to generate the new requirements file

Issues:
* Some versions are only apt-get installable (suds 0.4.1)
* Not all packages are installed in our standard dsvm-tempest env
* Some versions are lower then the minimum requirement we previously had
  because python is awful (libvirt-python)
* Doesn't pin transitive dependencies
* library versions can vanish from pypi and still break us
* Grenade and tempest-full pip-freezes are different and do not work with
each other, so some tweaking is required.

Taking the installed library version from pip-freeze and use that as a
version cap.

Previously used pip-freeze from a tempest-dsvm-neutron-full job.

Change-Id: Iaf48bb069fdd7a19d614ce44b86abd9977c5f0c0
2015-02-20 15:32:48 -08:00
Sean Dague
79387c1135 also install a new pbr into the venvs
Ensure we have an up to date pbr for running the tests.

Change-Id: If5229146ce66e79c05e688d2a340008c14f3f6ec
2015-02-13 10:07:43 -05:00
Ihar Hrachyshka
cc9e20ad30 Cleanup code related to pbr wheel installation from git
Since I8e453110655c0e74ff81b4cdc796c4935c75a553, we don't generate pbr
wheels from git, so there is no need for this either.

Change-Id: I0c70735556d1da68924518b9deb13f4ed082a79d
2015-02-11 10:04:22 +01:00
Sean Dague
0275f5f59b don't install pbr from upstream source
For requirements tests we should be using released pbr, not upstream
git.

Change-Id: I8e453110655c0e74ff81b4cdc796c4935c75a553
2015-02-10 17:19:12 -05:00
Jenkins
e3ac09e877 Merge "Add developer script to build wheels" 2015-01-29 21:50:51 +00:00
Jeremy Stanley
9f3770c349 Add a script to find cruft global requirements
Change-Id: Iee610bff2fedff9d54a1c505451507f3e6559af8
2015-01-17 20:27:19 +00:00
Sean Dague
b380f04a25 attempt to install all of global requirements
This adds a test to try to install all of the global-requirements.txt
into a venv to see if it works. If we can get this functioning, it
will help determine ordering issues and compatibility between all
components at an early stage.

Change-Id: Id147a7b31f96650159f5178ac6dbeec2736d73f6
2015-01-16 14:47:17 -05:00