This updates links going to git.openstack.org and review.openstack.org
to go to their respective opendev locations to avoid redirects.
Change-Id: I78e3bb5303718962f591117f9c0ee11f2314b128
Closes-Bug: #1833256
We know empirically that some legacy gate jobs pass and appear
to be running with python3 but actually pip was invoked with
PYTHON3_VERSION unset so that they are actually ran with python2
packages.
As a followup to this discussion [1], add a safety check in the
get_pip_command function to ensure that a python version has been
set when it is invoked.
[1] https://review.openstack.org/#/c/622415/4/inc/python@283
Change-Id: I3a08406fb7d68282c6b98abb33a625821510046a
If we are running with python3, just assume that any
package that is not blacklisted is available for py3
and just attempt to install it and let pip sort it out
whether it gets installed from a local or remote package.
Change-Id: Ic05d183e489320f6dfc721575d47e7e4d661f87c
Closes-Bug: #1820892
I7d16194d6ba1391ca31251d5b50cbb8de033fc38 added wrong behavour
on Fedora > 26 and Centos 7 when python3 disabled
pip should install packages in /usr/bin
Closes-Bug: #1820070
Change-Id: I3a8efbc8eb6e311db9c7347577c5d2047ba523a9
This makes the grep match in check_python3_support_for_package_local
the same as check_python3_support_for_package_remote.
Change I0349de2026c49279ba7f262d5e86d37018d66326 in grenade started
setting the PYTHON3_VERSION variable, and then we recently started
using bionic nodes everywhere which means we're running python 3.6.
The etcd3gw package has a python 3 and 3.5 classifier, but not 3.6:
https://pypi.org/project/etcd3gw/
The pip_install function code that is dealing with installing py3
packages is hitting a problem installing etcd3gw if the package is
local because of the more restrictive grep in the
check_python3_support_for_package_local function, and since
PYTHON3_VERSION=3.6 now, we don't install from py3 and install
etcd3gw on python 2.7 which makes services like cinder-volume and
cinder-backup, which use etcd3gw, fail when they are running under
python 3 (they get module import errors).
This simply removes the $ restriction on the grep. Looking at the
change that added those local/remote functions:
I243ea4b76f0d5ef57a03b5b0798a05468ee6de9b
There is no explanation for the difference, it just said:
Also, since not many packages are classified correctly, fallback
to looking for just "Programming Language :: Python :: 3" and
log a message for the package to highlight the problem.
So that's what this change does.
Note that alternatives would be:
1. Update the etcd3gw package to add the 3.6 classifier and do
a release (this should probably happen anyway).
2. Add etcd3gw to ENABLED_PYTHON3_PACKAGES but that would be a
short-term hack workaround.
Change-Id: Icd3768870ba0f1659bb2e6f002043d975047b73e
Closes-Bug: #1820892
when installing with python 3.6 on centos7 pip installs
packages to /usr/local/bin as it does on new versions
of fedora.
this change updates the check to include centos
Change-Id: I7d16194d6ba1391ca31251d5b50cbb8de033fc38
This adds a -bindep option to the key development library install
functions. With this option the bindep.txt file will be referenced
and the relevant packages installed.
Change-Id: I856f1f59fca49b6020920d8f859b797f3b904300
Given the file to be configured, if user "stack" even doesn't have
read access, the result of configuration is not expected. iniset with
"-sudo" option will always create the section and the option which we
want to configure for each calling, no matter whether this section and
this option exist in the file or not. The root cause is the calling of
grep and ini_has_option in iniset don't use the "sudo" option.
Change-Id: I9d21322046b7be411c4c7c28fefc24894fa2e131
Signed-off-by: Yi Wang <yi.c.wang@intel.com>
Remove the requirement that services explicitly enable python3 support
in order to be tested under python3 when running with python3
enabled. Keep the enable_python3_package() function for backwards
compatibility, for now, since it is called in some devstack plugins.
Explicitly add swift to the set of packages that should not be installed
using python3 by default until full support is available.
Change-Id: I8ab0a7c242bbf5bf3f091f5a85a98e2f4543f856
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
Old-style test fails on Ubuntu when python3 enabled, with:
.../devstack/inc/python: line 52: [: 16.04: integer expression expected
Use bash-style test, which doesn't attempt to evaluate the RHS if the
LHS evaluates to false
Change-Id: If18031ab98c9060e5825c3a8d3c647bd3705cd9c
Closes-Bug: #1796174
Ceph for example uses them. Creation already worked, but not
updates of existing keys.
Closes-Bug: 1774956
Change-Id: I20cb61c08079b9cd9ad56ac875525abf1442bff6
If a project shows up in zuul's required-projects list, add it
to LIBS_FROM_GIT automatically. This way, when a user specifies
that a job requires a zuul-project, it gets used in testing, but
otherwise, it doesn't (pypi is used instead).
Also add information about what happens behind the scenes for both
LIBS_FROM_GIT and plugin dependencies.
This moves the check performed in check_libs_from_git to
a helper function which is installed for most kinds of
installations. This means that if someone sets LIBS_FROM_GIT to
"foobar", devstack won't error anymore, as nothing is going to
try to install foobar, therefore the check won't run on that.
However, as we move to automated generation of the local config,
that error is not likely to happen. This check was originally
added due to an error in the upper-constraints file (where a
constraint name did not match a package name). This location of
the check would still catch that type of error.
Change-Id: Ifcf3ad008cf42d3d4762cfb3b6c31c93cfeb40db
python-openstacksdk does not match its pip name which is openstacksdk.
So setting python-openstacksdk in LIBS_FROM_GIT leads to devstack
thinking there is a problem.
Put in a workaround for now. It would be better to either:
a) rename python-openstacksdk repo to openstacksdk
b) rename the pip name for openstacksdk back to python-openstacksdk
c) add general support in the various GIT hashes for a pip name
Change-Id: I57cf95763d54ad2060a4ce2af91c3ba18ca04db0
The 'pip list' command prints the "safe name" which converts _'s to
-'s amongst other things; e.g. glance_store becomes
"glance-store 0.21.1.dev22 /opt/stack/glance_store"
Because people may use these more familiar "file system" names in
LIBS_FROM_GIT automatically convert names when checking if libraries
are installed.
Change-Id: I30524f80a341f38dfa794a8f629d859e85a4a448
In commit f0cd9a8b08d92524fc8e2c3f05d08cdebc638e2a we changed to
use column format, but it checks for zero length string and
check_libs_from_git fails.
Change-Id: I97b52b80efb33749647229a55147a08afa112dd2
The old code was strip()ing the version string instead of split()ing the
version string so we always got the first character of the version
string. This worked fine as long as the pip version was single digit but
as soon as it rolls over to '10.stuff' we will compare:
pip version 1 (instead of 10) > 6
Which fails bceause 1 is less than six. Instaed we really do want to
compare 10 > 6 so use split on '.' instead.
Change-Id: Ic7d0c04d7fa77774ab2d70fb9d11f182becec553
The pip list command supports a --format=columns option which outputs
things in space delimited columns. Switch to using that.
Change-Id: I5140a7d83bf567b1c3c67516112eb4c57074fa53
Strip the [<extras>] string from a <package_dir>[<extras>] argument
when looking for the package directory. Explain what the heck is
going on.
Change-Id: I79beb5c3e9e7c35c91cdd0d5a1d91532bebc4b6d
Closes-Bug: #1721638
As described in the change, "pip freeze" has issues with the way
zuulv3 clones repos without a remote. This is an attempt to use "pip
list" to check for local install
Change-Id: I33d25f86b6afcadb4b190a0f6c53311111c64521
lib/nova does a pip re-install of libvirt-python to rebuild the python
library incase the underlying libvirt version changed during package
installs. In offline mode, the underlying version of libvirt can't
have changed; so we have the situation that we've removed the libvirt
python bindings but can't reinstall them (because we're offline).
This fixes that particular situation, but skipping uninstalls in
offline mode seems generically OK.
Change-Id: I2b75d45d94d82f87d996c7570c125d46f5f99f6a
Closes-Bug: #1708369
Some URLs are broken, so fix them.
The others are redirect to new URLs,
so replace them with new ones.
The config options of nova serial console proxy
have been gathered in nova/conf/serial_console.py.
So the description in doc/source/guides/nova.rst
is fixed.
Change-Id: Ifd81cc09969341fbf8f135a913fc6003b94e0acc
To reduce the total number of invocations necessary for pip which isn't
the quickest thing ever (due to needing to evaluate constraints and deps
lists and what is currently installed) combine the main installation of
software with its test-requirements.txt file which should roughly halve
our pip invocations.
Change-Id: Ibcc3264136e66d34a879ad1c90a62e1bb6a84243
'pip uninstall' will hang running stack.sh if it has to
prompt the user for input, use -y.
Change-Id: Ic94639e444b87fd3538463d5a51c01a0208a2ab2
Closes-bug: #1691172
libvirt-python compiles against the currently installed libvirt. If
you upgrade that, it needs to rebuild, however it won't change
versions, so pip install just noops. Force an uninstall / reinstall of
it every time to handle potential upgrades of libvirt.
Change-Id: If34541b34aa6d55eedaf6c603fd1fe92eb887308
When variables use the 'declare' directive, it is by default a local
variable. While other variables have global scope.
For example:
declare -A AN_ARRAY # local in scope
foo=1 # global in scope
This causes errors to occur as some of the variables will be local only
and others will be global.
Update the code, as appropriate, so that variables using the 'declare'
directive also include the '-g' flag to have them also be global. Not
every instance of a declared variable has been updated.
Closes-Bug: #1669509
Co-Authored-By: John L. Villalovos <john.l.villalovos@intel.com>
Change-Id: I2180b68fe861ad19c6d4ec0df0f9f8a528347862
An initial install for devstack-tools, this will need to use all the
fun pip extra variables for installation, however the current
pip_install always prefers python2, and we only want to do python3
here.
Change-Id: I3dcdb35130f76fad81cb7b0d4001b7e96efbbd84
When installing a library from source and python 3 is enabled, first run
the installation process with python 2 enabled to ensure the library is
also installed under python 2 for any services not yet running under
3. The python 3 version is installed second so that command line tools
installed with libraries are installed under python 3 when python 3 is
enabled.
Change-Id: Ibb0f7a68d21081bf7652a0c1515080c0c54888ca
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
Add variables ENABLED_PYTHON3_PACKAGES and DISABLED_PYTHON3_PACKAGES to
work like ENABLED_SERVICES and DISABLED_SERVICES and to manage which
packages are installed using Python 3. Move the list of whitelisted
packages in pip_install to the default for ENABLED_PYTHON3_PACKAGES,
except swift which is not enabled by default for now.
Add enable_python3_package and disable_python3_package functions to make
editing the variables from local.conf easier.
Add python3_enabled_for and python3_disabled_for functions to check the
settings against packages being installed by pip.
Update pip_install to check if python3 is disabled for a service, then
see if it is explicitly enabled, and only then fall back to looking at
the classifiers in the packaging metadata.
Update pip_install messages to give more detail about why the choice
between python 2 and 3 is being made for a given package.
Change-Id: I69857d4e11f4767928614a3b637c894bcd03491f
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
Previous this was set the zake, but that was revert to missing
dependencies issue and because zake is a test fixture and not somthing
to deploy.
This change configures the Cinder dlm with this one is zookeeper.
And it installs tooz and the extra dependencies needed for the
zookeeper driver.
To do it, this commit have to introduce a new method for package
installation: 'pip_install_gr_extras package extra1,extra2'.
Change-Id: Idca310c08e345db59840eb31434c6cb1f849fa70
Really close to getting swift and keystone under uwsgi working, so
let's white list them. Won't affect any existing jobs, so we should
be good.
Change-Id: I51d56d16a5b175bd45dee09edc0b2748d72a5d06
Use trueorfalse to normalize the values for USE_PYTHON3
Install 3.5 instead of 3.4 When USE_PYTHON3 is specified.
Also, since not many packages are classified correctly, fallback
to looking for just "Programming Language :: Python :: 3" and
log a message for the package to highlight the problem.
Also special case some services that are *almost* ready
Depends-On: Id48e1b328230fcdf97ed1cb4b97f4c3f9cf6eb8a
Depends-On: Ib7d9aa0e0b74a936002e0eea0b3af05102b06a62
Change-Id: I243ea4b76f0d5ef57a03b5b0798a05468ee6de9b
When using local.conf in multinode envs not everything is going to be
defined in all places. Eventually we probably want to make it so we
have a host role for these sections or something. But for now warn
instead of die when we can't find a config var.
Change-Id: I6959099373f035fbfe9e540a44e4c52b8e7c95c0
Closes-Bug: #2000824
The current coding fails to process local.conf like
the following. Note: This example is taken from a
real use case. [1]
[[post-config|$NEUTRON_CONF]]
[qos]
notification_drivers = midonet
[[post-config|$NEUTRON_CONF]]
[quotas]
# x10 of default quotas (at the time of writing)
quota_network=100
quota_subnet=100
quota_port=500
quota_router=100
quota_floatingip=500
quota_security_group=100
quota_security_group_rule=1000
[1] https://review.openstack.org/#/c/400627/
Closes-Bug: #1583214
Change-Id: Ie571b5fa5a33d9ed09f30ba7c7724b958ce17616
In the 25.0.0 release [1] of setuptools during any install
operation the package in not overwritten. If a package is
installed from another requirement via pip and then it is
installed again from git, it is not updated causing
check_libs_from_git to fail.
[1] https://setuptools.readthedocs.io/en/latest/history.html#v25-0-0
Change-Id: Ibaa1d4157816ea649f4452756fbde25951347001
Closes-Bug: #1605998
This adds a set of local.conf modifying functions which make it easier
for consuming projects like devstack-gate to programatically add
elements to local.conf structured files.
Change-Id: I3427968c2bd43aba12b3619acc27f73c74f0dabb
Co-Authored-By: fumihiko kakuma <kakuma@valinux.co.jp>
For client debugging that invokes multiple libs it can be useful
to have all libs directly in git and not listing all of them
in LIBS_FROM_GIT.
TrivialFix
Change-Id: Ie631cc4045231ebbe8177d2d113e47e4bf83f61c
