Currently facts are not gathered when role is run with tags
This patch fixes that behaviour and prevents role failure.
Change-Id: If27a6e57c56a0b294442b71aa777d759bc401868
This fixes bug when content slurp has been performed only for single
host. Others were having invalid contstraints file with single record
in it.
Change-Id: I7189e061f6c8ab3dfb69dda6a9e7f974f8bdc06b
We can't evaluate `venv_wheel_build_enable` without gathered facts about
repo containers. And default facts cache is 1day. After that time role
will fail to evaluate the variable.
Change-Id: I79a10a3f774246ab9c57313f19f3804f9422e08c
These varaibles are not required to be persisted as facts, so
convert them from facts to task vars to save a task execution.
Change-Id: Ibaae18199295761f8b52da31cccb9fa8ec40558b
Other tasks in the role will fail with similar errors to these
preflight checks if virtualenv is missing or variables are undefined.
As these tasks are run many times we should drop them and rely
on regular ansible error messages.
Change-Id: Iabec3f88c02bf4d44c09735a314985366daddffb
Skipping tasks take time. It's faster not to import things we don't need
We also move symlinking to the separate task that can be inlcuded.
Change-Id: I20a5cda421910df9aaca3aacb3f3e088a1fd2938
Removal of venv_build_host_wheel_path leads to all wheels for all
services being dropped which results in other role brokeness
since we build wheels only when requirements or constraints
file is changed. Also we technically don't need to clean it up, since
removal of just rest of the files will lead to wheels being rebuild.
This change should make usage venv_rebuild safe and not breaking wheels
for all services rather then single one.
Change-Id: I6bbe66b699ce5ab245bb9779b61b5c4625eba927
Related-Bug: #1914301
When wheels are built from source, the repo server wheel build requirements
will have an input of the form git+https://example.com/foo.git@branch#egg=foo.
The wheel build will result in a wheel with a version like 1.2.3dev4.
When installing the built wheel into the venv we must switch the source
spec in the requirements from the original git repo to just the
package name 'foo', as the constraints file generated from the wheel
build will specify the exact version of the built wheel 'foo==1.2.3dev4'.
The new pip resolver will not accept there being a source spec in the
input requirements as well as a contradictory constraint from the wheel
build output, this results in an error from pip.
Change-Id: I9be6bbf4a29a4da2ddf96dc0336bc2a7d8ec9281
This patch aims to remove empty records from venv_pip_packages list
to avoid pip error due to that.
This also brings _venv_pip_packages variable, which optimize process by
doing union and sort only once.
Change-Id: Ic94f5a00346e47c394bd2cefc1cfca4ed8c3bdef
If we do not plan to set any local facts, we should
not be creating this folder. If we do need to set the
facts, then the tasks will need to be run via sudo to
ensure that they has the appropriate rights to create
the folder/files under /etc.
If this change is not implemented, and the playbook
running this role is not executed as root, then the
task fails because it does not have rights to create
the folder - even when there are no facts to set.
Change-Id: Idcb093775507728a7c68a877a05b8895d4c8c39f
Simplify this code by making the assumption that we always have a
new enough version of virtualenv present. Centos-7 ships 15.1.0 so
it is now possible to remove some conditional logic as well.
This patch removes the version check and replaces it with a check
that the virtualenv command is present on the system patch and
is usable.
Change-Id: I1a8f4961358c9551c5493b332187b411177b2769
This patch fixes the following ansible warning:
[WARNING]: The value True (type bool) in a string field was converted to
u'True' (type string). If this does not look like what you expect, quote the
entire value to ensure it does not change.
Change-Id: I06cee8173aaf79c2a1bd288fa1a813117ab2b319
If a service is already started it will create files in the venv
__pycache__ directory. This role will then try to symlink the non empty
directory to the host, which will fail.
"the directory /openstack/venvs/nova-19.1.0.dev272/lib/python3.6/
site-packages/__pycache__ is not empty, refusing to convert it"
This patch stops any __pycache__ files from being symlinked to the
venv, these appear to be present in suse packages [1].
[1] http://paste.openstack.org/show/773974/
Change-Id: Ib718f5de4e5ae5b401858dea6437e0039b7082e0
Looking at the output of the tasks, or reviewing them with ARA,
is a bit confusing without the names explaining what they are
doing.
Change-Id: Idb940f011f3b83c222a5acfe6a9430027eb2a657
In the previous repo build process, we had global constraints which
override upper constraints and anything set in the roles. This was
essential for two purposes:
1. To enable us to pin things that were not in upper constraints. eg: pip,
setuptools, wheel
2. To enable us to pin things which were in upper constraints, but broken.
This would usually be a temporary measure until upper constraints was
fixed.
This patch implements a new variable 'venv_build_global_constraints' which
is a list of constraints to be applied globally for all venvs. This list
will be used to produce a file in the venv suffixed with
'-global-constraints.txt' and will be used on the pip command line when
building the wheels and when installing packages.
We also ensure that all constraints are used when both building and
installing pip, setuptools and wheel into the venv.
Change-Id: I9ae3ef19c863b9237a51d2fcd6f4ebce1a9ebad7
If the installation of Python packages fails for any reason, the
constraint and requirements file is already written and therefore
unless you delete the virtualenv, it would not rebuild things
again.
This patch will handle failure and delete those files, then provide
a warning for the user upon failing to inspect the needed file.
Closes-Bug: #1824189
Change-Id: I67db44ce10250908f23c26c90d271a780a875eb6
This meant that we can't actually run fact checks and the evaluation
of that bool fails again!
This reverts commit 092d874e6fe78937a308b686b56b835cc3bf23ea.
Change-Id: Iab9e396bef03417e2dda19efab7da7d2bf92ee8b
Instead of including a file, then running a conditional in the block
which gets applied inside the task, which then results in every task
running, but getting skipped (especially in scenarios where the build
host is the same as the current one), we simply include if we have to.
This saves around 20 seconds in every iteration of python_venv_build
on metal.
Change-Id: I8afb2970ee696f0fe9a9e250f19f1db465d63511
There are sometimes buggy releases of PyPI shipped by the
distributions which don't react well to pip.conf config
options such extra index URLs which make it impossible to
upgrade it.
This adds a a variable which allows working around this
issue by ignoring the site config to do the upgrade.
Change-Id: I5266d827f19e14c6313b11408913e2f754befaca
The python venv build process requires access to facts that from a
delegated host to build venvs. This change adds a fact gathering
loop to the group "repo_all" if it exists and defaults to the current
inventory hostname. This change will allow the build system to gather
all facts from all build targets as needed ensuring a successful
playbook run.
Change-Id: I8a6c60de99360570bd7ae1c1fd8055f99d770c9f
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
In I8b032f3b5d043e543c1d0fd0434d833385ceecc4 the ternary filter
for the distro package install was not closed properly.
Change-Id: Iae88db62aa3d1a1f955347f817b43a82e23789ac
Rather than adding two lists together then passing them through
the 'unique' filter to remove duplicates, we combine the two
actions by using the union filter.
Change-Id: I8b032f3b5d043e543c1d0fd0434d833385ceecc4
If a package (like stackviz) should be installed using
a tarball, rather than from a git source, then it is
better to be able to disable the wheel build process,
constraints usage, etc and just install it from the
tarball.
Change-Id: Id1dc504586a3a1bbd7a161b7367606ced3789043
Currently a configured pip.conf is required in order to make
use of the repo server in an OSA build. This ensures that it
is used, but not exclusively used, if it is available to source
python packages.
The implementation allows it to still respect any configured pip
configuration on the host, so we should be able to remove the OSA
pip.conf with this in place and still maintain the use of the local
repo.
Change-Id: Ia6f2cf86f77ee380fce2d1ecc89e1cb4341e39df
Currently the role expects all constraints to be applied
in pip arguments provided to the role, which means that
there is some pre-processing required outside the role
for this to happen.
In order to pave the way to replace repo_build with this
role, we need to be able to apply constraints and maintain
idempotency even when building from a git source.
To achieve this we ensure that we build the wheels in a
temporary location, then use the resulting wheels for the
specific service to build a service-specific set of
requirements and constraints. To enable idempotency, we
only rebuild the wheels if the requirements/constraints
change.
We use slurp to collect the constraints and re-implement
them when installing the venv on the target hosts. This
prevents us having to inform the venv build role about
the repo server URL. We may change this at a later date
in order to facilitate a centralised repo server for
multiple regions.
Change-Id: I7c467e3a9e6627b75664b94f6b8e3232975171a7
Before creating the venv, it is necessary to make sure that the
parent directory exists otherwise venv creation will fail.
Change-Id: I241e249c919bc52fcc0220737b9d96d9608afebf
Some python packages have C bindings which tend to be very
particular about the version of their underlying shared libraries.
To ensure things run smoothly for stable releases, we opt to
use the distro packages for these python packages and symlink the
appropriate python library files and their bindings into the venv.
This functionality is required for libvirt and ceph and is used
across multiple roles.
Change-Id: Ib5b7fa1d06abe1e1bb4f14aea7de4207b61aca88
There are some packages which absolutely must be there
for all wheel builds, or for installing without wheels.
Without them, pip is totally unable to compile the
package due to missing headers or tooling.
This patch adds a default, minimal, set of compilers
and python headers.
Rather than use include_vars, with_first_found as we
do in most other roles, we use vars/main and a dict
based on ansible_os_family. The role is often included
by other roles, and we'd rather not risk the search
path being incorrect (there are constant bugs related
to this in ansible). Using this mechanism takes away
the need for an include_vars task and avoids any pathing
issues.
Change-Id: I4ef11e47e4d3fe5adc65e9888e660a5a121d205b
Currently all packages provided to the role (build and install)
are installed on both the build host and the install host. This
was done as a temporary measure to allow us time to ensure that
each role seperates these, but now that there are some
installations using this it's causing conflicts in services and
packages.
We should rather do the right thing, and fix the roles which need
the packages separated.
If there is no wheel build host, the venv_build_distro_package_list
packages will need to be installed on the target host, otherwise
the sdist install will not work, so we cater for that in this patch
too.
Depends-On: https://review.openstack.org/612704
Depends-On: https://review.openstack.org/613256
Change-Id: I04100a1073dddb06775d8583104bfd6ef4b3213a
Only set the 'Build wheels for the packages to be installed into the
venv' task to changed when new wheels are built or saved.
Change-Id: Iec1818c54f4a40791cbb69d9cc726960c905c069
Add an option to define a default set of python packages to install
within a virtual environment. This can be used to install a package
wihin a virtual environment that may be outside of a normal package
list but for a given service.
Change-Id: Ic2dc024049062ad9be396a1f71435f661576e91b
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
When building from git sources or from master, pre-release
wheels may be built. We need to ensure that they are used
if they are present. For stable branches, we are still
protected by the use of the constraints to ensure that we
only get the packages we expect.
Change-Id: I68e6d29cfb6a144d119eabdeb7e23bff73c51555
Python venvs are not particularly portable. To overcome this
problem we have a bunch of stuff implemented to fix them
after we unpack them. This ranges from relatively simple
things like re-implementing python in the venv and changing
the shebangs for everything in it, to more complex things
like having to package a venv per distro/architecture.
All of this can be eliminated by simplifying the mechanism
into just creating the venv on the target host, and installing
the python packages into it. To help speed up the build, we
can simply build wheels before hand and store them on a web
server somewhere.
This patch implements the changes to:
1. Do away with packaging the venv.
2. Keep it simple. We install into the venv, and that's that.
3. Add a toggle to rebuild the venv if you'd like to.
4. Use import_tasks with tags for each stage so that it's
easy to skip a portion of what executes.
Change-Id: I708b5cf32e5cce6a18624d0b3be0cd4c828ad389
The --no-site-packages option is deprecated, and is only
necessary for virtualenv versions below 1.7.0. This patch
checks the version and adds it only when necessary.
Change-Id: I19b4acf148c28c96f58bcbda4b19153d56206008
to ensure that the venv is created with the right python
version, we pass the right parameter when creating it.
We remove it in the later task as the argument only
applies when actually creating it.
Change-Id: Ief645ad97a249fcb8545cd2d11edac6984d0014d
The variable name is venv_distro_cache_valid_time
rather than distro_cache_valid_time. Also, this
variable is only used by apt, not zypper.
Change-Id: I71682b7d6bf183967d06f485772760876e2c8df6
To ensure a uniform experience across distributions
and from build to build, we need to ensure that the
venv has a consistent version of pip, setuptools and
wheel. This patch ensures that these packages are
upgraded to the latest version allowed by the args
provided.
This prevents issues as discovered and solved by
https://review.openstack.org/596553
We also remove the virtualenv_site_packages argument
from the pip tasks as they are unnecessary given that
the virtualenv is created in an earlier task.
Change-Id: Ia9e2f695212b01a54f8e73fbd03faa8e9b492f93
Instead of requiring pip installed on the host, we can
use the pip in the virtualenv instead. This keeps the
host cleaner.
Change-Id: I8d6c77e2cfa2cbcd81df21c7ed22a0a344d3b55c
With the more recent versions of ansible, we should now use
"is" instead of the "|" sign for the tests.
This should fix it.
Change-Id: If684a4bf85249a4eb5372cb69365fb79423326db
