ensure-pip: Install backported pip for Xenial

It turns out the extant comment, removed here, is correct in
identifying the problem, but incorrect about the solution.

As noted the v8 pip included with Xenial doesn't fall back to PyPi
correctly when nodes are configured with mirrors.  However, the note
about virtualenv upgrading pip is incorrect.  This was not tested on
our "plain" nodes (this will be added by a follow-on
https://review.opendev.org/724776 when it can pass) so virtualenv was
picking up the pip installed by the pip-and-virtualenv element.

Installing pip from source doesn't really help; in fact it makes
things even more confusing because "python3 -m venv" still uses the
inbuilt pip from the python-pip-whl package [1].  e.g.

 root@ubuntu-xenial-plain:~# pip --version
 pip 20.1 from /usr/local/lib/python3.5/dist-packages/pip (python 3.5)
   ^ this is installed by get-pip.py
 root@ubuntu-xenial-plain:~# python3 -m venv test
 root@ubuntu-xenial-plain:~# ./test/bin/pip --version
 pip 8.1.1 from /root/test/lib/python3.5/site-packages (python 3.5)
   ^ it still deploys pip in the venv from the whl

and thus will *not* pick up the source pip install.  This is a problem
on our extant Xenial hosts, so clearly nobody is using it.  However,
as part of this work we want to standardise other tools we are
installing in zuul-jobs to use "python3 -m venv".  Thus we want all
our platforms need to support a working venv out of the box.

The solution proposed here is to install a backport of Bionic's pip 9
into Xenial when using this element.  This way, we are still shipping
packaged pip on the host and keeping our images as close to plain
vanilla upstream as possible, but with almost as small change as we
can manage to actually work in our environment.  Given the sunsetting
lifespan of Xenial, this should require not further maintenance until
we are no longer interested in the distro.

Because we skip the install phase on nodes with pre-installed pip, we
put in a work-around to set "ensure_pip_virtualenv_command" to
virtualenv on extant nodes that have been configured with
pip-and-virtualenv.  We can remove this when we have only "plain"
nodes (i.e. no pip-and-virtualenv element) and then we will
consistently be using venv's.

[1] https://packages.ubuntu.com/xenial/python-pip-whl

Change-Id: Id8347b6b09735659a7ed9bbe7f9d2798fbec9620

roles/ensure-pip/tasks/main.yaml

@@ -58,21 +58,29 @@
 - name: Set host default
   set_fact:
     _host_virtualenv: '{{ (_venv_probe.rc == 0) | ternary("/usr/bin/python3 -m venv", "virtualenv") }}'
-  when: ansible_distribution_release != 'xenial'
-
-# The pip included with Xenial (version ~8) has issues with our wheel
-# mirrors; it will not correctly look to upstream pypi when it can't
-# find the wheel in the configured mirror, so virutalenv creation
-# fails.  venv uses the system pip version; for this reason we need to
-# use virtualenv which does upgrade pip in the environment; but note
-# that on Xenial "virtualenv" is owned by the python2 package; so we
-# specify the command to python3 directly.
-- name: Set host default (Xenial)
-  set_fact:
-    _host_virtualenv: '/usr/bin/python3 -m virtualenv'
-  when: ansible_distribution_release == 'xenial'
 
 - name: Set ensure_pip_virtualenv_cmd
   set_fact:
     ensure_pip_virtualenv_command: '{{ ensure_pip_virtualenv_command | default(_host_virtualenv) }}'
     cacheable: true
+
+# NOTE(ianw): "python3 -m venv" is broken on Xenial images with
+# pip-and-virtualenv because the pip is too old to handle our mirrors.
+# We can't easily install our backport version because that element
+# has put all the packages on hold.  In this case, export
+# ensure_pip_virtualenv_command as virtualenv until we have got rid of
+# this element.
+- name: Xenial override
+  when: ansible_distribution_release == 'xenial'
+  block:
+
+    - name: Check if we have pip-and-virtualenv
+      command: grep -q 'pip-and-virtualenv' /etc/dib-manifests/dib_environment
+      failed_when: false
+      register: _pip_check
+      become: yes
+
+    - name: Override virtualenv
+      set_fact:
+        ensure_pip_virtualenv_command: 'virtualenv -p python3'
+      when: _pip_check.rc == 0

roles/ensure-pip/tasks/xenial.yaml

@@ -1,4 +1,10 @@
-# See notes in main.yaml about the virtualenv requirements
+# Pip 8 as shipped with Xenial doesn't correctly fall back to pypi
+# when setup to point to infra mirrors.  This PPA has a backport of of
+# the Bionic 9 pip which means we ship a working environment at least.
+- name: Install backport pip
+  apt_repository:
+    repo: ppa:openstack-ci-core/python-pip
+  become: yes
 
 - name: Install Python 3 pip
   package:
@@ -6,7 +12,6 @@
       - python3-pip
       - python3-setuptools
       - python3-venv
-      - python3-virtualenv
   become: yes
 
 - name: Install Python 2 pip
@@ -14,7 +19,13 @@
     name:
       - python-setuptools
       - python-pip
-      - python-virtualenv
   become: yes
-  when: (ensure_pip_from_packages_with_python2) or
-        (ansible_python.version.major == 2)
+  when: ensure_pip_from_packages_with_python2
+
+# Remove the PPA to avoid exposing future job apt-get updates to ppa
+# latency.  The packages remain.
+- name: Remove backport pip repo
+  apt_repository:
+    repo: ppa:openstack-ci-core/python-pip
+    state: absent
+  become: yes