Streamline integration test role and make it easier to run

- Rename the "integration-tests" role to "ara_tests" in order to be
  in line with the other roles
- Add an "ansible-integration" tox environment which takes care of
  installing Ansible and running a playbook which includes the ara_tests
  role.
- Add docs and improve existing docs

Change-Id: I0d272744bb27f2c923ef992d1b64de0032849f0e

.zuul.d/zuul.yaml

@@ -13,16 +13,16 @@
       tox_envlist: linters
 
 - job:
-    name: ara-1.0-integration-base
+    name: ara-integration-base
     parent: base
     vars:
       integration_ansible_name: "{{ ansible_user_dir }}/src/github.com/ansible/ansible"
       integration_ansible_version: null
-    run: playbooks/integration-tests.yaml
+    run: playbooks/ara_tests.yaml
-    post-run: tests/integration-post.yaml
+    post-run: tests/ara_tests_post.yaml
 
 - job:
-    name: ara-1.0-integration-fedora-devel
+    name: ara-integration-fedora-devel
     parent: ara-integration-base
     nodeset: fedora-latest
     required-projects:
@@ -30,16 +30,16 @@
         override-checkout: devel
 
 - job:
-    name: ara-1.0-integration-fedora-2.7
+    name: ara-integration-fedora-2.7
-    parent: ara-1.0-integration-base
+    parent: ara-integration-base
     nodeset: fedora-latest
     required-projects:
       - name: github.com/ansible/ansible
         override-checkout: stable-2.7
 
 - job:
-    name: ara-1.0-integration-ubuntu-2.6
+    name: ara-integration-ubuntu-2.6
-    parent: ara-1.0-integration-base
+    parent: ara-integration-base
     nodeset: ubuntu-bionic
     required-projects:
       - name: github.com/ansible/ansible
@@ -114,10 +114,10 @@
       - docs-on-readthedocs
     check:
       jobs:
-        - ara-1.0-integration-fedora-2.7
+        - ara-integration-fedora-2.7
-        - ara-1.0-integration-fedora-devel:
+        - ara-integration-fedora-devel:
             voting: false
-        - ara-1.0-integration-ubuntu-2.6
+        - ara-integration-ubuntu-2.6
         - ara-1.0-role-integration-ubuntu
         - ara-1.0-role-integration-fedora
         - ara-web-role-integration-ubuntu
@@ -126,8 +126,8 @@
         - ara-tox-py3
     gate:
       jobs:
-        - ara-1.0-integration-fedora-2.7
+        - ara-integration-fedora-2.7
-        - ara-1.0-integration-ubuntu-2.6
+        - ara-integration-ubuntu-2.6
         - ara-1.0-role-integration-ubuntu
         - ara-1.0-role-integration-fedora
         - ara-web-role-integration-ubuntu

README.rst

@@ -16,21 +16,24 @@ The project provides several distinct components in order to make this happen:
 Quickstart
 ==========
 
-Here's how you can get started from scratch with default settings::
+Here's how you can get started from scratch with default settings:
 
-    # Create a virtual environment
+.. code-block:: bash
-    python3 -m venv ~/.ara/venv
+
+    # Create a virtual environment and activate it so we don't conflict
+    # with system or distribution packages
+    python3 -m venv ~/.ara/virtualenv
+    source ~/.ara/virtualenv/bin/activate
 
     # Install Ansible and the required ARA projects
-    ~/.ara/venv/bin/pip install ansible
+    pip install ansible git+https://github.com/openstack/ara@feature/1.0
-    ~/.ara/venv/bin/pip install git+https://github.com/openstack/ara@feature/1.0
 
     # Tell Ansible to use the ARA callback plugin
     # "python -m ara.plugins" provides the path to the ARA plugins directory
-    export ANSIBLE_CALLBACK_PLUGINS="$(~/.ara/venv/bin/python -m ara.plugins)/callback"
+    export ANSIBLE_CALLBACK_PLUGINS="$(python -m ara.plugins)/callback"
 
     # Run your playbook as your normally would
-    ~/.ara/venv/bin/ansible-playbook playbook.yml
+    ansible-playbook playbook.yml
 
 The data will be saved in real time throughout the execution of the Ansible playbook.
 
@@ -72,12 +75,12 @@ In addition, you can also find ARA on Twitter: `@ARecordsAnsible <https://twitte
 .. _teamchat: https://github.com/dmsimard/teamchat
 .. _irc.freenode.net: https://webchat.freenode.net/
 
-Development
+Development and testing
-===========
+=======================
 
-**TL;DR**: Using tox is convenient for the time being::
+.. code-block:: bash
 
-  # Retrieve the source
+  # Retrieve the source and check out the 1.0 branch
   git clone https://github.com/openstack/ara
   cd ara
   git checkout feature/1.0
@@ -85,14 +88,27 @@ Development
   # Install tox from pip or from your distro packages
   pip install tox
 
+  # Run Ansible integration tests with the latest version of Ansible
+  tox -e ansible-integration
+
+  # Run integration tests with a specific version of Ansible
+  # Note: tox will always use the latest version of Ansible to run the playbook which runs the tests.
+  # For example, if the latest version of Ansible is 2.7.9, it will use Ansible 2.7.9
+  # to install Ansible==2.6.15 in a virtual environment and 2.6.15 is what will be tested.
+  tox -e ansible-integration -- -e ara_tests_ansible_version=2.6.15
+
+  # Run integration tests with Ansible from source
+  tox -e ansible-integration -- -e "ara_tests_ansible_name=git+https://github.com/ansible/ansible"
+
+  # Run unit tests
+  tox -e py3
+
+  # Run linters (pep8, black, isort)
+  tox -e linters
+
   # Run test server -> http://127.0.0.1:8000/api/v1/
   tox -e runserver
 
-  # Run actual tests or get coverage
-  tox -e linters
-  tox -e py3
-  tox -e cover
-
   # Build docs
   tox -e docs
 

doc/source/ansible-role-ara-tests.rst

@@ -0,0 +1,4 @@
+.. _ansible-role-ara-tests:
+
+.. include:: ../../roles/ara_tests/README.rst
+  :end-before: include_delimiter_end

doc/source/index.rst

@@ -20,3 +20,4 @@ Table of Contents
 
     ansible-role-ara-api <ansible-role-ara-api>
     ansible-role-ara-web <ansible-role-ara-web>
+    ansible-role-ara-tests <ansible-role-ara-tests>

playbooks/ara_tests.yaml

@@ -0,0 +1,5 @@
+- name: Run ARA smoke tests
+  hosts: all
+  gather_facts: yes
+  roles:
+    - ara_tests

playbooks/integration-tests.yaml

@@ -1,7 +0,0 @@
-- name: Run ARA integration tests
-  hosts: all
-  gather_facts: yes
-  tasks:
-    - name: Include the integration-tests role
-      include_role:
-        name: integration-tests

roles/ara_tests/README.rst

@@ -0,0 +1,53 @@
+ansible-role-ara-tests
+======================
+
+An Ansible role that installs ARA and Ansible to run quick and inexpensive
+tests that do not require superuser privileges.
+
+Role Variables
+--------------
+
+See `defaults/main.yaml <https://github.com/openstack/ara/blob/feature/1.0/roles/ara_tests/defaults/main.yaml>`_.
+
+.. literalinclude:: ../../roles/ara_tests/defaults/main.yaml
+   :language: yaml+jinja
+   :start-after: www.gnu.org
+
+TL;DR
+-----
+
+.. code-block:: yaml+jinja
+
+   - name: Test ARA with the latest version of Ansible
+     hosts: all
+     gather_facts: yes
+     roles:
+       - ara_tests
+
+What the role ends up doing by default:
+
+- Creates a directory to contain the files for the duration of the tests
+- Installs ARA from source and the latest version of Ansible in a virtualenv
+- Runs test playbooks designed to exercise different features of ARA
+
+.. _include_delimiter_end:
+
+Copyright
+---------
+
+.. code-block:: text
+
+   Copyright (c) 2019 Red Hat, Inc.
+
+   ARA Records Ansible is free software: you can redistribute it and/or modify
+   it under the terms of the GNU General Public License as published by
+   the Free Software Foundation, either version 3 of the License, or
+   (at your option) any later version.
+
+   ARA Records Ansible is distributed in the hope that it will be useful,
+   but WITHOUT ANY WARRANTY; without even the implied warranty of
+   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+   GNU General Public License for more details.
+
+   You should have received a copy of the GNU General Public License
+   along with ARA Records Ansible. If not, see <http://www.gnu.org/licenses/>.

roles/ara_tests/defaults/main.yaml

@@ -1,37 +1,37 @@
 ---
-#  Copyright (c) 2018 Red Hat, Inc.
+#  Copyright (c) 2019 Red Hat, Inc.
 #
 #  This file is part of ARA Records Ansible.
 #
-#  ARA is free software: you can redistribute it and/or modify
+#  ARA Records Ansible is free software: you can redistribute it and/or modify
 #  it under the terms of the GNU General Public License as published by
 #  the Free Software Foundation, either version 3 of the License, or
 #  (at your option) any later version.
 #
-#  ARA is distributed in the hope that it will be useful,
+#  ARA Records Ansible is distributed in the hope that it will be useful,
 #  but WITHOUT ANY WARRANTY; without even the implied warranty of
 #  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 #  GNU General Public License for more details.
 #
 #  You should have received a copy of the GNU General Public License
-#  along with ARA.  If not, see <http://www.gnu.org/licenses/>.
+#  along with ARA Records Ansible. If not, see <http://www.gnu.org/licenses/>.
 
 # Root directory where integration tests will prepare and store data
-integration_root: "/tmp/ara-integration-tests"
+ara_tests_root: "/tmp/ara-integration-tests"
 
 # Directory where the virtualenv will be created
-integration_virtualenv: "{{ integration_root }}/venv"
+ara_tests_virtualenv: "{{ ara_tests_root }}/virtualenv"
 
 # Directory where ARA_BASE_DIR will be set
-integration_data: "{{ integration_root }}/data"
+ara_tests_data: "{{ ara_tests_root }}/.ara"
 
 # Whether the root directory should be cleaned up between runs
-integration_cleanup: true
+ara_tests_cleanup: true
 
 # Name of the Ansible package
 # This can be "ansible" which will use pip or it could be something like
 # /home/user/git/ansible as well as git+https://github.com/ansible/ansible
-integration_ansible_name: ansible
+ara_tests_ansible_name: ansible
 
 # Version of Ansible from pypi to install
-integration_ansible_version: latest
+ara_tests_ansible_version: latest

roles/ara_tests/meta/main.yaml

@@ -0,0 +1,35 @@
+---
+#  Copyright (c) 2019 Red Hat, Inc.
+#
+#  This file is part of ARA Records Ansible.
+#
+#  ARA Records Ansible is free software: you can redistribute it and/or modify
+#  it under the terms of the GNU General Public License as published by
+#  the Free Software Foundation, either version 3 of the License, or
+#  (at your option) any later version.
+#
+#  ARA Records Ansible is distributed in the hope that it will be useful,
+#  but WITHOUT ANY WARRANTY; without even the implied warranty of
+#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#  GNU General Public License for more details.
+#
+#  You should have received a copy of the GNU General Public License
+#  along with ARA Records Ansible. If not, see <http://www.gnu.org/licenses/>.
+
+galaxy_info:
+  author: David Moreau-Simard
+  description: Self-contained role to test ARA with Ansible
+  license: GPLv3
+  min_ansible_version: 2.7
+  platforms:
+    - name: Fedora
+      versions:
+        - 29
+    - name: Ubuntu
+      versions:
+        - bionic
+  galaxy_tags:
+    - ansible
+    - ara
+
+dependencies: []

roles/ara_tests/tasks/main.yaml

@@ -18,35 +18,24 @@
 
 - name: Clean up integration test root
   file:
-    path: "{{ integration_root }}"
+    path: "{{ ara_tests_root }}"
     state: absent
-  when: integration_cleanup | bool
+  when: ara_tests_cleanup | bool
 
 - name: Create integration test root
   file:
-    path: "{{ integration_root }}"
+    path: "{{ ara_tests_root }}"
     state: directory
 
-# Zuul already prepares the src repository on the remote node
+- name: Set location of the git repository with Zuul
-- name: Symlink Zuul repositories to integration test root
+  set_fact:
-  file:
+    _ara_tests_repository: "{{ ansible_user_dir }}/src/git.openstack.org/openstack/ara"
-    src: "{{ ansible_user_dir }}/src/git.openstack.org/openstack/ara"
-    dest: "{{ integration_root }}/ara"
-    state: link
   when: zuul is defined
 
-- when: zuul is not defined
+- name: Set location of the git repository without Zuul
-  block:
+  set_fact:
-    # The expectation is that the integration tests are run from a checked out
+    _ara_tests_repository: "{{ lookup('pipe', 'git rev-parse --show-toplevel') }}"
-    # git repository. Use this repository for the tests so we are able to test
+  when: zuul is not defined
-    # local modifications. Synchronize is used in place of symlink so we are
-    # able to run integration tests on a remote machine if necessary.
-    # git rev-parse --show-toplevel returns the root git directory
-    - name: Synchronize ara-infra to integration root
-      synchronize:
-        src: "{{ lookup('pipe', 'git rev-parse --show-toplevel') }}"
-        dest: "{{ integration_root }}/"
-        delete: yes
 
 # If a version is not explicitly set we want to make sure to
 # completely omit the version argument to pip, as it will be coming
@@ -55,31 +44,31 @@
 # any version number, but also set the package state to "latest".
 - name: Set Ansible version for installation
   set_fact:
-    _install_ansible_version: "{{ integration_ansible_version }}"
+    _install_ansible_version: "{{ ara_tests_ansible_version }}"
-  when: integration_ansible_version not in ("", "latest")
+  when: ara_tests_ansible_version not in ("", "latest")
 
 - name: Set Ansible package state for installation
   set_fact:
     _install_ansible_state: latest
-  when: integration_ansible_version == "latest"
+  when: ara_tests_ansible_version == "latest"
 
 - name: Initialize virtual environment with Ansible
   pip:
-    name: "{{ integration_ansible_name }}"
+    name: "{{ ara_tests_ansible_name }}"
     version: "{{ _install_ansible_version | default(omit, True) }}"
     state: "{{ _install_ansible_state | default(omit, True) }}"
-    virtualenv: "{{ integration_virtualenv }}"
+    virtualenv: "{{ ara_tests_virtualenv }}"
     virtualenv_python: python3
 
 - name: Install ARA from source in virtual environment
   pip:
-    name: "{{ integration_root }}/ara"
+    name: "{{ _ara_tests_repository }}"
     state: present
-    virtualenv: "{{ integration_virtualenv }}"
+    virtualenv: "{{ ara_tests_virtualenv }}"
     virtualenv_python: python3
 
 - name: Get ARA plugins directory
-  command: "{{ integration_root }}/venv/bin/python -m ara.plugins"
+  command: "{{ ara_tests_virtualenv }}/bin/python -m ara.plugins"
   register: ara_plugins
 
 # These aren't in the same task (i.e, with loop) so we can tell individual test
@@ -90,31 +79,31 @@
     ANSIBLE_ACTION_PLUGINS: "{{ ara_plugins.stdout }}/action"
     ARA_DEBUG: true
     ARA_LOG_LEVEL: DEBUG
-    ARA_BASE_DIR: "{{ integration_data }}"
+    ARA_BASE_DIR: "{{ ara_tests_data }}"
     ARA_SECRET_KEY: testing
   vars:
-    ansible_playbook: "{{ integration_virtualenv }}/bin/ansible-playbook -vvv"
+    _ansible_playbook: "{{ ara_tests_virtualenv }}/bin/ansible-playbook -vvv"
-    test_root: "{{ integration_root }}/ara/tests/integration"
+    _test_root: "{{ _ara_tests_repository }}/tests/integration"
   block:
     # smoke.yaml tests setting ara_playbook_name in one of three plays
     - name: Run smoke.yaml integration test
-      command: "{{ ansible_playbook }} {{ test_root }}/smoke.yaml"
+      command: "{{ _ansible_playbook }} {{ _test_root }}/smoke.yaml"
 
     - name: Run hosts.yaml integration test
-      command: "{{ ansible_playbook }} {{ test_root }}/hosts.yaml"
+      command: "{{ _ansible_playbook }} {{ _test_root }}/hosts.yaml"
 
     - name: Run import.yaml integration test
-      command: "{{ ansible_playbook }} {{ test_root }}/import.yaml"
+      command: "{{ _ansible_playbook }} {{ _test_root }}/import.yaml"
 
     # Tests setting ara_playbook_name as an extra var
     - name: Run failed.yaml integration test
       command: >
-        {{ ansible_playbook }} {{ test_root }}/failed.yaml -e ara_playbook_name="Failed playbook"
+        {{ _ansible_playbook }} {{ _test_root }}/failed.yaml -e ara_playbook_name="Failed playbook"
       ignore_errors: yes
 
     - name: Run incomplete.yaml integration test
       shell: |
-        {{ ansible_playbook }} {{ test_root }}/incomplete.yaml &
+        {{ _ansible_playbook }} {{ _test_root }}/incomplete.yaml &
         sleep 5
         kill $!
       args:

roles/integration-tests/README.rst

@@ -1,47 +0,0 @@
-integration-tests
-=================
-
-Installs a specified version of ARA and Ansible into a virtual environment
-and runs integration tests that do not require superuser privileges.
-
-Requirements
-============
-
-Since the role is designed to run without superuser privileges, the following
-things should be installed in order to let the role use them:
-
-- git
-- python3
-- pip
-- virtualenv
-
-Variables
-=========
-
-From ``defaults/main.yaml``:
-
-Root directory where integration tests will prepare and store data::
-
-    integration_root: "/tmp/ara-integration-tests"
-
-Directory where the virtualenv will be created::
-
-    integration_virtualenv: "{{ integration_root }}/venv"
-
-Directory where ARA_BASE_DIR will be set::
-
-    integration_data: "{{ integration_root }}/data"
-
-Whether the root directory should be cleaned up between runs::
-
-    integration_cleanup: true
-
-Name of the Ansible package. This can be ``ansible`` which will use pip or it
-could be something like ``/home/user/git/ansible`` as well as
-``git+https://github.com/ansible/ansible``::
-
-    integration_ansible_name: ansible
-
-Version of Ansible from pypi to install::
-
-    integration_ansible_version: latest

tests/ara_tests_post.yaml

@@ -27,7 +27,7 @@
         recurse: yes
 
     - name: Recover integration test data
-      command: cp -rp /tmp/ara-integration-tests/data {{ ansible_user_dir }}/workspace/logs/data
+      command: cp -rp /tmp/ara-integration-tests/.ara {{ ansible_user_dir }}/workspace/logs/data
 
     - name: Upload log artifacts
       synchronize:

tox.ini

@@ -36,6 +36,10 @@ setenv =
   ARA_LOG_LEVEL=DEBUG
   ARA_BASE_DIR={toxinidir}/.tox/runserver/tmp/ara
 
+[testenv:ansible-integration]
+deps = ansible
+commands = ansible-playbook -i "localhost," --connection=local {toxinidir}/playbooks/ara_tests.yaml {posargs}
+
 [testenv:cover]
 commands =
     coverage erase