Initial commit to add skydive

This commit adds playbooks and roles to the ops tooling setup to
build, deploy, and operate environments with skydive within in
it.

Skydive is a network analyzer which will allow users to explore
their topology in real-time using a defined storage back-end for
captures, alerts, and more.

The initial implementation of skydive deploys agents throughout
the environment and wires them all back to a cluster of analyzers
which leverage elasticsearch for its persistent storage back-end.
Storage back-ends are load balanced from the within the analyzer
nodes using the traefik light-weight reverse proxy. This setup
gives skydive a fully fault tolerant deployment.

Tests have been added to ensure the binary installation process
is validated. While these jobs are non-voting today, they'll be
iterated on and made passing in the subsequent PRs. All jobs are
following the selective pattern which allows these tools to be
gated in the mono-repo without impacting all other tools within
the environment.

Change-Id: Iaa1152566f2b615d67a33dc94ebdbebb1b492a9d
Signed-off-by: Kevin Carter <kevin@cloudnull.com>

skydive/README.md

@@ -0,0 +1,94 @@
+# Skydive Ansible deployment
+
+These playbooks and roles will deploy skydive, a network
+topology and protocols analyzer.
+
+Official documentation for skydive can be found here:
+http://skydive.network/documentation/deployment#ansible
+
+----
+
+### Overview
+
+The playbooks provide a lot of optionality. All of the
+available options are within the role `defaults` or
+`vars` directories and commented as nessisary.
+
+The playbooks are roles contained within this repository
+will build or GET skydive depending on how the inventory
+is setup. If build services are specified, skydive will
+be built from source using the provided checkout
+(default HEAD). Once the build process is complete, all
+skydive created binaries will be fetched and deployed to
+the target agent and analyzer hosts.
+
+Skydive requires a persistent storage solution to store
+data about the environment and to run captures. These
+playbooks require access to an existing Elasticsearch
+cluster. The variable `skydive_elasticsearch_uri` must be
+set in a variable file, or on the CLI at the time of
+deployment. If this option is undefined the playbooks
+will not run.
+
+A user password for skydive and the cluster must be
+defined. This option can be set in a variable file or
+on the CLI. If this option is undefined the playbooks
+will not run.
+
+Once the playbooks have been executed, the UI and API
+can be accessed via a web browser or CLI on port `8082`.
+
+#### Balancing storage traffic
+
+Storage traffic is balanced on each analyzer node using
+a reverse proxy/load balancer application named
+[Traefik](https://docs.traefik.io). This system
+provides a hyper-light weight, API-able, load balancer.
+All storage traffic will be sent through Traefik to
+various servers within the backend. This provides access
+to a highly available cluster of Elasticsearch nodes as
+needed.
+
+#### Deploying binaries or building from source
+
+This deployment solution provides the ability to install
+skydive from source or from pre-constructed binaries. The
+build process is also available for the traefik loadbalancer.
+
+The in cluster build process is triggered by simply having
+designated build nodes within the inventory. If
+`skydive_build_nodes` or `traefik_build_nodes` is defined in
+inventory the build process for the selected solution will
+be triggered. Regardless of installation preference, the
+installation process is the same. The playbooks will `fetch`
+the binaries and then ship them out the designated nodes
+within inventory. A complete inventory example can be seen
+in the **inventory** directory.
+
+### Deployment Execution
+
+The following example will use a local inventory, and
+set the required options on the CLI.
+
+``` shell
+ansible-playbook -i inventory/inventory.yml \
+                 -e skydive_password=secrete \
+                 -e skydive_elasticsearch_servers="172.17.24.8,172.17.24.9" \
+                 site.yml
+```
+
+Tags are available for every playbook, use the `--list-tags`
+switch to see all available tags.
+
+
+#### Validating the skydive installation
+
+Post deployment, the skydive installation can be valided by
+simply running the `validateSkydive.yml` playbook.
+
+----
+
+TODOs:
+* Setup cert based agent/server auth
+* Add openstack integration
+** document openstack integration, what it adds to the admin service

skydive/ansible-role-requirements.yml

@@ -0,0 +1,9 @@
+---
+- name: config_template
+  scm: git
+  src: https://git.openstack.org/openstack/ansible-config_template
+  version: master
+- name: systemd_service
+  scm: git
+  src: https://git.openstack.org/openstack/ansible-role-systemd_service
+  version: master

skydive/bootstrap-embedded-ansible.sh

@@ -0,0 +1 @@
+../bootstrap-embedded-ansible/bootstrap-embedded-ansible.sh

skydive/buildSkydive.yml

@@ -0,0 +1,79 @@
+---
+# Copyright 2019, Rackspace US, Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+- name: Setup localhost
+  hosts: localhost
+  connection: local
+  tags:
+    - always
+
+
+- name: Gather facts
+  hosts: skydive_all
+  tasks:
+    - name: Gather facts on all hosts
+      setup: {}
+  tags:
+    - always
+
+
+- name: Classify skydive build servers
+  hosts: skydive_build_nodes
+  serial: 1
+  tasks:
+    - name: Group repo servers by architecture and os version
+      group_by:
+        key: skydive_build_nodes_{{ ansible_architecture }}
+  tags:
+    - always
+
+
+- name: Prepare group of master build servers
+  hosts: localhost
+  tasks:
+    - name: Prepare group of master build servers
+      add_host:
+        name: "{{ groups[item][0] }}"
+        groups: skydive_build_nodes_masters
+      with_items: "{{ groups | select('match', '^skydive_build_nodes_') | list }}"
+      changed_when: false
+  tags:
+    - always
+
+
+- name: Make skydive
+  hosts: skydive_build_nodes_masters
+  become: yes
+  vars:
+    skydive_build_version: HEAD
+    skydive_build_path: "/opt/skydive-build/{{ skydive_build_version }}"
+    skydive_build_go_path: "{{ skydive_build_path }}/go"
+  roles:
+    - role: make_skydive
+  post_tasks:
+    - name: Find skydive binaries
+      find:
+        paths: "{{ skydive_build_go_path }}/bin/"
+        recurse: no
+        patterns: "*skydive*"
+      register: files_to_copy
+    - name: Fetch skydive binaries
+      fetch:
+        src: "{{ item.path }}"
+        dest: "/tmp/skydive/{{ ansible_architecture }}/{{ item.path | basename }}"
+        flat: true
+      with_items: "{{ files_to_copy.files }}"
+  tags:
+    - skydive-make

skydive/buildTraefik.yml

@@ -0,0 +1,80 @@
+---
+# Copyright 2019, Rackspace US, Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+- name: Setup localhost
+  hosts: localhost
+  connection: local
+  tags:
+    - always
+
+
+- name: Gather facts
+  hosts: traefik_all
+  tasks:
+    - name: Gather facts on all hosts
+      setup: {}
+  tags:
+    - always
+
+
+- name: Classify traefik build servers
+  hosts: traefik_build_nodes
+  serial: 1
+  tasks:
+    - name: Group repo servers by architecture and os version
+      group_by:
+        key: traefik_build_nodes_{{ ansible_architecture }}
+  tags:
+    - always
+
+
+- name: Prepare group of master build servers
+  hosts: localhost
+  tasks:
+    - name: Prepare group of master build servers
+      add_host:
+        name: "{{ groups[item][0] }}"
+        groups: traefik_build_nodes_masters
+      with_items: "{{ groups | select('match', '^traefik_build_nodes_') | list }}"
+      changed_when: false
+  tags:
+    - always
+
+
+- name: Make traefik
+  hosts: traefik_build_nodes_masters
+  become: yes
+  vars:
+    traefik_build_version: HEAD
+    traefik_build_path: "/opt/traefik-build/{{ traefik_build_version }}"
+    traefik_build_go_path: "{{ traefik_build_path }}/go"
+    traefik_build_project_path: "{{ traefik_build_go_path }}/src"
+  roles:
+    - role: make_traefik
+  post_tasks:
+    - name: Find traefik binaries
+      find:
+        paths: "{{ traefik_build_project_path }}/github.com/containous/traefik"
+        recurse: no
+        patterns: "traefik"
+      register: files_to_copy
+    - name: Fetch traefik binaries
+      fetch:
+        src: "{{ item.path }}"
+        dest: "/tmp/traefik/{{ ansible_architecture }}/{{ item.path | basename }}"
+        flat: true
+      with_items: "{{ files_to_copy.files }}"
+  tags:
+    - traefik-make

skydive/installSkydive.yml

@@ -0,0 +1,139 @@
+---
+# Copyright 2019, Rackspace US, Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+- name: Deploy skydive binaries
+  hosts: skydive_agents:skydive_analyzers
+  become: yes
+  vars:
+    skydive_binary_version: "v0.21.0"
+    skydive_binary_url: "https://github.com/skydive-project/skydive/releases/download/{{ skydive_binary_version }}/skydive"
+    skydive_staging_node: "localhost"
+  pre_tasks:
+    - name: Create skydive temp path
+      file:
+        path: "/tmp/skydive/{{ ansible_architecture }}"
+        state: directory
+      delegate_to: "{{ skydive_staging_node }}"
+      become: false
+  tasks:
+    - name: Built skydive installation
+      block:
+        - name: Find skydive binaries
+          find:
+            paths: "/tmp/skydive/{{ ansible_architecture }}/"
+            recurse: no
+            patterns: "*skydive*"
+          register: files_to_copy
+          delegate_to: "{{ skydive_staging_node }}"
+          become: false
+        - name: Install built skydive
+          copy:
+            src: "{{ item.path }}"
+            dest: "/usr/local/bin/{{ item.path | basename }}"
+            mode: "0755"
+          with_items: "{{ files_to_copy.files }}"
+      when:
+        - ((groups['skydive_build_nodes'] | default([])) | length) > 0
+
+    - name: Upstream skydive installation
+      block:
+        - name: Get skydive binary
+          get_url:
+            url: "{{ skydive_binary_url }}"
+            dest: "/tmp/skydive/{{ ansible_architecture }}/{{ skydive_binary_url | basename }}"
+            mode: '0755'
+          delegate_to: "{{ skydive_staging_node }}"
+          become: false
+        - name: Install binary skydive
+          copy:
+            src: "/tmp/skydive/{{ ansible_architecture }}/{{ skydive_binary_url | basename }}"
+            dest: "/usr/local/bin/skydive"
+            mode: "0755"
+      when:
+        - ((groups['skydive_build_nodes'] | default([])) | length) < 1
+  tags:
+    - skydive-install
+
+
+- name: Deploy traefik binaries
+  hosts: skydive_analyzers
+  become: yes
+  vars:
+    traefik_binary_version: "v1.7.7"
+    traefik_binary_url: "https://github.com/containous/traefik/releases/download/{{ traefik_binary_version }}/traefik"
+    traefik_staging_node: "localhost"
+  pre_tasks:
+    - name: Create traefik temp path
+      file:
+        path: "/tmp/traefik/{{ ansible_architecture }}"
+        state: directory
+      delegate_to: "{{ traefik_staging_node }}"
+      become: false
+  tasks:
+    - name: Built traefik installation
+      block:
+        - name: Find traefik binaries
+          find:
+            paths: "/tmp/traefik/{{ ansible_architecture }}/"
+            recurse: no
+            patterns: "*traefik*"
+          register: files_to_copy
+          delegate_to: "{{ traefik_staging_node }}"
+          become: false
+        - name: Install built traefik
+          copy:
+            src: "{{ item.path }}"
+            dest: "/usr/local/bin/{{ item.path | basename }}"
+            mode: "0755"
+          with_items: "{{ files_to_copy.files }}"
+      when:
+        - ((groups['traefik_build_nodes'] | default([])) | length) > 0
+
+    - name: Upstream traefik installation
+      block:
+        - name: Get traefik binary
+          get_url:
+            url: "{{ traefik_binary_url }}"
+            dest: "/tmp/traefik/{{ ansible_architecture }}/{{ traefik_binary_url | basename }}"
+            mode: '0755'
+          delegate_to: "{{ traefik_staging_node }}"
+          become: false
+        - name: Install binary traefik
+          copy:
+            src: "/tmp/traefik/{{ ansible_architecture }}/{{ traefik_binary_url | basename }}"
+            dest: "/usr/local/bin/traefik"
+            mode: "0755"
+      when:
+        - ((groups['traefik_build_nodes'] | default([])) | length) < 1
+  tags:
+    - traefik-install
+
+
+- name: Configure Skydive analyzers
+  hosts: skydive_analyzers
+  become: yes
+  roles:
+    - role: skydive_analyzer
+  tags:
+    - skydive-analyzer-setup
+
+
+- name: Configure Skydive agents
+  hosts: skydive_agents
+  become: yes
+  roles:
+    - role: skydive_agent
+  tags:
+    - skydive-agent-setup

skydive/inventory/inventory.yml

@@ -0,0 +1,56 @@
+---
+
+all_systems:
+  vars: {}
+  children:
+    systems:
+      vars:
+        ansible_ssh_extra_args: >-
+          -o UserKnownHostsFile=/dev/null
+          -o StrictHostKeyChecking=no
+          -o ServerAliveInterval=64
+          -o ServerAliveCountMax=1024
+          -o Compression=no
+          -o TCPKeepAlive=yes
+          -o VerifyHostKeyDNS=no
+          -o ForwardX11=no
+          -o ForwardAgent=yes
+          -T
+        ansible_become: yes
+        ansible_become_user: "root"
+        ansible_user: "root"
+
+      children:
+        traefik_all:
+          children:
+            traefik_build_nodes:
+              hosts:
+                build1:
+                  ansible_host: "172.17.24.2"
+                  ansible_user: ubuntu
+
+        skydive_all:
+          children:
+            skydive_build_nodes:
+              hosts:
+                build1:
+                  ansible_host: "172.17.24.2"
+                  ansible_user: ubuntu
+
+            skydive_agents:
+              hosts:
+                agent1:
+                  ansible_host: "172.17.24.3"
+                  ansible_user: ubuntu
+                agent2:
+                  ansible_host: "172.17.24.4"
+                  ansible_user: centos
+                agent3:
+                  ansible_host: "172.17.24.5"
+                  ansible_user: opensuse
+
+            skydive_analyzers:
+              hosts:
+                analyzer1:
+                  ansible_host: "172.17.24.6"
+                  ansible_user: ubuntu

skydive/roles/go_install/meta/main.yml

@@ -15,19 +15,24 @@
 
 galaxy_info:
   author: OpenStack
-  description: Elastic v6.x go install role
+  description: Skydive go install role
   company: Rackspace
   license: Apache2
   min_ansible_version: 2.5
   platforms:
     - name: Ubuntu
       versions:
-        - trusty
         - xenial
         - bionic
+    - name: EL
+      versions:
+        - 7
+    - name: opensuse
+      versions:
+        - all
   categories:
     - cloud
     - development
-    - elasticsearch
+    - skydive
-    - elastic-stack
+    - networking
 dependencies: []

skydive/roles/go_install/tasks/go_install.yml

@@ -35,10 +35,16 @@
     dest: "/opt/go{{ go_download_version }}"
     remote_src: yes
 
-- name: Create go defaults file
+- name: Create go versioned default file
   copy:
     content: |
       GOROOT=/opt/go{{ go_download_version }}/go
       GOPATH=/usr/local
       PATH=${PATH}:${GOROOT}/bin
     dest: "/etc/default/go{{ go_download_version }}"
+
+- name: Create go default file
+  file:
+    src: "/etc/default/go{{ go_download_version }}"
+    dest: "/etc/default/go"
+    state: link

skydive/roles/make_skydive/defaults/main.yml

@@ -0,0 +1,29 @@
+---
+# Copyright 2019, Rackspace US, Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+skydive_build_version: HEAD
+
+skydive_build_path: "/opt/skydive-build/{{ skydive_build_version }}"
+
+skydive_build_go_path: "{{ skydive_build_path }}/go"
+
+skydive_build_project_path: "{{ skydive_build_go_path }}/src"
+
+skydive_build_use_flags: >-
+  WITH_EBPF=true
+  WITH_NEUTRON=true
+  WITH_LXD=true
+  WITH_PROF=true
+  WITH_CDD=true

skydive/roles/make_skydive/meta/main.yml

@@ -0,0 +1,39 @@
+---
+# Copyright 2019, Rackspace US, Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+galaxy_info:
+  author: OpenStack
+  description: Build skydive
+  company: Rackspace
+  license: Apache2
+  min_ansible_version: 2.5
+  platforms:
+    - name: Ubuntu
+      versions:
+        - xenial
+        - bionic
+    - name: EL
+      versions:
+        - 7
+    - name: opensuse
+      versions:
+        - all
+  categories:
+    - cloud
+    - development
+    - skydive
+    - networking
+dependencies:
+  - role: go_install

skydive/roles/make_skydive/tasks/main.yml

@@ -0,0 +1,65 @@
+---
+# Copyright 2019, Rackspace US, Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+- name: Gather variables for each operating system
+  include_vars: "{{ item }}"
+  with_first_found:
+    - "{{ ansible_distribution | lower }}-{{ ansible_distribution_version | lower }}.yml"
+    - "{{ ansible_distribution | lower }}-{{ ansible_distribution_major_version | lower }}.yml"
+    - "{{ ansible_os_family | lower }}-{{ ansible_distribution_major_version | lower }}.yml"
+    - "{{ ansible_distribution | lower }}.yml"
+    - "{{ ansible_os_family | lower }}-{{ ansible_distribution_version.split('.')[0] }}.yml"
+    - "{{ ansible_os_family | lower }}.yml"
+  tags:
+    - always
+
+- name: Ensure build libraries are installed
+  package:
+    name: "{{ sykdive_build_distro_packages }}"
+    state: "present"
+    update_cache: "{{ (ansible_pkg_mgr == 'apt') | ternary('yes', omit) }}"
+  register: _package_task
+  until: _package_task is success
+  retries: 3
+  delay: 2
+  tags:
+    - package_install
+
+- name: Ensure skydive-project directory exists
+  file:
+    path: "{{ skydive_build_project_path }}/github.com/skydive-project/skydive"
+    state: directory
+
+- name: Get the skydive archive
+  git:
+    dest: "{{ skydive_build_project_path }}/github.com/skydive-project/skydive"
+    repo: https://github.com/skydive-project/skydive.git
+    force: yes
+    track_submodules: yes
+    clone: yes
+    update: yes
+    recursive: yes
+    version: "{{ skydive_build_version }}"
+
+- name: Make install skydive
+  shell: |
+    source /etc/default/go
+    export PATH="{{ skydive_build_go_path }}/bin:${PATH}"
+    export GOPATH="{{ skydive_build_go_path }}"
+    make install {{ skydive_build_use_flags }}
+  args:
+    chdir: "{{ skydive_build_project_path }}/github.com/skydive-project/skydive"
+    executable: "/bin/bash"
+    creates: "{{ skydive_build_go_path }}/bin/skydive"

skydive/roles/make_skydive/vars/redhat.yml

@@ -0,0 +1,34 @@
+---
+# Copyright 2019, Rackspace US, Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+sykdive_build_distro_packages:
+  - bcc-devel
+  - clang
+  - gcc
+  - git
+  - kernel-devel
+  - kernel-headers
+  - llvm
+  - libpcap-devel
+  - libxml2-devel
+  - libvirt-devel
+  - make
+  - npm
+  - numactl-devel
+  - patch
+  - findutils
+  - protobuf-c-compiler
+  - protobuf-devel
+  - systemd-devel

skydive/roles/make_skydive/vars/suse.yml

@@ -0,0 +1,34 @@
+---
+# Copyright 2019, Rackspace US, Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+sykdive_build_distro_packages:
+  - bcc-devel
+  - clang
+  - findutils
+  - gcc
+  - git
+  - glibc-devel
+  - kernel-devel
+  - libpcap0.8-dev
+  - libprotobuf-dev
+  - libsystemd-dev
+  - libxml2-dev
+  - libvirt-dev
+  - llvm
+  - make
+  - npm
+  - numactl
+  - patch
+  - protobuf-compiler

skydive/roles/make_skydive/vars/ubuntu.yml

@@ -0,0 +1,34 @@
+---
+# Copyright 2019, Rackspace US, Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+sykdive_build_distro_packages:
+  - bcc
+  - clang
+  - findutils
+  - gcc
+  - git
+  - kernel-package
+  - linux-libc-dev
+  - libpcap0.8-dev
+  - libprotobuf-dev
+  - libsystemd-dev
+  - libxml2-dev
+  - libvirt-dev
+  - llvm
+  - make
+  - npm
+  - numactl
+  - patch
+  - protobuf-compiler

skydive/roles/make_traefik/defaults/main.yml

@@ -0,0 +1,22 @@
+---
+# Copyright 2019, Rackspace US, Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+traefik_build_version: HEAD
+
+traefik_build_path: "/opt/traefik-build/{{ traefik_build_version }}"
+
+traefik_build_go_path: "{{ traefik_build_path }}/go"
+
+traefik_build_project_path: "{{ traefik_build_go_path }}/src"

skydive/roles/make_traefik/meta/main.yml

@@ -0,0 +1,39 @@
+---
+# Copyright 2019, Rackspace US, Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+galaxy_info:
+  author: OpenStack
+  description: Build traefik
+  company: Rackspace
+  license: Apache2
+  min_ansible_version: 2.5
+  platforms:
+    - name: Ubuntu
+      versions:
+        - xenial
+        - bionic
+    - name: EL
+      versions:
+        - 7
+    - name: opensuse
+      versions:
+        - all
+  categories:
+    - cloud
+    - development
+    - traefik
+    - networking
+dependencies:
+  - role: go_install

skydive/roles/make_traefik/tasks/main.yml

@@ -0,0 +1,87 @@
+---
+# Copyright 2019, Rackspace US, Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+- name: Gather variables for each operating system
+  include_vars: "{{ item }}"
+  with_first_found:
+    - "{{ ansible_distribution | lower }}-{{ ansible_distribution_version | lower }}.yml"
+    - "{{ ansible_distribution | lower }}-{{ ansible_distribution_major_version | lower }}.yml"
+    - "{{ ansible_os_family | lower }}-{{ ansible_distribution_major_version | lower }}.yml"
+    - "{{ ansible_distribution | lower }}.yml"
+    - "{{ ansible_os_family | lower }}-{{ ansible_distribution_version.split('.')[0] }}.yml"
+    - "{{ ansible_os_family | lower }}.yml"
+  tags:
+    - always
+
+- name: Ensure build libraries are installed
+  package:
+    name: "{{ sykdive_build_distro_packages }}"
+    state: "present"
+    update_cache: "{{ (ansible_pkg_mgr == 'apt') | ternary('yes', omit) }}"
+  register: _package_task
+  until: _package_task is success
+  retries: 3
+  delay: 2
+  tags:
+    - package_install
+
+- name: Ensure traefik-project directory exists
+  file:
+    path: "{{ traefik_build_project_path }}/github.com/containous/traefik"
+    state: directory
+
+- name: Get the traefik archive
+  git:
+    dest: "{{ traefik_build_project_path }}/github.com/containous/traefik"
+    repo: https://github.com/containous/traefik.git
+    force: yes
+    track_submodules: yes
+    clone: yes
+    update: yes
+    recursive: yes
+    version: "{{ traefik_build_version }}"
+
+- name: Go get traefik bin-data
+  shell: |
+    source /etc/default/go
+    export PATH="{{ traefik_build_go_path }}/bin:${PATH}"
+    export GOPATH="{{ traefik_build_go_path }}"
+    go get github.com/containous/go-bindata/...
+  args:
+    chdir: "{{ traefik_build_project_path }}/github.com/containous/traefik"
+    executable: "/bin/bash"
+    creates: "{{ traefik_build_project_path }}/github.com/containous/traefik/traefik"
+
+- name: Go generate traefik
+  shell: |
+    source /etc/default/go
+    export PATH="{{ traefik_build_go_path }}/bin:${PATH}"
+    export GOPATH="{{ traefik_build_go_path }}"
+    go generate
+  args:
+    chdir: "{{ traefik_build_project_path }}/github.com/containous/traefik"
+    executable: "/bin/bash"
+    creates: "{{ traefik_build_project_path }}/github.com/containous/traefik/traefik"
+
+- name: Go build traefik
+  shell: |
+    source /etc/default/go
+    export PATH="{{ traefik_build_go_path }}/bin:${PATH}"
+    export GOPATH="{{ traefik_build_go_path }}"
+    go build ./cmd/traefik
+  args:
+    chdir: "{{ traefik_build_project_path }}/github.com/containous/traefik"
+    executable: "/bin/bash"
+    creates: "{{ traefik_build_project_path }}/github.com/containous/traefik/traefik"

skydive/roles/make_traefik/vars/redhat.yml

@@ -0,0 +1,16 @@
+---
+# Copyright 2019, Rackspace US, Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+sykdive_build_distro_packages: []

skydive/roles/make_traefik/vars/suse.yml

@@ -0,0 +1,16 @@
+---
+# Copyright 2019, Rackspace US, Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+sykdive_build_distro_packages: []

skydive/roles/make_traefik/vars/ubuntu.yml

@@ -0,0 +1,16 @@
+---
+# Copyright 2019, Rackspace US, Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+sykdive_build_distro_packages: []

skydive/roles/skydive_agent/defaults/main.yml

@@ -0,0 +1,16 @@
+---
+# Copyright 2019, Rackspace US, Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+skydive_agent_service_state: started

skydive/roles/skydive_agent/meta/main.yml

@@ -0,0 +1,38 @@
+---
+# Copyright 2019, Rackspace US, Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+galaxy_info:
+  author: Sylvain Baubeau
+  description: Skydive Agent
+  company: Red Hat
+  license: Apache License, Version 2.0
+  min_ansible_version: 2.3
+  platforms:
+    - name: Ubuntu
+      versions:
+        - xenial
+        - bionic
+    - name: EL
+      versions:
+        - 7
+    - name: opensuse
+      versions:
+        - all
+  categories:
+    - cloud
+    - system
+
+dependencies:
+  - role: skydive_common

skydive/roles/skydive_agent/tasks/main.yml

@@ -0,0 +1,23 @@
+---
+# Copyright 2019, Rackspace US, Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+- name: Refresh local facts
+  setup:
+    filter: ansible_local
+    gather_subset: "!all"
+  tags:
+    - always
+
+- include_tasks: skydive_agent_setup.yml

skydive/roles/skydive_agent/tasks/skydive_agent_setup.yml

@@ -0,0 +1,51 @@
+---
+# Copyright 2019, Rackspace US, Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+- name: Refresh local facts
+  setup:
+    filter: ansible_local
+    gather_subset: "!all"
+  tags:
+    - always
+
+- name: Run the systemd service role
+  include_role:
+    name: systemd_service
+  vars:
+    systemd_service_enabled: true
+    systemd_service_restart_changed: false
+    systemd_services:
+      - service_name: "skydive-agent"
+        execstarts:
+          - /usr/local/bin/skydive agent
+
+- name: Force handlers
+  meta: flush_handlers
+
+- name: Set skydive service state (upstart)
+  service:
+    name: "skydive-agent"
+    state: "{{ (ansible_local['skydive']['skydive']['needs_restart_agent'] | bool) | ternary('restarted', skydive_agent_service_state) }}"
+    enabled: "{{ skydive_agent_service_state in ['running', 'started', 'restarted'] }}"
+  when:
+    - ansible_service_mgr == 'upstart'
+
+- name: Set skydive service state (systemd)
+  systemd:
+    name: "skydive-agent"
+    state: "{{ (ansible_local['skydive']['skydive']['needs_restart_agent'] | bool) | ternary('restarted', skydive_agent_service_state) }}"
+    enabled: "{{ skydive_agent_service_state in ['running', 'started', 'restarted'] }}"
+  when:
+    - ansible_service_mgr == 'systemd'

skydive/roles/skydive_analyzer/defaults/main.yml

@@ -0,0 +1,27 @@
+---
+# Copyright 2019, Rackspace US, Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+skydive_analyzer_service_state: started
+
+skydive_username: skydive
+
+# The skydive user name is required
+# skydive_password: secrete
+
+# Set basic authentication users and passwords into a basic auth file.
+# This is optional and will only be created if user defined.
+# skydive_basic_auth_users:
+#   skydive: secrete
+skydive_basic_auth_users: {}

skydive/roles/skydive_analyzer/meta/main.yml

@@ -0,0 +1,58 @@
+---
+# Copyright 2019, Rackspace US, Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+galaxy_info:
+  author: OpenStack
+  description: Skydive Analyzer configuration
+  company: Rackspace
+  license: Apache License, Version 2.0
+  min_ansible_version: 2.5
+  platforms:
+    - name: Ubuntu
+      versions:
+        - xenial
+        - bionic
+    - name: EL
+      versions:
+        - 7
+    - name: opensuse
+      versions:
+        - all
+  categories:
+    - cloud
+    - development
+    - skydive
+    - networking
+dependencies:
+  - role: traefik_common
+    traefik_basic_auth_users: "{{ _skydive_basic_auth_users | combine(skydive_basic_auth_users) }}"
+    traefik_dashboard_enabled: true
+    traefik_destinations:
+      elasticsearch:
+        proto: "{{ skydive_elasticsearch_proto | default('http') }}"
+        port: "19200"
+        bind: "127.0.0.1"
+        servers: |-
+          {% set nodes = [] %}
+          {% for target in skydive_elasticsearch_servers.split(',') %}
+          {%   set node = {} %}
+          {%   set _ = node.__setitem__('name', 'elasticsearch' ~ loop.index) %}
+          {%   set _ = node.__setitem__('address', target) %}
+          {%   set _ = node.__setitem__('weight', (100 - loop.index)) %}
+          {%   set _ = node.__setitem__('port', (skydive_elasticsearch_port | default('9200'))) %}
+          {%   set _ = nodes.append(node) %}
+          {% endfor %}
+          {{ nodes }}
+  - role: skydive_common

skydive/roles/skydive_analyzer/tasks/main.yml

@@ -0,0 +1,23 @@
+---
+# Copyright 2019, Rackspace US, Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+- name: Refresh local facts
+  setup:
+    filter: ansible_local
+    gather_subset: "!all"
+  tags:
+    - always
+
+- include_tasks: skydive_analyzer_setup.yml

skydive/roles/skydive_analyzer/tasks/skydive_analyzer_setup.yml

@@ -0,0 +1,44 @@
+---
+# Copyright 2019, Rackspace US, Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+- name: Run the systemd service role
+  include_role:
+    name: systemd_service
+  vars:
+    systemd_service_enabled: true
+    systemd_service_restart_changed: false
+    systemd_services:
+      - service_name: "skydive-analyzer"
+        execstarts:
+          - /usr/local/bin/skydive analyzer
+
+- name: Force handlers
+  meta: flush_handlers
+
+- name: Set skydive service state (upstart)
+  service:
+    name: "skydive-analyzer"
+    state: "{{ (ansible_local['skydive']['skydive']['needs_restart_analyzers'] | bool) | ternary('restarted', skydive_analyzer_service_state) }}"
+    enabled: "{{ skydive_analyzer_service_state in ['running', 'started', 'restarted'] }}"
+  when:
+    - ansible_service_mgr == 'upstart'
+
+- name: Set skydive service state (systemd)
+  systemd:
+    name: "skydive-analyzer"
+    state: "{{ (ansible_local['skydive']['skydive']['needs_restart_analyzers'] | bool) | ternary('restarted', skydive_analyzer_service_state) }}"
+    enabled: "{{ skydive_analyzer_service_state in ['running', 'started', 'restarted'] }}"
+  when:
+    - ansible_service_mgr == 'systemd'

skydive/roles/skydive_analyzer/vars/main.yml

@@ -0,0 +1,18 @@
+---
+# Copyright 2019, Rackspace US, Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Inject the required basic authentication information
+_skydive_basic_auth_users:
+  "{{ skydive_username }}": "{{ skydive_password }}"

skydive/roles/skydive_common/defaults/main.yml

@@ -0,0 +1,137 @@
+---
+# Copyright 2019, Rackspace US, Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Set the analyzer port
+skydive_analyzer_port: 8082
+
+# Set the agent api port
+skydive_agent_port: 8081
+
+# Set the flow protocol
+skydive_flow_protocol: udp
+
+# Set a particulare network interface used for skydive traffic
+skydive_network_device: "{{ ansible_default_ipv4['interface'] }}"
+
+# The skydive elasticsearch uri(s) is required
+# Set the elasticsearch URI(s), the system will attempt to connect to the URI.
+# If this URI is unreachable the deployment will fail. If there is more than
+# one elasticsearch ingest node or smart loadbancer use comma delimation to
+# list multiple nodes.
+skydive_elasticsearch_proto: "http"
+
+# The skydive elasticsearch port is set to 9200 which is the default, however, to
+# ensure there's no conflict in thee event an elasticsearch installation is on the
+# same host as the analyzer, the frontend port will always be 19200. The port binding
+# will be to localhost, and via traefik it will be load balanced to the cluster.
+skydive_elasticsearch_port: "9200"
+
+# The elastic search servers must be defined, each hostname, or IP address is
+# separated by commas.
+# skydive_elasticsearch_servers: "server1,server2"
+
+# Setup skydive to use embedded etcd. When set, skydive will run etcd from
+# the analyzer nodes.
+skydive_etcd_embedded: yes
+skydive_etcd_port: 12379
+skydive_etcd_listen_uri: "0.0.0.0:{{ skydive_etcd_port }}"
+skydive_etcd_scheme: http
+
+# If embedded etcd is disabled the etcd server must be defined.
+# skydive_etcd_servers:
+#   - http://127.0.0.1:12379
+skydive_etcd_servers: []
+
+# If not using the embedded etcd the peers must be defined.
+# Hash of peers for etcd clustering between analyzers.
+# each entry is composed of the peer name and the endpoints for this peer.
+# skydive_etcd_peers:
+#   analyzer1: http://172.17.0.2:12380
+#   analyzer2: http://172.17.0.3:12380
+
+# The fabric setup will auto detect all intefaces throughout the cluster
+# where agents will be installed. A user can define the fabric as needed.
+# Review the skydive documentation for more information on setting up
+# fabric.
+# skydive_fabric:
+#   - TOR[Name=tor] -> TOR_PORT1[Name=port1]
+#   - TOR1[Name=tor1] -> TOR1_PORT1[Name=port1]
+#   - TOR1[Name=tor1] -> [color=red] TOR1_PORT2[Name=port2, MTU=1500]
+#   - TOR_PORT1 --> TOR1_PORT1
+#   - TOR1_PORT2 --> *[Type=host]/eth0
+
+# Path to the openvswitch db socket on a local system running skydive
+skydive_ovs_db_socket: /var/run/openvswitch/db.sock
+
+# Path to the docker socket on a local system running skydive
+skydive_docker_socket: /var/run/docker.sock
+
+# Set the skydive collection probes, available options are: ovsdb, docker, neutron, opencontrail, socketinfo, lxd, lldp, runc
+skydive_probes:
+  - "socketinfo"
+  - "lldp"
+  - "lxd"
+
+skydive_username: skydive
+
+# The skydive user name is required
+# skydive_password: secrete
+
+# The cluster username and password can be defined independently from the normal system user.
+skydive_cluster_username: "{{ skydive_username }}"
+skydive_cluster_password: "{{ skydive_password }}"
+
+skydive_auth_type: mybasic
+skydive_basic_auth_file: /var/lib/skydive/skydive.secret
+
+# Set basic authentication users and passwords into a basic auth file.
+# This is optional and will only be created if user defined.
+# skydive_basic_auth_users:
+#   skydive: secrete
+skydive_basic_auth_users: {}
+
+# Skydive openstack setup
+skydive_os_service_username: "{{ skydive_username }}.service"
+skydive_os_service_password: "{{ skydive_password }}"
+skydive_os_service_tenant_name: service
+skydive_os_service_domain_name: Default
+skydive_os_service_region_name: RegionOne
+skydive_os_service_endpoint_type: internal
+skydive_os_service_insecure: true
+
+skydive_os_auth_url: null
+skydive_auth_os_tenant_name: "{{ skydive_username }}"
+skydive_auth_os_domain_name: Default
+skydive_auth_os_domain_id: default
+skydive_auth_os_user_role: admin
+
+
+os_auth_url:
+os_username:
+os_password:
+os_tenant_name: admin
+os_user_domain_name: Default
+os_project_domain_name: Default
+os_identity_api_version: 3
+
+# Role of the user created that will be used for the probe
+# authentication
+skydive_os_service_user_role: admin
+
+
+# Configuration overrides can be set using a config template.
+# `config_template` provides an interface that will inser any
+#  option into the compatible configuration file using a deep merge.
+skydive_config_overrides: {}

skydive/roles/skydive_common/meta/main.yml

@@ -0,0 +1,40 @@
+---
+# Copyright 2019, Rackspace US, Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+galaxy_info:
+  author: OpenStack
+  description: Skydive common configuration
+  company: Rackspace
+  license: Apache License, Version 2.0
+  min_ansible_version: 2.5
+  platforms:
+    - name: Ubuntu
+      versions:
+        - xenial
+        - bionic
+    - name: EL
+      versions:
+        - 7
+    - name: opensuse
+      versions:
+        - all
+  categories:
+    - cloud
+    - development
+    - skydive
+    - networking
+
+dependencies:
+  - role: config_template

skydive/roles/skydive_common/tasks/main.yml

@@ -0,0 +1,71 @@
+---
+# Copyright 2019, Rackspace US, Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+- name: Check for skydive password
+  fail:
+    msg: >-
+      The skydive password is undefined. Set the `skydive_password` option before continuing.
+  when:
+    - skydive_password is undefined
+  tags:
+    - always
+
+- name: Check for skydive elasticsearch uri
+  fail:
+    msg: >-
+      The elasticsearch backend for skydive is undefined. Set the `skydive_elasticsearch_servers`
+      option before continuing.
+  when:
+    - skydive_elasticsearch_servers is undefined
+  tags:
+    - always
+
+- name: Check elasticsearch status
+  uri:
+    url: "{{ skydive_elasticsearch_proto }}://{{ skydive_elasticsearch_servers.split(',')[0] }}:{{ skydive_elasticsearch_port }}"
+    method: GET
+    status_code: "200"
+  register: response
+  changed_when: false
+  until: response is success
+  retries: 3
+  delay: 2
+  run_once: true
+
+- name: Gather variables for each operating system
+  include_vars: "{{ item }}"
+  with_first_found:
+    - "{{ ansible_distribution | lower }}-{{ ansible_distribution_version | lower }}.yml"
+    - "{{ ansible_distribution | lower }}-{{ ansible_distribution_major_version | lower }}.yml"
+    - "{{ ansible_os_family | lower }}-{{ ansible_distribution_major_version | lower }}.yml"
+    - "{{ ansible_distribution | lower }}.yml"
+    - "{{ ansible_os_family | lower }}-{{ ansible_distribution_version.split('.')[0] }}.yml"
+    - "{{ ansible_os_family | lower }}.yml"
+  tags:
+    - always
+
+- name: Ensure distro packages are installed
+  package:
+    name: "{{ sykdive_distro_packages }}"
+    state: "present"
+    update_cache: "{{ (ansible_pkg_mgr == 'apt') | ternary('yes', omit) }}"
+  register: _package_task
+  until: _package_task is success
+  retries: 3
+  delay: 2
+  tags:
+    - package_install
+
+- include_tasks: skydive_setup.yml

skydive/roles/skydive_common/tasks/skydive_setup.yml

@@ -0,0 +1,117 @@
+---
+# Copyright 2019, Rackspace US, Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+- name: create the system group
+  group:
+    name: "skydive"
+    state: "present"
+    system: "yes"
+
+- name: Create the skydive user
+  user:
+    name: "skydive"
+    group: "skydive"
+    comment: "skydive user"
+    shell: "/bin/false"
+    createhome: "yes"
+    home: "/usr/share/skydive"
+
+- name: Create skydive data path
+  file:
+    path: "{{ item }}"
+    state: directory
+    owner: "skydive"
+    group: "skydive"
+    mode: "0755"
+  with_items:
+    - "/var/lib/skydive"
+    - "/etc/skydive"
+
+- name: Check for ovsdb
+  stat:
+    path: "{{ skydive_ovs_db_socket }}"
+  register: stat_ovsdb
+
+- name: Check for docker
+  stat:
+    path: "{{ skydive_docker_socket }}"
+  register: stat_docker
+
+- name: Check for libvirt
+  stat:
+    path: /run/libvirt/libvirt-sock
+  register: stat_libvirt
+
+- name: Check for runc
+  stat:
+    path: /var/run/runc
+  register: stat_runc
+
+- name: Set stat fact exists
+  set_fact:
+    skydive_ovs_db_exists: "{{ stat_ovsdb.stat.exists }}"
+    skydive_docker_exists: "{{ stat_docker.stat.exists }}"
+    skydive_libvirt_exists: "{{ stat_libvirt.stat.exists }}"
+    skydive_runc_exists: "{{ stat_runc.stat.exists }}"
+
+- name: Create basic auth file
+  htpasswd:
+    path: "{{ skydive_basic_auth_file }}"
+    name: "{{ item.key }}"
+    password: "{{ item.value }}"
+    owner: root
+    group: skydive
+    mode: 0640
+  with_dict: "{{ _skydive_basic_auth_users | combine(skydive_basic_auth_users) }}"
+  register: skydive_basic_auth_create
+
+- name: Drop skydive conf files
+  config_template:
+    src: "skydive.yml.j2"
+    dest: "/etc/skydive/skydive.yml"
+    mode: "0640"
+    config_overrides: "{{ skydive_config_overrides }}"
+    config_type: yaml
+  register: skydive_template_create
+
+- name: Ensure local facts directory exists
+  file:
+    dest: "/etc/ansible/facts.d"
+    state: directory
+    group: "root"
+    owner: "root"
+    mode:  "0755"
+    recurse: no
+
+- name: Initialize local facts
+  ini_file:
+    dest: "/etc/ansible/facts.d/skydive.fact"
+    section: "skydive"
+    option: initialized
+    value: true
+
+- name: Set restart fact exists
+  set_fact:
+    skydive_needs_restart: "{{ (skydive_template_create is changed) or (skydive_basic_auth_create is changed) }}"
+
+- name: Set storage fact
+  ini_file:
+    dest: "/etc/ansible/facts.d/skydive.fact"
+    section: "skydive"
+    option: "{{ item.key }}"
+    value: "{{ item.value }}"
+  with_dict:
+    needs_restart_agent: "{{ (inventory_hostname in groups['skydive_agents']) and (skydive_needs_restart | bool) }}"
+    needs_restart_analyzers: "{{ (inventory_hostname in groups['skydive_analyzers']) and (skydive_needs_restart | bool) }}"

skydive/roles/skydive_common/templates/skydive.yml.j2

@@ -0,0 +1,519 @@
+---
+
+# Skydive config file
+
+# host_id is used to reference the agent, by default set to hostname
+host_id: {{ ansible_hostname }}
+
+tls:
+  # File path to X509 Certificate and Private Key to enable TLS communication
+  # Unique certificate per agent is recommended
+  # client_cert: /etc/ssl/certs/agent.domain.com.crt
+  # client_key:  /etc/ssl/certs/agent.domain.com.key
+
+  # server_cert: /etc/ssl/certs/analyzer.domain.com.crt
+  # server_key:  /etc/ssl/certs/analyzer.domain.com.key
+
+  # ca_cert: /etc/ssl/certs/ca.domain.com.crt
+
+http:
+  # define the Cookie HTTP Request Header
+  cookie:
+    # <name1>: <value1>
+    # <name2>: <value2>
+
+  rest:
+    # log the HTTP client request and response (to log level DEBUG)
+    # debug: false
+
+  ws:
+    # WebSocket delay between two pings.
+    # ping_delay: 2
+
+    # WebSocket Ping/Pong timeout in second.
+    # pong_timeout: 5
+
+    # maximum number of topology aggregated messages before sending
+    # bulk_maxmsgs: 100
+
+    # duration in seconds before flushing topology aggregated messages
+    # bulk_maxdelay: 2
+
+    # Maximum size of the message queue
+    # queue_size: 10000
+
+    # enable write compression
+    # enable_write_compression: true
+
+{% if inventory_hostname in groups['skydive_analyzers'] %}
+analyzer:
+  # address and port for the analyzer API, Format: addr:port.
+  # Default addr is 127.0.0.1
+  listen: {{ hostvars[inventory_hostname]["ansible_" ~ (skydive_network_device | replace('-', '_') | string)]['ipv4']['address'] ~ ':' ~ skydive_analyzer_port }}
+
+  auth:
+    # auth section for API request
+    api:
+      # Specify the name of the auth backend definition, see auth section.
+      backend: {{ skydive_auth_type }}
+
+    cluster:
+      # Specify the name of the auth backend definition, see auth section.
+      backend: {{ skydive_auth_type }}
+
+      # Specify username, password for cluster authentication. Used for analyzer/analyzer communication.
+      username: {{ skydive_cluster_username }}
+      password: {{ skydive_cluster_password }}
+
+  # Section defining things to be invoked on startup
+  startup:
+    # By default no capturing,  set filter to capture from selected nodes
+    # from the beginning automatically
+    # capture_gremlin: "G.V().has('Name', NE('lo'))"
+    # capture_bpf: "port 80"
+
+  # Flow storage engine
+  flow:
+    # Storage backend name: myelasticsearch, myorientdb
+    backend: myelasticsearch
+
+    # Max number of flows in write buffer (after which all flows accumulated are dropped)
+    # max_buffer_size: 100000
+
+  topology:
+    # Storage backend name: mymemory, myelasticsearch, myorientdb
+    backend: myelasticsearch
+
+    # Define static interfaces and links updating Skydive topology
+    # Can be useful to define external resources like : TOR, Router, etc.
+    #
+    # A description language similar to the dot language is used to define
+    # interfaces and links. An arrow (->) is used to define a link between
+    # two interfaces (parent -> child). An arrow with a single dash will
+    # create an OwnerShip and a L2 link between the parent and the child.
+    # An arrow with two dashes (-->) will only create a L2 link between the
+    # parent and the child.
+    #
+    # Square brackets after the arrow is used to define additional metadata
+    # of the link (->[key=value,..]). Each interface described will be
+    # created in the topology excepted interfaces with the local prefix.
+    # In that specific case the interface of the local host will be used.
+    # Attributes of interfaces are declared using square brackets ([]).
+    # The following example creates a TOR node linked to TOR_PORT1 linked
+    # (l2 only) to TOR1_PORT1 linked to the TOR1 node, linked to TOR1_PORT2,
+    # which is linked to the local interface eth0, with an l2 only link.
+    fabric: {{ skydive_fabric | to_json }}
+
+    # list of probes used by the analyzers
+    probes:
+      # - k8s
+      # - istio
+
+    k8s:
+      # EXPERIMENTAL: k8s probe is still under development and should not be used
+      # on production systems
+
+      # kubeconfig resolution order:
+      # - if config_file param is defined then use it;
+      # - else if $KUBECONFIG environment is define then use it;
+      # - else if $HOME/.kube/config file exists then use it;
+      # - else use empty configuration (for accessing from within the k8s cluster).
+
+      # specify the path of k8s configuration YAML file.
+      # config_file: /etc/skydive/kubeconfig
+
+      # list of (sub) probes comprising k8s probe.
+      # if list is empty then will resolve to all existing (sub) probes.
+      probes:
+        - cluster
+        - container
+        - cronjob
+        - deployment
+        - endpoints
+        - ingress
+        - job
+        - namespace
+        - networkpolicy
+        - node
+        - persistentvolume
+        - persistentvolumeclaim
+        - pod
+        - replicaset
+        - replicationcontroller
+        - service
+        - statefulset
+        - storageclass
+
+    istio:
+      # specify the path of istio configuration YAML file.
+      # config_file: /etc/skydive/kubeconfig
+
+      # EXPERIMENTAL: istio probe is still under development and should not be used
+      # on production systems
+      probes:
+        - destinationrule
+        - gateway
+        - quotaspec
+        - quotaspecbinding
+        - serviceentry
+        - virtualservice
+
+  replication:
+    # debug: false
+{% endif %}
+
+# list of analyzers used by analyzers and agents
+{% set analyzers = [] %}
+{% for node in groups['skydive_analyzers'] %}
+{%   set _ansible_interface_name = hostvars[node]['skydive_network_device'] | default(hostvars[node]['ansible_default_ipv4']['interface']) | replace('-', '_') %}
+{%   set _ = analyzers.append(hostvars[node]["ansible_" ~ _ansible_interface_name]['ipv4']['address'] ~ ':' ~ skydive_analyzer_port) %}
+{% endfor %}
+analyzers: {{ analyzers | to_json }}
+
+{% if inventory_hostname in groups['skydive_agents'] %}
+agent:
+  # address and port for the agent API, Format: addr:port.
+  # Default addr is 127.0.0.1
+  listen: {{ hostvars[inventory_hostname]["ansible_" ~ (skydive_network_device | replace('-', '_') | string)]['ipv4']['address'] ~ ':' ~ skydive_agent_port }}
+
+  auth:
+    # auth section for API request
+    api:
+      # Specify the name of the auth backend definition, see auth section.
+      backend: {{ skydive_auth_type }}
+
+    cluster:
+      # Specify username, password for cluster authentication. Used for agent/analyzer communication.
+      username: {{ skydive_cluster_username }}
+      password: {{ skydive_cluster_password }}
+
+  topology:
+    # Probes used to capture topology information like interfaces,
+    # bridges, namespaces, etc...
+    # Available: ovsdb, docker, neutron, opencontrail, socketinfo, lxd, lldp
+{% if skydive_docker_exists | bool %}
+{%   set _ = skydive_probes.append('docker') %}
+{% endif %}
+{% if skydive_ovs_db_exists | bool %}
+{%   set _ = skydive_probes.append('ovsdb') %}
+{% endif %}
+    probes: {{ skydive_probes | to_json }}
+
+    netlink:
+      # delay in seconds between two metric updates
+      # metrics_update: 30
+
+    # Define OpenStack Neutron credentials and the enpoint type
+    # used by the neutron probe
+    neutron:
+      # auth_url:
+      # username: neutron
+      # password: secret
+      # tenant_name: service
+      # region_name: RegionOne
+      # domain_name: Default
+      # ssl_insecure: false
+
+      # The endpoint_type value must be 'public', 'internal' or 'admin'
+      # endpoint_type: public
+
+    lldp:
+      # Interfaces to listen for LLDP frames. If no list is specified,
+      # use all interfaces
+      interfaces:
+
+{% if skydive_libvirt_exists | bool %}
+    libvirt:
+      url: qemu:///system
+{% endif %}
+
+{% if skydive_runc_exists | bool %}
+    runc:
+      run_path:
+        - /var/run/runc
+{% endif %}
+
+  capture:
+    # Period in second to get capture stats from the probe. Note this
+    # stats_update: 1
+
+  metadata:
+    # info: This is compute node
+{% endif %}
+
+dpdk:
+  # DPDK port listening flows from
+  ports:
+    # - 0
+    # - 1
+
+  # nb workers per port
+  # workers: 4
+
+  # debug message every n seconds
+  # debug: 1
+
+sflow:
+  # Default listening address is 127.0.0.1
+  # bind_address: 127.0.0.1
+
+  # Port min/max used when starting a sflow probe, a agent will be started
+  # with a port from this range
+  # port_min: 6345
+  # port_max: 6355
+
+{% if skydive_ovs_db_exists | bool %}
+ovs:
+  # ovsdb connection, Format supported :
+  # * addr:port
+  # * tcp://addr:port
+  # * unix:///var/run/openvswitch/db.sock
+  # If you use the tcp connection you need to authorize connexion to ovsdb agent
+  # at least locally
+  # % sudo ovs-appctl -t ovsdb-server ovsdb-server/add-remote ptcp:6400:127.0.0.1
+  ovsdb: unix://{{ skydive_ovs_db_socket }}
+
+  oflow:
+    # Enable the parsing of openflow rules (disabled by default)
+    # enable: false
+
+    # Openflow versions used by ovs-ofctl when queries are made to the
+    # switch. 1.0 should always be supported. 1.3 gives a nicer output and
+    # it is recommended to add it if it is supported.
+    # 1.4 can be broken on some switch, 1.5 and 1.6 are still considered
+    # as experimental.
+    # openflow_versions:
+    #  - OpenFlow10
+
+    # The probe can connect to remote bridge over TLS (ssl url).
+    # The default value is empty for those options.
+    # Path to the private key file (TLS connection)
+    # key: /etc/ssl/private/agent.key
+
+    # Path to the certificate associated to the key (TLS connection)
+    # cert: /etc/ssl/certs/agent.crt
+
+    # Path to certificate authority validating bridge connections (TLS connection)
+    # ca: /etc/ssl/certs/ca.crt
+
+    address:
+      # Map translating bridge names into URL for remote connection
+      # - bridge: ssl:xxx.yyy.zzz.ttt:port
+{% endif %}
+
+{% if skydive_docker_exists | bool %}
+docker:
+  url: unix://{{ skydive_docker_socket }}
+{% endif %}
+
+netns:
+  # allow to specify where the netns probe is watching network namespace
+  run_path: /var/run/netns
+
+opencontrail:
+  # Host address of the OpenContrail vrouter agent
+  # host: localhost
+
+  # TCP port of the OpenContrail vrouter agent
+  # port: 8086
+
+  # UDP dest port for MPLS traffic
+  # mpls_udp_port: 51234
+
+storage:
+  # Elasticsearch backend information.
+  myelasticsearch:
+    driver: elasticsearch
+    host: "127.0.0.1:19200"
+
+    # Define the maximum delay before flushing document
+    # bulk_maxdelay: 5
+
+    # If a limit is specified, when the index reaches it, it is rolled.
+    # index_entries_limit specifies the maximum number of entries allowed in an index.
+    # index_age_limit specifies the maximum age (in minutes) allowed for an index.
+    # For both limits, a value of 0 specifies that there is no limitation.
+    # index_entries_limit: 0
+    # index_age_limit: 0
+
+    # The number of indices to keep before deleting.
+    # A value of 0 specifies no limit (i.e. indices will never be deleted)
+    # indices_to_keep: 0
+
+  # OrientDB backend information.
+  myorientdb:
+    # driver: orientdb
+    # addr: http://127.0.0.1:2480
+    # database: Skydive
+    # username: root
+    # password: hello
+
+  # Memory backend
+  mymemory:
+    # driver: memory
+
+logging:
+  # level: INFO
+
+  # Default backend used: stderr
+  backends:
+    # - stderr
+    # - stdout
+    - file
+    # - syslog
+
+  # configuration of the 'file' backend
+  file:
+    path: /var/log/skydive.log
+
+  # configuration encoder could be for all backends or for specific one
+  # encoder: json
+  # color: false
+
+auth:
+  mybasic:
+    # Define a basic auth authentication backend
+    type: basic
+
+    # Specify the htpassword file to be used
+    file: {{ skydive_basic_auth_file }}
+
+    # Users can be declared in this section instead of using a file.
+    # users:
+      # user1: secret1
+      # user2: secret2
+
+  mykeystone:
+    # Define a basic auth authentication backend
+    type: keystone
+    auth_url: {{ skydive_os_auth_url }}
+
+    # define the tenant and the domain that the users have to belong to
+    tenant_name: {{ skydive_auth_os_tenant_name }}
+    domain_name: {{ skydive_auth_os_domain_name }}
+
+    # define which role an authenticated user will have. Only used for API authentication.
+    # two roles are predefined, admin and guest.
+    role: {{ skydive_auth_os_user_role }}
+
+etcd:
+  # server parameters
+  # when 'embedded' is set to true, the analyzer will start an embedded etcd server
+  embedded: {{ skydive_etcd_embedded }}
+  listen: {{ skydive_etcd_listen_uri }}
+
+  # maximum number of WAL and snapshot files. 0 means unlimited
+  # max_wal_files: 0
+  # max_snap_files: 0
+
+  # path where the etcd files will be stored.
+  # data_dir: /var/lib/skydive/etcd
+
+  # client parameters
+{% if skydive_etcd_servers %}
+  servers: {{ skydive_etcd_servers | to_json }}
+{% endif %}
+  # name to use for clustering, by default it is set to the host id
+  name: {{ inventory_hostname }}
+
+  # list of peers for etcd clustering between analyzers
+  # each entry is composed of the peer name and the endpoints for this peer
+{% set peers = {} %}
+{% for node in groups['skydive_analyzers'] %}
+{%   set _ansible_interface_name = hostvars[node]['skydive_network_device'] | default(hostvars[node]['ansible_default_ipv4']['interface']) | replace('-', '_') %}
+{%   set _ = peers.__setitem__(inventory_hostname, 'http://' ~ hostvars[node]["ansible_" ~ _ansible_interface_name]['ipv4']['address'] ~ ':' ~ skydive_etcd_port) %}
+{% endfor %}
+  peers: {{ skydive_etcd_peers | default(peers) | to_json }}
+
+  # client_timeout: 5
+
+flow:
+  # Without any new packets, a flow expires after flow.expire
+  # seconds
+  # expire: 600
+
+  # Seconds between flow updates (metrics, enhancements,...)
+  # update: 60
+
+  # Protocol to use to send flows to the analyzer: websocket or udp
+  protocol: {{ skydive_flow_protocol }}
+
+  # Define the layer key mode used by default for captures. The key mode defines
+  # the layers used to identify a unique flow.
+  # * L2, this mode includes layer 2 and beyond.
+  # * L3, this mode includes layer 3 and beyond and takes layer 2 if there is no layer 3.
+  # default_layer_key_mode: L2
+
+  # Set the application field according to the following port mapping
+  application_ports:
+    tcp:
+      # 80: HTTP
+      # 8080: HTTP
+      # 443: HTTPS
+      # 1194: OPENVPN
+    udp:
+      # 1194: OPENVPN
+
+ui:
+  # Specify the extra assets folder. Javascript and CSS files present in this
+  # folder will be added to the WebUI.
+  # extra_assets: /usr/share/skydive/assets
+
+  # select between light, dark themes
+  # theme: dark
+
+  # Settings specific to the topology view
+  topology:
+    # Pre-defined Gremlin expression used in the WebUI for Filtering and Highlighting.
+    # Note: Key should be in lower case
+    favorites:
+    #   namespaces: "g.V().Has('Type', 'netns').OutE().BothV()"
+    #   layer2: "g.E().Has('RelationType', 'layer2')"
+
+    # Highlight Gremlin expression used by default and applied on WebUI load.
+    # default_highlight: "layer2"
+    # Filter Gremlin expression used by default and applied on WebUI load.
+    # default_filter: "layer2"
+
+  # update rate of links in seconds
+  bandwidth_update_rate: 5
+
+  # 'absolute' - thresholds in Kbit
+  # 'relative' - thresholds in % relative to link speed reported by netlink
+  bandwidth_threshold: absolute
+  bandwidth_absolute_active: 1
+  bandwidth_absolute_warning: 10
+  bandwidth_absolute_alert: 100
+  bandwidth_relative_active: 0.1
+  bandwidth_relative_warning: 0.4
+  bandwidth_relative_alert: 0.8
+
+  # Enable/disable ssh to hosts
+  # ssh_enabled: false
+
+  # Enable/disable k8s related elements
+  # k8s_enabled: false
+
+  bpf:
+    # Pre-defined BPF filters
+    favorites:
+      # filter1: ip broadcast
+      # filter2: ip multicast
+
+rbac:
+  model:
+    # RBAC model
+    # request_definition:
+    # - sub, obj, act
+    # policy_definition:
+    # - sub, obj, act, eft
+    # role_definition:
+    # - _, _
+    # policy_effect:
+    # - some(where (p_eft == allow)) && !some(where (p_eft == deny))
+    # matchers:
+    # - g(r.sub, p.sub) && r.obj == p.obj && r.act == p.act
+  policy:
+    # additional RBAC policy:
+    # - p, myuser, capture, write, deny
+    # - g, myuser, myrole

skydive/roles/skydive_common/vars/main.yml

@@ -0,0 +1,45 @@
+---
+# Copyright 2019, Rackspace US, Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Dynamic construction of the system fabric for all hosts within the agents group
+skydive_fabric: |-
+  {% set fabric = [] %}
+  {% set nodes = [] %}
+  {% for node in groups['skydive_analyzers'] %}
+  {%   set agents_loop = loop %}
+  {%   for interface in (hostvars[node]['ansible_interfaces'] | map('replace', '-','_') | list) %}
+  {%     if interface != 'lo' %}
+  {%       set ansible_interface_name = "ansible_" ~ interface %}
+  {%       set port_entry = "TOR[Name=TOR] -> TOR_PORT" ~ agents_loop.index ~ "[Name=port" ~ agents_loop.index ~ "]" %}
+  {%       if hostvars[node][ansible_interface_name] is defined %}
+  {%         set interface_data = hostvars[node][ansible_interface_name] %}
+  {%         if interface_data['mtu'] is defined %}
+  {%           set port_entry = "TOR[Name=TOR] -> [color=red] TOR_PORT" ~ agents_loop.index ~ "[Name=port" ~ agents_loop.index ~ ",MTU=" ~ interface_data['mtu'] ~ "]" %}
+  {%         endif %}
+  {%       endif %}
+  {%       set _ = fabric.append((port_entry)) %}
+  {%       if not interface in nodes %}
+  {%         set host_entry = "TOR_PORT" ~ agents_loop.index ~ "-> *[Type=host,Name=" ~ hostvars[node]['ansible_hostname'] ~ "/" ~ interface %}
+  {%         set _ = fabric.append((host_entry)) %}
+  {%       endif %}
+  {%       set _ = nodes.append(interface) %}
+  {%     endif %}
+  {%   endfor %}
+  {% endfor %}
+  {{ fabric }}
+
+# Inject the required basic authentication information
+_skydive_basic_auth_users:
+  "{{ skydive_username }}": "{{ skydive_password }}"

skydive/roles/skydive_common/vars/redhat.yml

@@ -0,0 +1,17 @@
+---
+# Copyright 2019, Rackspace US, Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+sykdive_distro_packages:
+  - python2-passlib

skydive/roles/skydive_common/vars/suse.yml

@@ -0,0 +1,17 @@
+---
+# Copyright 2019, Rackspace US, Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+sykdive_distro_packages:
+  - python-passlib

skydive/roles/skydive_common/vars/ubuntu.yml

@@ -0,0 +1,17 @@
+---
+# Copyright 2019, Rackspace US, Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+sykdive_distro_packages:
+  - python-passlib

skydive/roles/traefik_common/defaults/main.yml

@@ -0,0 +1,47 @@
+---
+# Copyright 2019, Rackspace US, Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+traefik_analyzer_service_state: restarted
+
+# traefik_destinations:
+#   env_name:
+#     proto: http
+#     port: 9200
+#     bind: 127.0.0.1
+#     servers:
+#       - name: server1
+#         address: 10.0.0.1
+#         weight: 1
+#       - name: server2
+#         address: 10.0.0.2
+#         weight: 2
+traefik_destinations: {}
+
+# Path to the docker socket on a local or remote system running traefik
+traefik_docker_socket: "unix://var/run/docker.sock"
+
+traffic_dashboard_bind: "{{ ansible_default_ipv4['address'] }}"
+
+traefik_dashboard_port: "8090"
+
+traefik_dashboard_enabled: false
+
+# Set basic authentication users and passwords into a basic auth file.
+# This is optional and will only be created if user defined.
+# traefik_basic_auth_users:
+#   traefik: secrete
+traefik_basic_auth_users: {}
+
+traefik_basic_auth_file: /var/lib/traefik/traefik.secret

skydive/roles/traefik_common/meta/main.yml

@@ -0,0 +1,39 @@
+---
+# Copyright 2019, Rackspace US, Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+galaxy_info:
+  author: OpenStack
+  description: Traefik common configuration
+  company: Rackspace
+  license: Apache License, Version 2.0
+  min_ansible_version: 2.5
+  platforms:
+    - name: Ubuntu
+      versions:
+        - xenial
+        - bionic
+    - name: EL
+      versions:
+        - 7
+    - name: opensuse
+      versions:
+        - all
+  categories:
+    - cloud
+    - development
+    - traefik
+    - networking
+
+dependencies: []

skydive/roles/traefik_common/tasks/main.yml

@@ -0,0 +1,40 @@
+---
+# Copyright 2019, Rackspace US, Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+- name: Gather variables for each operating system
+  include_vars: "{{ item }}"
+  with_first_found:
+    - "{{ ansible_distribution | lower }}-{{ ansible_distribution_version | lower }}.yml"
+    - "{{ ansible_distribution | lower }}-{{ ansible_distribution_major_version | lower }}.yml"
+    - "{{ ansible_os_family | lower }}-{{ ansible_distribution_major_version | lower }}.yml"
+    - "{{ ansible_distribution | lower }}.yml"
+    - "{{ ansible_os_family | lower }}-{{ ansible_distribution_version.split('.')[0] }}.yml"
+    - "{{ ansible_os_family | lower }}.yml"
+  tags:
+    - always
+
+- name: Ensure distro packages are installed
+  package:
+    name: "{{ traefik_distro_packages }}"
+    state: "present"
+    update_cache: "{{ (ansible_pkg_mgr == 'apt') | ternary('yes', omit) }}"
+  register: _package_task
+  until: _package_task is success
+  retries: 3
+  delay: 2
+  tags:
+    - package_install
+
+- import_tasks: traefik_setup.yml

skydive/roles/traefik_common/tasks/traefik_setup.yml

@@ -0,0 +1,98 @@
+---
+# Copyright 2019, Rackspace US, Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+- name: create the system group
+  group:
+    name: "traefik"
+    state: "present"
+    system: "yes"
+
+- name: Create the traefik user
+  user:
+    name: "traefik"
+    group: "traefik"
+    comment: "traefik user"
+    shell: "/bin/false"
+    createhome: "yes"
+    home: "/usr/share/traefik"
+
+- name: Create traefik data path
+  file:
+    path: "{{ item }}"
+    state: directory
+    owner: "traefik"
+    group: "traefik"
+    mode: "0755"
+  with_items:
+    - "/var/lib/traefik"
+    - "/etc/traefik"
+
+- name: Check for docker
+  stat:
+    path: "{{ traefik_docker_socket }}"
+  register: stat_docker
+
+- name: Set stat fact exists
+  set_fact:
+    traefik_docker_exists: "{{ stat_docker.stat.exists }}"
+
+- name: Create basic auth file
+  htpasswd:
+    path: "{{ traefik_basic_auth_file }}"
+    name: "{{ item.key }}"
+    password: "{{ item.value }}"
+    owner: root
+    group: traefik
+    mode: 0640
+  with_dict: "{{ traefik_basic_auth_users }}"
+  when:
+    - traefik_dashboard_enabled | bool
+    - traefik_basic_auth_users
+
+- name: Drop traefik conf files
+  template:
+    src: "traefik.toml.j2"
+    dest: "/etc/traefik/traefik.toml"
+    mode: "0640"
+
+- name: Run the systemd service role
+  include_role:
+    name: systemd_service
+  vars:
+    systemd_service_enabled: true
+    systemd_service_restart_changed: false
+    systemd_services:
+      - service_name: "traefik"
+        execstarts:
+          - /usr/local/bin/traefik --file.directory="/etc/traefik"
+
+- name: Force handlers
+  meta: flush_handlers
+
+- name: Set traefik service state (upstart)
+  service:
+    name: "traefik"
+    state: "{{ traefik_analyzer_service_state }}"
+    enabled: "{{ traefik_analyzer_service_state in ['running', 'started', 'restarted'] }}"
+  when:
+    - ansible_service_mgr == 'upstart'
+
+- name: Set traefik service state (systemd)
+  systemd:
+    name: "traefik"
+    state: "{{ traefik_analyzer_service_state }}"
+    enabled: "{{ traefik_analyzer_service_state in ['running', 'started', 'restarted'] }}"
+  when:
+    - ansible_service_mgr == 'systemd'

skydive/roles/traefik_common/templates/traefik.toml.j2

@@ -0,0 +1,205 @@
+################################################################
+# Global configuration
+################################################################
+
+# Enable debug mode
+#
+# Optional
+# Default: false
+#
+debug = false
+
+# Log level
+#
+# Optional
+# Default: "ERROR"
+#
+logLevel = "INFO"
+
+# Entrypoints to be used by frontends that do not specify any entrypoint.
+# Each frontend can specify its own entrypoints.
+#
+# Optional
+# Default: ["http"]
+#
+# defaultEntryPoints = ["http", "https"]
+
+################################################################
+# Entrypoints configuration
+################################################################
+
+# Entrypoints definition
+#
+# Optional
+# Default:
+[entryPoints]
+  [entryPoints.traefik]
+    address = "{{ traffic_dashboard_bind }}:{{ traefik_dashboard_port }}"
+{% if traefik_dashboard_enabled | bool %}
+    [entryPoints.traefik.auth]
+      [entryPoints.traefik.auth.basic]
+        removeHeader = true
+        usersFile = "{{ traefik_basic_auth_file }}"
+{% endif %}
+
+{% for key, value in traefik_destinations.items() %}
+  [entryPoints.{{ key }}-{{ value.proto }}]
+    address = "{{ value.bind | default('127.0.0.1') }}:{{ value.port }}"
+{% endfor %}
+
+[file]
+watch = true
+
+  ################################################################
+  # Backends configuration
+  ################################################################
+  [backends]
+  {% for key, value in traefik_destinations.items() %}
+    [backends.{{ key }}]
+  {%   for server in value.servers %}
+      [backends.{{ key }}.loadbalancer.stickiness]
+      [backends.{{ key }}.servers.{{ server.name }}]
+      url = "{{ server.proto | default(value.proto) }}://{{ server.address }}:{{ server.port | default(value.port) }}"
+      weight = {{ server.weight | default(100)  }}
+  {%   endfor %}
+  {% endfor %}
+
+  ################################################################
+  # Frontends configuration
+  ################################################################
+  [frontends]
+  {% for key, value in traefik_destinations.items() %}
+    [frontends.{{ key }}]
+      entryPoints = ["{{ key }}-{{ value.proto }}"]
+      backend = "{{ key }}"
+      passHostHeader = true
+
+  {% endfor %}
+
+################################################################
+# Traefik logs configuration
+################################################################
+
+# Traefik logs
+# Enabled by default and log to stdout
+#
+# Optional
+#
+[traefikLog]
+
+# Sets the filepath for the traefik log. If not specified, stdout will be used.
+# Intermediate directories are created if necessary.
+#
+# Optional
+# Default: os.Stdout
+#
+# filePath = "/var/log/traefik/traefik.log"
+
+# Format is either "json" or "common".
+#
+# Optional
+# Default: "common"
+#
+# format = "common"
+
+################################################################
+# Access logs configuration
+################################################################
+
+# Enable access logs
+# By default it will write to stdout and produce logs in the textual
+# Common Log Format (CLF), extended with additional fields.
+#
+# Optional
+#
+[accessLog]
+
+# Sets the file path for the access log. If not specified, stdout will be used.
+# Intermediate directories are created if necessary.
+#
+# Optional
+# Default: os.Stdout
+#
+# filePath = "/var/log/traefik/access.log"
+
+# Format is either "json" or "common".
+#
+# Optional
+# Default: "common"
+#
+# format = "common"
+
+################################################################
+# API and dashboard configuration
+################################################################
+
+# Enable API and dashboard
+[api]
+  # Name of the related entry point
+  #
+  # Optional
+  # Default: "traefik"
+  #
+  entryPoint = "traefik"
+
+  # Enabled Dashboard
+  #
+  # Optional
+  # Default: true
+  #
+  dashboard = true
+
+  # Enable debug mode.
+  # This will install HTTP handlers to expose Go expvars under /debug/vars and
+  # pprof profiling data under /debug/pprof/.
+  # Additionally, the log level will be set to DEBUG.
+  #
+  # Optional
+  # Default: false
+  #
+  debug = true
+
+################################################################
+# Ping configuration
+################################################################
+
+# Enable ping
+# [ping]
+
+  # Name of the related entry point
+  #
+  # Optional
+  # Default: "traefik"
+  #
+  # entryPoint = "traefik"
+
+{% if traefik_docker_exists | bool %}
+################################################################
+# Docker configuration backend
+################################################################
+
+# Enable Docker configuration backend
+[docker]
+
+# Docker server endpoint. Can be a tcp or a unix socket endpoint.
+#
+# Required
+# Default: "unix:///var/run/docker.sock"
+#
+endpoint = "{{ traefik_docker_socket }}"
+
+# Default domain used.
+# Can be overridden by setting the "traefik.domain" label on a container.
+#
+# Optional
+# Default: ""
+#
+# domain = "docker.localhost"
+
+# Expose containers by default in traefik
+#
+# Optional
+# Default: true
+#
+# exposedByDefault = true
+{% endif %}

skydive/roles/traefik_common/vars/redhat.yml

@@ -0,0 +1,17 @@
+---
+# Copyright 2019, Rackspace US, Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+traefik_distro_packages:
+  - python2-passlib

skydive/roles/traefik_common/vars/suse.yml

@@ -0,0 +1,17 @@
+---
+# Copyright 2019, Rackspace US, Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+traefik_distro_packages:
+  - python-passlib

skydive/roles/traefik_common/vars/ubuntu.yml

@@ -0,0 +1,17 @@
+---
+# Copyright 2019, Rackspace US, Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+traefik_distro_packages:
+  - python-passlib

skydive/site.yml

@@ -0,0 +1,22 @@
+---
+# Copyright 2019, Rackspace US, Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+- import_playbook: buildSkydive.yml
+
+- import_playbook: buildTraefik.yml
+
+- import_playbook: installSkydive.yml
+
+- import_playbook: validateSkydive.yml

skydive/tests/_container-setup.yml

@@ -0,0 +1,73 @@
+---
+# Copyright 2018, Rackspace US, Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+- name: Setup host for nspawn
+  hosts: localhost
+  connection: local
+  become: true
+  vars:
+    nspawn_networks:
+      nspawn_address:
+        bridge: "nspawn0"
+        private_device: true
+        enable_dhcp: true
+        dhcp_range: 10.100.101.2,10.100.101.129
+        address: 10.100.101.1
+        netmask: 255.255.255.0
+        macvlan_mode: bridge
+
+  pre_tasks:
+    - name: Ensure root ssh key
+      user:
+        name: "{{ ansible_env.USER | default('root') }}"
+        generate_ssh_key: "yes"
+        ssh_key_bits: 2048
+        ssh_key_file: ".ssh/id_rsa"
+
+    - name: Get root ssh key
+      slurp:
+        src: '~/.ssh/id_rsa.pub'
+      register: _root_ssh_key
+
+    - name: Prepare container ssh key fact
+      set_fact:
+        nspawn_container_ssh_key: "{{ _root_ssh_key['content'] | b64decode }}"
+
+    - name: Ensure public ssh key is in authorized_keys
+      authorized_key:
+        user: "{{ ansible_env.USER | default('root') }}"
+        key: "{{ nspawn_container_ssh_key }}"
+        manage_dir: no
+
+  roles:
+    - role: "nspawn_hosts"
+
+
+- name: Create container(s)
+  hosts: all_containers
+  gather_facts: false
+  become: true
+  pre_tasks:
+    - name: Show container facts
+      debug:
+        var: hostvars
+
+  roles:
+    - role: "nspawn_container_create"
+
+  post_tasks:
+    - name: Rescan quotas
+      command: "btrfs quota rescan -w /var/lib/machines"
+      delegate_to: "{{ physical_host }}"

skydive/tests/_key-setup.yml

@@ -0,0 +1,41 @@
+---
+# Copyright 2018, Rackspace US, Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+- name: Setup host keys
+  hosts: physical_hosts
+  connection: local
+  become: true
+  tasks:
+    - name: Ensure root ssh key
+      user:
+        name: "{{ ansible_env.USER | default('root') }}"
+        generate_ssh_key: "yes"
+        ssh_key_bits: 2048
+        ssh_key_file: ".ssh/id_rsa"
+
+    - name: Get root ssh key
+      slurp:
+        src: '~/.ssh/id_rsa.pub'
+      register: _root_ssh_key
+
+    - name: Prepare container ssh key fact
+      set_fact:
+        nspawn_container_ssh_key: "{{ _root_ssh_key['content'] | b64decode }}"
+
+    - name: Ensure public ssh key is in authorized_keys
+      authorized_key:
+        user: "{{ ansible_env.USER | default('root') }}"
+        key: "{{ nspawn_container_ssh_key }}"
+        manage_dir: no

skydive/tests/ansible-role-requirements.yml

@@ -0,0 +1,9 @@
+---
+- name: config_template
+  scm: git
+  src: https://git.openstack.org/openstack/ansible-config_template
+  version: master
+- name: systemd_service
+  scm: git
+  src: https://git.openstack.org/openstack/ansible-role-systemd_service
+  version: master

skydive/tests/functional.yml

@@ -0,0 +1,158 @@
+---
+# Copyright 2018, Rackspace US, Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+- import_playbook: run-setup.yml
+
+- name: Basic setup
+  hosts: "all"
+  become: true
+
+  environment:
+    # ZUUL_PROJECT is used by tests/get-ansible-role-requirements to
+    # determine when CI provided repos should be used.
+    ZUUL_PROJECT: "{{ zuul.project.short_name }}"
+    ANSIBLE_PACKAGE: "{{ ansible_package | default('') }}"
+    ANSIBLE_HOST_KEY_CHECKING: "False"
+    ANSIBLE_LOG_PATH: "/tmp/skydive-logs/ansible-skydive-test.log"
+    ANSIBLE_ACTION_PLUGINS: "${HOME}/ansible_venv/repositories/roles/config_template/action"
+    ANSIBLE_CONNECTION_PLUGINS: "${HOME}/ansible_venv/repositories/roles/plugins/connection"
+    ANSIBLE_ROLES_PATH: "${HOME}/ansible_venv/repositories/roles"
+
+  vars:
+    inventory_file: "inventory/test-metal-inventory.yml"
+
+  pre_tasks:
+    - name: Create swap file
+      command: "dd if=/dev/zero of=/swap.img bs=1M count=4096"
+      args:
+        creates: /swap.img
+      register: swap_create
+
+    - name: Format the swap file
+      command: mkswap /swap.img
+      when:
+        - swap_create is changed
+      tags:
+        - swap-format
+        - skip_ansible_lint
+
+    - name: Enable swap file
+      command: swapon /swap.img
+      failed_when: false
+      tags:
+        - swap-format
+        - skip_ansible_lint
+
+    - name: Set system swappiness
+      sysctl:
+        name: vm.swappiness
+        value: 10
+        state: present
+        reload: "yes"
+        sysctl_file: /etc/sysctl.d/99-skydive.conf
+
+    - name: Create tmp skydive dir
+      file:
+        path: "/tmp/skydive-logs"
+        state: directory
+
+    - name: Flush iptables rules
+      command: "{{ item }}"
+      args:
+        creates: "/tmp/skydive-logs/iptables.flushed"
+      with_items:
+        - "iptables -F"
+        - "iptables -X"
+        - "iptables -t nat -F"
+        - "iptables -t nat -X"
+        - "iptables -t mangle -F"
+        - "iptables -t mangle -X"
+        - "iptables -P INPUT ACCEPT"
+        - "iptables -P FORWARD ACCEPT"
+        - "iptables -P OUTPUT ACCEPT"
+        - "touch /tmp/skydive-logs/iptables.flushed"
+
+    - name: First ensure apt cache is always refreshed
+      apt:
+        update_cache: yes
+      when:
+        - ansible_pkg_mgr == 'apt'
+
+  tasks:
+    - name: Run embedded ansible installation
+      become: yes
+      become_user: root
+      command: "./bootstrap-embedded-ansible.sh"
+      args:
+        chdir: "src/{{ current_test_repo }}/skydive"
+
+    - name: Run ansible-galaxy (tests)
+      become: yes
+      become_user: root
+      command: "${HOME}/ansible_venv/bin/ansible-galaxy install --force --ignore-errors --roles-path=${HOME}/ansible_venv/repositories/roles -r ansible-role-requirements.yml"
+      args:
+        chdir: "src/{{ current_test_repo }}/skydive/tests"
+
+    - name: Run ansible-galaxy (skydive)
+      become: yes
+      become_user: root
+      command: "${HOME}/ansible_venv/bin/ansible-galaxy install --force --ignore-errors --roles-path=${HOME}/ansible_venv/repositories/roles -r ansible-role-requirements.yml"
+      args:
+        chdir: "src/{{ current_test_repo }}/skydive"
+
+    - name: Run environment setup
+      become: yes
+      become_user: root
+      command: "${HOME}/ansible_venv/bin/ansible-playbook -i {{ inventory_file }} -e @test-vars.yml _key-setup.yml"
+      environment:
+        ANSIBLE_LOG_PATH: "/tmp/skydive-logs/ansible-skydive-test-deployment.log"
+      args:
+        chdir: "src/{{ current_test_repo }}/skydive/tests"
+
+    - name: Get the ops repo
+      git:
+        dest: "/opt/openstack-ansible-ops"
+        repo: https://git.openstack.org/openstack/openstack-ansible-ops
+        force: yes
+        track_submodules: yes
+        clone: yes
+        update: yes
+        recursive: yes
+        version: "HEAD"
+
+    - name: Run ansible-galaxy (elk_metrics_6x)
+      become: yes
+      become_user: root
+      command: "${HOME}/ansible_venv/bin/ansible-galaxy install --force --ignore-errors --roles-path=${HOME}/ansible_venv/repositories/roles -r ansible-role-requirements.yml"
+      args:
+        chdir: "/opt/openstack-ansible-ops/elk_metrics_6x"
+
+    - name: Deploy elk_metrics_6x
+      become: yes
+      become_user: root
+      command: "${HOME}/ansible_venv/bin/ansible-playbook -i tests/{{ inventory_file }} -e @tests/test-vars.yml installElastic.yml"
+      environment:
+        ANSIBLE_LOG_PATH: "/tmp/skydive-logs/ansible-skydive-test-deployment.log"
+      args:
+        chdir: "/opt/openstack-ansible-ops/elk_metrics_6x"
+
+    - name: Run functional test
+      become: yes
+      become_user: root
+      command: "${HOME}/ansible_venv/bin/ansible-playbook -i tests/{{ inventory_file }} -e @tests/test-vars.yml site.yml"
+      environment:
+        ANSIBLE_LOG_PATH: "/tmp/skydive-logs/ansible-skydive-test-deployment.log"
+      args:
+        chdir: "src/{{ current_test_repo }}/skydive"

skydive/tests/inventory/test-metal-inventory.yml

@@ -0,0 +1,51 @@
+---
+all_systems:
+  vars:
+    ansible_ssh_extra_args: >-
+      -o UserKnownHostsFile=/dev/null
+      -o StrictHostKeyChecking=no
+      -o ServerAliveInterval=64
+      -o ServerAliveCountMax=1024
+      -o Compression=no
+      -o TCPKeepAlive=yes
+      -o VerifyHostKeyDNS=no
+      -o ForwardX11=no
+      -o ForwardAgent=yes
+      -T
+    ansible_become: yes
+    ansible_become_user: "root"
+    ansible_user: "root"
+  children:
+    systems:
+      children:
+        hosts:
+          children:
+            physical_hosts:
+              hosts:
+                localhost:
+                  ansible_host: 127.0.0.1
+                  ansible_user: root
+              vars:
+                physical_host: localhost
+
+        elk_all:
+          children:
+            elastic-logstash:
+              hosts:
+                localhost: {}
+
+        traefik_all:
+          children:
+            traefik_build_nodes: {}
+
+        skydive_all:
+          children:
+            skydive_build_nodes: {}
+
+            skydive_agents:
+              hosts:
+                localhost: {}
+
+            skydive_analyzers:
+              hosts:
+                localhost: {}

skydive/tests/manual-test.rc

@@ -0,0 +1,16 @@
+export ANSIBLE_HOST_KEY_CHECKING="False"
+export ANSIBLE_ROLES_PATH="${HOME}/ansible_venv/repositories/roles"
+export ANSIBLE_ACTION_PLUGINS="${HOME}/ansible_venv/repositories/roles/config_template/action"
+export ANSIBLE_CONNECTION_PLUGINS="${HOME}/ansible_venv/repositories/roles/plugins/connection"
+export ANSIBLE_LOG_PATH="/tmp/skydive-logs/ansible-skydive-test.log"
+
+if [[ ! -d "/tmp/skydive-logs" ]]; then
+  mkdir -pv "/tmp/skydive-logs"
+  chmod 0777 "/tmp/skydive-logs"
+fi
+
+echo "To build a test environment run the following:"
+echo -e "# ${HOME}/ansible_venv/bin/ansible-playbook -i tests/inventory/test-container-inventory.yml tests/test.yml --limit localhost\n"
+
+echo "Run manual functional tests by executing the following:"
+echo -e "# ${HOME}/ansible_venv/bin/ansible-playbook -i tests/inventory/test-container-inventory.yml site.yml\n"

skydive/tests/post-run.yml

@@ -0,0 +1,27 @@
+---
+# Copyright 2018, Rackspace US, Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+- import_playbook: run-setup.yml
+
+- name: Run post tasks
+  hosts: "all"
+  tasks:
+    - name: Copy logs back to the executor
+      synchronize:
+        src: "/tmp/skydive-logs"
+        dest: "{{ zuul.executor.log_root }}/"
+        mode: pull
+        rsync_opts:
+          - "--quiet"

skydive/tests/run-cleanup.sh

@@ -0,0 +1,56 @@
+#!/usr/bin/env bash
+# Copyright 2018, Rackspace US, Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+set -e
+
+export TEST_DIR="$(readlink -f $(dirname ${0})/../../)"
+
+# Stop beat processes
+pushd "${TEST_DIR}/skydive"
+  for i in $(ls -1 install*beat.yml); do
+    LOWER_BEAT="$(echo "${i}" | tr '[:upper:]' '[:lower:]')"
+    BEAT_PARTIAL="$(echo ${LOWER_BEAT} | awk -F'.' '{print $1}')"
+    BEAT="$(echo ${BEAT_PARTIAL} | awk -F'install' '{print $2}')"
+    echo "Stopping ${BEAT}"
+    (systemctl stop "${BEAT}" || true) &
+    apt remove --purge -y "${BEAT}" || true
+    if [[ -d "/etc/${BEAT}" ]]; then
+      rm -rf "/etc/${BEAT}"
+    fi
+    if [[ -d "/var/lib/${BEAT}" ]]; then
+      rm -rf "/var/lib/${BEAT}"
+    fi
+    if [[ -d "/etc/systemd/system/${BEAT}.service.d" ]]; then
+      rm -rf "/etc/systemd/system/${BEAT}.service.d"
+    fi
+  done
+popd
+
+for i in $(grep -lri elastic /etc/apt/sources.list.d/); do
+  rm "${i}"
+done
+
+# Stop and remove containers
+for i in {1..3}; do
+  if machinectl list-images | grep -v ubuntu | awk '/sub/ {print $1}' | xargs -n 1 machinectl kill; then
+    sleep 1
+  fi
+done
+
+for i in {1..3}; do
+  if machinectl list-images | grep -v ubuntu | awk '/sub/ {print $1}' | xargs -n 1 machinectl remove; then
+    sleep 1
+  fi
+done

skydive/tests/run-setup.yml

@@ -0,0 +1,49 @@
+---
+# Copyright 2018, Rackspace US, Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+- name: Zuul facts
+  hosts: "all"
+  tasks:
+    - name: Set zuul fact
+      set_fact:
+        zuul:
+          project:
+            canonical_name: "openstack-ansible-ops"
+            short_name: "ops"
+          executor:
+            log_root: "{{ ansible_env.HOME }}/skydive-test-logs"
+      when:
+        - zuul is not defined
+
+    - name: Print zuul fact
+      debug: var=zuul
+
+    - name: Set current test repo (cross-repo)
+      set_fact:
+        current_test_repo: "git.openstack.org/{{ osa_test_repo }}"
+      when:
+        - osa_test_repo is defined
+
+    - name: Set current test repo (non-cross-repo)
+      set_fact:
+        current_test_repo: "{{ zuul.project.canonical_name }}"
+      when:
+        - osa_test_repo is not defined
+
+  post_tasks:
+    - name: Ensure the log directory exists
+      file:
+        path: "/tmp/skydive-logs"
+        state: directory

skydive/tests/run-tests.sh

@@ -0,0 +1,46 @@
+#!/usr/bin/env bash
+# Copyright 2018, Rackspace US, Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+set -ve
+
+export TEST_DIR="$(readlink -f $(dirname ${0})/../../)"
+
+pushd "${HOME}"
+  if [[ ! -d "src" ]]; then
+    mkdir src
+  fi
+  pushd src
+    ln -sf "${TEST_DIR}"
+  popd
+popd
+
+source "${TEST_DIR}/skydive/tests/manual-test.rc"
+
+source "${TEST_DIR}/skydive/bootstrap-embedded-ansible.sh"
+deactivate
+
+${HOME}/ansible_venv/bin/ansible-galaxy install --force \
+                                                --roles-path="${HOME}/ansible_venv/repositories/roles" \
+                                                --role-file="${TEST_DIR}/skydive/tests/ansible-role-requirements.yml"
+
+if [[ ! -e "${TEST_DIR}/skydive/tests/src" ]]; then
+  ln -s ${TEST_DIR}/../ ${TEST_DIR}/skydive/tests/src
+fi
+
+${HOME}/ansible_venv/bin/ansible-playbook -i 'localhost,' \
+                                          -vv \
+                                          -e ansible_connection=local \
+                                          -e test_clustered_skydive=${CLUSTERED:-no} \
+                                          ${TEST_DIR}/skydive/tests/test.yml

skydive/tests/test-vars.yml

@@ -0,0 +1,27 @@
+---
+# Copyright 2018, Rackspace US, Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+physical_host: localhost
+
+# NOTE(cloudnull): Test configs used to minimize the impact of a
+#                  multi-node install with limited resources.
+q_storage: 1
+q_mem: 512
+h_mem: 512
+
+osa_test_repo: "openstack/openstack-ansible-ops"
+
+skydive_password: "secrete"
+skydive_elasticsearch_servers: "127.0.0.1"

skydive/tests/test.yml

@@ -12,3 +12,5 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
+
+- import_playbook: functional.yml

skydive/validateSkydive.yml

@@ -0,0 +1,56 @@
+---
+# Copyright 2019, Rackspace US, Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+- name: Configure Skydive analyzers
+  hosts: skydive_analyzers[0]
+  vars:
+    skydive_username: skydive
+    skydive_analyzer_port: 8082
+    skydive_network_device: "{{ ansible_default_ipv4['interface'] | replace('-', '_') }}"
+    skydive_analyzer_uri: "{{ hostvars[inventory_hostname]['ansible_' ~ skydive_network_device]['ipv4']['address'] ~ ':' ~ skydive_analyzer_port }}"
+  tasks:
+    - name: Check API login
+      uri:
+        url: "http://{{ skydive_analyzer_uri }}/login"
+        status_code: "200"
+        method: POST
+        body: "username={{ skydive_username }}&password={{ skydive_password }}"
+        headers:
+          Content-Type: "application/x-www-form-urlencoded"
+      register: skydive_login
+      until:
+        - skydive_login is success
+      retries: 10
+      delay: 10
+
+    - name: Check API status
+      uri:
+        url: "http://{{ skydive_analyzer_uri }}/api/status"
+        method: GET
+        return_content: true
+        headers:
+          Cookie: "{{ skydive_login.set_cookie | regex_replace(',', ';') }}"
+      register: skydive_response
+      changed_when: false
+      until:
+        - skydive_response is success
+        - skydive_response.json is defined
+      retries: 10
+      delay: 5
+      run_once: true
+
+    - name: Show Skydive agents
+      debug:
+        var: "{{ skydive_response.json | to_json }}"

zuul.d/jobs.yaml

@@ -116,3 +116,34 @@
     vars:
       osa_test_repo: "openstack/openstack-ansible-ops"
       test_clustered_kolide: true
+
+- job:
+    name: "openstack-ansible-ops:skydive-ubuntu-xenial"
+    parent: base
+    description: "Runs a gate test on the skydive project."
+    run: "skydive/tests/test.yml"
+    post-run: "skydive/tests/post-run.yml"
+    voting: false
+    files:
+      - ^skydive/.*
+      - ^bootstrap-embedded-ansible/.*
+
+- job:
+    name: "openstack-ansible-ops:skydive-ubuntu-bionic"
+    parent: "openstack-ansible-ops:skydive-ubuntu-xenial"
+    nodeset: ubuntu-bionic
+
+- job:
+    name: "openstack-ansible-ops:skydive-centos-7"
+    parent: "openstack-ansible-ops:skydive-ubuntu-xenial"
+    nodeset: centos-7
+
+- job:
+    name: "openstack-ansible-ops:skydive-opensuse-423"
+    parent: "openstack-ansible-ops:skydive-ubuntu-xenial"
+    nodeset: opensuse-423
+
+# - job:
+#     name: "openstack-ansible-ops:skydive-opensuse-15"
+#     parent: "openstack-ansible-ops:skydive-ubuntu-xenial"
+#     nodeset: opensuse-15

zuul.d/project.yaml

@@ -33,8 +33,14 @@
         # - openstack-ansible-ops:osquery-ubuntu-xenial-clustered
         - openstack-ansible-ops:osquery-ubuntu-bionic
         # - openstack-ansible-ops:osquery-ubuntu-bionic-clustered
+        - openstack-ansible-ops:skydive-ubuntu-xenial
+        - openstack-ansible-ops:skydive-ubuntu-bionic
+        - openstack-ansible-ops:skydive-centos-7
+        - openstack-ansible-ops:skydive-opensuse-423
+        # - openstack-ansible-ops:skydive-opensuse-15
     gate:
       jobs:
         - openstack-ansible-linters
         - openstack-ansible-ops:elk_metrics_6x-ubuntu-bionic
         - openstack-ansible-ops:osquery-ubuntu-bionic
+        - openstack-ansible-ops:skydive-ubuntu-bionic