abd6661b4e
This change implements namespaced variables and conditionals in needed services. This will ensure systems running these playbooks are able to be deployed in isolation without making osa specific assumptions. Change-Id: Ia20b8514144f0b0bf925d405f06ef2ddc28f1003 Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
240 lines
7.3 KiB
ReStructuredText
240 lines
7.3 KiB
ReStructuredText
Install OSQuery and Kolide fleet
|
|
################################
|
|
:tags: openstack, ansible
|
|
|
|
About this repository
|
|
---------------------
|
|
|
|
This set of playbooks will deploy osquery and kolide-fleet. If this is being
|
|
deployed as part of an OpenStack all of the inventory needs will be provided for.
|
|
|
|
|
|
**These playbooks require Ansible 2.4+.**
|
|
|
|
Highlevel overview of Osquery & Kolide Fleet infrastructure these playbooks will
|
|
build and operate against.
|
|
|
|
.. image:: assets/overview-osquery.png
|
|
:scale: 50 %
|
|
:alt: Osquery & Kolide Fleet Architecture Diagram
|
|
:align: center
|
|
|
|
|
|
OpenStack-Ansible Integration
|
|
-----------------------------
|
|
|
|
These playbooks can be used as standalone inventory or as an integrated part of
|
|
an OpenStack-Ansible deployment. For a simple example of standalone inventory
|
|
see ``inventory.example.yml``.
|
|
|
|
|
|
Setup | system configuration
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
|
|
Clone the osquery-osa repo
|
|
|
|
.. code-block:: bash
|
|
|
|
cd /opt
|
|
git clone https://github.com/openstack/openstack-ansible-ops
|
|
|
|
|
|
Copy the env.d file into place
|
|
|
|
.. code-block:: bash
|
|
|
|
cd /opt/openstack-ansible-ops/osquery
|
|
cp env.d/fleet.yml /etc/openstack_deploy/env.d/
|
|
|
|
|
|
Copy the conf.d file into place
|
|
|
|
.. code-block:: bash
|
|
|
|
cp conf.d/fleet.yml /etc/openstack_deploy/conf.d/
|
|
|
|
In **fleet.yml**, list your logging hosts under fleet-logstash_hosts to create
|
|
the kolide fleet cluster in multiple containers and one logging host under
|
|
`fleet_hosts` to create the fleet container
|
|
|
|
.. code-block:: bash
|
|
|
|
vi /etc/openstack_deploy/conf.d/fleet.yml
|
|
|
|
|
|
Create the containers
|
|
|
|
.. code-block:: bash
|
|
|
|
cd /opt/openstack-ansible/playbooks
|
|
openstack-ansible lxc-containers-create.yml --limit fleet_all
|
|
|
|
|
|
Update the `/etc/hosts` file *(optional)*
|
|
|
|
.. code-block:: bash
|
|
|
|
cd /opt/openstack-ansible/playbooks
|
|
openstack-ansible openstack-hosts-setup.yml
|
|
|
|
|
|
Create an haproxy entry for kolide-fleet service 8443
|
|
|
|
Add the following configuration item to the `haproxy_extra_services` variable
|
|
within a **user** defined variable file.
|
|
|
|
.. code-block:: yaml
|
|
|
|
haproxy_extra_services:
|
|
- service:
|
|
haproxy_service_name: kolide-fleet
|
|
haproxy_ssl: False
|
|
haproxy_backend_nodes: "{{ groups['kolide-fleet_all'] | default([]) }}"
|
|
haproxy_port: 6443 # This is set using the "kolide_fleet_port" variable
|
|
haproxy_check_port: 443 # This is set using the "kolide_fleet_port" variable
|
|
haproxy_backend_port: 443 # This is set using the "kolide_fleet_port" variable
|
|
haproxy_balance_type: tcp
|
|
|
|
|
|
With the appropriate haproxy configuration in place, setup haproxy to begin
|
|
load balancing the traffic.
|
|
|
|
.. code-block:: bash
|
|
|
|
cd /opt/openstack-ansible/playbooks/
|
|
openstack-ansible haproxy-install.yml
|
|
|
|
|
|
Deploying | Installing with embedded Ansible
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
|
|
If this is being executed on a system that already has Ansible installed but is
|
|
incompatible with these playbooks the script `bootstrap-embedded-ansible.sh` can
|
|
be sourced to grab an embedded version of Ansible prior to executing the
|
|
playbooks.
|
|
|
|
.. code-block:: bash
|
|
|
|
source bootstrap-embedded-ansible.sh
|
|
|
|
|
|
Deploying | Manually resolving the dependencies
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
|
|
This playbook has external role dependencies. If Ansible is not installed with
|
|
the `bootstrap-ansible.sh` script these dependencies can be resolved with the
|
|
``ansible-galaxy`` command and the ``ansible-role-requirements.yml`` file.
|
|
|
|
* Example galaxy execution
|
|
|
|
.. code-block:: bash
|
|
|
|
ansible-galaxy install -r ansible-role-requirements.yml --roles-path=~/ansible_venv/repositories/roles
|
|
|
|
|
|
In the even that some of the modules are alread installed execute the following
|
|
|
|
.. code-block:: bash
|
|
|
|
ansible-galaxy install -r ansible-role-requirements.yml --ignore-errors --roles-path=~/ansible_venv/repositories/roles
|
|
|
|
|
|
Once the dependencies are set make sure to set the action plugin path to the
|
|
location of the config_template action directory. This can be done using the
|
|
environment variable `ANSIBLE_ACTION_PLUGINS` or through the use of an
|
|
`ansible.cfg` file.
|
|
|
|
|
|
Deploying | The environment
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
|
|
Create some basic passwords keys that are needed by fleet
|
|
|
|
.. code-block:: bash
|
|
|
|
echo "kolide_fleet_db_password: $(openssl rand -base64 16)" >> /etc/openstack_deploy/user_secrets.yml
|
|
echo "kolide_fleet_jwt_key: $(openssl rand -base64 32)" >> /etc/openstack_deploy/user_secrets.yml
|
|
echo "kolide_fleet_admin_password: $(openssl rand -base64 16)" >> /etc/openstack_deploy/user_secrets.yml
|
|
# NOTICE: This may already be defined
|
|
echo "kolide_galera_root_password: $(openssl rand -base64 16)" >> /etc/openstack_deploy/user_secrets.yml
|
|
|
|
|
|
Install master/data Fleet nodes on the elastic-logstash containers,
|
|
deploy logstash, deploy Kibana, and then deploy all of the service beats.
|
|
|
|
.. code-block:: bash
|
|
|
|
cd /opt/openstack-ansible-ops/osquery
|
|
ansible-playbook site.yml -e@/etc/openstack_deploy/user_secrets.yml
|
|
|
|
|
|
If the `installOSquery.yml` playbook is executed with a limit, a single
|
|
kolide-fleet host must be part of the limit. This requirement exists because
|
|
the nodes running osquery require certificates to authenticate to the
|
|
kolide-fleet cluster. Should a node within the kolide-fleet cluster not be
|
|
part of the limit the playbooks will not be able to fetch the required
|
|
certificates.
|
|
|
|
.. code-block:: bash
|
|
|
|
ansible-playbook installOSquery.yml $USER_VARS --limit 'host1,host2,kolide-fleet_all[0]'
|
|
|
|
|
|
* The `openstack-ansible` command can be used if the version of ansible on the
|
|
system is greater than **2.5**. This will automatically pick up the necessary
|
|
group_vars for hosts in an OSA deployment.
|
|
|
|
* If required add ``-e@/opt/openstack-ansible/inventory/group_vars/all/all.yml``
|
|
to import sufficient OSA group variables to define the OpenStack release.
|
|
|
|
* Alternatively if using the embedded ansible, create a symlink to include all
|
|
of the OSA group_vars. These are not available by default with the embedded
|
|
ansible and can be symlinked into the ops repo.
|
|
|
|
.. code-block:: bash
|
|
|
|
ln -s /opt/openstack-ansible/inventory/group_vars /opt/openstack-ansible-ops/osquery/group_vars
|
|
|
|
|
|
The individual playbooks found within this repository can be independently run
|
|
at anytime.
|
|
|
|
|
|
Local testing
|
|
-------------
|
|
|
|
To test these playbooks within a local environment you will need a single server
|
|
with at leasts 8GiB of RAM and 40GiB of storage on root. Running an `m1.medium`
|
|
(openstack) flavor size is generally enough to get an environment online.
|
|
|
|
To run the local functional tests execute the `run-tests.sh` script out of the
|
|
tests directory. This will create a single node kolide-fleet cluster and install
|
|
osquery on the local host.
|
|
|
|
.. code-block:: bash
|
|
|
|
CLUSTERED=yes tests/run-tests.sh
|
|
|
|
|
|
To rerun the playbooks after a test build, source the `tests/manual-test.rc`
|
|
file and follow the onscreen instructions.
|
|
|
|
To clean-up a test environment and start from a bare server slate the
|
|
`run-cleanup.sh` script can be used. This script is disruptive and will purge
|
|
all `osquery` related services within the local test environment.
|
|
|
|
.. code-block:: bash
|
|
|
|
tests/run-cleanup.sh
|
|
|
|
|
|
Architecture | Data flow
|
|
^^^^^^^^^^^^^^^^^^^^^^^^
|
|
|
|
This diagram outlines the data flow from within an osquery deployment.
|
|
|
|
.. image:: assets/architecture-osquery.png
|
|
:scale: 50 %
|
|
:alt: Kolide & Osquery Data Flow Diagram
|
|
:align: center
|