openstack/openstack-ansible-ops
Code Issues Proposed changes
ede1af70b9
openstack-ansible-ops/bowling_ball/README.rst
Nolan Brubaker 0ebc0430d7 Add rolling downtime simulation tools 
These tools help simulate rolling downtime by taking down service
containers in a controlled fashion. While this can be done manually,
using a script to inspect the inventory for targets is much easier.
Having the rolling downtime happen automatically is also a bit less
error prone.

Tests for things like response times will be placed in the tests
directory, allowing for scripts to be written against different services
to try different things (like adding resources while rolling downtime
happens).

There is no central orchestration for the two components currently, in
order to keep things simple. Restarting containers and running the tests
is therefore best accomplished in a pair of tmux sessions.

Change-Id: I2d8e3484bbb1a71d2cd0c4124f824d49f76e5c2c
2017-03-09 16:29:13 -05:00

1.9 KiB
Raw Blame History

Bowling Ball - OpenStack-Ansible Rolling Downtime Simulator

date

2017-03-09

tags

rackspace, openstack, ansible

category

*openstack, *nix

About

This project aims to test for issues with rolling downtime on OpenStack-Ansible deployments. It's comprised of two main components:

  • The rolling_restart.py script
  • The tests directory

The rolling_restart.py script will stop containers from a specified group in a rolling fashion - node 1 will stop, then start, then node 2, then node 3 and so on. This script runs from the deployment host.

The tests directory contains scripts to generate traffic against the target services. These vary per service, but attempt to apply usage to a system that will be restarted by rolling_restart.py in order to measure the effects. These scripts run from a utility container.

Usage

  1. Start your test script from the utility container. keystone.py will request a session and a list of projects on an infinite loop, for example.

  2. From the deployment node, run rolling_restart.py in the playbooks directory (necessary to find the inventory script). Specify the service you're targeting with the -s parameter.

    rolling_restart.py -s keystone_container

    You can specify a wait time in seconds between stopping and starting individual nodes.

    rolling_restart.py -s keystone_container -w 60

Assumptions

These tools are currently coupled to OSA, and they assume paths to files as specified by the multi-node-aio scripts.

Container stopping and starting is done with an ansible command, and the physical host to target is derivced from the current inventory.

rolling_restart.py must currently be run from the playbooks directory. This will be fixed later.

You must source openrc before running keystone.py.

Why the name?

It sets 'em up and knocks em down.