performa/shaker
Code Issues Proposed changes
Files
66d1bc035dd024169da9bb44706fe0259081d886
shaker/doc/source/catalog.rst
Oded Le'Sage 5a25825b73 Enhance Iperf executor to accept "unmapped" args 
Issue: The current iperf executor is setup nicely to handle the
majority, let's say 90%, of the test cases a user might want to run.
However, it still leaves a bit to be desired for the remaining 10%. For
example we use a custom qcow2 image for our tests  and the verison of
iperf in this image (and the latest iperf which shaker doesn't have) you
can specify --sctp as an argument to run this type of traffic. This is
not possible with the current executor, but most users may not want or
care about this ability.

This commit proposes to extend the current executor to accept any
argument as part of an "advanced" mode, rather than mapping every
argument that may or may not used or exists only in certain versions.
This allows different version of iperf (e.g. --sctp capable) of
executing a supported argument without having to directly add support
for this new argument into the executor

I know that any argument could be passed to iperf using the script
executor, but this has the downside of not providing the nice graphs.
I believe this commit is a good compromise between general
useability (pre-mapping arguments) and granular control (using any
agrument that is not mapped). This will also be useful for other
arguments that may not be widely used, or are newly introduced, such
as --zerocopy or --omit along with other sctp related argument like
--xbind or --nstreams.

This would be for advanced users familiar with iperf and would not
impact current tests or general use. The only "safety net" provided is
not allowing mutually exclusive arguments with time to be specified with
the time argument. Since time is currently added by default, even if we
were to create as mapping for say --blockcount(-k), we would have to
handle removing the time argument for a correct run. Arguments that
don't mix will lead to an iperf error, but this error is correctly
logged and handled. Overriding time if a mutually exclusive argument is
used seems like a fair way to handle the situation while allowing other
possible argument combinations.

Change-Id: I072278d0ab3912aac375605dc36987a7f8c20461
2018-11-07 16:10:57 -06:00

28 KiB
Raw Blame History

Scenario Catalog

Scenarios

OpenStack instances metadata query

In this scenario Shaker launches ten instances on a single compute node and asks instances to retrieve the metadata. The scenario is used to load metadata processes.

To use this scenario specify parameter --scenario misc/instance_metadata. Scenario source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/misc/instance_metadata.yaml

OpenStack L2 Cross-AZ

In this scenario Shaker launches pairs of instances in the same tenant network. Every instance is hosted on a separate compute node, all available compute nodes are utilized. The master and slave instances are in different availability zones. The scenario is used to test throughput between nova and vcenter zones. The traffic goes within the tenant network (L2 domain).

To use this scenario specify parameter --scenario openstack/cross_az/full_l2. Scenario source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/openstack/cross_az/full_l2.yaml

OpenStack L3 East-West Cross-AZ

In this scenario Shaker launches pairs of instances, each instance on its own compute node. All available compute nodes are utilized. Instances are connected to one of 2 tenant networks, which plugged into single router. The traffic goes from one network to the other (L3 east-west). The master and slave instances are in different availability zones. The scenario is used to test throughput between nova and vcenter zones.

To use this scenario specify parameter --scenario openstack/cross_az/full_l3_east_west. Scenario source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/openstack/cross_az/full_l3_east_west.yaml

OpenStack L3 North-South Cross-AZ

In this scenario Shaker launches pairs of instances on different compute nodes. All available compute nodes are utilized. Instances are in different networks connected to different routers, master accesses slave by floating ip. The traffic goes from one network via external network to the other network. The master and slave instances are in different availability zones. The scenario is used to test throughput between nova and vcenter zones.

To use this scenario specify parameter --scenario openstack/cross_az/full_l3_north_south. Scenario source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/openstack/cross_az/full_l3_north_south.yaml

OpenStack L2 Cross-AZ Performance

In this scenario Shaker launches 1 pair of instances in the same tenant network. Each instance is hosted on a separate compute node. The master and slave instances are in different availability zones. The scenario is used to test throughput between nova and vcenter zones.

To use this scenario specify parameter --scenario openstack/cross_az/perf_l2. Scenario source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/openstack/cross_az/perf_l2.yaml

OpenStack L3 East-West Cross-AZ Performance

In this scenario Shaker launches 1 pair of instances, each instance on its own compute node. Instances are connected to one of 2 tenant networks, which plugged into single router. The traffic goes from one network to the other (L3 east-west). The master and slave instances are in different availability zones. The scenario is used to test throughput between nova and vcenter zones.

To use this scenario specify parameter --scenario openstack/cross_az/perf_l3_east_west. Scenario source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/openstack/cross_az/perf_l3_east_west.yaml

OpenStack L3 North-South Cross-AZ Performance

In this scenario Shaker launches 1 pair of instances on different compute nodes. Instances are in different networks connected to different routers, master accesses slave by floating ip. The traffic goes from one network via external network to the other network. The master and slave instances are in different availability zones. The scenario is used to test throughput between nova and vcenter zones.

To use this scenario specify parameter --scenario openstack/cross_az/perf_l3_north_south. Scenario source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/openstack/cross_az/perf_l3_north_south.yaml

OpenStack L2 Cross-AZ UDP

In this scenario Shaker launches pairs of instances in the same tenant network. Every instance is hosted on a separate compute node. The load is generated by UDP traffic. The master and slave instances are in different availability zones. The scenario is used to test throughput between nova and vcenter zones.

To use this scenario specify parameter --scenario openstack/cross_az/udp_l2. Scenario source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/openstack/cross_az/udp_l2.yaml

OpenStack L2 Cross-AZ UDP Jumbo

In this scenario Shaker launches pairs of instances in the same tenant network. Every instance is hosted on a separate compute node. The load is generated by UDP traffic and jumbo packets. The master and slave instances are in different availability zones. The scenario is used to test throughput between nova and vcenter zones.

To use this scenario specify parameter --scenario openstack/cross_az/udp_l2_mss8950. Scenario source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/openstack/cross_az/udp_l2_mss8950.yaml

OpenStack L3 East-West Cross-AZ UDP

In this scenario Shaker launches pairs of instances, each instance on its own compute node. Instances are connected to one of 2 tenant networks, which plugged into single router. The traffic goes from one network to the other (L3 east-west). The load is generated by UDP traffic. The master and slave instances are in different availability zones. The scenario is used to test throughput between nova and vcenter zones.

To use this scenario specify parameter --scenario openstack/cross_az/udp_l3_east_west. Scenario source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/openstack/cross_az/udp_l3_east_west.yaml

OpenStack L2 Dense

In this scenario Shaker launches several pairs of instances on a single compute node. Instances are plugged into the same tenant network. The traffic goes within the tenant network (L2 domain).

To use this scenario specify parameter --scenario openstack/dense_l2. Scenario source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/openstack/dense_l2.yaml

OpenStack L3 East-West Dense

In this scenario Shaker launches pairs of instances on the same compute node. Instances are connected to different tenant networks connected to one router. The traffic goes from one network to the other (L3 east-west).

To use this scenario specify parameter --scenario openstack/dense_l3_east_west. Scenario source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/openstack/dense_l3_east_west.yaml

OpenStack L3 North-South Dense

In this scenario Shaker launches pairs of instances on the same compute node. Instances are connected to different tenant networks, each connected to own router. Instances in one of networks have floating IPs. The traffic goes from one network via external network to the other network.

To use this scenario specify parameter --scenario openstack/dense_l3_north_south. Scenario source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/openstack/dense_l3_north_south.yaml

OpenStack L3 North-South Dense to external target

In this scenario Shaker launches instances on one compute node in a tenant network connected to external network. The traffic is sent to and from external host. The host name needs to be provided as command-line parameter, e.g. --matrix "{host: 172.10.1.2}".

To use this scenario specify parameter --scenario openstack/external/dense_l3_north_south_no_fip. Scenario source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/openstack/external/dense_l3_north_south_no_fip.yaml

OpenStack L3 North-South Dense to external target with floating IP

In this scenario Shaker launches instances on one compute node in a tenant network connected to external network. All instances have floating IPs. The traffic is sent to and from external host. The host name needs to be provided as command-line parameter, e.g. --matrix "{host: 172.10.1.2}".

To use this scenario specify parameter --scenario openstack/external/dense_l3_north_south_with_fip. Scenario source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/openstack/external/dense_l3_north_south_with_fip.yaml

OpenStack L3 North-South to external target

In this scenario Shaker launches instances in a tenant network connected to external network. Every instance is hosted on dedicated compute node. All available compute nodes are utilized. The traffic is sent to and from external host (L3 north-south). The host name needs to be provided as command-line parameter, e.g. --matrix "{host: 172.10.1.2}".

To use this scenario specify parameter --scenario openstack/external/full_l3_north_south_no_fip. Scenario source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/openstack/external/full_l3_north_south_no_fip.yaml

OpenStack L3 North-South to external target with floating IP

In this scenario Shaker launches instances in a tenant network connected to external network. Every instance is hosted on dedicated compute node. All available compute nodes are utilized. All instances have floating IPs. The traffic is sent to and from external host (L3 north-south). The host name needs to be provided as command-line parameter, e.g. --matrix "{host: 172.10.1.2}".

To use this scenario specify parameter --scenario openstack/external/full_l3_north_south_with_fip. Scenario source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/openstack/external/full_l3_north_south_with_fip.yaml

OpenStack L3 North-South Performance to external target

In this scenario Shaker launches instance in a tenant network connected to external network. The traffic is sent to and from external host. By default one of public iperf3 servers is used, to override this the target host can be provided as command-line parameter, e.g. --matrix "{host: 172.10.1.2}".

To use this scenario specify parameter --scenario openstack/external/perf_l3_north_south_no_fip. Scenario source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/openstack/external/perf_l3_north_south_no_fip.yaml

OpenStack L3 North-South performance to external target with floating IP

In this scenario Shaker launches instance in a tenant network connected to external network. The instance has floating IP. The traffic is sent to and from external host. By default one of public iperf3 servers is used, to override this the target host can be provided as command-line parameter, e.g. --matrix "{host: 172.10.1.2}".

To use this scenario specify parameter --scenario openstack/external/perf_l3_north_south_with_fip. Scenario source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/openstack/external/perf_l3_north_south_with_fip.yaml

OpenStack L2

In this scenario Shaker launches pairs of instances in the same tenant network. Every instance is hosted on a separate compute node, all available compute nodes are utilized. The traffic goes within the tenant network (L2 domain).

To use this scenario specify parameter --scenario openstack/full_l2. Scenario source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/openstack/full_l2.yaml

OpenStack L3 East-West

In this scenario Shaker launches pairs of instances, each instance on its own compute node. All available compute nodes are utilized. Instances are connected to one of 2 tenant networks, which plugged into single router. The traffic goes from one network to the other (L3 east-west).

To use this scenario specify parameter --scenario openstack/full_l3_east_west. Scenario source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/openstack/full_l3_east_west.yaml

OpenStack L3 North-South

In this scenario Shaker launches pairs of instances on different compute nodes. All available compute nodes are utilized. Instances are in different networks connected to different routers, master accesses slave by floating ip. The traffic goes from one network via external network to the other network.

To use this scenario specify parameter --scenario openstack/full_l3_north_south. Scenario source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/openstack/full_l3_north_south.yaml

OpenStack L2 Performance

In this scenario Shaker launches 1 pair of instances in the same tenant network. Each instance is hosted on a separate compute node. The traffic goes within the tenant network (L2 domain).

To use this scenario specify parameter --scenario openstack/perf_l2. Scenario source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/openstack/perf_l2.yaml

OpenStack L3 East-West Performance

In this scenario Shaker launches 1 pair of instances, each instance on its own compute node. Instances are connected to one of 2 tenant networks, which plugged into single router. The traffic goes from one network to the other (L3 east-west).

To use this scenario specify parameter --scenario openstack/perf_l3_east_west. Scenario source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/openstack/perf_l3_east_west.yaml

OpenStack L3 North-South Performance

In this scenario Shaker launches 1 pair of instances on different compute nodes. Instances are in different networks connected to different routers, master accesses slave by floating ip. The traffic goes from one network via external network to the other network.

To use this scenario specify parameter --scenario openstack/perf_l3_north_south. Scenario source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/openstack/perf_l3_north_south.yaml

OpenStack L2 QoS Performance

In this scenario Shaker launches 1 pair of instances in the same tenant network. Each instance is hosted on a separate compute node. The traffic goes within the tenant network (L2 domain). Neutron QoS feature is used to limit traffic throughput to 10 Mbit/s.

To use this scenario specify parameter --scenario openstack/qos/perf_l2. Scenario source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/openstack/qos/perf_l2.yaml

OpenStack L2 UDP

In this scenario Shaker launches pairs of instances in the same tenant network. Every instance is hosted on a separate compute node. The traffic goes within the tenant network (L2 domain). The load is generated by UDP traffic.

To use this scenario specify parameter --scenario openstack/udp_l2. Scenario source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/openstack/udp_l2.yaml

OpenStack L3 East-West UDP

In this scenario Shaker launches pairs of instances, each instance on its own compute node. Instances are connected to one of 2 tenant networks, which plugged into single router. The traffic goes from one network to the other (L3 east-west). The load is generated by UDP traffic.

To use this scenario specify parameter --scenario openstack/udp_l3_east_west. Scenario source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/openstack/udp_l3_east_west.yaml

OpenStack L3 North-South UDP

In this scenario Shaker launches pairs of instances on different compute nodes. Instances are in different networks connected to different routers, master accesses slave by floating ip. The traffic goes from one network via external network to the other network. The load is generated by UDP traffic.

To use this scenario specify parameter --scenario openstack/udp_l3_north_south. Scenario source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/openstack/udp_l3_north_south.yaml

Ping

This scenario uses ping to measure the latency between the local host and the remote. The remote host can be provided via command-line, it defaults to 8.8.8.8. The scenario verifies SLA and expects the latency to be at most 30ms. The destination host can be overridden by command-line parameter, e.g. --matrix "{host: 172.10.1.2}".

To use this scenario specify parameter --scenario spot/ping. Scenario source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/spot/ping.yaml

TCP bandwidth

This scenario uses iperf3 to measure TCP throughput between local host and ping.online.net (or against hosts provided via CLI). SLA check is verified and expects the speed to be at least 90Mbit and at most 20 retransmitts. The destination host can be overridden by command-line parameter, e.g. --matrix "{host: 172.10.1.2}".

To use this scenario specify parameter --scenario spot/tcp. Scenario source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/spot/tcp.yaml

UDP bandwidth

This scenario uses iperf3 to measure UDP throughput between local host and ping.online.net (or against hosts provided via CLI). SLA check is verified and requires at least 10 000 packets per second. The destination host can be overridden by command-line parameter, e.g. --matrix "{host: 172.10.1.2}".

To use this scenario specify parameter --scenario spot/udp. Scenario source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/spot/udp.yaml

Sample TCP Test with Advanced Iperf Arguments

This test definition demonstrates the use of advanced arguments with iperf. In this scenario Shaker launches pairs of instances in the same tenant network. Every instance is hosted on a separate compute node, 1 compute node is utilized. The traffic goes within the tenant network (L2 domain) and uses arguments not directly mapped by the iperf executor.

To use this scenario specify parameter --scenario test/sample_with_advanced_iperf. Scenario source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/test/sample_with_advanced_iperf.yaml

Sample TCP Test with Environment File

This test definition demonstrates the use of an environment file. In this scenario Shaker launches pairs of instances in the same tenant network. Every instance is hosted on a separate compute node, 1 compute node is utilized. The traffic goes within the tenant network (L2 domain)

To use this scenario specify parameter --scenario test/sample_with_env. Scenario source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/test/sample_with_env.yaml

Sample TCP Test with Support Stacks

This test definition demonstrates the use of support stacks In this scenario Shaker launches pairs of instances in the same tenant network. Each test VM is also connected to a previously launched support network. The support neworks are part of their own support heat stack. Every instance is hosted on a separate compute node, 1 compute node is utilized. The traffic goes within the tenant network (L2 domain)

To use this scenario specify parameter --scenario test/sample_with_support_stacks. Scenario source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/test/sample_with_support_stacks.yaml

Static agents

In this scenario Shaker runs tests in spot mode. The scenario can be used for Shaker integration testing.

To use this scenario specify parameter --scenario test/spot. Scenario source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/test/spot.yaml

Static agents

In this scenario Shaker runs tests on pre-deployed static agents. The scenario can be used for Shaker integration testing.

To use this scenario specify parameter --scenario test/static_agent. Scenario source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/test/static_agent.yaml

Paired static agents

In this scenario Shaker runs tests on pre-deployed pair of static agents. The scenario can be used for Shaker integration testing.

To use this scenario specify parameter --scenario test/static_agents_pair. Scenario source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/test/static_agents_pair.yaml

Heat Templates

misc/instance_metadata

Heat template creates a new Neutron network, a router to the external network, plugs instances into this network and assigns floating ips

Template source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/misc/instance_metadata.hot

openstack/cross_az/l2

This Heat template creates a new Neutron network, a router to the external network and plugs instances into this new network. All instances are located in the same L2 domain.

Template source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/openstack/cross_az/l2.hot

openstack/cross_az/l3_east_west

This Heat template creates a pair of networks plugged into the same router. Master instances and slave instances are connected into different networks.

Template source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/openstack/cross_az/l3_east_west.hot

openstack/cross_az/l3_north_south

This Heat template creates a new Neutron network plus a north_router to the external network. The template also assigns floating IP addresses to each instance so they are routable from the external network.

Template source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/openstack/cross_az/l3_north_south.hot

openstack/external/l3_north_south_no_fip

This Heat template creates a new Neutron network plugged into a router connected to the external network, and boots an instance in that network.

Template source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/openstack/external/l3_north_south_no_fip.hot

openstack/external/l3_north_south_with_fip

This Heat template creates a new Neutron network plugged into a router connected to the external network, and boots an instance in that network. The instance has floating IP.

Template source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/openstack/external/l3_north_south_with_fip.hot

openstack/l2

This Heat template creates a new Neutron network, a router to the external network and plugs instances into this new network. All instances are located in the same L2 domain.

Template source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/openstack/l2.hot

openstack/l3_east_west

This Heat template creates a pair of networks plugged into the same router. Master instances and slave instances are connected into different networks.

Template source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/openstack/l3_east_west.hot

openstack/l3_north_south

This Heat template creates a new Neutron network plus a north_router to the external network. The template also assigns floating IP addresses to each instance so they are routable from the external network.

Template source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/openstack/l3_north_south.hot

openstack/qos/l2_qos

This Heat template creates a new Neutron network, a router to the external network and plugs instances into this new network. All instances are located in the same L2 domain.

Template source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/openstack/qos/l2_qos.hot

test/l2_with_env

This Heat template creates a new Neutron network, a router to the external network and plugs instances into this new network. All instances are located in the same L2 domain.

Template source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/test/l2_with_env.hot

test/templates/l2_with_support

This Heat template creates a new Neutron network, a router to the external network and plugs instances into this new network. All instances are located in the same L2 domain. The VMs are also connected to support networks that should exist before this template is spun up.

Template source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/test/templates/l2_with_support.hot

test/templates/support_network

This Heat template creates a new Neutron network. This is used to demonstrate a support stack in Shaker.

Template source is available at: https://github.com/openstack/shaker/blob/master/shaker/scenarios/test/templates/support_network.hot