=========
utilities
=========

This file serves as documentation for the components and features included
on the utilities repository.

PCI IRQ Affinity Agent
----------------------

While in OpenStack it is possible to enable instances to use PCI devices,
the interrupts generated by these devices may be handled by host CPUs that
are unrelated to the instance, and this can lead to a performance that is
lower than it could be if the device interrupts were handled by the instance
CPUs.

The agent only acts over instances with dedicated vCPUs. For instances using
shared vCPUs no action will be taken by the agent.

The expected outcome from the agent operation is achieving a higher
performance by assigning the instances core to handle the interrupts from
PCI devices used by these instances and avoid interrupts consuming excessive
cycles from the platform cores.

Agent operation
~~~~~~~~~~~~~~~

The agent operates by listening to RabbitMQ notifications from Nova. When
an instance is created or moved to the host, the agent checks for an specific
flavor spec (detailed below) and if it does then it queries libvirt to map
the instance vCPUs into pCPUs from the host.

Once the agent has the CPU mapping, it determines the IRQ for each PCI
device used by the instance, and then it loops over all PCI devices and
determines which host NUMA node is associated with the device, the pCPUs
that are associated with the NUMA node and finally set the CPU affinity
for the IRQs of the PCI device based on the pCPU list.

There is also a periodic audit that runs every minute and loops over the
existing IRQs, so that if there are new IRQs that weren't mapped before
the agent maps them, and if there are PCI devices that aren't associated
to an instance that they were before, their IRQ affinity is reset to the
default value.

Flavor spec
~~~~~~~~~~~

The PCI IRQ Affinity Agent uses a specific flavor spec for PCI interrupt
affining, that is used to determine which vCPUs assigned to the instance
must handle the interrupts from the PCI devices:

- ``hw:pci_irq_affinity_mask=<vcpus_cpulist>``

Where ``vcpus_cpulist`` can assume a comma-separated list of values that
can be expressed as:

- ``int``: the vCPU expressed by ``int`` will be assigned to handle the
  interruptions from the PCI devices
- ``int1-int2``: the vCPUs between ``int1`` and ``int2`` (inclusive) will
  be used to handle the interruptions from the PCI devices
- ``^int``: the vCPU expressed by ``int`` will not be assigned to handle the
  interruptions from the PCI devices and shall be used to exclude a vCPU
  that was included in a previous range

**NOTE**: ``int`` must be a value between ``0`` and ``flavor.vcpus - 1``

Example: ``hw_pci_irq_affinity_mask=1-4,^3,6`` means that vCPUs with indexes
``1,2,4 and 6`` from the vCPU list that Nova allocates to the instance will
be assigned to handle interruptions from the PCI devices.

Limitations
~~~~~~~~~~~

- No CPU affining is performed for instances using shared CPUs (i.e.,
  when using flavor spec ``hw:cpu_policy=shared``)
- No CPU affining will be performed when invalid ranges are specified on
  the flavor spec, the agent instead will log error messages indicating
  the problem

Agent packaging
~~~~~~~~~~~~~~~

The agent code resides on the ``starlingx/utilities`` repo, along with the
spec and docker_image files that are used to build a CentOS image with the
agent wheel installed on it.

The agent is deployed by Armada along with the other OpenStack helm charts;
refer to `PCI IRQ Affinity Agent helm chart`_ on
``starlingx/openstack-armada-app`` repository.

.. _`PCI IRQ Affinity Agent helm chart`: https://opendev.org/starlingx
   /openstack-armada-app/src/branch/master/stx-openstack-helm
   /stx-openstack-helm/helm-charts/pci-irq-affinity-agent