From 7f7c4e2a486f48006bad059fd524b4ffcfcf7222 Mon Sep 17 00:00:00 2001 From: Mark Goddard Date: Tue, 29 Sep 2020 12:09:03 +0100 Subject: [PATCH] Add documentation for routed control plane networks Kayobe currently supports definition of various different networks - public, internal, tunnel, etc. These typically map to a VLAN or flat network, with an IP subnet. When a cloud exceeds the size of a single VLAN/subnet, this approach no longer works. One way to resolve this is to have multiple subnets that map to a single logical network, and provide routing between them. This is a similar concept to neutron's routed networks, but for the control plane. This change provides documentation for the currently tested parts of this feature. Change-Id: Ic06c6d4fff0fa568eb9ed3a9c30ce21c7699d965 Story: 2008180 Task: 40938 --- doc/source/conf.py | 1 + doc/source/configuration/reference/index.rst | 1 + .../configuration/reference/kolla-ansible.rst | 2 + .../routed-control-plane-networks.rst | 332 ++++++++++++++++++ ...ntrol-plane-networks-8c1b297398b57064.yaml | 7 + 5 files changed, 343 insertions(+) create mode 100644 doc/source/configuration/reference/routed-control-plane-networks.rst create mode 100644 releasenotes/notes/routed-control-plane-networks-8c1b297398b57064.yaml diff --git a/doc/source/conf.py b/doc/source/conf.py index 22ec97df0..97764ddd4 100644 --- a/doc/source/conf.py +++ b/doc/source/conf.py @@ -106,4 +106,5 @@ openstackdocs_projects = [ 'ironic-python-agent', 'kolla', 'kolla-ansible', + 'neutron', ] diff --git a/doc/source/configuration/reference/index.rst b/doc/source/configuration/reference/index.rst index d68b71b12..e4f6d99a0 100644 --- a/doc/source/configuration/reference/index.rst +++ b/doc/source/configuration/reference/index.rst @@ -12,6 +12,7 @@ options. ansible physical-network network + routed-control-plane-networks hosts kolla kolla-ansible diff --git a/doc/source/configuration/reference/kolla-ansible.rst b/doc/source/configuration/reference/kolla-ansible.rst index 17251ada6..ef1302aec 100644 --- a/doc/source/configuration/reference/kolla-ansible.rst +++ b/doc/source/configuration/reference/kolla-ansible.rst @@ -225,6 +225,8 @@ OpenStack services. This is not usually advisable in production. --- kolla_openstack_logging_debug: true +.. _configuration-kolla-ansible-api-addresses: + API Addresses ------------- diff --git a/doc/source/configuration/reference/routed-control-plane-networks.rst b/doc/source/configuration/reference/routed-control-plane-networks.rst new file mode 100644 index 000000000..00e6ef156 --- /dev/null +++ b/doc/source/configuration/reference/routed-control-plane-networks.rst @@ -0,0 +1,332 @@ +============================= +Routed Control Plane Networks +============================= + +This section describes configuration for routed control plane networks. This is +an advanced concept and generally applies only to larger deployments that +exceed the reasonable size of a broadcast domain. + +Concept +======= + +Kayobe currently supports the definition of various different networks - +public, internal, tunnel, etc. These typically map to a VLAN or flat network, +with an associated IP subnet. When a cloud exceeds the reasonable size of a +single VLAN/subnet, or is physically distributed, this approach no longer +works. + +One way to resolve this is to have multiple subnets that map to a single +logical network, and provide routing between them. This is a similar concept to +Neutron's :neutron-doc:`routed provider networks +`, but for the control plane networks. + +Limitations +=========== + +There are currently a few limitations to using routed control plane networks. +Only the following networks have been tested: + +* ``admin_oc`` +* ``internal`` +* ``tunnel`` +* ``storage`` +* ``storage_mgmt`` + +Additionally, only compute nodes and storage nodes have been tested with routed +control plane networks - controllers were always placed on the same set of +networks during testing. + +Bare metal provisioning (of the overcloud or baremetal compute) has not been +tested with routed control plane networks, and would not be expected to work +without taking additional steps. + +Configuration +============= + +The approach to configuring Kayobe for routed control plane networks is as +follows: + +* create groups in the inventory for the different sets of networks +* place hosts in the appropriate groups +* move ``vip_address`` and ``fqdn`` network attributes to `global variables + ` +* move global network name configuration to group variables +* add new networks to configuration +* add network interface group variables + +Example +======= + +In this example, we initially have a number of different logical networks: + +* ``public_0`` + + * ``10.0.0.0/24`` + * VLAN 100 + +* ``internal_0`` + + * ``10.0.1.0/24`` + * VLAN 101 + +* ``tunnel_0`` + + * ``10.0.2.0/24`` + * VLAN 102 + +* ``storage_0`` + + * ``10.0.3.0/24`` + * VLAN 103 + +* ``storage_mgmt_0`` + + * ``10.0.4.0/24`` + * VLAN 104 + +Initially the following hosts are connected to these networks: + +* ``controllers[0:2]``: ``public_0``, ``internal_0``, ``tunnel_0``, + ``storage_0`` +* ``compute[0:127]``: ``internal_0``, ``tunnel_0``, ``storage_0`` +* ``storage[0:63]``: ``internal_0``, ``storage_0``, ``storage_mgmt_0`` + +Now consider that we wish to add 128 compute nodes and 64 storage nodes. This +would exceed size of the current subnets. We could increase the subnet sizes, +but there are good reasons to keep broadcast domains reasonably small. + +To resolve this, we can add some more networks: + +* ``internal_1`` + + * ``10.1.1.0/24`` + * VLAN 111 + +* ``tunnel_1`` + + * ``10.1.2.0/24`` + * VLAN 112 + +* ``storage_1`` + + * ``10.1.3.0/24`` + * VLAN 113 + +* ``storage_mgmt_1`` + + * ``10.1.4.0/24`` + * VLAN 114 + +The network must provide routes between the following networks: + +* ``internal_0`` and ``internal_1`` +* ``tunnel_0`` and ``tunnel_1`` +* ``storage_0`` and ``storage_1`` +* ``storage_mgmt_0`` and ``storage_mgmt_1`` + +Now we can connect the new hosts to these networks: + +* ``compute[128:255]``: ``internal_1``, ``tunnel_1``, ``storage_1`` +* ``storage[64:127]``: ``internal_1``, ``storage_1``, ``storage_mgmt_1`` + +Inventory +--------- + +To model this change we could use an inventory such as the following: + +.. code-block:: console + :caption: ``inventory/hosts`` + + localhost ansible_connection=local + + [controllers] + controller[0:2] + + [compute] + compute[0:255] + + [storage] + storage[0:127] + + [network-0] + controller[0:2] + + [compute-network-0] + compute[0:127] + + [storage-network-0] + storage[0:63] + + [network-0:children] + compute-network-0 + storage-network-0 + + [network-1] + + [compute-network-1] + compute[128:255] + + [storage-network-1] + storage[64:127] + + [network-1:children] + compute-network-1 + storage-network-1 + +Kolla API addresses +------------------- + +Remove all variables defining ``vip_address`` or ``fqdn`` network attributes +from ``networks.yml``, and move the configuration to the `API address variables +` in ``kolla.yml``. + +Network names +------------- + +To move global network name configuration to group variables, the following +variables should be commented out in ``networks.yml``: + +.. code-block:: yaml + :caption: ``networks.yml`` + + #admin_oc_net_name: + #internal_net_name: + #tunnel_net_name: + #storage_net_name: + #storage_mgmt_net_name: + +Create group variable files in ``inventory/group_vars/network-0`` and +``inventory/group_vars/network-1``: + +.. code-block:: yaml + :caption: ``inventory/group_vars/network-0`` + + admin_oc_net_name: internal_0 + internal_net_name: internal_0 + tunnel_net_name: tunnel_0 + storage_net_name: storage_0 + storage_mgmt_net_name: storage_mgmt_0 + +.. code-block:: yaml + :caption: ``inventory/group_vars/network-1`` + + admin_oc_net_name: internal_1 + internal_net_name: internal_1 + tunnel_net_name: tunnel_1 + storage_net_name: storage_1 + storage_mgmt_net_name: storage_mgmt_1 + +Networks +-------- + +Now, ensure both sets of networks are defined in ``networks.yml``. Static +routes are added between the pairs of networks here, although these will depend +on your routing configuration. Other network attributes may be necessary, we +are including ``cidr``, ``vlan`` and ``routes`` only here for brevity: + +.. code-block:: yaml + :caption: ``networks.yml`` + + public_0_cidr: 10.0.0.0/24 + public_0_vlan: 100 + + internal_0_cidr: 10.0.1.0/24 + internal_0_vlan: 101 + internal_0_routes: + - cidr: "{{ internal_1_cidr }}" + gateway: 10.0.1.1 + + internal_1_cidr: 10.1.1.0/24 + internal_1_vlan: 111 + internal_1_routes: + - cidr: "{{ internal_0_cidr }}" + gateway: 10.1.1.1 + + tunnel_0_cidr: 10.0.2.0/24 + tunnel_0_vlan: 102 + tunnel_0_routes: + - cidr: "{{ tunnel_1_cidr }}" + gateway: 10.0.2.1 + + tunnel_1_cidr: 10.1.2.0/24 + tunnel_1_vlan: 112 + tunnel_1_routes: + - cidr: "{{ tunnel_0_cidr }}" + gateway: 10.1.2.1 + + storage_0_cidr: 10.0.3.0/24 + storage_0_vlan: 103 + storage_0_routes: + - cidr: "{{ storage_1_cidr }}" + gateway: 10.0.3.1 + + storage_1_cidr: 10.1.3.0/24 + storage_1_vlan: 113 + storage_1_routes: + - cidr: "{{ storage_0_cidr }}" + gateway: 10.1.3.1 + + storage_mgmt_0_cidr: 10.0.4.0/24 + storage_mgmt_0_vlan: 104 + storage_mgmt_0_routes: + - cidr: "{{ storage_mgmt_1_cidr }}" + gateway: 10.0.4.1 + + storage_mgmt_1_cidr: 10.1.4.0/24 + storage_mgmt_1_vlan: 114 + storage_mgmt_1_routes: + - cidr: "{{ storage_mgmt_0_cidr }}" + gateway: 10.1.4.1 + +Network interfaces +------------------ + +Since there are now differently named networks, the network interface variables +are named differently. This means that we must provide a group variables file +for each set of networks and each type of host. For example: + +.. code-block:: yaml + :caption: ``inventory/group_vars/compute-network-0/network-interfaces`` + + internal_0_interface: eth0.101 + tunnel_0_interface: eth0.102 + storage_0_interface: eth0.103 + +.. code-block:: yaml + :caption: ``inventory/group_vars/compute-network-1/network-interfaces`` + + internal_1_interface: eth0.111 + tunnel_1_interface: eth0.112 + storage_1_interface: eth0.113 + +.. code-block:: yaml + :caption: ``inventory/group_vars/storage-network-0/network-interfaces`` + + internal_0_interface: eth0.101 + storage_0_interface: eth0.103 + storage_mgmt_0_interface: eth0.104 + +.. code-block:: yaml + :caption: ``inventory/group_vars/storage-network-1/network-interfaces`` + + internal_1_interface: eth0.111 + storage_1_interface: eth0.113 + storage_mgmt_1_interface: eth0.114 + +The normal interface configuration group variables files +``inventory/group_vars/compute/network-interfaces`` and +``inventory/group_vars/storage/network-interfaces`` should be +removed. + +Group variables for controller network interfaces may be placed in +``inventory/group_vars/controllers/network-interfaces`` as normal. + +Alternative approach +==================== + +There is an alternative approach which has not been tested, but may be of +interest. Rather than having differently named networks (e.g. ``internal_0`` +and ``internal_1``), it should be possible to use the same name everywhere +(e.g. ``internal``), but define the network attributes in group variables. This +approach may be a little less verbose, and allows the same group variables file +to set the network interfaces as normal (e.g. via ``internal_interface``). diff --git a/releasenotes/notes/routed-control-plane-networks-8c1b297398b57064.yaml b/releasenotes/notes/routed-control-plane-networks-8c1b297398b57064.yaml new file mode 100644 index 000000000..a780ce4fb --- /dev/null +++ b/releasenotes/notes/routed-control-plane-networks-8c1b297398b57064.yaml @@ -0,0 +1,7 @@ +--- +features: + - | + Adds initial support for routed control plane networks. This feature allows + scaling clouds beyond the limit of a single broadcast domain or subnet. See + `story 2008180 `__ for + details.