Scalable and Highly Available DHCP Agents

Hosts for Demo
Host	Description
OpenStack Controller host - controlnode	Runs the Neutron service, Keystone and all of the Nova services that are required to deploy VMs. The node must have at least one network interface, this should be connected to the "Management Network". Note nova-network should not be running since it is replaced by Neutron.
HostA	Runs Nova compute, the Neutron L2 agent and DCHP agent
HostB	Same as HostA

Configuration controlnode - Neutron Server Neutron configuration file /etc/neutron/neutron.conf: [DEFAULT] core_plugin = neutron.plugins.linuxbridge.lb_neutron_plugin.LinuxBridgePluginV2 rabbit_host = controlnode allow_overlapping_ips = True host = controlnode agent_down_time = 5 Update the plugin configuration file /etc/neutron/plugins/linuxbridge/linuxbridge_conf.ini: [vlans] tenant_network_type = vlan network_vlan_ranges = physnet1:1000:2999 [database] sql_connection = mysql://root:root@127.0.0.1:3306/neutron_linux_bridge reconnect_interval = 2 [linux_bridge] physical_interface_mappings = physnet1:eth0 HostA and HostB - L2 Agent Neutron configuration file /etc/neutron/neutron.conf: [DEFAULT] rabbit_host = controlnode rabbit_password = openstack # host = HostB on hostb host = HostA Update the plugin configuration file /etc/neutron/plugins/linuxbridge/linuxbridge_conf.ini: [vlans] tenant_network_type = vlan network_vlan_ranges = physnet1:1000:2999 [database] sql_connection = mysql://root:root@127.0.0.1:3306/neutron_linux_bridge reconnect_interval = 2 [linux_bridge] physical_interface_mappings = physnet1:eth0 Update the nova configuration file /etc/nova/nova.conf: [DEFAULT] network_api_class=nova.network.neutronv2.api.API neutron_admin_username=neutron neutron_admin_password=servicepassword neutron_admin_auth_url=http://controlnode:35357/v2.0/ neutron_auth_strategy=keystone neutron_admin_tenant_name=servicetenant neutron_url=http://100.1.1.10:9696/ firewall_driver=nova.virt.firewall.NoopFirewallDriver HostA and HostB - DHCP Agent Update the DHCP configuration file /etc/neutron/dhcp_agent.ini: [DEFAULT] interface_driver = neutron.agent.linux.interface.BridgeInterfaceDriver

Commands in agent management and scheduler extensions The following commands require the tenant running the command to have an admin role. Please ensure that the following environment variables are set. These are used by the various clients to access Keystone. export OS_USERNAME=admin export OS_PASSWORD=adminpassword export OS_TENANT_NAME=admin export OS_AUTH_URL=http://controlnode:5000/v2.0/ Settings We need some VMs and a neutron network to experiment. Here they are: $ nova list +--------------------------------------+-----------+--------+---------------+ | ID | Name | Status | Networks | +--------------------------------------+-----------+--------+---------------+ | c394fcd0-0baa-43ae-a793-201815c3e8ce | myserver1 | ACTIVE | net1=10.0.1.3 | | 2d604e05-9a6c-4ddb-9082-8a1fbdcc797d | myserver2 | ACTIVE | net1=10.0.1.4 | | c7c0481c-3db8-4d7a-a948-60ce8211d585 | myserver3 | ACTIVE | net1=10.0.1.5 | +--------------------------------------+-----------+--------+---------------+ $ neutron net-list +--------------------------------------+------+--------------------------------------+ | id | name | subnets | +--------------------------------------+------+--------------------------------------+ | 89dca1c6-c7d4-4f7a-b730-549af0fb6e34 | net1 | f6c832e3-9968-46fd-8e45-d5cf646db9d1 | +--------------------------------------+------+--------------------------------------+ Manage agents in neutron deployment Every agent which supports these extensions will register itself with the neutron server when it starts up. List all agents: $ neutron agent-list +--------------------------------------+--------------------+-------+-------+----------------+ | id | agent_type | host | alive | admin_state_up | +--------------------------------------+--------------------+-------+-------+----------------+ | 1b69828d-6a9b-4826-87cd-1757f0e27f31 | Linux bridge agent | HostA | :-) | True | | a0c1c21c-d4f4-4577-9ec7-908f2d48622d | DHCP agent | HostA | :-) | True | | ed96b856-ae0f-4d75-bb28-40a47ffd7695 | Linux bridge agent | HostB | :-) | True | | f28aa126-6edb-4ea5-a81e-8850876bc0a8 | DHCP agent | HostB | :-) | True | +--------------------------------------+--------------------+-------+-------+----------------+ Just as shown, we have four agents now, and they have reported their state. The 'alive' will be ':-)' if the agent reported its state within the period defined by the option 'agent_down_time' in neutron server's neutron.conf. Otherwise the 'alive' is 'xxx'. List the DHCP agents hosting a given network In some deployments, one DHCP agent is not enough to hold all the network data. In addition, we should have backup for it even when the deployment is small one. The same network can be assigned to more than one DHCP agent and one DHCP agent can host more than one network. Let's first go with command that lists DHCP agents hosting a given network. $ neutron dhcp-agent-list-hosting-net net1 +--------------------------------------+-------+----------------+-------+ | id | host | admin_state_up | alive | +--------------------------------------+-------+----------------+-------+ | a0c1c21c-d4f4-4577-9ec7-908f2d48622d | HostA | True | :-) | +--------------------------------------+-------+----------------+-------+ List the networks hosted by a given DHCP agent. This command is to show which networks a given dhcp agent is managing. $ neutron net-list-on-dhcp-agent a0c1c21c-d4f4-4577-9ec7-908f2d48622d +--------------------------------------+------+---------------------------------------------------+ | id | name | subnets | +--------------------------------------+------+---------------------------------------------------+ | 89dca1c6-c7d4-4f7a-b730-549af0fb6e34 | net1 | f6c832e3-9968-46fd-8e45-d5cf646db9d1 10.0.1.0/24 | +--------------------------------------+------+---------------------------------------------------+ Show the agent detail information. The agent-list command gives very general information about agents. To obtain the detailed information of an agent, we can use agent-show. $ neutron agent-show a0c1c21c-d4f4-4577-9ec7-908f2d48622d +---------------------+----------------------------------------------------------+ | Field | Value | +---------------------+----------------------------------------------------------+ | admin_state_up | True | | agent_type | DHCP agent | | alive | False | | binary | neutron-dhcp-agent | | configurations | { | | | "subnets": 1, | | | "use_namespaces": true, | | | "dhcp_driver": "neutron.agent.linux.dhcp.Dnsmasq", | | | "networks": 1, | | | "dhcp_lease_time": 120, | | | "ports": 3 | | | } | | created_at | 2013-03-16T01:16:18.000000 | | description | | | heartbeat_timestamp | 2013-03-17T01:37:22.000000 | | host | HostA | | id | 58f4ce07-6789-4bb3-aa42-ed3779db2b03 | | started_at | 2013-03-16T06:48:39.000000 | | topic | dhcp_agent | +---------------------+----------------------------------------------------------+ In the above output, 'heartbeat_timestamp' is the time on neutron server. So we don't need all agents synced to neutron server's time for this extension to run well. 'configurations' is about the agent's static configuration or run time data. We can see that this agent is a DHCP agent, and it is hosting one network, one subnet and 3 ports. Different type of agents has different detail. Below is information for a 'Linux bridge agent' $ neutron agent-show ed96b856-ae0f-4d75-bb28-40a47ffd7695 +---------------------+--------------------------------------+ | Field | Value | +---------------------+--------------------------------------+ | admin_state_up | True | | binary | neutron-linuxbridge-agent | | configurations | { | | | "physnet1": "eth0", | | | "devices": "4" | | | } | | created_at | 2013-03-16T01:49:52.000000 | | description | | | disabled | False | | group | agent | | heartbeat_timestamp | 2013-03-16T01:59:45.000000 | | host | HostB | | id | ed96b856-ae0f-4d75-bb28-40a47ffd7695 | | topic | N/A | | started_at | 2013-03-16T06:48:39.000000 | | type | Linux bridge agent | +---------------------+--------------------------------------+ Just as shown, we can see bridge-mapping and the number of VM's virtual network devices on this L2 agent. Manage assignment of networks to DHCP agent We have shown net-list-on-dhcp-agent and dhcp-agent-list-hosting-net commands. Now let's look at how to add a network to a DHCP agent and remove one from it. Default scheduling. When a network is created and one port is created on it, we will try to schedule it to an active DHCP agent. If there are many active DHCP agents, we select one randomly. (We can design more sophisticated scheduling algorithm just like we do in nova-schedule later.) $ neutron net-create net2 $ neutron subnet-create net2 9.0.1.0/24 --name subnet2 $ neutron port-create net2 $ neutron dhcp-agent-list-hosting-net net2 +--------------------------------------+-------+----------------+-------+ | id | host | admin_state_up | alive | +--------------------------------------+-------+----------------+-------+ | a0c1c21c-d4f4-4577-9ec7-908f2d48622d | HostA | True | :-) | +--------------------------------------+-------+----------------+-------+ We can see it is allocated to DHCP agent on HostA. If we want to validate the behavior via dnsmasq, don't forget to create a subnet for the network since DHCP agent starts the dnsmasq service only if there is a DHCP enabled subnet on it. Assign a network to a given DHCP agent. We have two DHCP agents, and we want another DHCP agent to host the network too. $ neutron dhcp-agent-network-add f28aa126-6edb-4ea5-a81e-8850876bc0a8 net2 Added network net2 to dhcp agent $ neutron dhcp-agent-list-hosting-net net2 +--------------------------------------+-------+----------------+-------+ | id | host | admin_state_up | alive | +--------------------------------------+-------+----------------+-------+ | a0c1c21c-d4f4-4577-9ec7-908f2d48622d | HostA | True | :-) | | f28aa126-6edb-4ea5-a81e-8850876bc0a8 | HostB | True | :-) | +--------------------------------------+-------+----------------+-------+ We can see both DHCP agents are hosting 'net2' network. Remove a network from a given DHCP agent. This command is the sibling command for the previous one. Let's remove 'net2' from HostA's DHCP agent. $ neutron dhcp-agent-network-remove a0c1c21c-d4f4-4577-9ec7-908f2d48622d net2 Removed network net2 to dhcp agent $ neutron dhcp-agent-list-hosting-net net2 +--------------------------------------+-------+----------------+-------+ | id | host | admin_state_up | alive | +--------------------------------------+-------+----------------+-------+ | f28aa126-6edb-4ea5-a81e-8850876bc0a8 | HostB | True | :-) | +--------------------------------------+-------+----------------+-------+ We can see now only HostB's DHCP agent is hosting 'net2' network. HA of DHCP agents First we will boot a VM on net2, then we let both DHCP agents host 'net2'. After that, we fail the agents in turn and to see if the VM can still get the wanted IP during that time. Boot a VM on net2. $ neutron net-list +--------------------------------------+------+--------------------------------------------------+ | id | name | subnets | +--------------------------------------+------+--------------------------------------------------+ | 89dca1c6-c7d4-4f7a-b730-549af0fb6e34 | net1 | f6c832e3-9968-46fd-8e45-d5cf646db9d1 10.0.1.0/24| | 9b96b14f-71b8-4918-90aa-c5d705606b1a | net2 | 6979b71a-0ae8-448c-aa87-65f68eedcaaa 9.0.1.0/24 | +--------------------------------------+------+--------------------------------------------------+ $ nova boot --image tty --flavor 1 myserver4 \ --nic net-id=9b96b14f-71b8-4918-90aa-c5d705606b1a $ nova list +--------------------------------------+-----------+--------+---------------+ | ID | Name | Status | Networks | +--------------------------------------+-----------+--------+---------------+ | c394fcd0-0baa-43ae-a793-201815c3e8ce | myserver1 | ACTIVE | net1=10.0.1.3 | | 2d604e05-9a6c-4ddb-9082-8a1fbdcc797d | myserver2 | ACTIVE | net1=10.0.1.4 | | c7c0481c-3db8-4d7a-a948-60ce8211d585 | myserver3 | ACTIVE | net1=10.0.1.5 | | f62f4731-5591-46b1-9d74-f0c901de567f | myserver4 | ACTIVE | net2=9.0.1.2 | +--------------------------------------+-----------+--------+---------------+ Make sure both DHCP agents hosting 'net2'. We can use commands shown before to assign the network to agents. $ neutron dhcp-agent-list-hosting-net net2 +--------------------------------------+-------+----------------+-------+ | id | host | admin_state_up | alive | +--------------------------------------+-------+----------------+-------+ | a0c1c21c-d4f4-4577-9ec7-908f2d48622d | HostA | True | :-) | | f28aa126-6edb-4ea5-a81e-8850876bc0a8 | HostB | True | :-) | +--------------------------------------+-------+----------------+-------+ To test the HA Log in to the 'myserver4' VM, and run 'udhcpc', 'dhclient' or other DHCP client. Stop the DHCP agent on HostA (Beside stopping the neutron-dhcp-agent binary, we must make sure dnsmasq processes are gone too.) Run a DHCP client in VM. We can see it can get the wanted IP. Stop the DHCP agent on HostB too. Run 'udhcpc' in VM. We can see it cannot get the wanted IP. Start DHCP agent on HostB. We can see VM can get the wanted IP again. Disable and remove an agent An admin user wants to disable an agent if there is a system upgrade planned, whatever hardware or software. Some agents which support scheduling support disable or enable too, such as L3 agent and DHCP agent. Once the agent is disabled, the scheduler will not schedule new resources to the agent. After the agent is disabled, we can remove the agent safely. We should remove the resources on the agent before we delete the agent itself. To run the commands below, we need first stop the DHCP agent on HostA. $ neutron agent-update --admin-state-up False a0c1c21c-d4f4-4577-9ec7-908f2d48622d $ neutron agent-list +--------------------------------------+--------------------+-------+-------+----------------+ | id | agent_type | host | alive | admin_state_up | +--------------------------------------+--------------------+-------+-------+----------------+ | 1b69828d-6a9b-4826-87cd-1757f0e27f31 | Linux bridge agent | HostA | :-) | True | | a0c1c21c-d4f4-4577-9ec7-908f2d48622d | DHCP agent | HostA | :-) | False | | ed96b856-ae0f-4d75-bb28-40a47ffd7695 | Linux bridge agent | HostB | :-) | True | | f28aa126-6edb-4ea5-a81e-8850876bc0a8 | DHCP agent | HostB | :-) | True | +--------------------------------------+--------------------+-------+-------+----------------+ $ neutron agent-delete a0c1c21c-d4f4-4577-9ec7-908f2d48622d Deleted agent: a0c1c21c-d4f4-4577-9ec7-908f2d48622d $ neutron agent-list +--------------------------------------+--------------------+-------+-------+----------------+ | id | agent_type | host | alive | admin_state_up | +--------------------------------------+--------------------+-------+-------+----------------+ | 1b69828d-6a9b-4826-87cd-1757f0e27f31 | Linux bridge agent | HostA | :-) | True | | ed96b856-ae0f-4d75-bb28-40a47ffd7695 | Linux bridge agent | HostB | :-) | True | | f28aa126-6edb-4ea5-a81e-8850876bc0a8 | DHCP agent | HostB | :-) | True | +--------------------------------------+--------------------+-------+-------+----------------+ After deletion, if we restart the DHCP agent, it will be on agent list again.