xenapi - cleanup network configuration

The Xenserver/XCP part of devstack was configuring the hypervisor's
connectivity to the outer world, by adding VLANs, and physical
interfaces to the virtual networks. It added a lot of complexity, and
made it hard to get started with XenServer. This patch removes that
extra complexity, so it is left as an exercise for the user.

Related to blueprint blueprint xenapi-devstack-cleanup

Change-Id: If3367335c3da8621d0afe1f6cae77511fbdbb3e2

tools/xen/README.md

@@ -8,6 +8,34 @@ The Openstack services are configured to run within a "privileged" virtual
 machine on the Xenserver host (called OS domU). The VM uses the XAPI toolstack
 to communicate with the host.
 
+The provided localrc helps to build a basic environment.
+The requirements are:
+ - An internet-enabled network with a DHCP server on it
+ - XenServer box plugged in to the same network
+This network will be used as the OpenStack management network. The VM Network
+and the Public Network will not be connected to any physical interfaces, only
+new virtual networks will be created by the `install_os_domU.sh` script.
+
+Steps to follow:
+ - Install XenServer
+ - Download Devstack to XenServer
+ - Customise `localrc`
+ - Start `install_os_domU.sh` script
+
+The `install_os_domU.sh` script will:
+ - Setup XenAPI plugins
+ - Create the named networks, if they don't exist
+ - Install an Ubuntu Virtual Machine, with 4 network interfaces:
+   - eth0 - internal xapi interface
+   - eth1 - VM interface, connected to `VM_BRIDGE_OR_NET_NAME` defaults to
+   `"OpenStack VM Network"`.
+   - eth2 - Management interface, connected to `MGT_BRIDGE_OR_NET_NAME`,
+     defaults to `xenbr0`, XenServer's bridge associated with the Hypervisors
+     `eth0`.
+   - eth3 - Public interface, connected to `PUB_BRIDGE_OR_NET_NAME` defaults to
+   `"OpenStack Public Network"`.
+ - Start devstack inside the created OpenStack VM
+
 ## Step 1: Install Xenserver
 Install XenServer 5.6+ on a clean box. You can get XenServer by signing
 up for an account on citrix.com, and then visiting:
@@ -15,13 +43,12 @@ https://www.citrix.com/English/ss/downloads/details.asp?downloadId=2311504&produ
 
 For details on installation, see: http://wiki.openstack.org/XenServer/Install
 
-Here are some sample Xenserver network settings for when you are just
+The XenServer IP configuration depends on your local network setup. If you are
-getting started (Settings like this have been used with a laptop + cheap wifi router):
+using dhcp, make a reservation for XenServer, so its IP address won't change
-
+over time. Make a note of the XenServer's IP address, as it has to be specified
-* XenServer Host IP: 192.168.1.10
+in `localrc`. The other option is to manually specify the IP setup for the
-* XenServer Netmask: 255.255.255.0
+XenServer box. Please make sure, that a gateway and a nameserver is configured,
-* XenServer Gateway: 192.168.1.1
+as `install_os_domU.sh` will connect to github.com to get source-code snapshots.
-* XenServer DNS: 192.168.1.1
 
 ## Step 2: Download devstack
 On your XenServer host, run the following commands as root:
@@ -32,45 +59,63 @@ On your XenServer host, run the following commands as root:
 
 ## Step 3: Configure your localrc inside the devstack directory
 Devstack uses a localrc for user-specific configuration.  Note that
-the XENAPI_PASSWORD must be your dom0 root password.
+the `XENAPI_PASSWORD` must be your dom0 root password.
 Of course, use real passwords if this machine is exposed.
 
     cat > ./localrc <<EOF
+    # Passwords
+    # NOTE: these need to be specified, otherwise devstack will try
+    # to prompt for these passwords, blocking the install process.
+
     MYSQL_PASSWORD=my_super_secret
     SERVICE_TOKEN=my_super_secret
     ADMIN_PASSWORD=my_super_secret
     SERVICE_PASSWORD=my_super_secret
     RABBIT_PASSWORD=my_super_secret
     SWIFT_HASH="66a3d6b56c1f479c8b4e70ab5c2000f5"
-    # This is the password for the OpenStack VM (for both stack and root users)
+    # This will be the password for the OpenStack VM (both stack and root users)
     GUEST_PASSWORD=my_super_secret
 
     # XenAPI parameters
-    # IMPORTANT: The following must be set to your dom0 root password!
+    # NOTE: The following must be set to your XenServer root password!
+
     XENAPI_PASSWORD=my_xenserver_root_password
+
     XENAPI_CONNECTION_URL="http://address_of_your_xenserver"
     VNCSERVER_PROXYCLIENT_ADDRESS=address_of_your_xenserver
 
-    # Do not download the usual images yet!
+    # Do not download the usual images
     IMAGE_URLS=""
     # Explicitly set virt driver here
     VIRT_DRIVER=xenserver
-    # Explicitly set multi-host
+    # Explicitly enable multi-host
     MULTI_HOST=1
     # Give extra time for boot
     ACTIVE_TIMEOUT=45
-    # Host Interface, i.e. the interface on the nova vm you want to expose the
+    # Host Interface, i.e. the interface on the OpenStack vm you want to expose
-    # services on. Usually eth2 (management network) or eth3 (public network) and
+    # the services on. The default is eth3, which means the public network, but
-    # not eth0 (private network with XenServer host) or eth1 (VM traffic network)
+    # as the public network is going to be virtual, we are setting the services
-    # The default is eth3.
+    # to listen on the management network, which defaults to 'xenbr0', the
-    # HOST_IP_IFACE=eth3
+    # XenServer's network.
+    HOST_IP_IFACE=eth2
+
+    # Use DHCP server to configure the Management IP of OpenStack VM
+    MGT_IP="dhcp"
 
     # Settings for netinstalling Ubuntu
-    # UBUNTU_INST_RELEASE=precise
+    UBUNTU_INST_RELEASE=precise
 
-    # First time Ubuntu network install params
+    # First time Ubuntu network install params, use the DHCP server on the
-    # UBUNTU_INST_IFACE="eth3"
+    # management network
-    # UBUNTU_INST_IP="dhcp"
+    UBUNTU_INST_IFACE="eth2"
+    UBUNTU_INST_IP="dhcp"
+
+    # NOTE: the value of FLAT_NETWORK_BRIDGE will automatically be determined
+    # by install_os_domU.sh script.
+
+    # Public IP address is aligned with the devstack defaults (see FLOATING_RANGE)
+    PUB_IP=172.24.4.10
+    PUB_NETMASK=255.255.255.0
     EOF
 
 ## Step 4: Run `./install_os_domU.sh` from the `tools/xen` directory

tools/xen/functions

@@ -93,3 +93,84 @@ function find_ip_by_name() {
         fi
     done
 }
+
+function _create_new_network() {
+    local name_label
+    name_label=$1
+
+    xe network-create name-label="$name_label"
+}
+
+function _multiple_networks_with_name() {
+    local name_label
+    name_label=$1
+
+    # A comma indicates multiple matches
+    xe network-list name-label="$name_label" --minimal | grep -q ","
+}
+
+function _network_exists() {
+    local name_label
+    name_label=$1
+
+    ! [ -z $(xe network-list name-label="$name_label" --minimal) ]
+}
+
+function _bridge_exists() {
+    local bridge
+    bridge=$1
+
+    ! [ -z $(xe network-list bridge="$bridge" --minimal) ]
+}
+
+
+function setup_network() {
+    local bridge_or_net_name
+    bridge_or_net_name=$1
+
+    if ! _bridge_exists "$bridge_or_net_name"; then
+        if _network_exists "$bridge_or_net_name"; then
+            if _multiple_networks_with_name "$bridge_or_net_name"; then
+                cat >&2 << EOF
+ERROR: Multiple networks found matching name-label to "$bridge_or_net_name"
+please review your XenServer network configuration / localrc file.
+EOF
+                exit 1
+            fi
+        else
+            _create_new_network "$bridge_or_net_name"
+        fi
+    fi
+}
+
+function bridge_for() {
+    local bridge_or_net_name
+    bridge_or_net_name=$1
+
+    if _bridge_exists "$bridge_or_net_name"; then
+        echo "$bridge_or_net_name"
+    else
+        xe network-list name-label="$bridge_or_net_name" params=bridge --minimal
+    fi
+}
+
+function xenapi_ip_on() {
+    local bridge_or_net_name
+    bridge_or_net_name=$1
+
+    ifconfig $(bridge_for "$bridge_or_net_name") | grep "inet addr" | cut -d ":" -f2 | sed "s/ .*//"
+}
+
+function xenapi_is_listening_on() {
+    local bridge_or_net_name
+    bridge_or_net_name=$1
+
+    ! [ -z $(xenapi_ip_on "$bridge_or_net_name") ]
+}
+
+function parameter_is_specified() {
+    local parameter_name
+    parameter_name=$1
+
+    compgen -v | grep "$parameter_name"
+}

tools/xen/install_os_domU.sh

@@ -67,97 +67,28 @@ create_directory_for_kernels
 #
 # Configure Networking
 #
+setup_network "$VM_BRIDGE_OR_NET_NAME"
+setup_network "$MGT_BRIDGE_OR_NET_NAME"
+setup_network "$PUB_BRIDGE_OR_NET_NAME"
 
-# Helper to create networks
+if parameter_is_specified "FLAT_NETWORK_BRIDGE"; then
-# Uses echo trickery to return network uuid
+    cat >&2 << EOF
-function create_network() {
+ERROR: FLAT_NETWORK_BRIDGE is specified in localrc file
-    br=$1
+This is considered as an error, as its value will be derived from the
-    dev=$2
+VM_BRIDGE_OR_NET_NAME variable's value.
-    vlan=$3
+EOF
-    netname=$4
+    exit 1
-    if [ -z $br ]
-    then
-        pif=$(xe_min pif-list device=$dev VLAN=$vlan)
-        if [ -z $pif ]
-        then
-            net=$(xe network-create name-label=$netname)
-        else
-            net=$(xe_min network-list  PIF-uuids=$pif)
-        fi
-        echo $net
-        return 0
-    fi
-    if [ ! $(xe_min network-list  params=bridge | grep -w --only-matching $br) ]
-    then
-        echo "Specified bridge $br does not exist"
-        echo "If you wish to use defaults, please keep the bridge name empty"
-        exit 1
-    else
-        net=$(xe_min network-list  bridge=$br)
-        echo $net
-    fi
-}
-
-function errorcheck() {
-    rc=$?
-    if [ $rc -ne 0 ]
-    then
-        exit $rc
-    fi
-}
-
-# Create host, vm, mgmt, pub networks on XenServer
-VM_NET=$(create_network "$VM_BR" "$VM_DEV" "$VM_VLAN" "vmbr")
-errorcheck
-MGT_NET=$(create_network "$MGT_BR" "$MGT_DEV" "$MGT_VLAN" "mgtbr")
-errorcheck
-PUB_NET=$(create_network "$PUB_BR" "$PUB_DEV" "$PUB_VLAN" "pubbr")
-errorcheck
-
-# Helper to create vlans
-function create_vlan() {
-    dev=$1
-    vlan=$2
-    net=$3
-    # VLAN -1 refers to no VLAN (physical network)
-    if [ $vlan -eq -1 ]
-    then
-        return
-    fi
-    if [ -z $(xe_min vlan-list  tag=$vlan) ]
-    then
-        pif=$(xe_min pif-list  network-uuid=$net)
-        # We created a brand new network this time
-        if [ -z $pif ]
-        then
-            pif=$(xe_min pif-list  device=$dev VLAN=-1)
-            xe vlan-create pif-uuid=$pif vlan=$vlan network-uuid=$net
-        else
-            echo "VLAN does not exist but PIF attached to this network"
-            echo "How did we reach here?"
-            exit 1
-        fi
-    fi
-}
-
-# Create vlans for vm and management
-create_vlan $PUB_DEV $PUB_VLAN $PUB_NET
-create_vlan $VM_DEV $VM_VLAN $VM_NET
-create_vlan $MGT_DEV $MGT_VLAN $MGT_NET
-
-# Get final bridge names
-if [ -z $VM_BR ]; then
-    VM_BR=$(xe_min network-list  uuid=$VM_NET params=bridge)
-fi
-if [ -z $MGT_BR ]; then
-    MGT_BR=$(xe_min network-list  uuid=$MGT_NET params=bridge)
-fi
-if [ -z $PUB_BR ]; then
-    PUB_BR=$(xe_min network-list  uuid=$PUB_NET params=bridge)
 fi
 
-# dom0 ip, XenAPI is assumed to be listening
+if ! xenapi_is_listening_on "$MGT_BRIDGE_OR_NET_NAME"; then
-HOST_IP=${HOST_IP:-`ifconfig xenbr0 | grep "inet addr" | cut -d ":" -f2 | sed "s/ .*//"`}
+    cat >&2 << EOF
+ERROR: XenAPI does not have an assigned IP address on the management network.
+please review your XenServer network configuration / localrc file.
+EOF
+    exit 1
+fi
+
+HOST_IP=$(xenapi_ip_on "$MGT_BRIDGE_OR_NET_NAME")
 
 # Set up ip forwarding, but skip on xcp-xapi
 if [ -a /etc/sysconfig/network ]; then
@@ -263,7 +194,15 @@ if [ -z "$templateuuid" ]; then
 
     # create a new VM with the given template
     # creating the correct VIFs and metadata
-    $THIS_DIR/scripts/install-os-vpx.sh -t "$UBUNTU_INST_TEMPLATE_NAME" -v $VM_BR -m $MGT_BR -p $PUB_BR -l $GUEST_NAME -r $OSDOMU_MEM_MB -k "flat_network_bridge=${VM_BR}"
+    FLAT_NETWORK_BRIDGE=$(bridge_for "$VM_BRIDGE_OR_NET_NAME")
+    $THIS_DIR/scripts/install-os-vpx.sh \
+        -t "$UBUNTU_INST_TEMPLATE_NAME" \
+        -v "$VM_BRIDGE_OR_NET_NAME" \
+        -m "$MGT_BRIDGE_OR_NET_NAME" \
+        -p "$PUB_BRIDGE_OR_NET_NAME" \
+        -l "$GUEST_NAME" \
+        -r "$OSDOMU_MEM_MB" \
+        -k "flat_network_bridge=${FLAT_NETWORK_BRIDGE}"
 
     # wait for install to finish
     wait_for_VM_to_halt

tools/xen/scripts/install-os-vpx.sh

@@ -235,7 +235,7 @@ create_gi_vif()
 create_vm_vif()
 {
   local v="$1"
-  echo "Installing management interface on $BRIDGE_V."
+  echo "Installing VM interface on $BRIDGE_V."
   local out_network_uuid=$(find_network "$BRIDGE_V")
   create_vif "$v" "$out_network_uuid" "1" >/dev/null
 }

tools/xen/xenrc

@@ -16,6 +16,16 @@ VDI_MB=${VDI_MB:-5000}
 OSDOMU_MEM_MB=1024
 OSDOMU_VDI_GB=8
 
+# Network mapping. Specify bridge names or network names. Network names may
+# differ across localised versions of XenServer. If a given bridge/network
+# was not found, a new network will be created with the specified name.
+
+# The management network is specified by the bridge name. xenbr0 is usually
+# the name of the bridge of the network associated with the hypervisor's eth0.
+MGT_BRIDGE_OR_NET_NAME="xenbr0"
+VM_BRIDGE_OR_NET_NAME="OpenStack VM Network"
+PUB_BRIDGE_OR_NET_NAME="OpenStack Public Network"
+
 # VM Password
 GUEST_PASSWORD=${GUEST_PASSWORD:-secrete}
 
@@ -28,28 +38,17 @@ HOST_IP_IFACE=${HOST_IP_IFACE:-eth3}
 # Our nova host's network info
 #
 
-# A host-only ip that let's the interface come up, otherwise unused
+# Management network
-VM_IP=${VM_IP:-10.255.255.255}
 MGT_IP=${MGT_IP:-172.16.100.55}
-PUB_IP=${PUB_IP:-192.168.1.55}
+MGT_NETMASK=${MGT_NETMASK:-255.255.255.0}
+
+# VM Network
+VM_IP=${VM_IP:-10.255.255.255}
+VM_NETMASK=${VM_NETMASK:-255.255.255.0}
 
 # Public network
+PUB_IP=${PUB_IP:-192.168.1.55}
 PUB_NETMASK=${PUB_NETMASK:-255.255.255.0}
-PUB_BR=${PUB_BR:-"xenbr0"}
-PUB_VLAN=${PUB_VLAN:--1}
-PUB_DEV=${PUB_DEV:-eth0}
-
-# VM network params
-VM_NETMASK=${VM_NETMASK:-255.255.255.0}
-VM_BR=${VM_BR:-""}
-VM_VLAN=${VM_VLAN:-100}
-VM_DEV=${VM_DEV:-eth0}
-
-# MGMT network params
-MGT_NETMASK=${MGT_NETMASK:-255.255.255.0}
-MGT_BR=${MGT_BR:-""}
-MGT_VLAN=${MGT_VLAN:-101}
-MGT_DEV=${MGT_DEV:-eth0}
 
 # Decide if you should enable eth0,
 # the guest installer network