The change fix multi-node doc to use SERVICE_HOST. It resolves duplicate IP address. Closes-Bug: #1485159 Change-Id: If86393e9a37bcb911a9aa125829cd8ce684edd9f
11 KiB
Multi-Node Lab
Here is OpenStack in a realistic test configuration with multiple physical servers.
Prerequisites Linux & Network
Minimal Install
You need to have a system with a fresh install of Linux. You can download the Minimal CD for Ubuntu releases since DevStack will download & install all the additional dependencies. The netinstall ISO is available for Fedora and CentOS/RHEL.
Install a couple of packages to bootstrap configuration:
apt-get install -y git sudo || yum install -y git sudo
Network Configuration
The first iteration of the lab uses OpenStack's FlatDHCP network controller so only a single network will be required. It should be on its own subnet without DHCP; the host IPs and floating IP pool(s) will come out of this block. This example uses the following:
- Gateway: 192.168.42.1
- Physical nodes: 192.168.42.11-192.168.42.99
- Floating IPs: 192.168.42.128-192.168.42.254
Configure each node with a static IP. For Ubuntu edit
/etc/network/interfaces
:
auto eth0
iface eth0 inet static
address 192.168.42.11
netmask 255.255.255.0
gateway 192.168.42.1
For Fedora and CentOS/RHEL edit
/etc/sysconfig/network-scripts/ifcfg-eth0
:
BOOTPROTO=static
IPADDR=192.168.42.11
NETMASK=255.255.255.0
GATEWAY=192.168.42.1
Installation shake and bake
Add the DevStack User
OpenStack runs as a non-root user that has sudo access to root. There
is nothing special about the name, we'll use stack
here.
Every node must use the same name and preferably uid. If you created a
user during the OS install you can use it and give it sudo privileges
below. Otherwise create the stack user:
groupadd stack
useradd -g stack -s /bin/bash -d /opt/stack -m stack
This user will be making many changes to your system during installation and operation so it needs to have sudo privileges to root without a password:
echo "stack ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers
From here on use the stack
user. Logout
and login as the stack
user.
Set Up Ssh
Set up the stack user on each node with an ssh key for access:
mkdir ~/.ssh; chmod 700 ~/.ssh
echo "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyYjfgyPazTvGpd8OaAvtU2utL8W6gWC4JdRS1J95GhNNfQd657yO6s1AH5KYQWktcE6FO/xNUC2reEXSGC7ezy+sGO1kj9Limv5vrvNHvF1+wts0Cmyx61D2nQw35/Qz8BvpdJANL7VwP/cFI/p3yhvx2lsnjFE3hN8xRB2LtLUopUSVdBwACOVUmH2G+2BWMJDjVINd2DPqRIA4Zhy09KJ3O1Joabr0XpQL0yt/I9x8BVHdAx6l9U0tMg9dj5+tAjZvMAFfye3PJcYwwsfJoFxC8w/SLtqlFX7Ehw++8RtvomvuipLdmWCy+T9hIkl+gHYE4cS3OIqXH7f49jdJf jesse@spacey.local" > ~/.ssh/authorized_keys
Download DevStack
Grab the latest version of DevStack:
git clone https://git.openstack.org/openstack-dev/devstack
cd devstack
Up to this point all of the steps apply to each node in the cluster. From here on there are some differences between the cluster controller (aka 'head node') and the compute nodes.
Configure Cluster Controller
The cluster controller runs all OpenStack services. Configure the
cluster controller's DevStack in local.conf
:
[[local|localrc]]
HOST_IP=192.168.42.11
FLAT_INTERFACE=eth0
FIXED_RANGE=10.4.128.0/20
FIXED_NETWORK_SIZE=4096
FLOATING_RANGE=192.168.42.128/25
MULTI_HOST=1
LOGFILE=/opt/stack/logs/stack.sh.log
ADMIN_PASSWORD=labstack
MYSQL_PASSWORD=supersecret
RABBIT_PASSWORD=supersecrete
SERVICE_PASSWORD=supersecrete
SERVICE_TOKEN=xyzpdqlazydog
In the multi-node configuration the first 10 or so IPs in the private
subnet are usually reserved. Add this to local.sh
to have
it run after every stack.sh
run:
for i in `seq 2 10`; do /opt/stack/nova/bin/nova-manage fixed reserve 10.4.128.$i; done
Fire up OpenStack:
./stack.sh
A stream of activity ensues. When complete you will see a summary of
stack.sh
's work, including the relevant URLs, accounts and
passwords to poke at your shiny new OpenStack. The most recent log file
is available in stack.sh.log
.
Configure Compute Nodes
The compute nodes only run the OpenStack worker services. For
additional machines, create a local.conf
with:
[[local|localrc]]
HOST_IP=192.168.42.12 # change this per compute node
FLAT_INTERFACE=eth0
FIXED_RANGE=10.4.128.0/20
FIXED_NETWORK_SIZE=4096
FLOATING_RANGE=192.168.42.128/25
MULTI_HOST=1
LOGFILE=/opt/stack/logs/stack.sh.log
ADMIN_PASSWORD=labstack
MYSQL_PASSWORD=supersecret
RABBIT_PASSWORD=supersecrete
SERVICE_PASSWORD=supersecrete
SERVICE_TOKEN=xyzpdqlazydog
DATABASE_TYPE=mysql
SERVICE_HOST=192.168.42.11
MYSQL_HOST=$SERVICE_HOST
RABBIT_HOST=$SERVICE_HOST
GLANCE_HOSTPORT=$SERVICE_HOST:9292
ENABLED_SERVICES=n-cpu,n-net,n-api-meta,c-vol
NOVA_VNC_ENABLED=True
NOVNCPROXY_URL="http://$SERVICE_HOST:6080/vnc_auto.html"
VNCSERVER_LISTEN=$HOST_IP
VNCSERVER_PROXYCLIENT_ADDRESS=$VNCSERVER_LISTEN
Note: the n-api-meta
service is a
version of the api server that only serves the metadata service. It's
needed because the computes created won't have a routing path to the
metadata service on the controller.
Fire up OpenStack:
./stack.sh
A stream of activity ensues. When complete you will see a summary of
stack.sh
's work, including the relevant URLs, accounts and
passwords to poke at your shiny new OpenStack. The most recent log file
is available in stack.sh.log
.
Cleaning Up After DevStack
Shutting down OpenStack is now as simple as running the included
unstack.sh
script:
./unstack.sh
A more aggressive cleanup can be performed using
clean.sh
. It removes certain troublesome packages and
attempts to leave the system in a state where changing the database or
queue manager can be reliably performed.
./clean.sh
Sometimes running instances are not cleaned up. DevStack attempts to do this when it runs but there are times it needs to still be done by hand:
sudo rm -rf /etc/libvirt/qemu/inst*
sudo virsh list | grep inst | awk '{print $1}' | xargs -n1 virsh destroy
Options pimp your stack
Additional Users
DevStack creates two OpenStack users (admin
and
demo
) and two projects (also admin
and
demo
). admin
is exactly what it sounds like, a
privileged administrative account that is a member of both the
admin
and demo
projects. demo
is
a normal user account that is only a member of the demo
project. Creating additional OpenStack users can be done through the
dashboard, sometimes it is easier to do them in bulk from a script,
especially since they get blown away every time stack.sh
runs. The following steps are ripe for scripting:
# Get admin creds
. openrc admin admin
# List existing projects
openstack project list
# List existing users
openstack user list
# Add a user and project
NAME=bob
PASSWORD=BigSecrete
PROJECT=$NAME
openstack project create $PROJECT
openstack user create $NAME --password=$PASSWORD --project $PROJECT
openstack role add Member --user $NAME --project $PROJECT
# The Member role is created by stack.sh
# openstack role list
Swift
Swift, OpenStack Object Storage, requires a significant amount of
resources and is disabled by default in DevStack. The support in
DevStack is geared toward a minimal installation but can be used for
testing. To implement a true multi-node test of swift, additional steps
will be required. Enabling it is as simple as enabling the
swift
service in local.conf
:
enable_service s-proxy s-object s-container s-account
Swift, OpenStack Object Storage, will put its data files in
SWIFT_DATA_DIR
(default
/opt/stack/data/swift
). The size of the data 'partition'
created (really a loop-mounted file) is set by
SWIFT_LOOPBACK_DISK_SIZE
. The Swift config files are
located in SWIFT_CONF_DIR
(default
/etc/swift
). All of these settings can be overridden in
(wait for it...) local.conf
.
Volumes
DevStack will automatically use an existing LVM volume group named
stack-volumes
to store cloud-created volumes. If
stack-volumes
doesn't exist, DevStack will set up a 10Gb
loop-mounted file to contain it. This obviously limits the number and
size of volumes that can be created inside OpenStack. The size can be
overridden by setting VOLUME_BACKING_FILE_SIZE
in
local.conf
.
stack-volumes
can be pre-created on any physical volume
supported by Linux's LVM. The name of the volume group can be changed by
setting VOLUME_GROUP
in localrc
.
stack.sh
deletes all logical volumes in
VOLUME_GROUP
that begin with
VOLUME_NAME_PREFIX
as part of cleaning up from previous
runs. It is recommended to not use the root volume group as
VOLUME_GROUP
.
The details of creating the volume group depends on the server hardware involved but looks something like this:
pvcreate /dev/sdc
vgcreate stack-volumes /dev/sdc
Syslog
DevStack is capable of using rsyslog
to aggregate
logging across the cluster. It is off by default; to turn it on set
SYSLOG=True
in local.conf
.
SYSLOG_HOST
defaults to HOST_IP
; on the
compute nodes it must be set to the IP of the cluster controller to send
syslog output there. In the example above, add this to the compute node
local.conf
:
SYSLOG_HOST=192.168.42.11
Using Alternate Repositories/Branches
The git repositories for all of the OpenStack services are defined in
stackrc
. Since this file is a part of the DevStack package
changes to it will probably be overwritten as updates are applied. Every
setting in stackrc
can be redefined in
local.conf
.
To change the repository or branch that a particular OpenStack
service is created from, simply change the value of *_REPO
or *_BRANCH
corresponding to that service.
After making changes to the repository or branch, if
RECLONE
is not set in localrc
it may be
necessary to remove the corresponding directory from
/opt/stack
to force git to re-clone the repository.
For example, to pull nova, OpenStack Compute, from a proposed release candidate in the primary nova repository:
NOVA_BRANCH=rc-proposed
To pull glance, OpenStack Image service, from an experimental fork:
GLANCE_BRANCH=try-something-big
GLANCE_REPO=https://github.com/mcuser/glance.git
Notes stuff you might need to know
Reset the Bridge
How to reset the bridge configuration:
sudo brctl delif br100 eth0.926
sudo ip link set dev br100 down
sudo brctl delbr br100
Set MySQL Password
If you forgot to set the root password you can do this:
mysqladmin -u root -pnova password 'supersecret'