openstack/swift
Code Issues Proposed changes

Tweak Swift AIO docs, update CentOS and Fedora

Browse Source 
The Swift AIO documentation is out of date for CentOS and Fedora.

This patch updates the documentation to ensure that the instructions are
clear and accurate.

It also cleans up a few other sections along the way and adds some new
headings to make it easier to read. Some explanatory notes are added,
such as the need for XFS storage and test directories. XFS tmp loopback
device is moved out of the user's home directory to /srv to match
storage.

Change-Id: Ieb0341536b7149c99139a1cf620828eba25f4bc6
Signed-off-by: Chris Smart <chris.smart@humanservices.gov.au>
This commit is contained in:
Chris Smart 2019-11-02 20:48:47 +11:00
parent dd235b6ac0
commit a505d02b37
1 changed files with 196 additions and 122 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

316
doc/source/development_saio.rst
View File

@ -16,10 +16,11 @@ This section documents setting up a virtual machine for doing Swift
development. The virtual machine will emulate running a four node Swift development. The virtual machine will emulate running a four node Swift
cluster. To begin: cluster. To begin:
* Get a linux system server image, this guide will cover: * Get a Linux system server image, this guide will cover:
* Ubuntu 14.04, 16.04 LTS * Ubuntu 14.04, 16.04 LTS
* Fedora/CentOS * CentOS 7
* Fedora
* OpenSuse * OpenSuse
- Create guest virtual machine from the image. - Create guest virtual machine from the image.
@ -38,6 +39,14 @@ is ``swift``, which may not exist on your system. These instructions are
intended to allow a developer to use his/her username for intended to allow a developer to use his/her username for
``<your-user-name>:<your-group-name>``. ``<your-user-name>:<your-group-name>``.
.. note::
For OpenSuse users, a user's primary group is ``users``, so you have 2 options:
* Change ``${USER}:${USER}`` to ``${USER}:users`` in all references of this guide; or
* Create a group for your username and add yourself to it::
sudo groupadd ${USER} && sudo gpasswd -a ${USER} ${USER} && newgrp ${USER}
----------------------- -----------------------
Installing dependencies Installing dependencies
----------------------- -----------------------
@ -54,9 +63,12 @@ Installing dependencies
python-netifaces python-pip python-dnspython \ python-netifaces python-pip python-dnspython \
python-mock python-mock
* On ``yum`` based systems:: * On ``CentOS`` (requires additional repositories)::
sudo yum update sudo yum update
sudo yum install epel-release
sudo yum-config-manager --enable epel extras
sudo yum install centos-release-openstack-train
sudo yum install curl gcc memcached rsync sqlite xfsprogs git-core \ sudo yum install curl gcc memcached rsync sqlite xfsprogs git-core \
libffi-devel xinetd liberasurecode-devel \ libffi-devel xinetd liberasurecode-devel \
openssl-devel python-setuptools \ openssl-devel python-setuptools \
@ -66,6 +78,18 @@ Installing dependencies
python-netifaces python-pip python-dns \ python-netifaces python-pip python-dns \
python-mock python-mock
* On ``Fedora``::
sudo dnf update
sudo dnf install curl gcc memcached rsync-daemon sqlite xfsprogs git-core \
libffi-devel xinetd liberasurecode-devel \
openssl-devel python-setuptools \
python-coverage python-devel python-nose \
pyxattr python-eventlet \
python-greenlet python-paste-deploy \
python-netifaces python-pip python-dns \
python-mock
* On ``OpenSuse``:: * On ``OpenSuse``::
sudo zypper install curl gcc memcached rsync sqlite3 xfsprogs git-core \ sudo zypper install curl gcc memcached rsync sqlite3 xfsprogs git-core \
@ -81,7 +105,13 @@ Installing dependencies
dependencies. Later in the process setuptools/distribute or pip will install dependencies. Later in the process setuptools/distribute or pip will install
and/or upgrade packages. and/or upgrade packages.
Next, choose either :ref:`partition-section` or :ref:`loopback-section`. -------------------
Configuring storage
-------------------
Swift requires some space on XFS filesystems to store data and run tests.
Choose either :ref:`partition-section` or :ref:`loopback-section`.
.. _partition-section: .. _partition-section:
@ -91,54 +121,29 @@ Using a partition for storage
If you are going to use a separate partition for Swift data, be sure to add If you are going to use a separate partition for Swift data, be sure to add
another device when creating the VM, and follow these instructions: another device when creating the VM, and follow these instructions:
#. Set up a single partition:: .. note::
The disk does not have to be ``/dev/sdb1`` (for example, it could be
``/dev/vdb1``) however the mount point should still be ``/mnt/sdb1``.
#. Set up a single partition on the device (this will wipe the drive)::
sudo parted /dev/sdb mklabel msdos mkpart p xfs 0% 100%
#. Create an XFS file system on the partition::
sudo fdisk /dev/sdb
sudo mkfs.xfs /dev/sdb1 sudo mkfs.xfs /dev/sdb1
#. Edit ``/etc/fstab`` and add:: #. Edit ``/etc/fstab`` and add::
/dev/sdb1 /mnt/sdb1 xfs noatime,nodiratime,logbufs=8 0 0 /dev/sdb1 /mnt/sdb1 xfs noatime,nodiratime,logbufs=8 0 0
#. Create the mount point and the individualized links:: #. Create the Swift data mount point and test that mounting works::
sudo mkdir /mnt/sdb1 sudo mkdir /mnt/sdb1
sudo mount /mnt/sdb1 sudo mount -a
sudo mkdir /mnt/sdb1/1 /mnt/sdb1/2 /mnt/sdb1/3 /mnt/sdb1/4
sudo chown ${USER}:${USER} /mnt/sdb1/*
sudo mkdir /srv
for x in {1..4}; do sudo ln -s /mnt/sdb1/$x /srv/$x; done
sudo mkdir -p /srv/1/node/sdb1 /srv/1/node/sdb5 \
/srv/2/node/sdb2 /srv/2/node/sdb6 \
/srv/3/node/sdb3 /srv/3/node/sdb7 \
/srv/4/node/sdb4 /srv/4/node/sdb8 \
/var/run/swift
sudo chown -R ${USER}:${USER} /var/run/swift
# **Make sure to include the trailing slash after /srv/$x/**
for x in {1..4}; do sudo chown -R ${USER}:${USER} /srv/$x/; done
.. note::
For OpenSuse users, a user's primary group is ``users``, so you have 2 options:
* Change ``${USER}:${USER}`` to ``${USER}:users`` in all references of this guide; or
* Create a group for your username and add yourself to it::
sudo groupadd ${USER} && sudo gpasswd -a ${USER} ${USER}
.. note::
We create the mount points and mount the storage disk under
/mnt/sdb1. This disk will contain one directory per simulated swift node,
each owned by the current swift user.
We then create symlinks to these directories under /srv.
If the disk sdb is unmounted, files will not be written under
/srv/\*, because the symbolic link destination /mnt/sdb1/* will not
exist. This prevents disk sync operations from writing to the root
partition in the event a drive is unmounted.
#. Next, skip to :ref:`common-dev-section`. #. Next, skip to :ref:`common-dev-section`.
.. _loopback-section: .. _loopback-section:
Using a loopback device for storage Using a loopback device for storage
@ -149,7 +154,7 @@ these instructions:
#. Create the file for the loopback device:: #. Create the file for the loopback device::
sudo mkdir /srv sudo mkdir -p /srv
sudo truncate -s 1GB /srv/swift-disk sudo truncate -s 1GB /srv/swift-disk
sudo mkfs.xfs /srv/swift-disk sudo mkfs.xfs /srv/swift-disk
@ -160,75 +165,115 @@ these instructions:
/srv/swift-disk /mnt/sdb1 xfs loop,noatime,nodiratime,logbufs=8 0 0 /srv/swift-disk /mnt/sdb1 xfs loop,noatime,nodiratime,logbufs=8 0 0
#. Create the mount point and the individualized links:: #. Create the Swift data mount point and test that mounting works::
sudo mkdir /mnt/sdb1 sudo mkdir /mnt/sdb1
sudo mount /mnt/sdb1 sudo mount -a
sudo mkdir /mnt/sdb1/1 /mnt/sdb1/2 /mnt/sdb1/3 /mnt/sdb1/4
sudo chown ${USER}:${USER} /mnt/sdb1/*
for x in {1..4}; do sudo ln -s /mnt/sdb1/$x /srv/$x; done
sudo mkdir -p /srv/1/node/sdb1 /srv/1/node/sdb5 \
/srv/2/node/sdb2 /srv/2/node/sdb6 \
/srv/3/node/sdb3 /srv/3/node/sdb7 \
/srv/4/node/sdb4 /srv/4/node/sdb8 \
/var/run/swift
sudo chown -R ${USER}:${USER} /var/run/swift
# **Make sure to include the trailing slash after /srv/$x/**
for x in {1..4}; do sudo chown -R ${USER}:${USER} /srv/$x/; done
.. note::
For OpenSuse users, a user's primary group is ``users``, so you have 2 options:
* Change ``${USER}:${USER}`` to ``${USER}:users`` in all references of this guide; or
* Create a group for your username and add yourself to it::
sudo groupadd ${USER} && sudo gpasswd -a ${USER} ${USER}
.. note::
We create the mount points and mount the loopback file under
/mnt/sdb1. This file will contain one directory per simulated swift node,
each owned by the current swift user.
We then create symlinks to these directories under /srv.
If the loopback file is unmounted, files will not be written under
/srv/\*, because the symbolic link destination /mnt/sdb1/* will not
exist. This prevents disk sync operations from writing to the root
partition in the event a drive is unmounted.
.. _common-dev-section: .. _common-dev-section:
Common Post-Device Setup Common Post-Device Setup
======================== ========================
Add the following lines to ``/etc/rc.local`` (before the ``exit 0``):: #. Create the individualized data links::
sudo mkdir /mnt/sdb1/1 /mnt/sdb1/2 /mnt/sdb1/3 /mnt/sdb1/4
sudo chown ${USER}:${USER} /mnt/sdb1/*
for x in {1..4}; do sudo ln -s /mnt/sdb1/$x /srv/$x; done
sudo mkdir -p /srv/1/node/sdb1 /srv/1/node/sdb5 \
/srv/2/node/sdb2 /srv/2/node/sdb6 \
/srv/3/node/sdb3 /srv/3/node/sdb7 \
/srv/4/node/sdb4 /srv/4/node/sdb8
sudo mkdir -p /var/run/swift
sudo mkdir -p /var/cache/swift /var/cache/swift2 \
/var/cache/swift3 /var/cache/swift4
sudo chown -R ${USER}:${USER} /var/run/swift
sudo chown -R ${USER}:${USER} /var/cache/swift*
# **Make sure to include the trailing slash after /srv/$x/**
for x in {1..4}; do sudo chown -R ${USER}:${USER} /srv/$x/; done
.. note::
We create the mount points and mount the loopback file under
/mnt/sdb1. This file will contain one directory per simulated Swift node,
each owned by the current Swift user.
We then create symlinks to these directories under /srv.
If the disk sdb or loopback file is unmounted, files will not be written under
/srv/\*, because the symbolic link destination /mnt/sdb1/* will not
exist. This prevents disk sync operations from writing to the root
partition in the event a drive is unmounted.
#. Restore appropriate permissions on reboot.
* On traditional Linux systems, add the following lines to ``/etc/rc.local`` (before the ``exit 0``)::
mkdir -p /var/cache/swift /var/cache/swift2 /var/cache/swift3 /var/cache/swift4 mkdir -p /var/cache/swift /var/cache/swift2 /var/cache/swift3 /var/cache/swift4
chown <your-user-name>:<your-group-name> /var/cache/swift* chown <your-user-name>:<your-group-name> /var/cache/swift*
mkdir -p /var/run/swift mkdir -p /var/run/swift
chown <your-user-name>:<your-group-name> /var/run/swift chown <your-user-name>:<your-group-name> /var/run/swift
Note that on some systems you might have to create ``/etc/rc.local``. * On CentOS and Fedora we can use systemd (rc.local is deprecated)::
On Fedora 19 or later, you need to place these in ``/etc/rc.d/rc.local``. cat << EOF |sudo tee /etc/tmpfiles.d/swift.conf
d /var/cache/swift 0755 ${USER} ${USER} - -
d /var/cache/swift2 0755 ${USER} ${USER} - -
d /var/cache/swift3 0755 ${USER} ${USER} - -
d /var/cache/swift4 0755 ${USER} ${USER} - -
d /var/run/swift 0755 ${USER} ${USER} - -
EOF
On OpenSuse you need to place these in ``/etc/init.d/boot.local``. * On OpenSuse place the lines in ``/etc/init.d/boot.local``.
.. note::
On some systems the rc file might need to be an executable shell script.
Creating an XFS tmp dir Creating an XFS tmp dir
----------------------- -----------------------
Tests require having an XFS directory available in ``/tmp`` or in the Tests require having a directory available on an XFS filesystem. By default the
``TMPDIR`` environment variable. To set up ``/tmp`` with an XFS filesystem, tests use ``/tmp``, however this can be pointed elsewhere with the ``TMPDIR``
do the following:: environment variable.
cd ~ .. note::
truncate -s 1GB xfs_file # create 1GB fil for XFS in your home directory If your root filesystem is XFS, you can skip this section if ``/tmp`` is
mkfs.xfs xfs_file just a directory and not a mounted tmpfs. Or you could simply point to any
sudo mount -o loop,noatime,nodiratime xfs_file /tmp existing directory owned by your user by specifying it with the ``TMPDIR``
environment variable.
If your root filesystem is not XFS, you should create a loopback device,
format it with XFS and mount it. You can mount it over ``/tmp`` or to
another location and specify it with the ``TMPDIR`` environment variable.
* Create the file for the tmp loopback device::
sudo mkdir -p /srv
sudo truncate -s 1GB /srv/swift-tmp # create 1GB file for XFS in /srv
sudo mkfs.xfs /srv/swift-tmp
* To mount the tmp loopback device at ``/tmp``, do the following::
sudo mount -o loop,noatime,nodiratime /srv/swift-tmp /tmp
sudo chmod -R 1777 /tmp sudo chmod -R 1777 /tmp
To persist this, edit and add the following to ``/etc/fstab``:: * To persist this, edit and add the following to ``/etc/fstab``::
/home/<your-user-name>/xfs_file /tmp xfs rw,noatime,nodiratime,attr2,inode64,noquota 0 0 /srv/swift-tmp /tmp xfs rw,noatime,nodiratime,attr2,inode64,noquota 0 0
* To mount the tmp loopback at an alternate location (for example, ``/mnt/tmp``),
do the following::
sudo mkdir -p /mnt/tmp
sudo mount -o loop,noatime,nodiratime /srv/swift-tmp /mnt/tmp
sudo chown ${USER}:${USER} /mnt/tmp
* To persist this, edit and add the following to ``/etc/fstab``::
/srv/swift-tmp /mnt/tmp xfs rw,noatime,nodiratime,attr2,inode64,noquota 0 0
* Set your ``TMPDIR`` environment dir so that Swift looks in the right location::
export TMPDIR=/mnt/tmp
echo "export TMPDIR=/mnt/tmp" >> $HOME/.bashrc
---------------- ----------------
Getting the code Getting the code
@ -247,11 +292,11 @@ Getting the code
cd $HOME/python-swiftclient; sudo pip install -r requirements.txt; sudo python setup.py develop; cd - cd $HOME/python-swiftclient; sudo pip install -r requirements.txt; sudo python setup.py develop; cd -
#. Check out the swift repo:: #. Check out the Swift repo::
git clone https://github.com/openstack/swift.git git clone https://github.com/openstack/swift.git
#. Build a development installation of swift:: #. Build a development installation of Swift::
cd $HOME/swift; sudo pip install --no-binary cryptography -r requirements.txt; sudo python setup.py develop; cd - cd $HOME/swift; sudo pip install --no-binary cryptography -r requirements.txt; sudo python setup.py develop; cd -
@ -260,12 +305,12 @@ Getting the code
wheel/binary won't work; thus we use ``--no-binary cryptography`` to build ``cryptography`` wheel/binary won't work; thus we use ``--no-binary cryptography`` to build ``cryptography``
locally. locally.
Fedora 19 or later users might have to perform the following if development Fedora users might have to perform the following if development
installation of swift fails:: installation of Swift fails::
sudo pip install -U xattr sudo pip install -U xattr
#. Install swift's test dependencies:: #. Install Swift's test dependencies::
cd $HOME/swift; sudo pip install -r test-requirements.txt cd $HOME/swift; sudo pip install -r test-requirements.txt
@ -284,21 +329,26 @@ Setting up rsync
.. literalinclude:: /../saio/rsyncd.conf .. literalinclude:: /../saio/rsyncd.conf
:language: ini :language: ini
#. On Ubuntu, edit the following line in ``/etc/default/rsync``:: #. Enable rsync daemon
* On Ubuntu, edit the following line in ``/etc/default/rsync``::
RSYNC_ENABLE=true RSYNC_ENABLE=true
On Fedora, edit the following line in ``/etc/xinetd.d/rsync``:: .. note::
You might have to create the file to perform the edits.
disable = no * On CentOS and Fedora, enable the systemd service::
One might have to create the above files to perform the edits. sudo systemctl enable rsyncd
* On OpenSuse, nothing needs to happen here.
On OpenSuse, nothing needs to happen here.
#. On platforms with SELinux in ``Enforcing`` mode, either set to ``Permissive``:: #. On platforms with SELinux in ``Enforcing`` mode, either set to ``Permissive``::
sudo setenforce Permissive sudo setenforce Permissive
sudo sed -i 's/^SELINUX=.*/SELINUX=permissive/g' /etc/selinux/config
Or just allow rsync full access:: Or just allow rsync full access::
@ -315,16 +365,9 @@ Setting up rsync
sudo systemctl enable rsync sudo systemctl enable rsync
sudo systemctl start rsync sudo systemctl start rsync
* On Fedora, run:: * On CentOS, Fedora and OpenSuse, run::
sudo systemctl restart xinetd.service sudo systemctl start rsyncd
sudo systemctl enable rsyncd.service
sudo systemctl start rsyncd.service
* On OpenSuse, run::
sudo systemctl enable rsyncd.service
sudo systemctl start rsyncd.service
* On other xinetd based systems simply run:: * On other xinetd based systems simply run::
@ -360,8 +403,8 @@ On non-Ubuntu distros you need to ensure memcached is running::
or:: or::
sudo systemctl enable memcached.service sudo systemctl enable memcached
sudo systemctl start memcached.service sudo systemctl start memcached
The tempauth middleware stores tokens in memcached. If memcached is not The tempauth middleware stores tokens in memcached. If memcached is not
running, tokens cannot be validated, and accessing Swift becomes impossible. running, tokens cannot be validated, and accessing Swift becomes impossible.
@ -370,16 +413,23 @@ running, tokens cannot be validated, and accessing Swift becomes impossible.
Optional: Setting up rsyslog for individual logging Optional: Setting up rsyslog for individual logging
--------------------------------------------------- ---------------------------------------------------
#. Install the swift rsyslogd configuration:: Fedora and OpenSuse may not have rsyslog installed, in which case you will need
to install it if you want to use individual logging.
sudo cp $HOME/swift/doc/saio/rsyslog.d/10-swift.conf /etc/rsyslog.d/ #. Install rsyslogd
Note: OpenSuse may have the systemd logger installed, so if you want this
to work, you need to install rsyslog:: * On Fedora::
sudo dnf install rsyslog
* On OpenSuse::
sudo zypper install rsyslog sudo zypper install rsyslog
sudo systemctl start rsyslog.service
sudo systemctl enable rsyslog.service #. Install the Swift rsyslogd configuration::
sudo cp $HOME/swift/doc/saio/rsyslog.d/10-swift.conf /etc/rsyslog.d/
Be sure to review that conf file to determine if you want all the logs Be sure to review that conf file to determine if you want all the logs
in one file vs. all the logs separated out, and if you want hourly logs in one file vs. all the logs separated out, and if you want hourly logs
@ -410,11 +460,12 @@ Optional: Setting up rsyslog for individual logging
sudo chmod -R g+w /var/log/swift sudo chmod -R g+w /var/log/swift
sudo service rsyslog restart sudo service rsyslog restart
* On Fedora and OpenSuse:: * On CentOS, Fedora and OpenSuse::
sudo chown -R root:adm /var/log/swift sudo chown -R root:adm /var/log/swift
sudo chmod -R g+w /var/log/swift sudo chmod -R g+w /var/log/swift
sudo systemctl restart rsyslog.service sudo systemctl restart rsyslog
sudo systemctl enable rsyslog
--------------------- ---------------------
Configuring each node Configuring each node
@ -544,11 +595,6 @@ Setting up scripts for running Swift
.. literalinclude:: /../saio/bin/resetswift .. literalinclude:: /../saio/bin/resetswift
:language: bash :language: bash
If you are using a loopback device add an environment var to
substitute ``/dev/sdb1`` with ``/srv/swift-disk``::
echo "export SAIO_BLOCK_DEVICE=/srv/swift-disk" >> $HOME/.bashrc
If you did not set up rsyslog for individual logging, remove the ``find If you did not set up rsyslog for individual logging, remove the ``find
/var/log/swift...`` line:: /var/log/swift...`` line::
@ -564,6 +610,10 @@ Setting up scripts for running Swift
.. literalinclude:: /../../test/sample.conf .. literalinclude:: /../../test/sample.conf
:language: ini :language: ini
-----------------------------------------
Configure environment variables for Swift
-----------------------------------------
#. Add an environment variable for running tests below:: #. Add an environment variable for running tests below::
echo "export SWIFT_TEST_CONFIG_FILE=/etc/swift/test.conf" >> $HOME/.bashrc echo "export SWIFT_TEST_CONFIG_FILE=/etc/swift/test.conf" >> $HOME/.bashrc
@ -572,10 +622,30 @@ Setting up scripts for running Swift
echo "export PATH=${PATH}:$HOME/bin" >> $HOME/.bashrc echo "export PATH=${PATH}:$HOME/bin" >> $HOME/.bashrc
#. If you are using a loopback device for Swift Storage, add an environment var
to substitute ``/dev/sdb1`` with ``/srv/swift-disk``::
echo "export SAIO_BLOCK_DEVICE=/srv/swift-disk" >> $HOME/.bashrc
#. If you are using a device other than ``/dev/sdb1`` for Swift storage (for
example, ``/dev/vdb1``), add an environment var to substitute it::
echo "export SAIO_BLOCK_DEVICE=/dev/vdb1" >> $HOME/.bashrc
#. If you are using a location other than ``/tmp`` for Swift tmp data (for
example, ``/mnt/tmp``), add ``TMPDIR`` environment var to set it::
export TMPDIR=/mnt/tmp
echo "export TMPDIR=/mnt/tmp" >> $HOME/.bashrc
#. Source the above environment variables into your current environment:: #. Source the above environment variables into your current environment::
. $HOME/.bashrc . $HOME/.bashrc
--------------------------
Constructing initial rings
--------------------------
#. Construct the initial rings using the provided script:: #. Construct the initial rings using the provided script::
remakerings remakerings
@ -627,11 +697,15 @@ Setting up scripts for running Swift
#. Read more about Storage Policies and your SAIO :doc:`policies_saio` #. Read more about Storage Policies and your SAIO :doc:`policies_saio`
-------------
Testing Swift
-------------
#. Verify the unit tests run:: #. Verify the unit tests run::
$HOME/swift/.unittests $HOME/swift/.unittests
Note that the unit tests do not require any swift daemons running. Note that the unit tests do not require any Swift daemons running.
#. Start the "main" Swift daemon processes (proxy, account, container, and #. Start the "main" Swift daemon processes (proxy, account, container, and
object):: object)::