openstack/swift
Code Issues Proposed changes

Merge "docs: clean up SAIO formatting"

Browse Source
This commit is contained in:
Zuul 2019-03-05 23:57:37 +00:00 committed by Gerrit Code Review
commit f9329ca5ff
3 changed files with 332 additions and 291 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

613
doc/source/development_saio.rst
View File

@ -76,9 +76,10 @@ Installing dependencies
python2-netifaces python2-pip python2-dnspython \ python2-netifaces python2-pip python2-dnspython \
python2-mock python2-mock
Note: This installs necessary system dependencies and *most* of the python .. note::
dependencies. Later in the process setuptools/distribute or pip will install This installs necessary system dependencies and *most* of the python
and/or upgrade packages. dependencies. Later in the process setuptools/distribute or pip will install
and/or upgrade packages.
Next, choose either :ref:`partition-section` or :ref:`loopback-section`. Next, choose either :ref:`partition-section` or :ref:`loopback-section`.
@ -90,50 +91,52 @@ Using a partition for storage
If you are going to use a separate partition for Swift data, be sure to add If you are going to use a separate partition for Swift data, be sure to add
another device when creating the VM, and follow these instructions: another device when creating the VM, and follow these instructions:
#. Set up a single partition:: #. Set up a single partition::
sudo fdisk /dev/sdb sudo fdisk /dev/sdb
sudo mkfs.xfs /dev/sdb1 sudo mkfs.xfs /dev/sdb1
#. Edit ``/etc/fstab`` and add:: #. Edit ``/etc/fstab`` and add::
/dev/sdb1 /mnt/sdb1 xfs noatime,nodiratime,nobarrier,logbufs=8 0 0 /dev/sdb1 /mnt/sdb1 xfs noatime,nodiratime,nobarrier,logbufs=8 0 0
#. Create the mount point and the individualized links:: #. Create the mount point and the individualized links::
sudo mkdir /mnt/sdb1 sudo mkdir /mnt/sdb1
sudo mount /mnt/sdb1 sudo mount /mnt/sdb1
sudo mkdir /mnt/sdb1/1 /mnt/sdb1/2 /mnt/sdb1/3 /mnt/sdb1/4 sudo mkdir /mnt/sdb1/1 /mnt/sdb1/2 /mnt/sdb1/3 /mnt/sdb1/4
sudo chown ${USER}:${USER} /mnt/sdb1/* sudo chown ${USER}:${USER} /mnt/sdb1/*
sudo mkdir /srv sudo mkdir /srv
for x in {1..4}; do sudo ln -s /mnt/sdb1/$x /srv/$x; done for x in {1..4}; do sudo ln -s /mnt/sdb1/$x /srv/$x; done
sudo mkdir -p /srv/1/node/sdb1 /srv/1/node/sdb5 \ sudo mkdir -p /srv/1/node/sdb1 /srv/1/node/sdb5 \
/srv/2/node/sdb2 /srv/2/node/sdb6 \ /srv/2/node/sdb2 /srv/2/node/sdb6 \
/srv/3/node/sdb3 /srv/3/node/sdb7 \ /srv/3/node/sdb3 /srv/3/node/sdb7 \
/srv/4/node/sdb4 /srv/4/node/sdb8 \ /srv/4/node/sdb4 /srv/4/node/sdb8 \
/var/run/swift /var/run/swift
sudo chown -R ${USER}:${USER} /var/run/swift sudo chown -R ${USER}:${USER} /var/run/swift
# **Make sure to include the trailing slash after /srv/$x/** # **Make sure to include the trailing slash after /srv/$x/**
for x in {1..4}; do sudo chown -R ${USER}:${USER} /srv/$x/; done for x in {1..4}; do sudo chown -R ${USER}:${USER} /srv/$x/; done
Note: For OpenSuse users, a user's primary group is `users`, so you have 2 options: .. note::
For OpenSuse users, a user's primary group is ``users``, so you have 2 options:
* Change `${USER}:${USER}` to `${USER}:users` in all references of this guide; or * Change ``${USER}:${USER}`` to ``${USER}:users`` in all references of this guide; or
* Create a group for your username and add yourself to it:: * Create a group for your username and add yourself to it::
sudo groupadd ${USER} && sudo gpasswd -a ${USER} ${USER} sudo groupadd ${USER} && sudo gpasswd -a ${USER} ${USER}
Note: We create the mount points and mount the storage disk under .. note::
/mnt/sdb1. This disk will contain one directory per simulated swift node, We create the mount points and mount the storage disk under
each owned by the current swift user. /mnt/sdb1. This disk will contain one directory per simulated swift node,
each owned by the current swift user.
We then create symlinks to these directories under /srv. We then create symlinks to these directories under /srv.
If the disk sdb is unmounted, files will not be written under If the disk sdb is unmounted, files will not be written under
/srv/\*, because the symbolic link destination /mnt/sdb1/* will not /srv/\*, because the symbolic link destination /mnt/sdb1/* will not
exist. This prevents disk sync operations from writing to the root exist. This prevents disk sync operations from writing to the root
partition in the event a drive is unmounted. partition in the event a drive is unmounted.
#. Next, skip to :ref:`common-dev-section`. #. Next, skip to :ref:`common-dev-section`.
.. _loopback-section: .. _loopback-section:
@ -144,51 +147,53 @@ Using a loopback device for storage
If you want to use a loopback device instead of another partition, follow If you want to use a loopback device instead of another partition, follow
these instructions: these instructions:
#. Create the file for the loopback device:: #. Create the file for the loopback device::
sudo mkdir /srv sudo mkdir /srv
sudo truncate -s 1GB /srv/swift-disk sudo truncate -s 1GB /srv/swift-disk
sudo mkfs.xfs /srv/swift-disk sudo mkfs.xfs /srv/swift-disk
Modify size specified in the ``truncate`` command to make a larger or Modify size specified in the ``truncate`` command to make a larger or
smaller partition as needed. smaller partition as needed.
#. Edit `/etc/fstab` and add:: #. Edit `/etc/fstab` and add::
/srv/swift-disk /mnt/sdb1 xfs loop,noatime,nodiratime,nobarrier,logbufs=8 0 0 /srv/swift-disk /mnt/sdb1 xfs loop,noatime,nodiratime,nobarrier,logbufs=8 0 0
#. Create the mount point and the individualized links:: #. Create the mount point and the individualized links::
sudo mkdir /mnt/sdb1 sudo mkdir /mnt/sdb1
sudo mount /mnt/sdb1 sudo mount /mnt/sdb1
sudo mkdir /mnt/sdb1/1 /mnt/sdb1/2 /mnt/sdb1/3 /mnt/sdb1/4 sudo mkdir /mnt/sdb1/1 /mnt/sdb1/2 /mnt/sdb1/3 /mnt/sdb1/4
sudo chown ${USER}:${USER} /mnt/sdb1/* sudo chown ${USER}:${USER} /mnt/sdb1/*
for x in {1..4}; do sudo ln -s /mnt/sdb1/$x /srv/$x; done for x in {1..4}; do sudo ln -s /mnt/sdb1/$x /srv/$x; done
sudo mkdir -p /srv/1/node/sdb1 /srv/1/node/sdb5 \ sudo mkdir -p /srv/1/node/sdb1 /srv/1/node/sdb5 \
/srv/2/node/sdb2 /srv/2/node/sdb6 \ /srv/2/node/sdb2 /srv/2/node/sdb6 \
/srv/3/node/sdb3 /srv/3/node/sdb7 \ /srv/3/node/sdb3 /srv/3/node/sdb7 \
/srv/4/node/sdb4 /srv/4/node/sdb8 \ /srv/4/node/sdb4 /srv/4/node/sdb8 \
/var/run/swift /var/run/swift
sudo chown -R ${USER}:${USER} /var/run/swift sudo chown -R ${USER}:${USER} /var/run/swift
# **Make sure to include the trailing slash after /srv/$x/** # **Make sure to include the trailing slash after /srv/$x/**
for x in {1..4}; do sudo chown -R ${USER}:${USER} /srv/$x/; done for x in {1..4}; do sudo chown -R ${USER}:${USER} /srv/$x/; done
Note: For OpenSuse users, a user's primary group is `users`, so you have 2 options: .. note::
For OpenSuse users, a user's primary group is ``users``, so you have 2 options:
* Change `${USER}:${USER}` to `${USER}:users` in all references of this guide; or * Change ``${USER}:${USER}`` to ``${USER}:users`` in all references of this guide; or
* Create a group for your username and add yourself to it:: * Create a group for your username and add yourself to it::
sudo groupadd ${USER} && sudo gpasswd -a ${USER} ${USER} sudo groupadd ${USER} && sudo gpasswd -a ${USER} ${USER}
Note: We create the mount points and mount the loopback file under .. note::
/mnt/sdb1. This file will contain one directory per simulated swift node, We create the mount points and mount the loopback file under
each owned by the current swift user. /mnt/sdb1. This file will contain one directory per simulated swift node,
each owned by the current swift user.
We then create symlinks to these directories under /srv. We then create symlinks to these directories under /srv.
If the loopback file is unmounted, files will not be written under If the loopback file is unmounted, files will not be written under
/srv/\*, because the symbolic link destination /mnt/sdb1/* will not /srv/\*, because the symbolic link destination /mnt/sdb1/* will not
exist. This prevents disk sync operations from writing to the root exist. This prevents disk sync operations from writing to the root
partition in the event a drive is unmounted. partition in the event a drive is unmounted.
.. _common-dev-section: .. _common-dev-section:
@ -229,117 +234,120 @@ To persist this, edit and add the following to ``/etc/fstab``::
Getting the code Getting the code
---------------- ----------------
#. Check out the python-swiftclient repo:: #. Check out the python-swiftclient repo::
cd $HOME; git clone https://github.com/openstack/python-swiftclient.git cd $HOME; git clone https://github.com/openstack/python-swiftclient.git
#. Build a development installation of python-swiftclient:: #. Build a development installation of python-swiftclient::
cd $HOME/python-swiftclient; sudo python setup.py develop; cd - cd $HOME/python-swiftclient; sudo python setup.py develop; cd -
Ubuntu 12.04 users need to install python-swiftclient's dependencies before the installation of Ubuntu 12.04 users need to install python-swiftclient's dependencies before the installation of
python-swiftclient. This is due to a bug in an older version of setup tools:: python-swiftclient. This is due to a bug in an older version of setup tools::
cd $HOME/python-swiftclient; sudo pip install -r requirements.txt; sudo python setup.py develop; cd - cd $HOME/python-swiftclient; sudo pip install -r requirements.txt; sudo python setup.py develop; cd -
#. Check out the swift repo:: #. Check out the swift repo::
git clone https://github.com/openstack/swift.git git clone https://github.com/openstack/swift.git
#. Build a development installation of swift:: #. Build a development installation of swift::
cd $HOME/swift; sudo pip install --no-binary cryptography -r requirements.txt; sudo python setup.py develop; cd - cd $HOME/swift; sudo pip install --no-binary cryptography -r requirements.txt; sudo python setup.py develop; cd -
Note: Due to a difference in libssl.so naming in OpenSuse to other Linux distros the wheel/binary wont work so the .. note::
cryptography must be built, thus the ``--no-binary cryptography``. Due to a difference in how ``libssl.so`` is named in OpenSuse vs. other Linux distros the
wheel/binary won't work; thus we use ``--no-binary cryptography`` to build ``cryptography``
locally.
Fedora 19 or later users might have to perform the following if development Fedora 19 or later users might have to perform the following if development
installation of swift fails:: installation of swift fails::
sudo pip install -U xattr sudo pip install -U xattr
#. Install swift's test dependencies:: #. Install swift's test dependencies::
cd $HOME/swift; sudo pip install -r test-requirements.txt cd $HOME/swift; sudo pip install -r test-requirements.txt
---------------- ----------------
Setting up rsync Setting up rsync
---------------- ----------------
#. Create ``/etc/rsyncd.conf``:: #. Create ``/etc/rsyncd.conf``::
sudo cp $HOME/swift/doc/saio/rsyncd.conf /etc/ sudo cp $HOME/swift/doc/saio/rsyncd.conf /etc/
sudo sed -i "s/<your-user-name>/${USER}/" /etc/rsyncd.conf sudo sed -i "s/<your-user-name>/${USER}/" /etc/rsyncd.conf
Here is the default ``rsyncd.conf`` file contents maintained in the repo Here is the default ``rsyncd.conf`` file contents maintained in the repo
that is copied and fixed up above: that is copied and fixed up above:
.. literalinclude:: /../saio/rsyncd.conf .. literalinclude:: /../saio/rsyncd.conf
:language: ini
#. On Ubuntu, edit the following line in ``/etc/default/rsync``:: #. On Ubuntu, edit the following line in ``/etc/default/rsync``::
RSYNC_ENABLE=true RSYNC_ENABLE=true
On Fedora, edit the following line in ``/etc/xinetd.d/rsync``:: On Fedora, edit the following line in ``/etc/xinetd.d/rsync``::
disable = no disable = no
One might have to create the above files to perform the edits. One might have to create the above files to perform the edits.
On OpenSuse, nothing needs to happen here. On OpenSuse, nothing needs to happen here.
#. On platforms with SELinux in ``Enforcing`` mode, either set to ``Permissive``:: #. On platforms with SELinux in ``Enforcing`` mode, either set to ``Permissive``::
sudo setenforce Permissive sudo setenforce Permissive
Or just allow rsync full access:: Or just allow rsync full access::
sudo setsebool -P rsync_full_access 1 sudo setsebool -P rsync_full_access 1
#. Start the rsync daemon #. Start the rsync daemon
* On Ubuntu 14.04, run:: * On Ubuntu 14.04, run::
sudo service rsync restart sudo service rsync restart
* On Ubuntu 16.04, run:: * On Ubuntu 16.04, run::
sudo systemctl enable rsync sudo systemctl enable rsync
sudo systemctl start rsync sudo systemctl start rsync
* On Fedora, run:: * On Fedora, run::
sudo systemctl restart xinetd.service sudo systemctl restart xinetd.service
sudo systemctl enable rsyncd.service sudo systemctl enable rsyncd.service
sudo systemctl start rsyncd.service sudo systemctl start rsyncd.service
* On OpenSuse, run:: * On OpenSuse, run::
sudo systemctl enable rsyncd.service sudo systemctl enable rsyncd.service
sudo systemctl start rsyncd.service sudo systemctl start rsyncd.service
* On other xinetd based systems simply run:: * On other xinetd based systems simply run::
sudo service xinetd restart sudo service xinetd restart
#. Verify rsync is accepting connections for all servers:: #. Verify rsync is accepting connections for all servers::
rsync rsync://pub@localhost/ rsync rsync://pub@localhost/
You should see the following output from the above command:: You should see the following output from the above command::
account6012 account6012
account6022 account6022
account6032 account6032
account6042 account6042
container6011 container6011
container6021 container6021
container6031 container6031
container6041 container6041
object6010 object6010
object6020 object6020
object6030 object6030
object6040 object6040
------------------ ------------------
Starting memcached Starting memcached
@ -362,50 +370,51 @@ running, tokens cannot be validated, and accessing Swift becomes impossible.
Optional: Setting up rsyslog for individual logging Optional: Setting up rsyslog for individual logging
--------------------------------------------------- ---------------------------------------------------
#. Install the swift rsyslogd configuration:: #. Install the swift rsyslogd configuration::
sudo cp $HOME/swift/doc/saio/rsyslog.d/10-swift.conf /etc/rsyslog.d/ sudo cp $HOME/swift/doc/saio/rsyslog.d/10-swift.conf /etc/rsyslog.d/
Note: OpenSuse may have the systemd logger installed, so if you want this Note: OpenSuse may have the systemd logger installed, so if you want this
to work, you need to install rsyslog:: to work, you need to install rsyslog::
sudo zypper install rsyslog sudo zypper install rsyslog
sudo systemctl start rsyslog.service sudo systemctl start rsyslog.service
sudo systemctl enable rsyslog.service sudo systemctl enable rsyslog.service
Be sure to review that conf file to determine if you want all the logs Be sure to review that conf file to determine if you want all the logs
in one file vs. all the logs separated out, and if you want hourly logs in one file vs. all the logs separated out, and if you want hourly logs
for stats processing. For convenience, we provide its default contents for stats processing. For convenience, we provide its default contents
below: below:
.. literalinclude:: /../saio/rsyslog.d/10-swift.conf .. literalinclude:: /../saio/rsyslog.d/10-swift.conf
:language: ini
#. Edit ``/etc/rsyslog.conf`` and make the following change (usually in the #. Edit ``/etc/rsyslog.conf`` and make the following change (usually in the
"GLOBAL DIRECTIVES" section):: "GLOBAL DIRECTIVES" section)::
$PrivDropToGroup adm $PrivDropToGroup adm
#. If using hourly logs (see above) perform:: #. If using hourly logs (see above) perform::
sudo mkdir -p /var/log/swift/hourly sudo mkdir -p /var/log/swift/hourly
Otherwise perform:: Otherwise perform::
sudo mkdir -p /var/log/swift sudo mkdir -p /var/log/swift
#. Setup the logging directory and start syslog: #. Setup the logging directory and start syslog:
* On Ubuntu:: * On Ubuntu::
sudo chown -R syslog.adm /var/log/swift sudo chown -R syslog.adm /var/log/swift
sudo chmod -R g+w /var/log/swift sudo chmod -R g+w /var/log/swift
sudo service rsyslog restart sudo service rsyslog restart
* On Fedora and OpenSuse:: * On Fedora and OpenSuse::
sudo chown -R root:adm /var/log/swift sudo chown -R root:adm /var/log/swift
sudo chmod -R g+w /var/log/swift sudo chmod -R g+w /var/log/swift
sudo systemctl restart rsyslog.service sudo systemctl restart rsyslog.service
--------------------- ---------------------
Configuring each node Configuring each node
@ -415,89 +424,106 @@ After performing the following steps, be sure to verify that Swift has access
to resulting configuration files (sample configuration files are provided with to resulting configuration files (sample configuration files are provided with
all defaults in line-by-line comments). all defaults in line-by-line comments).
#. Optionally remove an existing swift directory:: #. Optionally remove an existing swift directory::
sudo rm -rf /etc/swift sudo rm -rf /etc/swift
#. Populate the ``/etc/swift`` directory itself:: #. Populate the ``/etc/swift`` directory itself::
cd $HOME/swift/doc; sudo cp -r saio/swift /etc/swift; cd - cd $HOME/swift/doc; sudo cp -r saio/swift /etc/swift; cd -
sudo chown -R ${USER}:${USER} /etc/swift sudo chown -R ${USER}:${USER} /etc/swift
#. Update ``<your-user-name>`` references in the Swift config files:: #. Update ``<your-user-name>`` references in the Swift config files::
find /etc/swift/ -name \*.conf | xargs sudo sed -i "s/<your-user-name>/${USER}/" find /etc/swift/ -name \*.conf | xargs sudo sed -i "s/<your-user-name>/${USER}/"
The contents of the configuration files provided by executing the above The contents of the configuration files provided by executing the above
commands are as follows: commands are as follows:
#. ``/etc/swift/swift.conf`` #. ``/etc/swift/swift.conf``
.. literalinclude:: /../saio/swift/swift.conf .. literalinclude:: /../saio/swift/swift.conf
:language: ini
#. ``/etc/swift/proxy-server.conf`` #. ``/etc/swift/proxy-server.conf``
.. literalinclude:: /../saio/swift/proxy-server.conf .. literalinclude:: /../saio/swift/proxy-server.conf
:language: ini
#. ``/etc/swift/object-expirer.conf`` #. ``/etc/swift/object-expirer.conf``
.. literalinclude:: /../saio/swift/object-expirer.conf .. literalinclude:: /../saio/swift/object-expirer.conf
:language: ini
#. ``/etc/swift/container-reconciler.conf`` #. ``/etc/swift/container-reconciler.conf``
.. literalinclude:: /../saio/swift/container-reconciler.conf .. literalinclude:: /../saio/swift/container-reconciler.conf
:language: ini
#. ``/etc/swift/container-sync-realms.conf`` #. ``/etc/swift/container-sync-realms.conf``
.. literalinclude:: /../saio/swift/container-sync-realms.conf .. literalinclude:: /../saio/swift/container-sync-realms.conf
:language: ini
#. ``/etc/swift/account-server/1.conf`` #. ``/etc/swift/account-server/1.conf``
.. literalinclude:: /../saio/swift/account-server/1.conf .. literalinclude:: /../saio/swift/account-server/1.conf
:language: ini
#. ``/etc/swift/container-server/1.conf`` #. ``/etc/swift/container-server/1.conf``
.. literalinclude:: /../saio/swift/container-server/1.conf .. literalinclude:: /../saio/swift/container-server/1.conf
:language: ini
#. ``/etc/swift/object-server/1.conf`` #. ``/etc/swift/object-server/1.conf``
.. literalinclude:: /../saio/swift/object-server/1.conf .. literalinclude:: /../saio/swift/object-server/1.conf
:language: ini
#. ``/etc/swift/account-server/2.conf`` #. ``/etc/swift/account-server/2.conf``
.. literalinclude:: /../saio/swift/account-server/2.conf .. literalinclude:: /../saio/swift/account-server/2.conf
:language: ini
#. ``/etc/swift/container-server/2.conf`` #. ``/etc/swift/container-server/2.conf``
.. literalinclude:: /../saio/swift/container-server/2.conf .. literalinclude:: /../saio/swift/container-server/2.conf
:language: ini
#. ``/etc/swift/object-server/2.conf`` #. ``/etc/swift/object-server/2.conf``
.. literalinclude:: /../saio/swift/object-server/2.conf .. literalinclude:: /../saio/swift/object-server/2.conf
:language: ini
#. ``/etc/swift/account-server/3.conf`` #. ``/etc/swift/account-server/3.conf``
.. literalinclude:: /../saio/swift/account-server/3.conf .. literalinclude:: /../saio/swift/account-server/3.conf
:language: ini
#. ``/etc/swift/container-server/3.conf`` #. ``/etc/swift/container-server/3.conf``
.. literalinclude:: /../saio/swift/container-server/3.conf .. literalinclude:: /../saio/swift/container-server/3.conf
:language: ini
#. ``/etc/swift/object-server/3.conf`` #. ``/etc/swift/object-server/3.conf``
.. literalinclude:: /../saio/swift/object-server/3.conf .. literalinclude:: /../saio/swift/object-server/3.conf
:language: ini
#. ``/etc/swift/account-server/4.conf`` #. ``/etc/swift/account-server/4.conf``
.. literalinclude:: /../saio/swift/account-server/4.conf .. literalinclude:: /../saio/swift/account-server/4.conf
:language: ini
#. ``/etc/swift/container-server/4.conf`` #. ``/etc/swift/container-server/4.conf``
.. literalinclude:: /../saio/swift/container-server/4.conf .. literalinclude:: /../saio/swift/container-server/4.conf
:language: ini
#. ``/etc/swift/object-server/4.conf`` #. ``/etc/swift/object-server/4.conf``
.. literalinclude:: /../saio/swift/object-server/4.conf .. literalinclude:: /../saio/swift/object-server/4.conf
:language: ini
.. _setup_scripts: .. _setup_scripts:
@ -505,139 +531,146 @@ commands are as follows:
Setting up scripts for running Swift Setting up scripts for running Swift
------------------------------------ ------------------------------------
#. Copy the SAIO scripts for resetting the environment:: #. Copy the SAIO scripts for resetting the environment::
mkdir -p $HOME/bin mkdir -p $HOME/bin
cd $HOME/swift/doc; cp saio/bin/* $HOME/bin; cd - cd $HOME/swift/doc; cp saio/bin/* $HOME/bin; cd -
chmod +x $HOME/bin/* chmod +x $HOME/bin/*
#. Edit the ``$HOME/bin/resetswift`` script #. Edit the ``$HOME/bin/resetswift`` script
The template ``resetswift`` script looks like the following: The template ``resetswift`` script looks like the following:
.. literalinclude:: /../saio/bin/resetswift .. literalinclude:: /../saio/bin/resetswift
:language: bash
If you are using a loopback device add an environment var to If you are using a loopback device add an environment var to
substitute ``/dev/sdb1`` with ``/srv/swift-disk``:: substitute ``/dev/sdb1`` with ``/srv/swift-disk``::
echo "export SAIO_BLOCK_DEVICE=/srv/swift-disk" >> $HOME/.bashrc echo "export SAIO_BLOCK_DEVICE=/srv/swift-disk" >> $HOME/.bashrc
If you did not set up rsyslog for individual logging, remove the ``find If you did not set up rsyslog for individual logging, remove the ``find
/var/log/swift...`` line:: /var/log/swift...`` line::
sed -i "/find \/var\/log\/swift/d" $HOME/bin/resetswift sed -i "/find \/var\/log\/swift/d" $HOME/bin/resetswift
#. Install the sample configuration file for running tests:: #. Install the sample configuration file for running tests::
cp $HOME/swift/test/sample.conf /etc/swift/test.conf cp $HOME/swift/test/sample.conf /etc/swift/test.conf
The template ``test.conf`` looks like the following: The template ``test.conf`` looks like the following:
.. literalinclude:: /../../test/sample.conf .. literalinclude:: /../../test/sample.conf
:language: ini
#. Add an environment variable for running tests below:: #. Add an environment variable for running tests below::
echo "export SWIFT_TEST_CONFIG_FILE=/etc/swift/test.conf" >> $HOME/.bashrc echo "export SWIFT_TEST_CONFIG_FILE=/etc/swift/test.conf" >> $HOME/.bashrc
#. Be sure that your ``PATH`` includes the ``bin`` directory:: #. Be sure that your ``PATH`` includes the ``bin`` directory::
echo "export PATH=${PATH}:$HOME/bin" >> $HOME/.bashrc echo "export PATH=${PATH}:$HOME/bin" >> $HOME/.bashrc
#. Source the above environment variables into your current environment:: #. Source the above environment variables into your current environment::
. $HOME/.bashrc . $HOME/.bashrc
#. Construct the initial rings using the provided script:: #. Construct the initial rings using the provided script::
remakerings remakerings
The ``remakerings`` script looks like the following: The ``remakerings`` script looks like the following:
.. literalinclude:: /../saio/bin/remakerings .. literalinclude:: /../saio/bin/remakerings
:language: bash
You can expect the output from this command to produce the following. Note You can expect the output from this command to produce the following. Note
that 3 object rings are created in order to test storage policies and EC in that 3 object rings are created in order to test storage policies and EC in
the SAIO environment. The EC ring is the only one with all 8 devices. the SAIO environment. The EC ring is the only one with all 8 devices.
There are also two replication rings, one for 3x replication and another There are also two replication rings, one for 3x replication and another
for 2x replication, but those rings only use 4 devices:: for 2x replication, but those rings only use 4 devices:
Device d0r1z1-127.0.0.1:6010R127.0.0.1:6010/sdb1_"" with 1.0 weight got id 0
Device d1r1z2-127.0.0.2:6020R127.0.0.2:6020/sdb2_"" with 1.0 weight got id 1
Device d2r1z3-127.0.0.3:6030R127.0.0.3:6030/sdb3_"" with 1.0 weight got id 2
Device d3r1z4-127.0.0.4:6040R127.0.0.4:6040/sdb4_"" with 1.0 weight got id 3
Reassigned 3072 (300.00%) partitions. Balance is now 0.00. Dispersion is now 0.00
Device d0r1z1-127.0.0.1:6010R127.0.0.1:6010/sdb1_"" with 1.0 weight got id 0
Device d1r1z2-127.0.0.2:6020R127.0.0.2:6020/sdb2_"" with 1.0 weight got id 1
Device d2r1z3-127.0.0.3:6030R127.0.0.3:6030/sdb3_"" with 1.0 weight got id 2
Device d3r1z4-127.0.0.4:6040R127.0.0.4:6040/sdb4_"" with 1.0 weight got id 3
Reassigned 2048 (200.00%) partitions. Balance is now 0.00. Dispersion is now 0.00
Device d0r1z1-127.0.0.1:6010R127.0.0.1:6010/sdb1_"" with 1.0 weight got id 0
Device d1r1z1-127.0.0.1:6010R127.0.0.1:6010/sdb5_"" with 1.0 weight got id 1
Device d2r1z2-127.0.0.2:6020R127.0.0.2:6020/sdb2_"" with 1.0 weight got id 2
Device d3r1z2-127.0.0.2:6020R127.0.0.2:6020/sdb6_"" with 1.0 weight got id 3
Device d4r1z3-127.0.0.3:6030R127.0.0.3:6030/sdb3_"" with 1.0 weight got id 4
Device d5r1z3-127.0.0.3:6030R127.0.0.3:6030/sdb7_"" with 1.0 weight got id 5
Device d6r1z4-127.0.0.4:6040R127.0.0.4:6040/sdb4_"" with 1.0 weight got id 6
Device d7r1z4-127.0.0.4:6040R127.0.0.4:6040/sdb8_"" with 1.0 weight got id 7
Reassigned 6144 (600.00%) partitions. Balance is now 0.00. Dispersion is now 0.00
Device d0r1z1-127.0.0.1:6011R127.0.0.1:6011/sdb1_"" with 1.0 weight got id 0
Device d1r1z2-127.0.0.2:6021R127.0.0.2:6021/sdb2_"" with 1.0 weight got id 1
Device d2r1z3-127.0.0.3:6031R127.0.0.3:6031/sdb3_"" with 1.0 weight got id 2
Device d3r1z4-127.0.0.4:6041R127.0.0.4:6041/sdb4_"" with 1.0 weight got id 3
Reassigned 3072 (300.00%) partitions. Balance is now 0.00. Dispersion is now 0.00
Device d0r1z1-127.0.0.1:6012R127.0.0.1:6012/sdb1_"" with 1.0 weight got id 0
Device d1r1z2-127.0.0.2:6022R127.0.0.2:6022/sdb2_"" with 1.0 weight got id 1
Device d2r1z3-127.0.0.3:6032R127.0.0.3:6032/sdb3_"" with 1.0 weight got id 2
Device d3r1z4-127.0.0.4:6042R127.0.0.4:6042/sdb4_"" with 1.0 weight got id 3
Reassigned 3072 (300.00%) partitions. Balance is now 0.00. Dispersion is now 0.00
#. Read more about Storage Policies and your SAIO :doc:`policies_saio` .. code-block:: console
#. Verify the unit tests run:: Device d0r1z1-127.0.0.1:6010R127.0.0.1:6010/sdb1_"" with 1.0 weight got id 0
Device d1r1z2-127.0.0.2:6020R127.0.0.2:6020/sdb2_"" with 1.0 weight got id 1
Device d2r1z3-127.0.0.3:6030R127.0.0.3:6030/sdb3_"" with 1.0 weight got id 2
Device d3r1z4-127.0.0.4:6040R127.0.0.4:6040/sdb4_"" with 1.0 weight got id 3
Reassigned 3072 (300.00%) partitions. Balance is now 0.00. Dispersion is now 0.00
Device d0r1z1-127.0.0.1:6010R127.0.0.1:6010/sdb1_"" with 1.0 weight got id 0
Device d1r1z2-127.0.0.2:6020R127.0.0.2:6020/sdb2_"" with 1.0 weight got id 1
Device d2r1z3-127.0.0.3:6030R127.0.0.3:6030/sdb3_"" with 1.0 weight got id 2
Device d3r1z4-127.0.0.4:6040R127.0.0.4:6040/sdb4_"" with 1.0 weight got id 3
Reassigned 2048 (200.00%) partitions. Balance is now 0.00. Dispersion is now 0.00
Device d0r1z1-127.0.0.1:6010R127.0.0.1:6010/sdb1_"" with 1.0 weight got id 0
Device d1r1z1-127.0.0.1:6010R127.0.0.1:6010/sdb5_"" with 1.0 weight got id 1
Device d2r1z2-127.0.0.2:6020R127.0.0.2:6020/sdb2_"" with 1.0 weight got id 2
Device d3r1z2-127.0.0.2:6020R127.0.0.2:6020/sdb6_"" with 1.0 weight got id 3
Device d4r1z3-127.0.0.3:6030R127.0.0.3:6030/sdb3_"" with 1.0 weight got id 4
Device d5r1z3-127.0.0.3:6030R127.0.0.3:6030/sdb7_"" with 1.0 weight got id 5
Device d6r1z4-127.0.0.4:6040R127.0.0.4:6040/sdb4_"" with 1.0 weight got id 6
Device d7r1z4-127.0.0.4:6040R127.0.0.4:6040/sdb8_"" with 1.0 weight got id 7
Reassigned 6144 (600.00%) partitions. Balance is now 0.00. Dispersion is now 0.00
Device d0r1z1-127.0.0.1:6011R127.0.0.1:6011/sdb1_"" with 1.0 weight got id 0
Device d1r1z2-127.0.0.2:6021R127.0.0.2:6021/sdb2_"" with 1.0 weight got id 1
Device d2r1z3-127.0.0.3:6031R127.0.0.3:6031/sdb3_"" with 1.0 weight got id 2
Device d3r1z4-127.0.0.4:6041R127.0.0.4:6041/sdb4_"" with 1.0 weight got id 3
Reassigned 3072 (300.00%) partitions. Balance is now 0.00. Dispersion is now 0.00
Device d0r1z1-127.0.0.1:6012R127.0.0.1:6012/sdb1_"" with 1.0 weight got id 0
Device d1r1z2-127.0.0.2:6022R127.0.0.2:6022/sdb2_"" with 1.0 weight got id 1
Device d2r1z3-127.0.0.3:6032R127.0.0.3:6032/sdb3_"" with 1.0 weight got id 2
Device d3r1z4-127.0.0.4:6042R127.0.0.4:6042/sdb4_"" with 1.0 weight got id 3
Reassigned 3072 (300.00%) partitions. Balance is now 0.00. Dispersion is now 0.00
$HOME/swift/.unittests
Note that the unit tests do not require any swift daemons running. #. Read more about Storage Policies and your SAIO :doc:`policies_saio`
#. Start the "main" Swift daemon processes (proxy, account, container, and #. Verify the unit tests run::
object)::
startmain $HOME/swift/.unittests
(The "``Unable to increase file descriptor limit. Running as non-root?``" Note that the unit tests do not require any swift daemons running.
warnings are expected and ok.)
The ``startmain`` script looks like the following: #. Start the "main" Swift daemon processes (proxy, account, container, and
object)::
.. literalinclude:: /../saio/bin/startmain startmain
#. Get an ``X-Storage-Url`` and ``X-Auth-Token``:: (The "``Unable to increase file descriptor limit. Running as non-root?``"
warnings are expected and ok.)
curl -v -H 'X-Storage-User: test:tester' -H 'X-Storage-Pass: testing' http://127.0.0.1:8080/auth/v1.0 The ``startmain`` script looks like the following:
#. Check that you can ``GET`` account:: .. literalinclude:: /../saio/bin/startmain
:language: bash
curl -v -H 'X-Auth-Token: <token-from-x-auth-token-above>' <url-from-x-storage-url-above> #. Get an ``X-Storage-Url`` and ``X-Auth-Token``::
#. Check that ``swift`` command provided by the python-swiftclient package works:: curl -v -H 'X-Storage-User: test:tester' -H 'X-Storage-Pass: testing' http://127.0.0.1:8080/auth/v1.0
swift -A http://127.0.0.1:8080/auth/v1.0 -U test:tester -K testing stat #. Check that you can ``GET`` account::
#. Verify the functional tests run:: curl -v -H 'X-Auth-Token: <token-from-x-auth-token-above>' <url-from-x-storage-url-above>
$HOME/swift/.functests #. Check that ``swift`` command provided by the python-swiftclient package works::
(Note: functional tests will first delete everything in the configured swift -A http://127.0.0.1:8080/auth/v1.0 -U test:tester -K testing stat
accounts.)
#. Verify the probe tests run:: #. Verify the functional tests run::
$HOME/swift/.probetests $HOME/swift/.functests
(Note: probe tests will reset your environment as they call ``resetswift`` (Note: functional tests will first delete everything in the configured
for each test.) accounts.)
#. Verify the probe tests run::
$HOME/swift/.probetests
(Note: probe tests will reset your environment as they call ``resetswift``
for each test.)
---------------- ----------------
Debugging Issues Debugging Issues

8
swift/common/bufferedhttp.py
View File

@ -178,6 +178,14 @@ class BufferedHTTPConnection(HTTPConnection):
return ret return ret
def putrequest(self, method, url, skip_host=0, skip_accept_encoding=0): def putrequest(self, method, url, skip_host=0, skip_accept_encoding=0):
'''Send a request to the server.
:param method: specifies an HTTP request method, e.g. 'GET'.
:param url: specifies the object being requested, e.g. '/index.html'.
:param skip_host: if True does not add automatically a 'Host:' header
:param skip_accept_encoding: if True does not add automatically an
'Accept-Encoding:' header
'''
self._method = method self._method = method
self._path = url self._path = url
return HTTPConnection.putrequest(self, method, url, skip_host, return HTTPConnection.putrequest(self, method, url, skip_host,

2
swift/common/middleware/s3api/controllers/s3_acl.py
View File

@ -13,7 +13,7 @@
# See the License for the specific language governing permissions and # See the License for the specific language governing permissions and
# limitations under the License. # limitations under the License.
from urllib import quote from six.moves.urllib.parse import quote
from swift.common.utils import public from swift.common.utils import public
from swift.common.middleware.s3api.controllers.base import Controller from swift.common.middleware.s3api.controllers.base import Controller