diff --git a/doc/source/development_saio.rst b/doc/source/development_saio.rst index 4b6b6de830..3a1e505d93 100644 --- a/doc/source/development_saio.rst +++ b/doc/source/development_saio.rst @@ -76,9 +76,10 @@ Installing dependencies python2-netifaces python2-pip python2-dnspython \ python2-mock - Note: This installs necessary system dependencies and *most* of the python - dependencies. Later in the process setuptools/distribute or pip will install - and/or upgrade packages. +.. note:: + This installs necessary system dependencies and *most* of the python + dependencies. Later in the process setuptools/distribute or pip will install + and/or upgrade packages. Next, choose either :ref:`partition-section` or :ref:`loopback-section`. @@ -90,50 +91,52 @@ Using a partition for storage If you are going to use a separate partition for Swift data, be sure to add another device when creating the VM, and follow these instructions: - #. Set up a single partition:: +#. Set up a single partition:: - sudo fdisk /dev/sdb - sudo mkfs.xfs /dev/sdb1 + sudo fdisk /dev/sdb + sudo mkfs.xfs /dev/sdb1 - #. Edit ``/etc/fstab`` and add:: +#. Edit ``/etc/fstab`` and add:: - /dev/sdb1 /mnt/sdb1 xfs noatime,nodiratime,nobarrier,logbufs=8 0 0 + /dev/sdb1 /mnt/sdb1 xfs noatime,nodiratime,nobarrier,logbufs=8 0 0 - #. Create the mount point and the individualized links:: +#. Create the mount point and the individualized links:: - sudo mkdir /mnt/sdb1 - sudo mount /mnt/sdb1 - sudo mkdir /mnt/sdb1/1 /mnt/sdb1/2 /mnt/sdb1/3 /mnt/sdb1/4 - sudo chown ${USER}:${USER} /mnt/sdb1/* - sudo mkdir /srv - for x in {1..4}; do sudo ln -s /mnt/sdb1/$x /srv/$x; done - sudo mkdir -p /srv/1/node/sdb1 /srv/1/node/sdb5 \ - /srv/2/node/sdb2 /srv/2/node/sdb6 \ - /srv/3/node/sdb3 /srv/3/node/sdb7 \ - /srv/4/node/sdb4 /srv/4/node/sdb8 \ - /var/run/swift - sudo chown -R ${USER}:${USER} /var/run/swift - # **Make sure to include the trailing slash after /srv/$x/** - for x in {1..4}; do sudo chown -R ${USER}:${USER} /srv/$x/; done + sudo mkdir /mnt/sdb1 + sudo mount /mnt/sdb1 + sudo mkdir /mnt/sdb1/1 /mnt/sdb1/2 /mnt/sdb1/3 /mnt/sdb1/4 + sudo chown ${USER}:${USER} /mnt/sdb1/* + sudo mkdir /srv + for x in {1..4}; do sudo ln -s /mnt/sdb1/$x /srv/$x; done + sudo mkdir -p /srv/1/node/sdb1 /srv/1/node/sdb5 \ + /srv/2/node/sdb2 /srv/2/node/sdb6 \ + /srv/3/node/sdb3 /srv/3/node/sdb7 \ + /srv/4/node/sdb4 /srv/4/node/sdb8 \ + /var/run/swift + sudo chown -R ${USER}:${USER} /var/run/swift + # **Make sure to include the trailing slash after /srv/$x/** + for x in {1..4}; do sudo chown -R ${USER}:${USER} /srv/$x/; done - Note: For OpenSuse users, a user's primary group is `users`, so you have 2 options: + .. note:: + For OpenSuse users, a user's primary group is ``users``, so you have 2 options: - * Change `${USER}:${USER}` to `${USER}:users` in all references of this guide; or - * Create a group for your username and add yourself to it:: + * Change ``${USER}:${USER}`` to ``${USER}:users`` in all references of this guide; or + * Create a group for your username and add yourself to it:: - sudo groupadd ${USER} && sudo gpasswd -a ${USER} ${USER} + sudo groupadd ${USER} && sudo gpasswd -a ${USER} ${USER} - Note: We create the mount points and mount the storage disk under - /mnt/sdb1. This disk will contain one directory per simulated swift node, - each owned by the current swift user. + .. note:: + We create the mount points and mount the storage disk under + /mnt/sdb1. This disk will contain one directory per simulated swift node, + each owned by the current swift user. - We then create symlinks to these directories under /srv. - If the disk sdb is unmounted, files will not be written under - /srv/\*, because the symbolic link destination /mnt/sdb1/* will not - exist. This prevents disk sync operations from writing to the root - partition in the event a drive is unmounted. + We then create symlinks to these directories under /srv. + If the disk sdb is unmounted, files will not be written under + /srv/\*, because the symbolic link destination /mnt/sdb1/* will not + exist. This prevents disk sync operations from writing to the root + partition in the event a drive is unmounted. - #. Next, skip to :ref:`common-dev-section`. +#. Next, skip to :ref:`common-dev-section`. .. _loopback-section: @@ -144,51 +147,53 @@ Using a loopback device for storage If you want to use a loopback device instead of another partition, follow these instructions: - #. Create the file for the loopback device:: +#. Create the file for the loopback device:: - sudo mkdir /srv - sudo truncate -s 1GB /srv/swift-disk - sudo mkfs.xfs /srv/swift-disk + sudo mkdir /srv + sudo truncate -s 1GB /srv/swift-disk + sudo mkfs.xfs /srv/swift-disk - Modify size specified in the ``truncate`` command to make a larger or - smaller partition as needed. + Modify size specified in the ``truncate`` command to make a larger or + smaller partition as needed. - #. Edit `/etc/fstab` and add:: +#. Edit `/etc/fstab` and add:: - /srv/swift-disk /mnt/sdb1 xfs loop,noatime,nodiratime,nobarrier,logbufs=8 0 0 + /srv/swift-disk /mnt/sdb1 xfs loop,noatime,nodiratime,nobarrier,logbufs=8 0 0 - #. Create the mount point and the individualized links:: +#. Create the mount point and the individualized links:: - sudo mkdir /mnt/sdb1 - sudo mount /mnt/sdb1 - sudo mkdir /mnt/sdb1/1 /mnt/sdb1/2 /mnt/sdb1/3 /mnt/sdb1/4 - sudo chown ${USER}:${USER} /mnt/sdb1/* - for x in {1..4}; do sudo ln -s /mnt/sdb1/$x /srv/$x; done - sudo mkdir -p /srv/1/node/sdb1 /srv/1/node/sdb5 \ - /srv/2/node/sdb2 /srv/2/node/sdb6 \ - /srv/3/node/sdb3 /srv/3/node/sdb7 \ - /srv/4/node/sdb4 /srv/4/node/sdb8 \ - /var/run/swift - sudo chown -R ${USER}:${USER} /var/run/swift - # **Make sure to include the trailing slash after /srv/$x/** - for x in {1..4}; do sudo chown -R ${USER}:${USER} /srv/$x/; done + sudo mkdir /mnt/sdb1 + sudo mount /mnt/sdb1 + sudo mkdir /mnt/sdb1/1 /mnt/sdb1/2 /mnt/sdb1/3 /mnt/sdb1/4 + sudo chown ${USER}:${USER} /mnt/sdb1/* + for x in {1..4}; do sudo ln -s /mnt/sdb1/$x /srv/$x; done + sudo mkdir -p /srv/1/node/sdb1 /srv/1/node/sdb5 \ + /srv/2/node/sdb2 /srv/2/node/sdb6 \ + /srv/3/node/sdb3 /srv/3/node/sdb7 \ + /srv/4/node/sdb4 /srv/4/node/sdb8 \ + /var/run/swift + sudo chown -R ${USER}:${USER} /var/run/swift + # **Make sure to include the trailing slash after /srv/$x/** + for x in {1..4}; do sudo chown -R ${USER}:${USER} /srv/$x/; done - Note: For OpenSuse users, a user's primary group is `users`, so you have 2 options: + .. note:: + For OpenSuse users, a user's primary group is ``users``, so you have 2 options: - * Change `${USER}:${USER}` to `${USER}:users` in all references of this guide; or - * Create a group for your username and add yourself to it:: + * Change ``${USER}:${USER}`` to ``${USER}:users`` in all references of this guide; or + * Create a group for your username and add yourself to it:: - sudo groupadd ${USER} && sudo gpasswd -a ${USER} ${USER} + sudo groupadd ${USER} && sudo gpasswd -a ${USER} ${USER} - Note: We create the mount points and mount the loopback file under - /mnt/sdb1. This file will contain one directory per simulated swift node, - each owned by the current swift user. + .. note:: + We create the mount points and mount the loopback file under + /mnt/sdb1. This file will contain one directory per simulated swift node, + each owned by the current swift user. - We then create symlinks to these directories under /srv. - If the loopback file is unmounted, files will not be written under - /srv/\*, because the symbolic link destination /mnt/sdb1/* will not - exist. This prevents disk sync operations from writing to the root - partition in the event a drive is unmounted. + We then create symlinks to these directories under /srv. + If the loopback file is unmounted, files will not be written under + /srv/\*, because the symbolic link destination /mnt/sdb1/* will not + exist. This prevents disk sync operations from writing to the root + partition in the event a drive is unmounted. .. _common-dev-section: @@ -229,117 +234,120 @@ To persist this, edit and add the following to ``/etc/fstab``:: Getting the code ---------------- - #. Check out the python-swiftclient repo:: +#. Check out the python-swiftclient repo:: - cd $HOME; git clone https://github.com/openstack/python-swiftclient.git + cd $HOME; git clone https://github.com/openstack/python-swiftclient.git - #. Build a development installation of python-swiftclient:: +#. Build a development installation of python-swiftclient:: - cd $HOME/python-swiftclient; sudo python setup.py develop; cd - + cd $HOME/python-swiftclient; sudo python setup.py develop; cd - - Ubuntu 12.04 users need to install python-swiftclient's dependencies before the installation of - python-swiftclient. This is due to a bug in an older version of setup tools:: + Ubuntu 12.04 users need to install python-swiftclient's dependencies before the installation of + python-swiftclient. This is due to a bug in an older version of setup tools:: - cd $HOME/python-swiftclient; sudo pip install -r requirements.txt; sudo python setup.py develop; cd - + cd $HOME/python-swiftclient; sudo pip install -r requirements.txt; sudo python setup.py develop; cd - - #. Check out the swift repo:: +#. Check out the swift repo:: - git clone https://github.com/openstack/swift.git + git clone https://github.com/openstack/swift.git - #. Build a development installation of swift:: +#. Build a development installation of swift:: - cd $HOME/swift; sudo pip install --no-binary cryptography -r requirements.txt; sudo python setup.py develop; cd - + cd $HOME/swift; sudo pip install --no-binary cryptography -r requirements.txt; sudo python setup.py develop; cd - - Note: Due to a difference in libssl.so naming in OpenSuse to other Linux distros the wheel/binary wont work so the - cryptography must be built, thus the ``--no-binary cryptography``. + .. note:: + Due to a difference in how ``libssl.so`` is named in OpenSuse vs. other Linux distros the + wheel/binary won't work; thus we use ``--no-binary cryptography`` to build ``cryptography`` + locally. - Fedora 19 or later users might have to perform the following if development - installation of swift fails:: + Fedora 19 or later users might have to perform the following if development + installation of swift fails:: - sudo pip install -U xattr + sudo pip install -U xattr - #. Install swift's test dependencies:: +#. Install swift's test dependencies:: - cd $HOME/swift; sudo pip install -r test-requirements.txt + cd $HOME/swift; sudo pip install -r test-requirements.txt ---------------- Setting up rsync ---------------- - #. Create ``/etc/rsyncd.conf``:: +#. Create ``/etc/rsyncd.conf``:: - sudo cp $HOME/swift/doc/saio/rsyncd.conf /etc/ - sudo sed -i "s//${USER}/" /etc/rsyncd.conf + sudo cp $HOME/swift/doc/saio/rsyncd.conf /etc/ + sudo sed -i "s//${USER}/" /etc/rsyncd.conf - Here is the default ``rsyncd.conf`` file contents maintained in the repo - that is copied and fixed up above: + Here is the default ``rsyncd.conf`` file contents maintained in the repo + that is copied and fixed up above: - .. literalinclude:: /../saio/rsyncd.conf + .. literalinclude:: /../saio/rsyncd.conf + :language: ini - #. On Ubuntu, edit the following line in ``/etc/default/rsync``:: +#. On Ubuntu, edit the following line in ``/etc/default/rsync``:: - RSYNC_ENABLE=true + RSYNC_ENABLE=true - On Fedora, edit the following line in ``/etc/xinetd.d/rsync``:: + On Fedora, edit the following line in ``/etc/xinetd.d/rsync``:: - disable = no + disable = no - One might have to create the above files to perform the edits. + One might have to create the above files to perform the edits. - On OpenSuse, nothing needs to happen here. + On OpenSuse, nothing needs to happen here. - #. On platforms with SELinux in ``Enforcing`` mode, either set to ``Permissive``:: +#. On platforms with SELinux in ``Enforcing`` mode, either set to ``Permissive``:: - sudo setenforce Permissive + sudo setenforce Permissive - Or just allow rsync full access:: + Or just allow rsync full access:: - sudo setsebool -P rsync_full_access 1 + sudo setsebool -P rsync_full_access 1 - #. Start the rsync daemon +#. Start the rsync daemon - * On Ubuntu 14.04, run:: + * On Ubuntu 14.04, run:: - sudo service rsync restart + sudo service rsync restart - * On Ubuntu 16.04, run:: + * On Ubuntu 16.04, run:: - sudo systemctl enable rsync - sudo systemctl start rsync + sudo systemctl enable rsync + sudo systemctl start rsync - * On Fedora, run:: + * On Fedora, run:: - sudo systemctl restart xinetd.service - sudo systemctl enable rsyncd.service - sudo systemctl start rsyncd.service + sudo systemctl restart xinetd.service + sudo systemctl enable rsyncd.service + sudo systemctl start rsyncd.service - * On OpenSuse, run:: + * On OpenSuse, run:: - sudo systemctl enable rsyncd.service - sudo systemctl start rsyncd.service + sudo systemctl enable rsyncd.service + sudo systemctl start rsyncd.service - * On other xinetd based systems simply run:: + * On other xinetd based systems simply run:: - sudo service xinetd restart + sudo service xinetd restart - #. Verify rsync is accepting connections for all servers:: +#. Verify rsync is accepting connections for all servers:: - rsync rsync://pub@localhost/ + rsync rsync://pub@localhost/ - You should see the following output from the above command:: + You should see the following output from the above command:: - account6012 - account6022 - account6032 - account6042 - container6011 - container6021 - container6031 - container6041 - object6010 - object6020 - object6030 - object6040 + account6012 + account6022 + account6032 + account6042 + container6011 + container6021 + container6031 + container6041 + object6010 + object6020 + object6030 + object6040 ------------------ Starting memcached @@ -362,50 +370,51 @@ running, tokens cannot be validated, and accessing Swift becomes impossible. Optional: Setting up rsyslog for individual logging --------------------------------------------------- - #. Install the swift rsyslogd configuration:: +#. Install the swift rsyslogd configuration:: - sudo cp $HOME/swift/doc/saio/rsyslog.d/10-swift.conf /etc/rsyslog.d/ + sudo cp $HOME/swift/doc/saio/rsyslog.d/10-swift.conf /etc/rsyslog.d/ - Note: OpenSuse may have the systemd logger installed, so if you want this - to work, you need to install rsyslog:: + Note: OpenSuse may have the systemd logger installed, so if you want this + to work, you need to install rsyslog:: - sudo zypper install rsyslog - sudo systemctl start rsyslog.service - sudo systemctl enable rsyslog.service + sudo zypper install rsyslog + sudo systemctl start rsyslog.service + sudo systemctl enable rsyslog.service - Be sure to review that conf file to determine if you want all the logs - in one file vs. all the logs separated out, and if you want hourly logs - for stats processing. For convenience, we provide its default contents - below: + Be sure to review that conf file to determine if you want all the logs + in one file vs. all the logs separated out, and if you want hourly logs + for stats processing. For convenience, we provide its default contents + below: - .. literalinclude:: /../saio/rsyslog.d/10-swift.conf + .. literalinclude:: /../saio/rsyslog.d/10-swift.conf + :language: ini - #. Edit ``/etc/rsyslog.conf`` and make the following change (usually in the - "GLOBAL DIRECTIVES" section):: +#. Edit ``/etc/rsyslog.conf`` and make the following change (usually in the + "GLOBAL DIRECTIVES" section):: - $PrivDropToGroup adm + $PrivDropToGroup adm - #. If using hourly logs (see above) perform:: +#. If using hourly logs (see above) perform:: - sudo mkdir -p /var/log/swift/hourly + sudo mkdir -p /var/log/swift/hourly - Otherwise perform:: + Otherwise perform:: - sudo mkdir -p /var/log/swift + sudo mkdir -p /var/log/swift - #. Setup the logging directory and start syslog: +#. Setup the logging directory and start syslog: - * On Ubuntu:: + * On Ubuntu:: - sudo chown -R syslog.adm /var/log/swift - sudo chmod -R g+w /var/log/swift - sudo service rsyslog restart + sudo chown -R syslog.adm /var/log/swift + sudo chmod -R g+w /var/log/swift + sudo service rsyslog restart - * On Fedora and OpenSuse:: + * On Fedora and OpenSuse:: - sudo chown -R root:adm /var/log/swift - sudo chmod -R g+w /var/log/swift - sudo systemctl restart rsyslog.service + sudo chown -R root:adm /var/log/swift + sudo chmod -R g+w /var/log/swift + sudo systemctl restart rsyslog.service --------------------- Configuring each node @@ -415,89 +424,106 @@ After performing the following steps, be sure to verify that Swift has access to resulting configuration files (sample configuration files are provided with all defaults in line-by-line comments). - #. Optionally remove an existing swift directory:: +#. Optionally remove an existing swift directory:: - sudo rm -rf /etc/swift + sudo rm -rf /etc/swift - #. Populate the ``/etc/swift`` directory itself:: +#. Populate the ``/etc/swift`` directory itself:: - cd $HOME/swift/doc; sudo cp -r saio/swift /etc/swift; cd - - sudo chown -R ${USER}:${USER} /etc/swift + cd $HOME/swift/doc; sudo cp -r saio/swift /etc/swift; cd - + sudo chown -R ${USER}:${USER} /etc/swift - #. Update ```` references in the Swift config files:: +#. Update ```` references in the Swift config files:: - find /etc/swift/ -name \*.conf | xargs sudo sed -i "s//${USER}/" + find /etc/swift/ -name \*.conf | xargs sudo sed -i "s//${USER}/" The contents of the configuration files provided by executing the above commands are as follows: - #. ``/etc/swift/swift.conf`` +#. ``/etc/swift/swift.conf`` - .. literalinclude:: /../saio/swift/swift.conf + .. literalinclude:: /../saio/swift/swift.conf + :language: ini - #. ``/etc/swift/proxy-server.conf`` +#. ``/etc/swift/proxy-server.conf`` - .. literalinclude:: /../saio/swift/proxy-server.conf + .. literalinclude:: /../saio/swift/proxy-server.conf + :language: ini - #. ``/etc/swift/object-expirer.conf`` +#. ``/etc/swift/object-expirer.conf`` - .. literalinclude:: /../saio/swift/object-expirer.conf + .. literalinclude:: /../saio/swift/object-expirer.conf + :language: ini - #. ``/etc/swift/container-reconciler.conf`` +#. ``/etc/swift/container-reconciler.conf`` - .. literalinclude:: /../saio/swift/container-reconciler.conf + .. literalinclude:: /../saio/swift/container-reconciler.conf + :language: ini - #. ``/etc/swift/container-sync-realms.conf`` +#. ``/etc/swift/container-sync-realms.conf`` - .. literalinclude:: /../saio/swift/container-sync-realms.conf + .. literalinclude:: /../saio/swift/container-sync-realms.conf + :language: ini - #. ``/etc/swift/account-server/1.conf`` +#. ``/etc/swift/account-server/1.conf`` - .. literalinclude:: /../saio/swift/account-server/1.conf + .. literalinclude:: /../saio/swift/account-server/1.conf + :language: ini - #. ``/etc/swift/container-server/1.conf`` +#. ``/etc/swift/container-server/1.conf`` - .. literalinclude:: /../saio/swift/container-server/1.conf + .. literalinclude:: /../saio/swift/container-server/1.conf + :language: ini - #. ``/etc/swift/object-server/1.conf`` +#. ``/etc/swift/object-server/1.conf`` - .. literalinclude:: /../saio/swift/object-server/1.conf + .. literalinclude:: /../saio/swift/object-server/1.conf + :language: ini - #. ``/etc/swift/account-server/2.conf`` +#. ``/etc/swift/account-server/2.conf`` - .. literalinclude:: /../saio/swift/account-server/2.conf + .. literalinclude:: /../saio/swift/account-server/2.conf + :language: ini - #. ``/etc/swift/container-server/2.conf`` +#. ``/etc/swift/container-server/2.conf`` - .. literalinclude:: /../saio/swift/container-server/2.conf + .. literalinclude:: /../saio/swift/container-server/2.conf + :language: ini - #. ``/etc/swift/object-server/2.conf`` +#. ``/etc/swift/object-server/2.conf`` - .. literalinclude:: /../saio/swift/object-server/2.conf + .. literalinclude:: /../saio/swift/object-server/2.conf + :language: ini - #. ``/etc/swift/account-server/3.conf`` +#. ``/etc/swift/account-server/3.conf`` - .. literalinclude:: /../saio/swift/account-server/3.conf + .. literalinclude:: /../saio/swift/account-server/3.conf + :language: ini - #. ``/etc/swift/container-server/3.conf`` +#. ``/etc/swift/container-server/3.conf`` - .. literalinclude:: /../saio/swift/container-server/3.conf + .. literalinclude:: /../saio/swift/container-server/3.conf + :language: ini - #. ``/etc/swift/object-server/3.conf`` +#. ``/etc/swift/object-server/3.conf`` - .. literalinclude:: /../saio/swift/object-server/3.conf + .. literalinclude:: /../saio/swift/object-server/3.conf + :language: ini - #. ``/etc/swift/account-server/4.conf`` +#. ``/etc/swift/account-server/4.conf`` - .. literalinclude:: /../saio/swift/account-server/4.conf + .. literalinclude:: /../saio/swift/account-server/4.conf + :language: ini - #. ``/etc/swift/container-server/4.conf`` +#. ``/etc/swift/container-server/4.conf`` - .. literalinclude:: /../saio/swift/container-server/4.conf + .. literalinclude:: /../saio/swift/container-server/4.conf + :language: ini - #. ``/etc/swift/object-server/4.conf`` +#. ``/etc/swift/object-server/4.conf`` - .. literalinclude:: /../saio/swift/object-server/4.conf + .. literalinclude:: /../saio/swift/object-server/4.conf + :language: ini .. _setup_scripts: @@ -505,139 +531,146 @@ commands are as follows: Setting up scripts for running Swift ------------------------------------ - #. Copy the SAIO scripts for resetting the environment:: +#. Copy the SAIO scripts for resetting the environment:: - mkdir -p $HOME/bin - cd $HOME/swift/doc; cp saio/bin/* $HOME/bin; cd - - chmod +x $HOME/bin/* + mkdir -p $HOME/bin + cd $HOME/swift/doc; cp saio/bin/* $HOME/bin; cd - + chmod +x $HOME/bin/* - #. Edit the ``$HOME/bin/resetswift`` script +#. Edit the ``$HOME/bin/resetswift`` script - The template ``resetswift`` script looks like the following: + The template ``resetswift`` script looks like the following: - .. literalinclude:: /../saio/bin/resetswift + .. literalinclude:: /../saio/bin/resetswift + :language: bash - If you are using a loopback device add an environment var to - substitute ``/dev/sdb1`` with ``/srv/swift-disk``:: + If you are using a loopback device add an environment var to + substitute ``/dev/sdb1`` with ``/srv/swift-disk``:: - echo "export SAIO_BLOCK_DEVICE=/srv/swift-disk" >> $HOME/.bashrc + echo "export SAIO_BLOCK_DEVICE=/srv/swift-disk" >> $HOME/.bashrc - If you did not set up rsyslog for individual logging, remove the ``find - /var/log/swift...`` line:: + If you did not set up rsyslog for individual logging, remove the ``find + /var/log/swift...`` line:: - sed -i "/find \/var\/log\/swift/d" $HOME/bin/resetswift + sed -i "/find \/var\/log\/swift/d" $HOME/bin/resetswift - #. Install the sample configuration file for running tests:: +#. Install the sample configuration file for running tests:: - cp $HOME/swift/test/sample.conf /etc/swift/test.conf + cp $HOME/swift/test/sample.conf /etc/swift/test.conf - The template ``test.conf`` looks like the following: + The template ``test.conf`` looks like the following: - .. literalinclude:: /../../test/sample.conf + .. literalinclude:: /../../test/sample.conf + :language: ini - #. Add an environment variable for running tests below:: +#. Add an environment variable for running tests below:: - echo "export SWIFT_TEST_CONFIG_FILE=/etc/swift/test.conf" >> $HOME/.bashrc + echo "export SWIFT_TEST_CONFIG_FILE=/etc/swift/test.conf" >> $HOME/.bashrc - #. Be sure that your ``PATH`` includes the ``bin`` directory:: +#. Be sure that your ``PATH`` includes the ``bin`` directory:: - echo "export PATH=${PATH}:$HOME/bin" >> $HOME/.bashrc + echo "export PATH=${PATH}:$HOME/bin" >> $HOME/.bashrc - #. Source the above environment variables into your current environment:: +#. Source the above environment variables into your current environment:: - . $HOME/.bashrc + . $HOME/.bashrc - #. Construct the initial rings using the provided script:: +#. Construct the initial rings using the provided script:: - remakerings + remakerings - The ``remakerings`` script looks like the following: + The ``remakerings`` script looks like the following: - .. literalinclude:: /../saio/bin/remakerings + .. literalinclude:: /../saio/bin/remakerings + :language: bash - You can expect the output from this command to produce the following. Note - that 3 object rings are created in order to test storage policies and EC in - the SAIO environment. The EC ring is the only one with all 8 devices. - There are also two replication rings, one for 3x replication and another - for 2x replication, but those rings only use 4 devices:: - - Device d0r1z1-127.0.0.1:6010R127.0.0.1:6010/sdb1_"" with 1.0 weight got id 0 - Device d1r1z2-127.0.0.2:6020R127.0.0.2:6020/sdb2_"" with 1.0 weight got id 1 - Device d2r1z3-127.0.0.3:6030R127.0.0.3:6030/sdb3_"" with 1.0 weight got id 2 - Device d3r1z4-127.0.0.4:6040R127.0.0.4:6040/sdb4_"" with 1.0 weight got id 3 - Reassigned 3072 (300.00%) partitions. Balance is now 0.00. Dispersion is now 0.00 - Device d0r1z1-127.0.0.1:6010R127.0.0.1:6010/sdb1_"" with 1.0 weight got id 0 - Device d1r1z2-127.0.0.2:6020R127.0.0.2:6020/sdb2_"" with 1.0 weight got id 1 - Device d2r1z3-127.0.0.3:6030R127.0.0.3:6030/sdb3_"" with 1.0 weight got id 2 - Device d3r1z4-127.0.0.4:6040R127.0.0.4:6040/sdb4_"" with 1.0 weight got id 3 - Reassigned 2048 (200.00%) partitions. Balance is now 0.00. Dispersion is now 0.00 - Device d0r1z1-127.0.0.1:6010R127.0.0.1:6010/sdb1_"" with 1.0 weight got id 0 - Device d1r1z1-127.0.0.1:6010R127.0.0.1:6010/sdb5_"" with 1.0 weight got id 1 - Device d2r1z2-127.0.0.2:6020R127.0.0.2:6020/sdb2_"" with 1.0 weight got id 2 - Device d3r1z2-127.0.0.2:6020R127.0.0.2:6020/sdb6_"" with 1.0 weight got id 3 - Device d4r1z3-127.0.0.3:6030R127.0.0.3:6030/sdb3_"" with 1.0 weight got id 4 - Device d5r1z3-127.0.0.3:6030R127.0.0.3:6030/sdb7_"" with 1.0 weight got id 5 - Device d6r1z4-127.0.0.4:6040R127.0.0.4:6040/sdb4_"" with 1.0 weight got id 6 - Device d7r1z4-127.0.0.4:6040R127.0.0.4:6040/sdb8_"" with 1.0 weight got id 7 - Reassigned 6144 (600.00%) partitions. Balance is now 0.00. Dispersion is now 0.00 - Device d0r1z1-127.0.0.1:6011R127.0.0.1:6011/sdb1_"" with 1.0 weight got id 0 - Device d1r1z2-127.0.0.2:6021R127.0.0.2:6021/sdb2_"" with 1.0 weight got id 1 - Device d2r1z3-127.0.0.3:6031R127.0.0.3:6031/sdb3_"" with 1.0 weight got id 2 - Device d3r1z4-127.0.0.4:6041R127.0.0.4:6041/sdb4_"" with 1.0 weight got id 3 - Reassigned 3072 (300.00%) partitions. Balance is now 0.00. Dispersion is now 0.00 - Device d0r1z1-127.0.0.1:6012R127.0.0.1:6012/sdb1_"" with 1.0 weight got id 0 - Device d1r1z2-127.0.0.2:6022R127.0.0.2:6022/sdb2_"" with 1.0 weight got id 1 - Device d2r1z3-127.0.0.3:6032R127.0.0.3:6032/sdb3_"" with 1.0 weight got id 2 - Device d3r1z4-127.0.0.4:6042R127.0.0.4:6042/sdb4_"" with 1.0 weight got id 3 - Reassigned 3072 (300.00%) partitions. Balance is now 0.00. Dispersion is now 0.00 + You can expect the output from this command to produce the following. Note + that 3 object rings are created in order to test storage policies and EC in + the SAIO environment. The EC ring is the only one with all 8 devices. + There are also two replication rings, one for 3x replication and another + for 2x replication, but those rings only use 4 devices: - #. Read more about Storage Policies and your SAIO :doc:`policies_saio` + .. code-block:: console - #. Verify the unit tests run:: + Device d0r1z1-127.0.0.1:6010R127.0.0.1:6010/sdb1_"" with 1.0 weight got id 0 + Device d1r1z2-127.0.0.2:6020R127.0.0.2:6020/sdb2_"" with 1.0 weight got id 1 + Device d2r1z3-127.0.0.3:6030R127.0.0.3:6030/sdb3_"" with 1.0 weight got id 2 + Device d3r1z4-127.0.0.4:6040R127.0.0.4:6040/sdb4_"" with 1.0 weight got id 3 + Reassigned 3072 (300.00%) partitions. Balance is now 0.00. Dispersion is now 0.00 + Device d0r1z1-127.0.0.1:6010R127.0.0.1:6010/sdb1_"" with 1.0 weight got id 0 + Device d1r1z2-127.0.0.2:6020R127.0.0.2:6020/sdb2_"" with 1.0 weight got id 1 + Device d2r1z3-127.0.0.3:6030R127.0.0.3:6030/sdb3_"" with 1.0 weight got id 2 + Device d3r1z4-127.0.0.4:6040R127.0.0.4:6040/sdb4_"" with 1.0 weight got id 3 + Reassigned 2048 (200.00%) partitions. Balance is now 0.00. Dispersion is now 0.00 + Device d0r1z1-127.0.0.1:6010R127.0.0.1:6010/sdb1_"" with 1.0 weight got id 0 + Device d1r1z1-127.0.0.1:6010R127.0.0.1:6010/sdb5_"" with 1.0 weight got id 1 + Device d2r1z2-127.0.0.2:6020R127.0.0.2:6020/sdb2_"" with 1.0 weight got id 2 + Device d3r1z2-127.0.0.2:6020R127.0.0.2:6020/sdb6_"" with 1.0 weight got id 3 + Device d4r1z3-127.0.0.3:6030R127.0.0.3:6030/sdb3_"" with 1.0 weight got id 4 + Device d5r1z3-127.0.0.3:6030R127.0.0.3:6030/sdb7_"" with 1.0 weight got id 5 + Device d6r1z4-127.0.0.4:6040R127.0.0.4:6040/sdb4_"" with 1.0 weight got id 6 + Device d7r1z4-127.0.0.4:6040R127.0.0.4:6040/sdb8_"" with 1.0 weight got id 7 + Reassigned 6144 (600.00%) partitions. Balance is now 0.00. Dispersion is now 0.00 + Device d0r1z1-127.0.0.1:6011R127.0.0.1:6011/sdb1_"" with 1.0 weight got id 0 + Device d1r1z2-127.0.0.2:6021R127.0.0.2:6021/sdb2_"" with 1.0 weight got id 1 + Device d2r1z3-127.0.0.3:6031R127.0.0.3:6031/sdb3_"" with 1.0 weight got id 2 + Device d3r1z4-127.0.0.4:6041R127.0.0.4:6041/sdb4_"" with 1.0 weight got id 3 + Reassigned 3072 (300.00%) partitions. Balance is now 0.00. Dispersion is now 0.00 + Device d0r1z1-127.0.0.1:6012R127.0.0.1:6012/sdb1_"" with 1.0 weight got id 0 + Device d1r1z2-127.0.0.2:6022R127.0.0.2:6022/sdb2_"" with 1.0 weight got id 1 + Device d2r1z3-127.0.0.3:6032R127.0.0.3:6032/sdb3_"" with 1.0 weight got id 2 + Device d3r1z4-127.0.0.4:6042R127.0.0.4:6042/sdb4_"" with 1.0 weight got id 3 + Reassigned 3072 (300.00%) partitions. Balance is now 0.00. Dispersion is now 0.00 - $HOME/swift/.unittests - Note that the unit tests do not require any swift daemons running. +#. Read more about Storage Policies and your SAIO :doc:`policies_saio` - #. Start the "main" Swift daemon processes (proxy, account, container, and - object):: +#. Verify the unit tests run:: - startmain + $HOME/swift/.unittests - (The "``Unable to increase file descriptor limit. Running as non-root?``" - warnings are expected and ok.) + Note that the unit tests do not require any swift daemons running. - The ``startmain`` script looks like the following: +#. Start the "main" Swift daemon processes (proxy, account, container, and + object):: - .. literalinclude:: /../saio/bin/startmain + startmain - #. Get an ``X-Storage-Url`` and ``X-Auth-Token``:: + (The "``Unable to increase file descriptor limit. Running as non-root?``" + warnings are expected and ok.) - curl -v -H 'X-Storage-User: test:tester' -H 'X-Storage-Pass: testing' http://127.0.0.1:8080/auth/v1.0 + The ``startmain`` script looks like the following: - #. Check that you can ``GET`` account:: + .. literalinclude:: /../saio/bin/startmain + :language: bash - curl -v -H 'X-Auth-Token: ' +#. Get an ``X-Storage-Url`` and ``X-Auth-Token``:: - #. Check that ``swift`` command provided by the python-swiftclient package works:: + curl -v -H 'X-Storage-User: test:tester' -H 'X-Storage-Pass: testing' http://127.0.0.1:8080/auth/v1.0 - swift -A http://127.0.0.1:8080/auth/v1.0 -U test:tester -K testing stat +#. Check that you can ``GET`` account:: - #. Verify the functional tests run:: + curl -v -H 'X-Auth-Token: ' - $HOME/swift/.functests +#. Check that ``swift`` command provided by the python-swiftclient package works:: - (Note: functional tests will first delete everything in the configured - accounts.) + swift -A http://127.0.0.1:8080/auth/v1.0 -U test:tester -K testing stat - #. Verify the probe tests run:: +#. Verify the functional tests run:: - $HOME/swift/.probetests + $HOME/swift/.functests - (Note: probe tests will reset your environment as they call ``resetswift`` - for each test.) + (Note: functional tests will first delete everything in the configured + accounts.) + +#. Verify the probe tests run:: + + $HOME/swift/.probetests + + (Note: probe tests will reset your environment as they call ``resetswift`` + for each test.) ---------------- Debugging Issues diff --git a/swift/common/bufferedhttp.py b/swift/common/bufferedhttp.py index 402345b3d2..de76ec7eec 100644 --- a/swift/common/bufferedhttp.py +++ b/swift/common/bufferedhttp.py @@ -178,6 +178,14 @@ class BufferedHTTPConnection(HTTPConnection): return ret def putrequest(self, method, url, skip_host=0, skip_accept_encoding=0): + '''Send a request to the server. + + :param method: specifies an HTTP request method, e.g. 'GET'. + :param url: specifies the object being requested, e.g. '/index.html'. + :param skip_host: if True does not add automatically a 'Host:' header + :param skip_accept_encoding: if True does not add automatically an + 'Accept-Encoding:' header + ''' self._method = method self._path = url return HTTPConnection.putrequest(self, method, url, skip_host, diff --git a/swift/common/middleware/s3api/controllers/s3_acl.py b/swift/common/middleware/s3api/controllers/s3_acl.py index a99c85b708..2b21570bb5 100644 --- a/swift/common/middleware/s3api/controllers/s3_acl.py +++ b/swift/common/middleware/s3api/controllers/s3_acl.py @@ -13,7 +13,7 @@ # See the License for the specific language governing permissions and # limitations under the License. -from urllib import quote +from six.moves.urllib.parse import quote from swift.common.utils import public from swift.common.middleware.s3api.controllers.base import Controller