swift/doc/source/development_saio.rst

SAIO - Swift All In One

Instructions for setting up a development VM

This section documents setting up a virtual machine for doing Swift development. The virtual machine will emulate running a four node Swift cluster.

• Get either Ubuntu 12.04 LTS (Precise Pangolin) or Ubuntu 10.04 LTS (Lucid Lynx) server image.
• Create guest virtual machine from the Ubuntu image.

Additional information about setting up a Swift development snapshot on other distributions is available on the wiki at http://wiki.openstack.org/SAIOInstructions.

Installing dependencies and the core code

• As root on guest (you'll have to log in as you, then sudo su -):

  1. apt-get install python-software-properties
  2. add-apt-repository ppa:swift-core/release
  3. apt-get update
  4. apt-get install curl gcc git-core memcached python-coverage python-dev python-nose python-setuptools python-simplejson python-xattr sqlite3 xfsprogs python-eventlet python-greenlet python-pastedeploy python-netifaces python-pip
  5. pip install mock
  6. Install anything else you want, like screen, ssh, vim, etc.

• On Fedora, log in as root and do:

  1. yum install openstack-swift openstack-swift-proxy openstack-swift-account openstack-swift-container openstack-swift-object
  2. yum install xinetd rsync
  3. yum install memcached
  4. yum install python-netifaces python-nose python-mock

  This installs all necessary dependencies, and also creates user swift and group swift. So, swift:swift ought to be used in every place where this manual calls for <your-user-name>:<your-group-name>.

  Ensure that you are installing the version of Swift that corresponds to this document. If not, enable the correct update repositories.

Next, choose either partition-section or loopback-section.

Using a partition for storage

If you are going to use a separate partition for Swift data, be sure to add another device when creating the VM, and follow these instructions.

1. fdisk /dev/sdb (set up a single partition)

2. mkfs.xfs -i size=1024 /dev/sdb1

3. Edit /etc/fstab and add

   /dev/sdb1 /mnt/sdb1 xfs noatime,nodiratime,nobarrier,logbufs=8 0 0

4. mkdir /mnt/sdb1

5. mount /mnt/sdb1

6. mkdir /mnt/sdb1/1 /mnt/sdb1/2 /mnt/sdb1/3 /mnt/sdb1/4

7. chown <your-user-name>:<your-group-name> /mnt/sdb1/*

8. mkdir /srv

9. for x in {1..4}; do ln -s /mnt/sdb1/$x /srv/$x; done

10. mkdir -p /etc/swift/object-server /etc/swift/container-server /etc/swift/account-server /srv/1/node/sdb1 /srv/2/node/sdb2 /srv/3/node/sdb3 /srv/4/node/sdb4 /var/run/swift

11. chown -R <your-user-name>:<your-group-name> /etc/swift /srv/[1-4]/ /var/run/swift -- Make sure to include the trailing slash after /srv/[1-4]/

12. Add to /etc/rc.local (before the exit 0):

    mkdir -p /var/cache/swift /var/cache/swift2 /var/cache/swift3 /var/cache/swift4
    chown <your-user-name>:<your-group-name> /var/cache/swift*
    mkdir -p /var/run/swift
    chown <your-user-name>:<your-group-name> /var/run/swift

13. Next, skip to rsync-section.

Using a loopback device for storage

If you want to use a loopback device instead of another partition, follow these instructions.

1. mkdir /srv

2. truncate -s 1GB /srv/swift-disk

   (modify size to make a larger or smaller partition)

3. mkfs.xfs -i size=1024 /srv/swift-disk

4. Edit /etc/fstab and add

   /srv/swift-disk /mnt/sdb1 xfs loop,noatime,nodiratime,nobarrier,logbufs=8 0 0

5. mkdir /mnt/sdb1

6. mount /mnt/sdb1

7. mkdir /mnt/sdb1/1 /mnt/sdb1/2 /mnt/sdb1/3 /mnt/sdb1/4

8. chown <your-user-name>:<your-group-name> /mnt/sdb1/*

9. for x in {1..4}; do ln -s /mnt/sdb1/$x /srv/$x; done

10. mkdir -p /etc/swift/object-server /etc/swift/container-server /etc/swift/account-server /srv/1/node/sdb1 /srv/2/node/sdb2 /srv/3/node/sdb3 /srv/4/node/sdb4 /var/run/swift

11. chown -R <your-user-name>:<your-group-name> /etc/swift /srv/[1-4]/ /var/run/swift -- Make sure to include the trailing slash after /srv/[1-4]/

12. Add to /etc/rc.local (before the exit 0):

    mkdir -p /var/cache/swift /var/cache/swift2 /var/cache/swift3 /var/cache/swift4
    chown <your-user-name>:<your-group-name> /var/cache/swift*
    mkdir -p /var/run/swift
    chown <your-user-name>:<your-group-name> /var/run/swift

Setting up rsync

1. Create /etc/rsyncd.conf:

   uid = <Your user name>
   gid = <Your group name>
   log file = /var/log/rsyncd.log
   pid file = /var/run/rsyncd.pid
   address = 127.0.0.1
   
   [account6012]
   max connections = 25
   path = /srv/1/node/
   read only = false
   lock file = /var/lock/account6012.lock
   
   [account6022]
   max connections = 25
   path = /srv/2/node/
   read only = false
   lock file = /var/lock/account6022.lock
   
   [account6032]
   max connections = 25
   path = /srv/3/node/
   read only = false
   lock file = /var/lock/account6032.lock
   
   [account6042]
   max connections = 25
   path = /srv/4/node/
   read only = false
   lock file = /var/lock/account6042.lock
   
   
   [container6011]
   max connections = 25
   path = /srv/1/node/
   read only = false
   lock file = /var/lock/container6011.lock
   
   [container6021]
   max connections = 25
   path = /srv/2/node/
   read only = false
   lock file = /var/lock/container6021.lock
   
   [container6031]
   max connections = 25
   path = /srv/3/node/
   read only = false
   lock file = /var/lock/container6031.lock
   
   [container6041]
   max connections = 25
   path = /srv/4/node/
   read only = false
   lock file = /var/lock/container6041.lock
   
   [object6010]
   max connections = 25
   path = /srv/1/node/
   read only = false
   lock file = /var/lock/object6010.lock
   
   [object6020]
   max connections = 25
   path = /srv/2/node/
   read only = false
   lock file = /var/lock/object6020.lock
   
   [object6030]
   max connections = 25
   path = /srv/3/node/
   read only = false
   lock file = /var/lock/object6030.lock
   
   [object6040]
   max connections = 25
   path = /srv/4/node/
   read only = false
   lock file = /var/lock/object6040.lock

2. On Ubuntu, edit the following line in /etc/default/rsync:

   RSYNC_ENABLE=true

   On Fedora, edit the following line in /etc/xinetd.d/rsync:

   disable = no

3. On Ubuntu service rsync restart

Starting memcached

On Fedora, make sure that memcached runs, running this if necessary:

• systemctl enable memcached.service
• systemctl start memcached.service

If this is not done, tokens of tempauth expire immediately and accessing Swift with curl becomes impossible.

Optional: Setting up rsyslog for individual logging

1. Create /etc/rsyslog.d/10-swift.conf:

   # Uncomment the following to have a log containing all logs together
   #local1,local2,local3,local4,local5.*   /var/log/swift/all.log
   
   # Uncomment the following to have hourly proxy logs for stats processing
   #$template HourlyProxyLog,"/var/log/swift/hourly/%$YEAR%%$MONTH%%$DAY%%$HOUR%"
   #local1.*;local1.!notice ?HourlyProxyLog
   
   local1.*;local1.!notice /var/log/swift/proxy.log
   local1.notice           /var/log/swift/proxy.error
   local1.*                ~
   
   local2.*;local2.!notice /var/log/swift/storage1.log
   local2.notice           /var/log/swift/storage1.error
   local2.*                ~
   
   local3.*;local3.!notice /var/log/swift/storage2.log
   local3.notice           /var/log/swift/storage2.error
   local3.*                ~
   
   local4.*;local4.!notice /var/log/swift/storage3.log
   local4.notice           /var/log/swift/storage3.error
   local4.*                ~
   
   local5.*;local5.!notice /var/log/swift/storage4.log
   local5.notice           /var/log/swift/storage4.error
   local5.*                ~

2. Edit /etc/rsyslog.conf and make the following change:

   $PrivDropToGroup adm

3. mkdir -p /var/log/swift/hourly

4. chown -R syslog.adm /var/log/swift

5. chmod -R g+w /var/log/swift

6. service rsyslog restart

Getting the code and setting up test environment

Sample configuration files are provided with all defaults in line-by-line comments.

Do these commands as you on guest.

1. mkdir ~/bin

2. Check out the swift repo with git clone https://github.com/openstack/swift.git

3. Build a development installation of swift, for example: cd ~/swift; sudo python setup.py develop

4. Check out the python-swiftclient repo with git clone https://github.com/openstack/python-swiftclient.git

5. Build a development installation of python-swiftclient, for example: cd ~/python-swiftclient; sudo python setup.py develop

6. Edit ~/.bashrc and add to the end:

   export SWIFT_TEST_CONFIG_FILE=/etc/swift/test.conf
   export PATH=${PATH}:~/bin

7. . ~/.bashrc

Configuring each node

Sample configuration files are provided with all defaults in line-by-line comments.

1. Create `/etc/swift/proxy-server.conf`:

   [DEFAULT]
   bind_port = 8080
   user = <your-user-name>
   log_facility = LOG_LOCAL1
   eventlet_debug = true
   
   [pipeline:main]
   pipeline = healthcheck cache tempauth proxy-logging proxy-server
   
   [app:proxy-server]
   use = egg:swift#proxy
   allow_account_management = true
   account_autocreate = true
   
   [filter:tempauth]
   use = egg:swift#tempauth
   user_admin_admin = admin .admin .reseller_admin
   user_test_tester = testing .admin
   user_test2_tester2 = testing2 .admin
   user_test_tester3 = testing3
   
   [filter:healthcheck]
   use = egg:swift#healthcheck
   
   [filter:cache]
   use = egg:swift#memcache
   
   [filter:proxy-logging]
   use = egg:swift#proxy_logging

2. Create `/etc/swift/swift.conf`:

   [swift-hash]
   # random unique strings that can never change (DO NOT LOSE)
   swift_hash_path_prefix = changeme
   swift_hash_path_suffix = changeme

3. Create `/etc/swift/account-server/1.conf`:

   [DEFAULT]
   devices = /srv/1/node
   mount_check = false
   disable_fallocate = true
   bind_port = 6012
   user = <your-user-name>
   log_facility = LOG_LOCAL2
   recon_cache_path = /var/cache/swift
   eventlet_debug = true
   
   [pipeline:main]
   pipeline = recon account-server
   
   [app:account-server]
   use = egg:swift#account
   
   [filter:recon]
   use = egg:swift#recon
   
   [account-replicator]
   vm_test_mode = yes
   
   [account-auditor]
   
   [account-reaper]

4. Create `/etc/swift/account-server/2.conf`:

   [DEFAULT]
   devices = /srv/2/node
   mount_check = false
   disable_fallocate = true
   bind_port = 6022
   user = <your-user-name>
   log_facility = LOG_LOCAL3
   recon_cache_path = /var/cache/swift2
   eventlet_debug = true
   
   [pipeline:main]
   pipeline = recon account-server
   
   [app:account-server]
   use = egg:swift#account
   
   [filter:recon]
   use = egg:swift#recon
   
   [account-replicator]
   vm_test_mode = yes
   
   [account-auditor]
   
   [account-reaper]

5. Create `/etc/swift/account-server/3.conf`:

   [DEFAULT]
   devices = /srv/3/node
   mount_check = false
   disable_fallocate = true
   bind_port = 6032
   user = <your-user-name>
   log_facility = LOG_LOCAL4
   recon_cache_path = /var/cache/swift3
   eventlet_debug = true
   
   [pipeline:main]
   pipeline = recon account-server
   
   [app:account-server]
   use = egg:swift#account
   
   [filter:recon]
   use = egg:swift#recon
   
   [account-replicator]
   vm_test_mode = yes
   
   [account-auditor]
   
   [account-reaper]

6. Create `/etc/swift/account-server/4.conf`:

   [DEFAULT]
   devices = /srv/4/node
   mount_check = false
   disable_fallocate = true
   bind_port = 6042
   user = <your-user-name>
   log_facility = LOG_LOCAL5
   recon_cache_path = /var/cache/swift4
   eventlet_debug = true
   
   [pipeline:main]
   pipeline = recon account-server
   
   [app:account-server]
   use = egg:swift#account
   
   [filter:recon]
   use = egg:swift#recon
   
   [account-replicator]
   vm_test_mode = yes
   
   [account-auditor]
   
   [account-reaper]

7. Create `/etc/swift/container-server/1.conf`:

   [DEFAULT]
   devices = /srv/1/node
   mount_check = false
   disable_fallocate = true
   bind_port = 6011
   user = <your-user-name>
   log_facility = LOG_LOCAL2
   recon_cache_path = /var/cache/swift
   eventlet_debug = true
   
   [pipeline:main]
   pipeline = recon container-server
   
   [app:container-server]
   use = egg:swift#container
   
   [filter:recon]
   use = egg:swift#recon
   
   [container-replicator]
   vm_test_mode = yes
   
   [container-updater]
   
   [container-auditor]
   
   [container-sync]

8. Create `/etc/swift/container-server/2.conf`:

   [DEFAULT]
   devices = /srv/2/node
   mount_check = false
   disable_fallocate = true
   bind_port = 6021
   user = <your-user-name>
   log_facility = LOG_LOCAL3
   recon_cache_path = /var/cache/swift2
   eventlet_debug = true
   
   [pipeline:main]
   pipeline = recon container-server
   
   [app:container-server]
   use = egg:swift#container
   
   [filter:recon]
   use = egg:swift#recon
   
   [container-replicator]
   vm_test_mode = yes
   
   [container-updater]
   
   [container-auditor]
   
   [container-sync]

9. Create `/etc/swift/container-server/3.conf`:

   [DEFAULT]
   devices = /srv/3/node
   mount_check = false
   disable_fallocate = true
   bind_port = 6031
   user = <your-user-name>
   log_facility = LOG_LOCAL4
   recon_cache_path = /var/cache/swift3
   eventlet_debug = true
   
   [pipeline:main]
   pipeline = recon container-server
   
   [app:container-server]
   use = egg:swift#container
   
   [filter:recon]
   use = egg:swift#recon
   
   [container-replicator]
   vm_test_mode = yes
   
   [container-updater]
   
   [container-auditor]
   
   [container-sync]

10. Create `/etc/swift/container-server/4.conf`:

    [DEFAULT]
    devices = /srv/4/node
    mount_check = false
    disable_fallocate = true
    bind_port = 6041
    user = <your-user-name>
    log_facility = LOG_LOCAL5
    recon_cache_path = /var/cache/swift4
    eventlet_debug = true
    
    [pipeline:main]
    pipeline = recon container-server
    
    [app:container-server]
    use = egg:swift#container
    
    [filter:recon]
    use = egg:swift#recon
    
    [container-replicator]
    vm_test_mode = yes
    
    [container-updater]
    
    [container-auditor]
    
    [container-sync]

11. Create `/etc/swift/object-server/1.conf`:

    [DEFAULT]
    devices = /srv/1/node
    mount_check = false
    disable_fallocate = true
    bind_port = 6010
    user = <your-user-name>
    log_facility = LOG_LOCAL2
    recon_cache_path = /var/cache/swift
    eventlet_debug = true
    
    [pipeline:main]
    pipeline = recon object-server
    
    [app:object-server]
    use = egg:swift#object
    
    [filter:recon]
    use = egg:swift#recon
    
    [object-replicator]
    vm_test_mode = yes
    
    [object-updater]
    
    [object-auditor]

12. Create `/etc/swift/object-server/2.conf`:

    [DEFAULT]
    devices = /srv/2/node
    mount_check = false
    disable_fallocate = true
    bind_port = 6020
    user = <your-user-name>
    log_facility = LOG_LOCAL3
    recon_cache_path = /var/cache/swift2
    eventlet_debug = true
    
    [pipeline:main]
    pipeline = recon object-server
    
    [app:object-server]
    use = egg:swift#object
    
    [filter:recon]
    use = egg:swift#recon
    
    [object-replicator]
    vm_test_mode = yes
    
    [object-updater]
    
    [object-auditor]

13. Create `/etc/swift/object-server/3.conf`:

    [DEFAULT]
    devices = /srv/3/node
    mount_check = false
    disable_fallocate = true
    bind_port = 6030
    user = <your-user-name>
    log_facility = LOG_LOCAL4
    recon_cache_path = /var/cache/swift3
    eventlet_debug = true
    
    [pipeline:main]
    pipeline = recon object-server
    
    [app:object-server]
    use = egg:swift#object
    
    [filter:recon]
    use = egg:swift#recon
    
    [object-replicator]
    vm_test_mode = yes
    
    [object-updater]
    
    [object-auditor]

14. Create `/etc/swift/object-server/4.conf`:

    [DEFAULT]
    devices = /srv/4/node
    mount_check = false
    disable_fallocate = true
    bind_port = 6040
    user = <your-user-name>
    log_facility = LOG_LOCAL5
    recon_cache_path = /var/cache/swift4
    eventlet_debug = true
    
    [pipeline:main]
    pipeline = recon object-server
    
    [app:object-server]
    use = egg:swift#object
    
    [filter:recon]
    use = egg:swift#recon
    
    [object-replicator]
    vm_test_mode = yes
    
    [object-updater]
    
    [object-auditor]

Setting up scripts for running Swift

1. Create ~/bin/resetswift.

   If you are using a loopback device substitute /dev/sdb1 with /srv/swift-disk.

   If you did not set up rsyslog for individual logging, remove the find /var/log/swift... line:

   #!/bin/bash
   
   swift-init all stop
   find /var/log/swift -type f -exec rm -f {} \;
   sudo umount /mnt/sdb1
   sudo mkfs.xfs -f -i size=1024 /dev/sdb1
   sudo mount /mnt/sdb1
   sudo mkdir /mnt/sdb1/1 /mnt/sdb1/2 /mnt/sdb1/3 /mnt/sdb1/4
   sudo chown <your-user-name>:<your-group-name> /mnt/sdb1/*
   mkdir -p /srv/1/node/sdb1 /srv/2/node/sdb2 /srv/3/node/sdb3 /srv/4/node/sdb4
   sudo rm -f /var/log/debug /var/log/messages /var/log/rsyncd.log /var/log/syslog
   find /var/cache/swift* -type f -name *.recon -exec rm -f {} \;
   sudo service rsyslog restart
   sudo service memcached restart

2. Create `~/bin/remakerings`:

   #!/bin/bash
   
   cd /etc/swift
   
   rm -f *.builder *.ring.gz backups/*.builder backups/*.ring.gz
   
   swift-ring-builder object.builder create 18 3 1
   swift-ring-builder object.builder add z1-127.0.0.1:6010/sdb1 1
   swift-ring-builder object.builder add z2-127.0.0.1:6020/sdb2 1
   swift-ring-builder object.builder add z3-127.0.0.1:6030/sdb3 1
   swift-ring-builder object.builder add z4-127.0.0.1:6040/sdb4 1
   swift-ring-builder object.builder rebalance
   swift-ring-builder container.builder create 18 3 1
   swift-ring-builder container.builder add z1-127.0.0.1:6011/sdb1 1
   swift-ring-builder container.builder add z2-127.0.0.1:6021/sdb2 1
   swift-ring-builder container.builder add z3-127.0.0.1:6031/sdb3 1
   swift-ring-builder container.builder add z4-127.0.0.1:6041/sdb4 1
   swift-ring-builder container.builder rebalance
   swift-ring-builder account.builder create 18 3 1
   swift-ring-builder account.builder add z1-127.0.0.1:6012/sdb1 1
   swift-ring-builder account.builder add z2-127.0.0.1:6022/sdb2 1
   swift-ring-builder account.builder add z3-127.0.0.1:6032/sdb3 1
   swift-ring-builder account.builder add z4-127.0.0.1:6042/sdb4 1
   swift-ring-builder account.builder rebalance

3. Create `~/bin/startmain`:

   #!/bin/bash
   
   swift-init main start

4. Create `~/bin/startrest`:

   #!/bin/bash
   
   swift-init rest start

5. chmod +x ~/bin/*

6. remakerings

7. cp ~/swift/test/sample.conf /etc/swift/test.conf

8. cd ~/swift; ./.unittests

9. startmain (The Unable to increase file descriptor limit. Running as non-root? warnings are expected and ok.)

10. Get an X-Storage-Url and `X-Auth-Token`: curl -v -H 'X-Storage-User: test:tester' -H 'X-Storage-Pass: testing' http://127.0.0.1:8080/auth/v1.0

11. Check that you can GET account: curl -v -H 'X-Auth-Token: <token-from-x-auth-token-above>' <url-from-x-storage-url-above>

12. Check that swift works: swift -A http://127.0.0.1:8080/auth/v1.0 -U test:tester -K testing stat

13. cd ~/swift; ./.functests (Note: functional tests will first delete everything in the configured accounts.)

14. cd ~/swift; ./.probetests (Note: probe tests will reset your environment as they call resetswift for each test.)

Debugging Issues

If all doesn't go as planned, and tests fail, or you can't auth, or something doesn't work, here are some good starting places to look for issues:

1. Everything is logged using system facilities -- usually in /var/log/syslog, but possibly in /var/log/messages on e.g. Fedora -- so that is a good first place to look for errors (most likely python tracebacks).
2. Make sure all of the server processes are running. For the base functionality, the Proxy, Account, Container, and Object servers should be running.
3. If one of the servers are not running, and no errors are logged to syslog, it may be useful to try to start the server manually, for example: swift-object-server /etc/swift/object-server/1.conf will start the object server. If there are problems not showing up in syslog, then you will likely see the traceback on startup.
4. If you need to, you can turn off syslog for unit tests. This can be useful for environments where /dev/log is unavailable, or which cannot rate limit (unit tests generate a lot of logs very quickly). Open the file SWIFT_TEST_CONFIG_FILE points to, and change the value of fake_syslog to True.