26 Commits

Author SHA1 Message Date
Robert Francis
99b89aea10 Symlink implementation.
Add a symbolic link ("symlink") object support to Swift. This
object will reference another object. GET and HEAD
requests for a symlink object will operate on the referenced object.
DELETE and PUT requests for a symlink object will operate on the
symlink object, not the referenced object, and will delete or
overwrite it, respectively.
POST requests are *not* forwarded to the referenced object and should
be sent directly. POST requests sent to a symlink object will
result in a 307 Error.

Historical information on symlink design can be found here:
https://github.com/openstack/swift-specs/blob/master/specs/in_progress/symlinks.rst.
https://etherpad.openstack.org/p/swift_symlinks

Co-Authored-By: Thiago da Silva <thiago@redhat.com>
Co-Authored-By: Janie Richling <jrichli@us.ibm.com>
Co-Authored-By: Kazuhiro MIYAHARA <miyahara.kazuhiro@lab.ntt.co.jp>
Co-Authored-By: Kota Tsuyuzaki <tsuyuzaki.kota@lab.ntt.co.jp>

Change-Id: I838ed71bacb3e33916db8dd42c7880d5bb9f8e18
Signed-off-by: Thiago da Silva <thiago@redhat.com>
2017-12-13 21:26:12 +00:00
Tim Burke
4806434cb0 Move listing formatting out to proxy middleware
Make some json -> (text, xml) stuff in a common module, reference that in
account/container servers so we don't break existing clients (including
out-of-date proxies), but have the proxy controllers always force a json
listing.

This simplifies operations on listings (such as the ones already happening in
decrypter, or the ones planned for symlink and sharding) by only needing to
consider a single response type.

There is a downside of larger backend requests for text/plain listings, but
it seems like a net win?

Change-Id: Id3ce37aa0402e2d8dd5784ce329d7cb4fbaf700d
2017-09-15 06:38:26 +00:00
Matthew Oliver
e11a38c63a Bind SAIO services on different loopback addresses
Currently all devices in the ring and all services in a SAIO
all bind to the same loopback address 127.0.0.1. But this
breaks servers_per_port if you want to do any testing on that.

This change binds each service to a different loopback address
and updates the rings (remakerings) accordingly.

To make sure rysncd binds correctly the bind address needed
to be changed to listen on all addresses (0.0.0.0).

Change-Id: I7e77434f275df1e2699de495d8b622b90157a9d7
2017-06-27 16:08:08 -04:00
Tim Burke
4ee20dba48 Default object_post_as_copy to False
Additionally, emit deprecation warnings when running POST-as-COPY

Change-Id: I11324e711057f7332577fd38f9bff82bdc6aac90
2017-01-20 12:37:01 -05:00
Thiago da Silva
b36d8d9afe add object_post_as_copy to saio.
adding the option to the saio configuration file
to make it easier to test fast post.

Change-Id: I22b0e60c7d73580e3db6fcd48e6dfc79fab6032a
Signed-off-by: Thiago da Silva <thiago@redhat.com>
2016-12-06 13:02:23 -05:00
Ondřej Nový
19102c6e7f Apply bash error handling consistently in all bash scripts
Change-Id: I36dcff9cc2593bcfb6aebbe84b59d95c711eb3e4
2016-10-11 22:13:06 +02:00
Prashanth Pai
46d61a4dcd Refactor server side copy as middleware
Rewrite server side copy and 'object post as copy' feature as middleware to
simplify the PUT method in the object controller code. COPY is no longer
a verb implemented as public method in Proxy application.

The server side copy middleware is inserted to the left of dlo, slo and
versioned_writes middlewares in the proxy server pipeline. As a result,
dlo and slo copy_hooks are no longer required. SLO manifests are now
validated when copied so when copying a manifest to another account the
referenced segments must be readable in that account for the manifest
copy to succeed (previously this validation was not made, meaning the
manifest was copied but could be unusable if the segments were not
readable).

With this change, there should be no change in functionality or existing
behavior. This is asserted with (almost) no changes required to existing
functional tests.

Some notes (for operators):
* Middleware required to be auto-inserted before slo and dlo and
  versioned_writes
* Turning off server side copy is not configurable.
* object_post_as_copy is no longer a configurable option of proxy server
  but of this middleware. However, for smooth upgrade, config option set
  in proxy server app is also read.

DocImpact: Introducing server side copy as middleware

Co-Authored-By: Alistair Coles <alistair.coles@hpe.com>
Co-Authored-By: Thiago da Silva <thiago@redhat.com>

Change-Id: Ic96a92e938589a2f6add35a40741fd062f1c29eb
Signed-off-by: Prashanth Pai <ppai@redhat.com>
Signed-off-by: Thiago da Silva <thiago@redhat.com>
2016-05-11 14:55:51 -04:00
Christian Schwede
e46d6b1725 Add note on using printable chars for swift_hash_path_suffix/prefix
Using unprintable characters for swift_hash_path_prefix/suffix might lead to
hard problems when parsing of these values changes, for example due to newer
Python versions or changes in the parsers itself. Let's avoid this and add a
note that deployers should use printable strings for these values.

Change-Id: I976982b753b6af831ab91d7190f50f8f15bf73bf
2016-02-17 20:47:53 +01:00
gh159m
b5311f63db Removed default value for log_statsd_host
Multiple files and documents showed that log_statsd_host had
a default value, usually localhost.  This was incorrect, instead
setting a value for log_statsd_host enables statsd logging.

Removed any reference of log_statsd_host having a default value.

Also changed descriptions to show setting a value enables logging.

Change-Id: I3ca5c0e8b8e4981de3aa6db0c476072b5a59723d
Closes-Bug: #1542227
2016-02-10 10:36:59 -06:00
Ondřej Nový
16976a0f14 Changed EC backend from jerasure to liberasurecode in examples and docs
liberasurecode_rs_vand is build-in liberasurecode, so you don't need
another depedency libjerasure2.

liberasurecode_rs_vand is supported by pyeclib from 1.0.8
version, so bumping version up.

Closes-Bug: #1534325
Change-Id: If2d96875694df8fd48c5278395859aaa165cb566
2016-02-02 23:08:11 +01:00
Ondřej Nový
9cafa472a3 Autodetect systemctl in SAIO and use it on systemd distros
Change-Id: I84a9b27baac89327749d8774032860f8ad5166f2
2015-10-12 22:39:52 +02:00
Romain LE DISEZ
71f6fd025e Allows to configure the rsync modules where the replicators will send data
Currently, the rsync module where the replicators send data is static. It
forbids administrators to set rsync configuration based on their current
deployment or needs.

As an example, the rsyncd configuration example encourages to set a connections
limit for the modules account, container and object. It permits to protect
devices from excessives parallels connections, because it would impact
performances.

On a server with many devices, it is tempting to increase this number
proportionally, but nothing guarantees that the distribution of the connections
will be balanced. In the worst scenario, a single device can receive all the
connections, which is a severe impact on performances.

This commit adds a new option named 'rsync_module' to the *-replicator sections
of the *-server configuration file. This configuration variable can be
extrapolated with device attributes like ip, port, device, zone, ... by using
the format {NAME}. eg:
    rsync_module = {replication_ip}::object_{device}

With this configuration, an administrators can solve the problem of connections
distribution by creating one module per device in rsyncd configuration.

The default values are backward compatible:
    {replication_ip}::account
    {replication_ip}::container
    {replication_ip}::object

Option vm_test_mode is deprecated by this commit, but backward compatibility is
maintained. The option is only effective when rsync_module is not set. In that
case, {replication_port} is appended to the default value of rsync_module.

Change-Id: Iad91df50dadbe96c921181797799b4444323ce2e
2015-09-07 08:00:18 +02:00
Eran Rom
69e7424d3c Add container sync probe test to SAIO default set
SAIO Configuration and documentation changes enabling to run the
container sync probe test by default

Change-Id: Iccf59533d0d4fe72549d318339ab125d04dde006
Related-Bug: #1476623
2015-08-22 22:50:50 +03:00
Jenkins
0279411c58 Merge "versioned writes middleware" 2015-08-10 17:37:49 +00:00
Thiago da Silva
035a411660 versioned writes middleware
Rewrite object versioning as middleware to simplify the PUT method
in the object controller.

The functionality remains basically the
same with the only major difference being the ability to now
version slo manifest files. dlo manifests are still not
supported as part of this patch.

Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>

DocImpact
Change-Id: Ie899290b3312e201979eafefb253d1a60b65b837
Signed-off-by: Thiago da Silva <thiago@redhat.com>
Signed-off-by: Prashanth Pai <ppai@redhat.com>
2015-08-07 14:11:32 -04:00
Zhao Lei
4ac1fea5d1 Fix some spelling typo in comments
s/overide/override for object-expirer.conf and sample.
s/automaticaly/automatically for swift/proxy/controllers/obj.py

Change-Id: Ife107c7a1005a5d4959288db50a7f8f33c522dd4
Signed-off-by: Zhao Lei <zhaolei@cn.fujitsu.com>
2015-08-07 22:21:37 +08:00
Paul Luse
8f5d4d2455 Erasure Code Documentation
This patch adds all the relevant EC documentation to
the source tree. Notable additions are:
  - Updated SAIO documentation
  - Updates to existing swift documentation; and
  - Erasure Coding overview

Co-Authored-By: Alistair Coles <alistair.coles@hp.com>
Co-Authored-By: Thiago da Silva <thiago@redhat.com>
Co-Authored-By: John Dickinson <me@not.mn>
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Co-Authored-By: Tushar Gohad <tushar.gohad@intel.com>
Co-Authored-By: Samuel Merritt <sam@swiftstack.com>
Co-Authored-By: Christian Schwede <christian.schwede@enovance.com>
Co-Authored-By: Yuan Zhou <yuan.zhou@intel.com>
Change-Id: I0403016a4bb7dad9535891632753b0e5e9d402eb
Implements: blueprint swift-ec
Signed-off-by: Thiago da Silva <thiago@redhat.com>
2015-04-14 00:52:17 -07:00
Thiago da Silva
11a72a4a50 move slo, dlo after tempauth in pipeline
Noticed that slo and dlo middleware were placed before
tempauth, they should be placed after

DocImpact

Change-Id: Ia931e2280125d846f248b23e219aebad14c66210
Signed-off-by: Thiago da Silva <thiago@redhat.com>
2014-11-17 16:41:09 +00:00
Samuel Merritt
466403723c Make resetswift customizable via environment
Instead of recommending to edit resetswift to replace "/dev/sdb1" with
"/srv/swift-disk", use an environment variable instead. This way I can
set SAIO_BLOCK_DEVICE=/srv/swift-disk in my .bashrc, and then when I'm
testing out changes to resetswift, I don't need to remember to edit
the modified script, nor do I end up submitting changes with the wrong
default in there.

The variable defaults to /dev/sdb1, so if you use the script unmodified
and don't set SAIO_BLOCK_DEVICE, nothing changes for you.

Change-Id: I741a8c91c2c54a4f32bc391cd794ef4206402753
2014-11-10 15:07:44 -08:00
Andrew Hale
8e9b16a9ea Only bind SAIO daemons to localhost
The SAIO configs have no default bind_ip setting configured
which causes them to listen on all available IP addresses.
This can be dangerous on a test machine with public interfaces,
especially with the default passwords set. Its reasonable to
choose a more restrictive setup, especially in SAIO which uses
127.0.0.1 throughout ring-builder, example commands and the
probe tests.

Change-Id: I471c49705ce09e07ec7acc07ee42a1e220529b82
2014-09-01 18:11:15 +00:00
Paul Luse
e52e8bc917 Add Storage Policy Documentation
Add overview and example information for using Storage Policies.

DocImpact
Implements: blueprint storage-policies
Change-Id: I6f11f7a1bdaa6f3defb3baa56a820050e5f727f1
2014-06-19 10:18:34 -07:00
Peter Portante
2e6325f351 Enable object versions for SAIO by default
Change-Id: I4a20cacd6c8eb4d82a1f0b59a6b26340bc57184a
2014-03-19 16:20:25 -04:00
David Goetz
8d1278cae8 copy over swift.authorize stuff into subrequests
If auth is setup in the env then it needs to be copied over with the
make_request wsgi helper.  Also renamed make_request to
make_subrequest- when I grepped for make_request I got > 250 results,
this'll make it easier to find references to this function in the
future.

Updated docs and sample confs to show tempurl needs to be before dlo and
slo as well as auth.

Change-Id: I9750555727f520a7c9fedd5f4fd31ff0f63d8088
2014-03-07 11:08:37 -08:00
Samuel Merritt
6acea29fa6 Move all DLO functionality to middleware
This is for the same reason that SLO got pulled into middleware, which
includes stuff like automatic retry of GETs on broken connection and
the multi-ring storage policy stuff.

The proxy will automatically insert the dlo middleware at an
appropriate place in the pipeline the same way it does with the
gatekeeper middleware. Clusters will still support DLOs after upgrade
even with an old config file that doesn't mention dlo at all.

Includes support for reading config values from the proxy server's
config section so that upgraded clusters continue to work as before.

Bonus fix: resolve 'after' vs. 'after_fn' in proxy's required filters
list. Having two was confusing, so I kept the more-general one.

DocImpact

blueprint multi-ring-large-objects

Change-Id: Ib3b3830c246816dd549fc74be98b4bc651e7bace
2014-02-03 18:29:48 -08:00
Peter Portante
4b1fc88b1e Fix up SAIO to use new gatekeeper
Fix up the SAIO default proxy-server pipeline to use the new
gatekeeper and update to follow the pipeline ordering from proxy
sample.

Change-Id: Iaa4de7abd7fc557cafe6fe40668973e9fc142ca3
2014-01-19 21:57:10 -05:00
Peter Portante
ba5fe5f39e Use files in the source tree instead of cut/paste
Many of the large files are included in the tree and the script now
leverages a checked out swift tree to provide those files so that
users don't have to cut/paste text from the document. The contents of
those files are still included in the document for reference.

Updated to add sudo in appropriate places so that the entire script
can be run as the user instead of as root.

We also simplify the steps needed to get resetswift script working
(don't need to edit the user name).

Change-Id: Ie5b5a815870edcc205d273e35e0bbd2426d3b002
Signed-off-by: Peter Portante <peter.portante@redhat.com>
2013-12-09 20:55:08 -05:00