60 Commits

Author SHA1 Message Date
Gilles Biannic
a4cc353375 Make log format for requests configurable
Add the log_msg_template option in proxy-server.conf and log_format in
a/c/o-server.conf. It is a string parsable by Python's format()
function. Some fields containing user data might be anonymized by using
log_anonymization_method and log_anonymization_salt.

Change-Id: I29e30ef45fe3f8a026e7897127ffae08a6a80cd9
2019-05-02 17:43:25 -06:00
John Dickinson
02841ee0c6 fix documentation of default
Change-Id: I7c716dea5e0a5b8849b84b1bb25d5294591dcd51
2018-12-18 14:25:23 -08:00
Alistair Coles
904e7c97f1 Add more doc and test for cors_expose_headers option
In follow-up to the related change, mention the new
cors_expose_headers option (and other proxy-server.conf
options) in the CORS doc.

Add a test for the cors options being loaded into the
proxy server.

Improve CORS comments in docs.

Change-Id: I647d8f9e9cbd98de05443638628414b1e87d1a76
Related-Change: I5ca90a052f27c98a514a96ee2299bfa1b6d46334
2018-09-17 12:35:25 -07:00
FatemaKhalid
cfeb32c66b Adding keep_idle config value to socket
User can cofigure KEEPIDLE time for sockets in TCP connection.
The default value is the old value which is 600.

Change-Id: Ib7fb166deb8a87ae4e97ba0671048b1ec079a2ef
Closes-Bug:1759606
2018-09-15 01:30:53 +02:00
wangqi
708b24aef1 Deprecate auth_uri option
Option auth_uri from group keystone_authtoken is deprecated[1].
Use option www_authenticate_uri from group keystone_authtoken.

[1]https://review.openstack.org/#/c/508522/

Change-Id: I43bbc8b8c986e54a9a0829a0631d78d4077306f8
2018-04-18 02:07:11 +00:00
Alistair Coles
5c76b9e691 Add concurrent_gets to proxy.conf man page
Change-Id: Iab1beff4899d096936c0e5915f3ec32364b3e517
Closes-Bug: #1559347
2017-09-27 14:11:14 +01:00
Kota Tsuyuzaki
1e79f828ad Remove all post_as_copy related code and configes
It was deprecated and we discussed on this topic in Denver PTG
for Queen cycle. Main motivation for this work is that deprecated
post_as_copy option and its gate blocks future symlink work.

Change-Id: I411893db1565864ed5beb6ae75c38b982a574476
2017-09-16 05:50:41 +00:00
junboli
6998d804ff doc migration: update the doc link address[1/3]
Update the doc link brought by the swift doc migration.
Although we had some effort to fix these before, it still left lots
of bad doc link, I separate these changes into 3 patches aim to fix
all of these, this is the 1st patch for doc/manpages.

Change-Id: Ib49696706e61bbd36ae56b15b1d94aa4ce84531c
2017-09-05 19:13:47 +00:00
Romain LE DISEZ
420e73fabd Allow to configure the nameservers in cname_lookup
For various reasons, an operator might want to use specifics nameservers
instead of the systems ones to resolve CNAME in cname_lookup. This patch
creates a new configuration variable nameservers which accepts a list of
nameservers separated by commas. If not specified or empty, systems
namservers are used as previously.

Co-Authored-By: Tim Burke <tim.burke@gmail.com>
Change-Id: I34219e6ab7e45678c1a80ff76a1ac0730c64ddde
2017-06-01 14:02:08 -07:00
Alistair Coles
f02ec4de81 Add read and write affinity options to deployment guide
Add entries for these options in the deployment guide and
make the text in proxy-server.conf-sample and man page
consistent.

Change-Id: I5854ddb3e5864ddbeaf9ac2c930bfafdb47517c3
2017-05-18 10:42:44 -07:00
Jenkins
b43414c905 Merge "Accept storage_domain as a list in domain_remap" 2017-03-16 12:35:48 +00:00
Jenkins
1e9b8888bf Merge "Enable cluster-wide CORS Expose-Headers setting" 2017-03-13 19:24:20 +00:00
Monty Taylor
3c844d02b9
Replace references to swift.openstack.org
The policy of giving projects vanity domains stopped about 5 years ago.
swift.openstack.org is a redirect to the canonical location -
docs.openstack.org/developer/swift. While we are not aiming to remove
the redirect any time in the forseeable future due to existing published
links pointing to it, we should at the very least stop adding more of
those links to the world.

Change-Id: I10e92309f5d3b5f908fe4438f5cc0b184f161cba
2017-03-08 09:46:41 -06:00
Romain LE DISEZ
9b47de3095 Enable cluster-wide CORS Expose-Headers setting
An operator proposing a web UX to its customers might want to allow web
browser to access some headers by default (eg: X-Storage-Policy,
 X-Container-Read, ...). This commit adds a new setting to the
proxy-server to allow some headers to be added cluster-wide to the CORS
header Access-Control-Expose-Headers.

Change-Id: I5ca90a052f27c98a514a96ee2299bfa1b6d46334
2017-02-25 19:00:28 +01:00
Romain LE DISEZ
5c93d6f238 Accept storage_domain as a list in domain_remap
Middleware domain_remap can work with cname_lookup middleware. This last
middleware accept that storage_domain is a list of domains. To be
consistent, domain_remap should have the same behavior.

Closes-Bug: #1664647

Change-Id: Iacc6619968cc7c677bf63e0b8d101a20c86ce599
2017-02-18 10:41:27 +01:00
Tim Burke
4ee20dba48 Default object_post_as_copy to False
Additionally, emit deprecation warnings when running POST-as-COPY

Change-Id: I11324e711057f7332577fd38f9bff82bdc6aac90
2017-01-20 12:37:01 -05:00
Jenkins
6daa382c34 Merge "Revises 'url' to 'URL' and 'json' to 'JSON'" 2016-10-06 00:23:41 +00:00
Yushiro FURUKAWA
9b98c89983 Revises 'url' to 'URL' and 'json' to 'JSON'
Change-Id: I44743fbb9bcbce3a50ed6770264ba0f4b17803d7
2016-09-30 22:21:03 +09:00
Alistair Coles
18bb99971f Add more comment to authtoken sample options
Prior to the Mitaka release the install guides showed
services (including Swift) being in a default Keystone
domain which existed by default and has id=default. This
domain id is reflected in the proxy-server.conf-sample
authtoken options and also shown in man page and auth docs.

The Mitaka install guide shows a domain with *name* default
being created, and having a random UUID assigned, in which
services are created. This has caused confusion (see
discussion on linked bug report).

This patch does not change the sample options but does
add to the comments in order to emphasize that a user
may need to alter the options to match their Keystone
configuration.

Change-Id: I17bfcdbd983402eeb561bb704b8b1f1e27547c7d
Partial-Bug: #1604674
2016-09-21 15:48:11 +01:00
Peter Lisák
8bf2233b40 Documantation enhancements of nice/ionice feature
Based on comments from patch #238799.

Change-Id: I9455cf6dc7fd12fee62439ff3c5f3255287ab1be
2016-08-19 07:39:49 +02:00
Peter Lisák
ed772236c7 Change schedule priority of daemon/server in config
The goal is to modify schedule priority and I/O scheduling class and
priority of daemon/server via configuration.
Setting is optional, default keeps current behaviour.

Use case:
Prioritize object-server to object-auditor, because all user's requests
needed to be served in peak hours and audit could wait.

Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
DocImpact
Change-Id: I1018a18f4706daabdb84574ffd9a58d831e68396
2016-08-10 23:56:15 +02:00
Shashirekha Gundur
c5ff9932a4 NIT: fixing inconsistent naming of OpenStack Swift
Throughout the manpages maintaining references to OpenStack Swift.

Change-Id: I2a0c2658e10a92671bfc092c0a3abaddfd8cd7d9
Closes-Bug: #1609687
2016-08-05 13:58:25 +00:00
Alistair Coles
736de613f1 Docs: Container sync does not require POST-as-COPY
Updates docs to remove warnings that container sync only
works with object_post_as_copy=True. Since commit e91de49
container sync will also sync POST updates when using
object_post_as_copy=False.

Change-Id: I5cc3cc6e8f9ba2fef6f896f2b11d2a4e06825f7f
2016-03-22 11:36:32 +00:00
Jenkins
7cc2c783a4 Merge "Keystone middleware deprecated option is_admin removed" 2016-03-18 10:51:46 +00:00
Alistair Coles
6efee0ebb1 Make keystone middleware options consistent in docs
Bring overview_auth.rst and proxy server man page
up to date with changes made in [1]

[1] Change-Id: I373734933189c87c4094203b0752dd3762689034

Change-Id: Ia16f0c391e7c357ccb9c13945839dc5647e49a13
2016-03-16 11:43:20 +00:00
Ondřej Nový
335d58611d Keystone middleware deprecated option is_admin removed
It has been deprecated from Swift 1.8.0 (Grizzly)

Change-Id: Id6bc10c3e84262c0a9e6160a76af03c0ad363e9c
2016-02-11 10:52:47 +00:00
root
bcada66b90 Removed unused parameter in server.py
The variable max_large_object_get_time is no longer used and was
removed to reflect the change.

Change-Id: I43051181dcb38245de6d13fab63876e83f46fc39
Closes-Bug: #1538834
2016-02-10 14:26:10 -06:00
Ondřej Nový
ae632abbd8 Fixed manpages errors.
account-server.conf.5
105: warning: numeric expression expected (got `)')

container-server.conf.5
111: warning: numeric expression expected (got `)')

object-expirer.conf.5
79: warning: numeric expression expected (got `)')

object-server.conf.5
114: warning: numeric expression expected (got `)')

proxy-server.conf.5
121: warning: numeric expression expected (got `)')
331: warning: numeric expression expected (got `[')
1005: warning: macro `*' not defined

Change-Id: I203dcfde83035e3b1dcb91109b72b5d08bb7840e
2016-02-04 16:20:14 +01:00
Jenkins
eaf6af3179 Merge "Allow IPv6 addresses/hostnames in StatsD target" 2016-02-04 03:23:01 +00:00
Darrell Bishop
26327e1e8b Allow IPv6 addresses/hostnames in StatsD target
The log_statsd_host value can now be an IPv6 address or a hostname
which only resolves to an IPv6 address.  In both cases, the new
behavior is to use an AF_INET6 socket on which .sendto() is called
with the originally-configured hostname (or IP).  This means the
Swift process is not caching a DNS resolution for the lifetime of
the process (a good thing).

If a hostname resolves to both an IPv6 or IPv4 address, an AF_INET
socket is used (i.e. only the IPv4 address will receive the UDP
packet).

The old behavior is preserved: any invalid IP address literals and
failures in DNS resolution or actual StatsD packet sending do not
halt the process or bubble up; they are caught, logged, and
otherwise ignored.

Change-Id: Ibddddcf140e2e69b08edf3feed3e9a5fa17307cf
2016-02-03 00:26:31 -08:00
Thomas Goirand
c3886eea15 Fix a few English mistakes in man
These errors are producing lintian warnings, so fixing them
helps having less errors when checking for Debian packages.

Change-Id: Iff99a8d5f2276515f42d758d110a43cae757db28
2016-01-28 09:16:59 +00:00
Clay Gerrard
3347646023 fixups for ipv6 memcache_servers docs
Change-Id: I20d91c1e276014eaf210fa9eb43788bc17f4e8df
2016-01-12 21:08:58 -08:00
Peter Lisák
28c4b7310f Unification of manpages and conf-samples (default values, etc)
Change-Id: I47a3127ef698b4bd1537b1562901ee9c2b5924d4
2015-11-30 10:08:16 -08:00
Koert van der Veer
11e5c4adf0 Allow default reseller prefix in domain_remap middleware
Previously, the reseller prefix needed to be provided in the host name
even when the domain was unique to that reseller. With the
default_reseller_prefix, any domain which matches in this middleware,
will will be passed on with a reseller prefix, whether or not it was
provided.

Change-Id: I5aa5ce78ad1ee2e3660cce4c3e07306f8999f02a
Implements: blueprint domainremap-reseller-domains
2015-06-06 12:54:41 -07:00
David Goetz
172a9b369f Change black/white-listing to use sysmeta.
The way we do this now involves a conf change and a proxy
reload which is a pain. You can now just set these:

X-Account-Sysmeta-Global-Write-Ratelimit: WHITELIST

or

X-Account-Sysmeta-Global-Write-Ratelimit: BLACKLIST

NOTE:
The existing proxy config settings: account_whitelist
and account_blacklist will continue to work.

Change-Id: I532663f1d2c75d03170c5fdb9b330416822fbc88
2015-01-09 08:35:50 -08:00
replay
19a574f304 Fixes typo in man page
There is a simple typo in the man page of proxy-server.conf,
"client_timeout" is written as "client_timeoutt".
This commit fixes it.

Closes-Bug: #1326237
Change-Id: I98777f523906e4ed625de8f20a96979ea627aa1f
2014-06-04 09:52:07 +03:00
David Goetz
8d1278cae8 copy over swift.authorize stuff into subrequests
If auth is setup in the env then it needs to be copied over with the
make_request wsgi helper.  Also renamed make_request to
make_subrequest- when I grepped for make_request I got > 250 results,
this'll make it easier to find references to this function in the
future.

Updated docs and sample confs to show tempurl needs to be before dlo and
slo as well as auth.

Change-Id: I9750555727f520a7c9fedd5f4fd31ff0f63d8088
2014-03-07 11:08:37 -08:00
Tobias Stevenson
83a6ec1683 Man page lintian errors and warnings
Used groff to recreate the errors. I believe all the issues
except `binary-without-manpage` are solved. Would like
confirmation from someone using Lintian.

Closes-Bug: #1210114
Change-Id: I533205c53efdb7cdf3645cc3e3dc487f9ee5640a
2013-09-11 09:21:23 -05:00
Clay Gerrard
de3acec4bf Set default wsgi workers to cpu_count
Change the default value of wsgi workers from 1 to auto.  The new default
value for workers in the proxy, container, account & object wsgi servers will
spawn as many workers per process as you have cpu cores.

This will not be ideal for some configurations, but it's much more likely to
produce a successful out of the box deployment.

Inspect the number of cpu_cores using python's multiprocessing when available.
Multiprocessing was added in python 2.6, but I know I've compiled python
without it before on accident.  The cpu_count method seems to be pretty system
agnostic, but it says it can raise NotImplementedError or sometimes return 0.

Add a new utility method 'config_auto_int_value' to pull an integer out of the
config which has a dynamic default.

 * drive by s/container/proxy/ in proxy-server.conf.5
 * fix misplaced max_clients in *-server.conf-sample
 * update doc/development_saio to force workers = 1

DocImpact

Change-Id: Ifa563d22952c902ab8cbe1d339ba385413c54e95
2013-07-18 22:57:18 -07:00
Peter Portante
2d42b37303 Add the max_clients parameter to bound clients
The new max_clients parameter allows one full control over the maximum
number of client requests that will be handled by a given worker for
any of the proxy, account, container or object servers.

Lowering the number of clients handled per worker, and raising the
number of workers can lessen the impact that a CPU intensive, or
blocking, request can have on other requests served by the same
worker.

If the maximum number of clients is set to one, then a given worker
will not perform another accept(2) call while processing, allowing
other workers a chance to process it.

DocImpact
Signed-off-by: Peter Portante <peter.portante@redhat.com>

Change-Id: Ic01430f7a6c5ff48d7aa349dc86a5f8ac463a420
2013-04-26 10:29:57 -04:00
Marcelo Martins
1126e59c12 Adding a new optional variable called trans_id_suffix
The trans_id_suffix (default is empty) would be appended to the swift transaction
id allowing one to easily figure out from which cluster that X-Trans-Id
belongs to. This is very useful when one is managing more than one swift
cluster. Also updated sample and manpage to reflect the changes.

Change-Id: Icdf63643e9c1bde36a9ef5e3f41ee9fb20e55f5d
2013-04-10 06:37:32 -05:00
Pete Zaitcev
93ea7c63b1 Documentation fixups
These are mostly cosmetic fixes for irritating imperfections:
- "separated with commas" was duplicated, leave just one
- extra whitespace here and there, man pages are not PEP8, drop
- weird extra commas, drop
- Fedora logs to /var/log/messages
- "drive is has failed", drop "is"

Change-Id: I5ceba2e61b16db4855d76c92cbc83663b9b2a0da
2013-02-18 10:54:27 -07:00
annegentle
72428434f7 Replaces Copyright statements for LLC with Foundation, removes date.
Replaced GA code for cross-domain tracking.

Patchset addresses reviewer's comments
and follows new guidance from Foundation:
http://wiki.openstack.org/Documentation/Copyright

Adds current year to each Sphinx-built page.

Addresses only the docs copyright attribution, not code files.

Change-Id: Ib90fd1c92c8fafce2db821bc2b17cef1377cfc1e
2013-02-11 16:32:33 -06:00
gholt
85529531d6 Remove tempauth allowed_sync_hosts conf option
Seems we missed these references when committing
357b12dc2ba7b19c66196a573ccb2489d2104b93

DocImpact

Change-Id: Ia226ce1d63e52769bc067d50ec4704cea4e11c5c
2013-01-31 18:30:10 +00:00
Darrell Bishop
b8e3e9e1c2 Allow optional, temporary healthcheck failure.
A deployer may want to remove a Swift node from a load balancer for
maintenance or upgrade.  This patch provides an optional mechanism for
this.  The healthcheck filter config can specify "disable_path" which is
a filesystem path.  If a file is present at that location, the
healthcheck middleware returns a 503 with a body of "DISABLED BY FILE".

So a deployer can configure "disable_path" and then touch that
filesystem path, wait for the proxy to be removed from the load balancer
pool, perform maintenance/upgrade, and then remove the "disable_path"
file.

Also cleaned up the conf file man pages a bit.

Change-Id: I1759c78c74910a54c720f298d4d8e6fa57a4dab4
2012-12-04 09:14:27 -08:00
Vincent Untz
e1ff51c045 Do not use pickle for serialization in memcache, but JSON
We don't want to use pickle as it can execute arbitrary code. JSON is
safer. However, note that it supports serialization for only some
specific subset of object types; this should be enough for what we need,
though.

To avoid issues on upgrades (unability to read pickled values, and cache
poisoning for old servers not understanding JSON), we add a
memcache_serialization_support configuration option, with the following
values:

 0 = older, insecure pickle serialization
 1 = json serialization but pickles can still be read (still insecure)
 2 = json serialization only (secure and the default)

To avoid an instant full cache flush, existing installations should
upgrade with 0, then set to 1 and reload, then after some time (24
hours) set to 2 and reload. Support for 0 and 1 will be removed in
future versions.

Part of bug 1006414.

Change-Id: Id7d6d547b103b4f23ebf5be98b88f09ec6027ce4
2012-08-03 16:22:21 +02:00
Vincent Untz
faff4ae769 Forbid substrings based on a regexp in name_filter middleware
In comments from https://review.openstack.org/8798 it was raised that it
might make sense to forbid some substrings in the name_filter
middleware.

There is now a new forbidden_regexp option for the name_filter
middleware to specify which substrings to forbid. The default is
"/\./|/\.\./|/\.$|/\.\.$" (or in a non-regexp language: the /./ and /../
substrings as well as strings ending with /. or /..).

This can be useful for extra paranoia to avoid directory traversals
(bug 1005908), or for more general filtering.

Change-Id: I39bf2de45b9dc7d3ca4d350d24b3f2276e958a62
DocImpact: new forbidden_regexp option for the name_filter middleware
2012-07-19 14:13:47 +02:00
Vincent Untz
1125368624 Remove ambiguity in memcache_servers documentation
The documentation could be understood like the following:
memcache_servers from memcache.conf is always used if set, even if
memcache_servers in proxy-server.conf is set.

This is clearly not the case, as proxy-server.conf has a higher priority
if it memcache_servers is set there.

Change-Id: I967c7e80796a0e296c5c65bd097df1669d16203e
2012-06-28 16:09:10 +02:00
Jenkins
57008e553a Merge "Patch for Swift Solaris (Illumos) compability." 2012-06-27 16:41:43 +00:00
Victor Rodionov
13e4de1899 Patch for Swift Solaris (Illumos) compability.
* Add new configuration option log_address.

Change-Id: I636bd4116687629c997b70a0d804b7ed4bc46032
2012-06-19 15:38:56 +04:00