227 Commits

Author SHA1 Message Date
Dirk Mueller
8ab64b3236 Drop signing_dir option from configure_auth_token_middleware
This is no longer being used due to Keystone PKI tokens no longer
being implemented.

In order to not break backward compatibility we create a new function
that is to be used instead and deprecate the old one. Modify the old
function to ignore the 3rd argument and display a deprecation warning.
Adjust callers to no longer create and set that directory, calling the
new function instead.

Change-Id: Id0dec1ba72467cce5cacfcfdb2bc0af2bd3a3610
2019-06-28 16:28:03 +00:00
Kota Tsuyuzaki
070e4ee65e Deprecate swift3, use s3api in Swift repo
swift3 is no longer actively maintained in the upstream.
That has been moved to Swift repository as s3api so we should
use s3api middleware instead. As well as swift3, s3token is
also maintained in Swift upstream.

Change-Id: I4582d81da066ab53e6f11ad1df7af91425f2b0ca
2019-05-01 18:07:58 +00:00
Lee Yarwood
5d7d891380 Drop the deprecated and now removed barrier XFS mount options
Both barrier and nobarrier were deprecated with the 4.10 kernel [1] and
then removed [2] with the 4.19 kernel as now used by Fedora >= 28. Both
should be safe to drop at this point.

[1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=4cf4573
[2] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=1c02d50

Change-Id: I6871a7765e3e04122d8d546f43d36bb8415383fc
2019-01-04 20:53:29 +00:00
Zuul
5da7e4a22e Merge "Fix running with SERVICE_IP_VERSION=6" 2018-07-10 06:10:28 +00:00
Sumit Jamgade
1fccf0b339 swift expects an internal client config to start
using the sample form the source

Change-Id: I01874b650cd5d662ca2feabe58cc880155c9421e
2018-05-29 12:07:13 +02:00
Paul Belanger
a62ede7d50
Use NOVA_BIN_DIR / SWIFT_BIN_DIR for binaries
Fix a few path issues where we didn't properly use NOVA_BIN_DIR /
SWIFT_BIN_DIR.

This is part of the effort to start using a virtualenv for openstack
services.

Change-Id: I6eb383db65cc902c67c43e5cb1a16a9716a914b2
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2018-03-31 19:20:24 -04:00
Dirk Mueller
4404f680f2 Add an openSUSE Tumbleweed devstack non-voting check job
This adds the necessary fixes to pass a devstack run
on openSUSE Tumbleweed. Also removes opensuse 42.2 as it
is EOL for some time already and no longer actively tested
in the OpenStack infra.

Depends-On: I1b68c08c07cf6653ea58506f738cbe0054b38f3a
Change-Id: I2894482deef063fd02b0818c695a2ddbf6767039
2018-03-11 14:04:36 +01:00
Jens Harbott
dc7b429463 Fix running with SERVICE_IP_VERSION=6
- There are some locations where we need the raw IPv6 address instead of the
  url-quoted version enclosed in brackets.
- Make nova-api-metadata service listen on IPv6 when we need that.
- Use SERVICE_HOST instead of HOST_IP for TLS_IP.

Change-Id: Id074be38ee95754e88b7219de7d9beb06f796fad
Partial-Bug: 1656329
2018-03-11 08:53:41 +00:00
Ian Wienand
2346e9a8b5 Run swift-container-sync under run_process
All the evidence from [1] suggests that on opensuse swift-init is not
detaching the daemon process correctly.  It's possible there's a pipe
still in play that somehow holds our ansible-streamer open.

This is a minimal fix to avoid swift-init.  Although it's possible in
non-default paths to still use swift-init (and hence possibly hit
another variant of this issue), after discussions with swift
developers it was decided the intersection of tests running under our
current ansible, on suse, that would enable these services is
sufficiently small that this is the best course for now.

[1] https://storyboard.openstack.org/#!/story/2001528

Change-Id: I1b68c08c07cf6653ea58506f738cbe0054b38f3a
2018-02-23 08:26:34 +11:00
Ian Wienand
1f82f43016 Revert "Remove cache dirs from the services"
This reverts commit ef5ebed6c9ca3d9d47fd2a732a1542555a0f65ba.

The problem here is a backwards-incompatible change to
configure_auth_token_middleware.  Plugins are still passing a
"signing_dir" which is interpreted now as the "section" argument
... this leads to an interesting red-herring issue; because "v" is a
gnu sed command for checking the version, a signing_dir of "/var/..."
(as done in most plugins) gives the weird error:

 sed: -e expression #1, char 32: expected newer version of sed

I think we'll either need a new function, or dummy arguments to get
this back in.

Change-Id: I2098d4eb2747282622cf486fa7dbf216f932f58b
2017-10-04 09:54:43 +11:00
Jamie Lennox
ef5ebed6c9 Remove cache dirs from the services
PKI tokens have been actively deprecated from keystone and there are
deprecations being emitted from keystonemiddleware. Because of this we
no longer need an auth cache directory in the services where the PKI
certifcates used to be stored.

Remove the creation and use of all these AUTH_CACHE directories.

Change-Id: I5680376e70e74882e9fdb87ee1b95d5f40570ad7
2017-09-26 10:10:11 +10:00
Jenkins
86bdfffbe0 Merge "clean up screen and tail_log references" 2017-09-19 13:23:02 +00:00
Jenkins
dbd32d1bc4 Merge "Allow both Keystone and Tempauth reseller prefixes" 2017-09-12 08:55:31 +00:00
Sean Dague
0eebeb415a clean up screen and tail_log references
Change-Id: I6bcfa09931ed1f70e071ccb16688c15c7ef2898f
2017-09-01 15:08:17 -04:00
Clay Gerrard
0e58d22897 Create correct directory layout for swift on purpose.
The pre-existing configuration for swift on devstack set's the
*-server's devices option (the root of the servers list of devices) to:

    devices = /opt/stack/data/swift/1

where "1" is the node_number, and will be 2, 3, ... N if the devstack
machine is built with more than one swift node/device (pretty sure no
one does that on devstack ever).

The device(s) in the rings are named (perhaps confusingly similar to the
swift loopback image) just "sdb1", so all storage servers expect to have
a $STACK_USER writeable file system at:

    os.path.join(<devices_root>, "sdb1")

That directory does not exist when you start up a devstack [1].

Currently Swift's object-server's require that directory exist before
they write data into it (even with mount_check = false!).

Unfortunately however, with mount_check=false the account/container
servers are able to create the device directory when it does not exist
[2].  Which can lead to some unfortunate results with permissions on
some deployments using mount_check = false (e.g. testing or
containerized environments).  Fixing this issue [3] uncovered the
previously benign [4] mis-configuration in devstack.

Attempting

1. It was lost a long while ago I7c65303791689523f02e5ae44483a6c50b2eed1e
2. Essentially they want to:

    mkdir -p /opt/stack/data/swift/1/sdb1/containers/<part#>

... but end up creating the "sdb1" dir too!
3. I3362a6ebff423016bb367b4b6b322bb41ae08764
4. Benign because the object-server share their device with the
account-container devices and they would create the dirs before trying
to write an object.  It was incorrect, but worked by happenstance, which
is nearly as good as worked on purpose.

Change-Id: I52c4ecb70b1ae47e613ba243da5a4d94e5adedf2
2017-08-28 14:03:18 -07:00
Tim Burke
59fb961180 Allow both Keystone and Tempauth reseller prefixes
... to be used with domain_remap. Swift will start functionally testing
domain_remap in I63428132283986bda9e5c082ffe85741449b71ba.

Change-Id: I4c1ab06d040d91fd8c314d0aa2cecbbb00adf8ad
2017-08-15 15:13:24 -07:00
Tim Burke
ab70350403 When configuring temp urls, give Swift time to start up
Otherwise, we've seen intermittent "Unable to establish connection"
failures, with the main devstack log reporting things like

   2017-07-19 13:54:29.973 -> start proxy service
   2017-07-19 13:54:30.082 -> start OSC to store temp url key
   2017-07-19 13:54:31.908 -> OSC reports failure

Meanwhile, the s-proxy screen session tells us things like

   Jul 19 13:54:31.919988 -> start child worker
   Jul 19 13:54:32.206598 -> still loading the WSGI app

... and ports aren't actually bound until *after* the app is loaded.

Add a wait_for_service call to wait for the proxy to come up.

Change-Id: I1a722de31b144797230991700e110353a2d937dd
2017-07-20 11:12:00 -07:00
Jenkins
8f314400d8 Merge "Set specified header size when enabling tls-proxy" 2017-06-29 23:00:35 +00:00
Jenkins
3350542a44 Merge "Set swift functest config when using tls" 2017-06-14 16:09:11 +00:00
Clark Boylan
902158bb8f Don't treat service as enabled if in disabled list
The old implementation for is_$service_enabled simply checked if any of
the subservices were enabled and if so the service was considered to be
enabled. This makes disabling services complicated as it means you have
to list every single subservice which can and do change over time.

Instead also check if the generic service name is in the disabled
services list and if so don't treat the service as enabled.

Change-Id: I7fe4dfca2cd9c15069d50a04161a29c5638291cb
2017-06-07 17:23:38 +00:00
Clark Boylan
f4dbd12f78 Set specified header size when enabling tls-proxy
As part of getting swift's functional testing to work properly through
the tls-proxy we need to increase the allowed request header size in
apache. This was a non issue without tls proxy as requests hit the
eventlet webserver directly which was configured via the swift config
which sets this relatively large limit (by default devstack configures
swift to have a header size limit of 16384).

Now we pass in an optional parameter to start_tls_proxy that includes
the desired header size. lib/swift then passes in the value it also
configures in its swift.conf.

If not explicitly set we default to 8190 which is apache2's default.

Change-Id: Ib2811c8d3cbb49cf94b70294788526b15a798edd
2017-06-05 12:47:50 -07:00
Clark Boylan
e28db4c2f2 Set swift functest config when using tls
Because the swift functests (which use test.conf) run out of a
virtualenv they don't get access to the system wide trust of the
devstack CA. Handle this by explicitly configuring the cafile to trust
in the test.conf file.

We also set the web_front_end to apache2 as that is what is terminating
TLS for us. The tests handle different web server behaviors using this
flag.

Swift's functests will need to read these values in and properly
configure things on its end.

Change-Id: I4cdba36ccab6acd76205184882ee29e4f1e12333
2017-06-05 09:21:46 -07:00
Jenkins
bc8db65da2 Merge "Define a new function for notifications URL" 2017-04-25 17:47:35 +00:00
Thomas Herve
26e431dbd7 Define a new function for notifications URL
This defines a new function get_notification_url, which returns the URL
of RabbitMQ when you want connect to it, and uses in
ceilometermiddleware. This fixes an issue when we try to use AMQP for
RPC, but not for notifications.

Change-Id: I14450b2440806a17a90e5ddefc243868fdbe4f2c
2017-04-25 17:01:21 +02:00
Paul Belanger
37c7843aad
Increase rsyslog buffer sizes.
Swift proxy logs to syslog during the devstack-gate tempest runs. To
better capture the swift logs increase the rsyslog buffer size to 6k
bytes allowing for longer messages like tracebacks.

This was setup by openstack-infra previous during our diskimage
builds.

  I03e42964e14d9f930c07ed047851bdf775639c59

Change-Id: Iaa232335865410600c93f47d4777ed4f1bce08e2
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2017-04-19 13:20:47 -04:00
Tim Burke
f43ea47766 Send useful auth_port and auth_prefix to swift's test.conf
Until we can test with a version of swiftclient that knows how to eat
auth_uri, swift still needs a working gate.

Change-Id: I09f9ad5c87b542df962a79898e06fbf1e968b1e3
Related-Change: I46294fb24e3c23fa19fcfd7d6c9ee8a932354702
Related-Change: Ie427f3b0b9eb834ff940fa5d52444a5a6cdcab15
2017-04-18 21:58:10 -07:00
Sean Dague
f3b2f4c853 Remove USE_SSL support
tls-proxy is the way we're now doing a standard install using https
between services. There is a lot more work to make services directly
handle https, and having python daemons do that directly is a bit of
an anti pattern. Nothing currently tests this in project-config from
my recent grepping, so in the interest of long term maintenance,
delete it all.

Change-Id: I910df4ceab6f24f3d9c484e0433c93b06f17d6e1
2017-04-17 07:27:32 -04:00
Sean Dague
4222ee35f6 Make auth_uri available in the swift test setup
The swift functional tests use a config which requires keystone ports,
we're about to make those go away. This exposes the actual auth_uri to
swift for consumption.

Change-Id: I5868dfdb8e5f0972ba04e359d212b04351502436
2017-04-13 20:33:42 -04:00
Jordan Pittier
980d65f586 Swift: Optionally start only the necessary services
Currently Devstack starts all Swift services, including those
in charge of "consistency convergence" (remember Swift is eventually
consistent), data scrubbing, hard-deletion (*-reaper services)
cleanup.

But when running with Replication Factor 1 some of those services
are not needed at all. Besides, the fonctionnalities provided by
some of these services are not tested at all (neither in Tempest
nor in Swift functional tests).

Thus, in light of saving some Mo of RAM, this patch introduces a config
flag to start only a minimal set of Swift services, just what's required
to make all of our current tests pass.

The default value for this new config flag is set to start all services,
that is to maintain Devstack's current behavior.

For sake of completeness, here is the list of services that are not
going to be started is the config flag is toggled, and the associated RSS
according to our peakmem_tracker

40004  swift-object-replicator /etc/swift/object-server/1.conf
34320  swift-container-replicator /etc/swift/container-server/1.conf
33584  swift-object-auditor /etc/swift/object-server/1.conf
33328  swift-object-reconstructor /etc/swift/object-server/1.conf
31936  swift-object-updater /etc/swift/object-server/1.conf
31492  swift-account-reaper /etc/swift/account-server/1.conf
31076  swift-account-replicator /etc/swift/account-server/1.conf
29540  swift-container-updater /etc/swift/container-server/1.conf
29220  swift-account-auditor /etc/swift/account-server/1.conf
29036  swift-container-auditor /etc/swift/container-server/1.conf

So we are looking at saving at most ~350Mo of RAM (could be less
because RSS doesn't account for shared memory).

A follow-up patch will soon be proposed in devstack-gate to not run
those additional services in our Gate jobs.

Change-Id: I8a0d03ac0296a74e38efd185beb8513866eaf0c4
2017-03-29 13:26:52 +02:00
Sean Dague
5edae54855 initial work to enable systemd service running
During the PTG there was a discussion that the screen developer
workflow wasn't nearly as useful as it once was. There were now too
many services to see them all on one screen, and one of the most
common service restart scenarios was not restarting one service, but a
bunch to get code to take effect.

This implements a 3rd way of running services instead of direct
forking via bash, or running under screen, which is running as systemd
units.

Logging is adjusted because it's redundant to log datetime in oslo.log
when journald has that.

Swift needed to have services launched by absolute path to work.

This is disabled by default, but with instructions on using it. The
long term intent is to make this the way to run devstack, which would
be the same between both the gate and local use.

Some changes were also needed to run_process to pass the run User
in. A hack around the keystone uwsgi launcher was done at the same
time to remove a run_process feature that only keystone uwsgi uses.

Change-Id: I836bf27c4cfdc449628aa7641fb96a5489d5d4e7
2017-03-28 07:19:15 -04:00
Sean Dague
11eb2017ef simplify endpoints used in devstack
The proliferation of internal/admin endpoints is mostly legacy and
based on some specific deployment patterns. These are not used by
everyone, and for the devstack case aren't really that useful. We
should simplify our service catalog down to the minimum we need for
development.

Change-Id: Ided7a65c81b3a0b56f0184847fc82e17c29a771e
2017-02-13 16:16:59 -05:00
Jenkins
7ec59a017a Merge "Run Swift services under py35" 2017-01-06 00:33:27 +00:00
Davanum Srinivas
80f3635521 Run Swift services under py35
* iniuncomment followed by iniset for reseller_prefix just adds a
  duplicate line in the config file that configparser does not like
  so just remove the uncomment
* fall back to http:// url for glance->swift keystone authentication
* insecure flag to talk to swift

Depends-On: I51d56d16a5b175bd45dee09edc0b2748d72a5d06
Change-Id: I02ed01e20f8dce195c51273e8384130af53384ce
2017-01-05 09:10:35 -05:00
Matthew Treinish
85879f1f81
Use the installed swift scripts
This commit switches how scripts we use to launch the installed version
in the path. Previously the scripts were manually executed in the source
repo, but this has issues if you're trying to run with py3 in a system
where python == py2. Setuptools already does the shebang magic for us
at install time, so we just need to use the installed version of the
script.

Change-Id: Iaa4d80ec607a2aa200400330e16cad3a4ca782ac
2017-01-04 19:51:50 -05:00
Attila Fazekas
2bd8d90681 Objects versions new style config
allow_versions is going to be a deprecated option.
Setting up the object versining in the new way.

[1] http://docs.openstack.org/developer/swift/overview_object_versioning.html

Change-Id: Ia520fbb6eb535b08ce83c0cb4bea31e7bba55eb3
2016-10-13 10:08:09 +02:00
Gregory Haynes
4b49e409f8 Use apache for tls-proxy ssl termination
Stud is now abandonware (see https://github.com/bumptech/stud) and is
not packaged in xenial. Lets use Apache for SSL termination since its
there already.

Change-Id: Ifcba410f5969521e8b3d30f02795541c1661f83a
2016-09-20 08:14:11 -07:00
Jenkins
5fc597289b Merge "Enable Swift Account Management" 2016-06-15 12:48:09 +00:00
Brant Knudson
e86b91ba78 Reference keystone URIs rather than building URI
lib/keystone sets KEYSTONE_AUTH_URI and KEYSTONE_SERVICE_URI that
other projects should use rather than building the URL themselves.
This will allow us to more easily drop the port altogether.

Change-Id: I7467aae680215f3045d32a088af2187e1eba8169
2016-05-19 21:44:43 +00:00
Brian Ober
958c169d79 Enable Swift Account Management
This patch enables account management by default in Swift.  This will be
leveraged by Tempest test cases validating account management APIs.

Depends-On: Id29f5ca48f92cd139535be7064107b8a61b02856
Change-Id: Ic01432939ed9b4cf0cbf20e3244d4d76847f539f
2016-05-16 23:00:43 +00:00
Matt Riedemann
ae4578bed9 Revert "Don't check for existing endpoints"
This reverts commit 7d1ec430046256ac3f536f4920a43936e28ef0c4.

This broke the sahara and layer4 dsvm jobs. The layer4 job
is voting on tempest changes so tempest is also broken.

Change-Id: Ide69f10cd85bf7ff0d86bc8cba56dedd26850362
Partial-Bug: #1573868
2016-04-23 01:45:40 +00:00
Sean Dague
7d1ec43004 Don't check for existing endpoints
We really should only have code that create endpoints once, making all
osc calls get_or_set adds 3 seconds per call for no really good
reason.

This also stops creating the internal endpoints in the service
catalog. It's a pattern that we're trying not to propogate, so lets
not have it in devstack any more.

Change-Id: Ia8cefe43753900d62117beae330db46deb6a9fc9
2016-04-22 10:44:10 -04:00
Jamie Lennox
cbcbd8f33c Allow putting service users in a seperate domain
Make it possible to construct the service users in their own seperate
domain. Changing this away from Default will not work for everyone yet,
though it does work for basic service interaction however enabling it
will allow us to start testing and hopefully gating that services aren't
relying on v2 only concepts.

Change-Id: I7e73df5dd1caabf355783da2bc0f3007ade92fba
2016-04-18 09:59:52 +10:00
Jenkins
19c7842e4f Merge "Add variable SWIFT_STORAGE_IPS" 2016-04-14 19:50:17 +00:00
Christian Schwede
91d2245d6e Add variable SWIFT_STORAGE_IPS
If the variable SWIFT_STORAGE_IPS contains a space-separated list of
IPs, we can use this to create consistent rings across all proxy and
storage nodes.

Change-Id: If9307196dc7e74e4a842c95503958ae2d7f7acc7
2016-04-14 14:16:55 +02:00
Sean Dague
19dfa2af32 convert tenant -> project for internal variables
This sets all the internal variables and service users that are set to
use project instead of tenant for clarity.

Change-Id: I4aa833bac2ee2281c5f2881f7ae1fd8e7c759f74
2016-04-05 11:40:25 -04:00
Sean Dague
260583e25e Switch over swift to using $(project_id) in service catalog
This removes one more place where we use tenant_id in our code.

Change-Id: I628fb7c1dd6f22ce499fc5cbe97b79458cce966f
2016-04-05 11:58:21 +00:00
Matthew Oliver
7b85723c04 Set the Swift max_file_size to 1/2 the loopback
Currently Devstack (and devstack gate) uses the default
max_file_size of for Swift (5GB). However the loopback file
is only 1 or 2 GB is size.

We are looking at setting a default FALLOCATE_RESERVE in swift
https://review.openstack.org/#/c/288011.
Because of this our max file size test fails due to the max_file_size
and devstacks loopback size being too small.

This patch sets this to a more sane size by default inside DevStack.
See the gerrit link above for more details.

Tempest uses a loopback of 6GB, so in this case the swift default of 5GB
should be fine.

Change-Id: If09eab7d16ae67bd252020e00e8812ff252f065b
2016-03-08 12:04:15 +11:00
Sean Dague
7580a0c3e3 Replace TENANT => PROJECT phase 1
This replaces the use of TENANT variables with PROJECT ones during the
initial setup. The openrc will still export a OS_TENANT_NAME because
many tools (cinderclient, glanceclient amoung them) will not function
without it. We warn when we do that.

Change-Id: I824b1121842eb5821034071874bf1bb2d7c3631e
2016-02-22 11:55:17 +11:00
Sean Dague
985e958dda remove keystone templated catalog
This is just another code path for little benefit in devstack which is
going to rot out. We should be opinionated here and only support the
dynamic catalog.

Change-Id: I4e5c7e86aefe72fc21c77d423033e9b169318fec
2016-02-10 15:39:31 -05:00
Dmitry Tantsur
f90c0b7d29 Stop referencing no longer existing SERVICE_ENDPOINT variable
It was probably finally removed by one of recent refactoring changes.
Now ironic gate fails due to Swift trying to access it to override OS_AUTH_URL.
This change drops this override and just uses OS_AUTH_URL.

Closes-Bug: #1535245
Change-Id: I145bec110c4299e61f2bce49df41dcd82e5d462d
2016-01-18 10:53:01 +01:00