This is no longer being used due to Keystone PKI tokens no longer
being implemented.
In order to not break backward compatibility we create a new function
that is to be used instead and deprecate the old one. Modify the old
function to ignore the 3rd argument and display a deprecation warning.
Adjust callers to no longer create and set that directory, calling the
new function instead.
Change-Id: Id0dec1ba72467cce5cacfcfdb2bc0af2bd3a3610
swift3 is no longer actively maintained in the upstream.
That has been moved to Swift repository as s3api so we should
use s3api middleware instead. As well as swift3, s3token is
also maintained in Swift upstream.
Change-Id: I4582d81da066ab53e6f11ad1df7af91425f2b0ca
Fix a few path issues where we didn't properly use NOVA_BIN_DIR /
SWIFT_BIN_DIR.
This is part of the effort to start using a virtualenv for openstack
services.
Change-Id: I6eb383db65cc902c67c43e5cb1a16a9716a914b2
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
This adds the necessary fixes to pass a devstack run
on openSUSE Tumbleweed. Also removes opensuse 42.2 as it
is EOL for some time already and no longer actively tested
in the OpenStack infra.
Depends-On: I1b68c08c07cf6653ea58506f738cbe0054b38f3a
Change-Id: I2894482deef063fd02b0818c695a2ddbf6767039
- There are some locations where we need the raw IPv6 address instead of the
url-quoted version enclosed in brackets.
- Make nova-api-metadata service listen on IPv6 when we need that.
- Use SERVICE_HOST instead of HOST_IP for TLS_IP.
Change-Id: Id074be38ee95754e88b7219de7d9beb06f796fad
Partial-Bug: 1656329
All the evidence from [1] suggests that on opensuse swift-init is not
detaching the daemon process correctly. It's possible there's a pipe
still in play that somehow holds our ansible-streamer open.
This is a minimal fix to avoid swift-init. Although it's possible in
non-default paths to still use swift-init (and hence possibly hit
another variant of this issue), after discussions with swift
developers it was decided the intersection of tests running under our
current ansible, on suse, that would enable these services is
sufficiently small that this is the best course for now.
[1] https://storyboard.openstack.org/#!/story/2001528
Change-Id: I1b68c08c07cf6653ea58506f738cbe0054b38f3a
This reverts commit ef5ebed6c9ca3d9d47fd2a732a1542555a0f65ba.
The problem here is a backwards-incompatible change to
configure_auth_token_middleware. Plugins are still passing a
"signing_dir" which is interpreted now as the "section" argument
... this leads to an interesting red-herring issue; because "v" is a
gnu sed command for checking the version, a signing_dir of "/var/..."
(as done in most plugins) gives the weird error:
sed: -e expression #1, char 32: expected newer version of sed
I think we'll either need a new function, or dummy arguments to get
this back in.
Change-Id: I2098d4eb2747282622cf486fa7dbf216f932f58b
PKI tokens have been actively deprecated from keystone and there are
deprecations being emitted from keystonemiddleware. Because of this we
no longer need an auth cache directory in the services where the PKI
certifcates used to be stored.
Remove the creation and use of all these AUTH_CACHE directories.
Change-Id: I5680376e70e74882e9fdb87ee1b95d5f40570ad7
The pre-existing configuration for swift on devstack set's the
*-server's devices option (the root of the servers list of devices) to:
devices = /opt/stack/data/swift/1
where "1" is the node_number, and will be 2, 3, ... N if the devstack
machine is built with more than one swift node/device (pretty sure no
one does that on devstack ever).
The device(s) in the rings are named (perhaps confusingly similar to the
swift loopback image) just "sdb1", so all storage servers expect to have
a $STACK_USER writeable file system at:
os.path.join(<devices_root>, "sdb1")
That directory does not exist when you start up a devstack [1].
Currently Swift's object-server's require that directory exist before
they write data into it (even with mount_check = false!).
Unfortunately however, with mount_check=false the account/container
servers are able to create the device directory when it does not exist
[2]. Which can lead to some unfortunate results with permissions on
some deployments using mount_check = false (e.g. testing or
containerized environments). Fixing this issue [3] uncovered the
previously benign [4] mis-configuration in devstack.
Attempting
1. It was lost a long while ago I7c65303791689523f02e5ae44483a6c50b2eed1e
2. Essentially they want to:
mkdir -p /opt/stack/data/swift/1/sdb1/containers/<part#>
... but end up creating the "sdb1" dir too!
3. I3362a6ebff423016bb367b4b6b322bb41ae08764
4. Benign because the object-server share their device with the
account-container devices and they would create the dirs before trying
to write an object. It was incorrect, but worked by happenstance, which
is nearly as good as worked on purpose.
Change-Id: I52c4ecb70b1ae47e613ba243da5a4d94e5adedf2
... to be used with domain_remap. Swift will start functionally testing
domain_remap in I63428132283986bda9e5c082ffe85741449b71ba.
Change-Id: I4c1ab06d040d91fd8c314d0aa2cecbbb00adf8ad
Otherwise, we've seen intermittent "Unable to establish connection"
failures, with the main devstack log reporting things like
2017-07-19 13:54:29.973 -> start proxy service
2017-07-19 13:54:30.082 -> start OSC to store temp url key
2017-07-19 13:54:31.908 -> OSC reports failure
Meanwhile, the s-proxy screen session tells us things like
Jul 19 13:54:31.919988 -> start child worker
Jul 19 13:54:32.206598 -> still loading the WSGI app
... and ports aren't actually bound until *after* the app is loaded.
Add a wait_for_service call to wait for the proxy to come up.
Change-Id: I1a722de31b144797230991700e110353a2d937dd
The old implementation for is_$service_enabled simply checked if any of
the subservices were enabled and if so the service was considered to be
enabled. This makes disabling services complicated as it means you have
to list every single subservice which can and do change over time.
Instead also check if the generic service name is in the disabled
services list and if so don't treat the service as enabled.
Change-Id: I7fe4dfca2cd9c15069d50a04161a29c5638291cb
As part of getting swift's functional testing to work properly through
the tls-proxy we need to increase the allowed request header size in
apache. This was a non issue without tls proxy as requests hit the
eventlet webserver directly which was configured via the swift config
which sets this relatively large limit (by default devstack configures
swift to have a header size limit of 16384).
Now we pass in an optional parameter to start_tls_proxy that includes
the desired header size. lib/swift then passes in the value it also
configures in its swift.conf.
If not explicitly set we default to 8190 which is apache2's default.
Change-Id: Ib2811c8d3cbb49cf94b70294788526b15a798edd
Because the swift functests (which use test.conf) run out of a
virtualenv they don't get access to the system wide trust of the
devstack CA. Handle this by explicitly configuring the cafile to trust
in the test.conf file.
We also set the web_front_end to apache2 as that is what is terminating
TLS for us. The tests handle different web server behaviors using this
flag.
Swift's functests will need to read these values in and properly
configure things on its end.
Change-Id: I4cdba36ccab6acd76205184882ee29e4f1e12333
This defines a new function get_notification_url, which returns the URL
of RabbitMQ when you want connect to it, and uses in
ceilometermiddleware. This fixes an issue when we try to use AMQP for
RPC, but not for notifications.
Change-Id: I14450b2440806a17a90e5ddefc243868fdbe4f2c
Swift proxy logs to syslog during the devstack-gate tempest runs. To
better capture the swift logs increase the rsyslog buffer size to 6k
bytes allowing for longer messages like tracebacks.
This was setup by openstack-infra previous during our diskimage
builds.
I03e42964e14d9f930c07ed047851bdf775639c59
Change-Id: Iaa232335865410600c93f47d4777ed4f1bce08e2
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
Until we can test with a version of swiftclient that knows how to eat
auth_uri, swift still needs a working gate.
Change-Id: I09f9ad5c87b542df962a79898e06fbf1e968b1e3
Related-Change: I46294fb24e3c23fa19fcfd7d6c9ee8a932354702
Related-Change: Ie427f3b0b9eb834ff940fa5d52444a5a6cdcab15
tls-proxy is the way we're now doing a standard install using https
between services. There is a lot more work to make services directly
handle https, and having python daemons do that directly is a bit of
an anti pattern. Nothing currently tests this in project-config from
my recent grepping, so in the interest of long term maintenance,
delete it all.
Change-Id: I910df4ceab6f24f3d9c484e0433c93b06f17d6e1
The swift functional tests use a config which requires keystone ports,
we're about to make those go away. This exposes the actual auth_uri to
swift for consumption.
Change-Id: I5868dfdb8e5f0972ba04e359d212b04351502436
Currently Devstack starts all Swift services, including those
in charge of "consistency convergence" (remember Swift is eventually
consistent), data scrubbing, hard-deletion (*-reaper services)
cleanup.
But when running with Replication Factor 1 some of those services
are not needed at all. Besides, the fonctionnalities provided by
some of these services are not tested at all (neither in Tempest
nor in Swift functional tests).
Thus, in light of saving some Mo of RAM, this patch introduces a config
flag to start only a minimal set of Swift services, just what's required
to make all of our current tests pass.
The default value for this new config flag is set to start all services,
that is to maintain Devstack's current behavior.
For sake of completeness, here is the list of services that are not
going to be started is the config flag is toggled, and the associated RSS
according to our peakmem_tracker
40004 swift-object-replicator /etc/swift/object-server/1.conf
34320 swift-container-replicator /etc/swift/container-server/1.conf
33584 swift-object-auditor /etc/swift/object-server/1.conf
33328 swift-object-reconstructor /etc/swift/object-server/1.conf
31936 swift-object-updater /etc/swift/object-server/1.conf
31492 swift-account-reaper /etc/swift/account-server/1.conf
31076 swift-account-replicator /etc/swift/account-server/1.conf
29540 swift-container-updater /etc/swift/container-server/1.conf
29220 swift-account-auditor /etc/swift/account-server/1.conf
29036 swift-container-auditor /etc/swift/container-server/1.conf
So we are looking at saving at most ~350Mo of RAM (could be less
because RSS doesn't account for shared memory).
A follow-up patch will soon be proposed in devstack-gate to not run
those additional services in our Gate jobs.
Change-Id: I8a0d03ac0296a74e38efd185beb8513866eaf0c4
During the PTG there was a discussion that the screen developer
workflow wasn't nearly as useful as it once was. There were now too
many services to see them all on one screen, and one of the most
common service restart scenarios was not restarting one service, but a
bunch to get code to take effect.
This implements a 3rd way of running services instead of direct
forking via bash, or running under screen, which is running as systemd
units.
Logging is adjusted because it's redundant to log datetime in oslo.log
when journald has that.
Swift needed to have services launched by absolute path to work.
This is disabled by default, but with instructions on using it. The
long term intent is to make this the way to run devstack, which would
be the same between both the gate and local use.
Some changes were also needed to run_process to pass the run User
in. A hack around the keystone uwsgi launcher was done at the same
time to remove a run_process feature that only keystone uwsgi uses.
Change-Id: I836bf27c4cfdc449628aa7641fb96a5489d5d4e7
The proliferation of internal/admin endpoints is mostly legacy and
based on some specific deployment patterns. These are not used by
everyone, and for the devstack case aren't really that useful. We
should simplify our service catalog down to the minimum we need for
development.
Change-Id: Ided7a65c81b3a0b56f0184847fc82e17c29a771e
* iniuncomment followed by iniset for reseller_prefix just adds a
duplicate line in the config file that configparser does not like
so just remove the uncomment
* fall back to http:// url for glance->swift keystone authentication
* insecure flag to talk to swift
Depends-On: I51d56d16a5b175bd45dee09edc0b2748d72a5d06
Change-Id: I02ed01e20f8dce195c51273e8384130af53384ce
This commit switches how scripts we use to launch the installed version
in the path. Previously the scripts were manually executed in the source
repo, but this has issues if you're trying to run with py3 in a system
where python == py2. Setuptools already does the shebang magic for us
at install time, so we just need to use the installed version of the
script.
Change-Id: Iaa4d80ec607a2aa200400330e16cad3a4ca782ac
Stud is now abandonware (see https://github.com/bumptech/stud) and is
not packaged in xenial. Lets use Apache for SSL termination since its
there already.
Change-Id: Ifcba410f5969521e8b3d30f02795541c1661f83a
lib/keystone sets KEYSTONE_AUTH_URI and KEYSTONE_SERVICE_URI that
other projects should use rather than building the URL themselves.
This will allow us to more easily drop the port altogether.
Change-Id: I7467aae680215f3045d32a088af2187e1eba8169
This patch enables account management by default in Swift. This will be
leveraged by Tempest test cases validating account management APIs.
Depends-On: Id29f5ca48f92cd139535be7064107b8a61b02856
Change-Id: Ic01432939ed9b4cf0cbf20e3244d4d76847f539f
This reverts commit 7d1ec430046256ac3f536f4920a43936e28ef0c4.
This broke the sahara and layer4 dsvm jobs. The layer4 job
is voting on tempest changes so tempest is also broken.
Change-Id: Ide69f10cd85bf7ff0d86bc8cba56dedd26850362
Partial-Bug: #1573868
We really should only have code that create endpoints once, making all
osc calls get_or_set adds 3 seconds per call for no really good
reason.
This also stops creating the internal endpoints in the service
catalog. It's a pattern that we're trying not to propogate, so lets
not have it in devstack any more.
Change-Id: Ia8cefe43753900d62117beae330db46deb6a9fc9
Make it possible to construct the service users in their own seperate
domain. Changing this away from Default will not work for everyone yet,
though it does work for basic service interaction however enabling it
will allow us to start testing and hopefully gating that services aren't
relying on v2 only concepts.
Change-Id: I7e73df5dd1caabf355783da2bc0f3007ade92fba
If the variable SWIFT_STORAGE_IPS contains a space-separated list of
IPs, we can use this to create consistent rings across all proxy and
storage nodes.
Change-Id: If9307196dc7e74e4a842c95503958ae2d7f7acc7
This sets all the internal variables and service users that are set to
use project instead of tenant for clarity.
Change-Id: I4aa833bac2ee2281c5f2881f7ae1fd8e7c759f74
Currently Devstack (and devstack gate) uses the default
max_file_size of for Swift (5GB). However the loopback file
is only 1 or 2 GB is size.
We are looking at setting a default FALLOCATE_RESERVE in swift
https://review.openstack.org/#/c/288011.
Because of this our max file size test fails due to the max_file_size
and devstacks loopback size being too small.
This patch sets this to a more sane size by default inside DevStack.
See the gerrit link above for more details.
Tempest uses a loopback of 6GB, so in this case the swift default of 5GB
should be fine.
Change-Id: If09eab7d16ae67bd252020e00e8812ff252f065b
This replaces the use of TENANT variables with PROJECT ones during the
initial setup. The openrc will still export a OS_TENANT_NAME because
many tools (cinderclient, glanceclient amoung them) will not function
without it. We warn when we do that.
Change-Id: I824b1121842eb5821034071874bf1bb2d7c3631e
This is just another code path for little benefit in devstack which is
going to rot out. We should be opinionated here and only support the
dynamic catalog.
Change-Id: I4e5c7e86aefe72fc21c77d423033e9b169318fec
It was probably finally removed by one of recent refactoring changes.
Now ironic gate fails due to Swift trying to access it to override OS_AUTH_URL.
This change drops this override and just uses OS_AUTH_URL.
Closes-Bug: #1535245
Change-Id: I145bec110c4299e61f2bce49df41dcd82e5d462d