371 Commits

Author SHA1 Message Date
Zuul
4caa9c3580 Merge "Provide finite value for systemd's TimeoutStopSec" 2017-11-21 20:04:29 +00:00
Zuul
2647fc2ac1 Merge "Drop support for "kvmibm" distro" 2017-11-15 00:05:24 +00:00
Michał Dulko
3b815a3cb2 Provide finite value for systemd's TimeoutStopSec
This commit switches TimeoutStopSec in DevStack's systemd unit files
from "infinity" to "300". There are two motivations for that change:

* 5 minutes should be more than enough to stop a service.
* systemd included in CentOS 7 and RHEL 7 doesn't support "infinity" as
  a value, "0" should be provided instead. When "infinity" is set,
  systemd will kill the service instantly, leaving service children
  processes orphaned. Instead of differentiating here, we can just set a
  sane, finite number.

Closes-Bug: 1731275
Change-Id: I0a079ea9879fa4fbba23104c2f5ab6e0721a2a2a
2017-11-14 16:12:56 +01:00
Jenkins
e4eef81c2b Merge "Enable reloadable config in services" 2017-10-10 22:04:56 +00:00
Radoslav Gerganov
2b97a81640 Remove references to $USE_SCREEN in comments
The USE_SCREEN variable is not used any more

Change-Id: I29ad9cdb6c8498404502d91fbc4e1299bf4a633e
2017-10-10 16:53:18 +03:00
Markus Zoeller
b8335eebe8 Drop support for "kvmibm" distro
The IBM hypervisor distro "KVM for IBM z Systems" gets discontiued,
like announced in March 2017 [1]. The key dates are:

* 03/2017: announcement
* 08/2017: the last day to order (EOM)
* 03/2018: the End of Service (EOL)

As the CI which tests OpenStack with KVM on IBM Z doesn't rely on this
distro anymore and EOM has reached, we remove the Devstack support for
this distro.

This basically reverts commit a5ea08b of Dec 2015.

NOTE: This doesn't affect other distros which have KVM on Z support.

References:
[1] FAQ for KVM for IBM z Systems Delivery Strategy Change
    https://www-01.ibm.com/common/ssi/cgi-bin/ssialias?htmlfid=ZSQ03110USEN&

Change-Id: I009ae4779588615633bff81d0c47a1b879ec9279
2017-10-10 11:08:09 +02:00
Jenkins
7c9542eb02 Merge "Install minimal lsb-release package on openSUSE" 2017-09-28 12:16:50 +00:00
Jenkins
4a50053c74 Merge "Make is_ipv4_address a bit more robust" 2017-09-25 13:37:32 +00:00
Dirk Mueller
5ef8a125cc Install minimal lsb-release package on openSUSE
lsb-release is a dependency of "lsb", so it used to work
before just fine as well, but it was installing about 300MB
of "stuff" that we don't actually need..

Change-Id: I25c7c750cbaeb40bf4f2e8695608c4b1003289ea
2017-09-25 13:09:43 +02:00
Jenkins
5aff50b09d Merge "Display unaccounted time in "DevStack Component Timing"" 2017-09-21 19:42:27 +00:00
Jens Harbott
7617ac2085 Make is_ipv4_address a bit more robust
Still not ideal, but at least should avoid matching IPv6 addresses.

Change-Id: Ibb64263fdb0308f56c18518289501dd4642dcbad
2017-09-20 08:56:10 +00:00
John L. Villalovos
5ad4e58868 Display unaccounted time in "DevStack Component Timing"
In the "DevStack Component Timing" section, display the unaccounted
time.

Also add the units (seconds) to the output to make it clear to viewers.

Change-Id: Iaca82cc54a355f7077e20e548b771e53387f6628
2017-09-19 08:06:17 -07:00
Jenkins
3cc59ab38c Merge "Install neutron rpms/debs when lib/neutron is used" 2017-09-19 13:23:09 +00:00
Jenkins
b2503f4692 Merge "Ignore stderr when checking for Python version" 2017-09-19 01:51:10 +00:00
Jenkins
cd06b2ee81 Merge "Enable graceful shutdown for services" 2017-09-19 01:49:24 +00:00
Javier Pena
6bd4924cec Ignore stderr when checking for Python version
https://review.openstack.org/504171 prevented Python version detection
from failing when python3 is not installed. However, "which python3"
returns a message in stderr when python3 is not there, and this output
can make diskimage-builder get confused when parsing
source-repository-images.

Change-Id: Idb649dc341ede73c39954b0432ef3cacf379ed37
2017-09-15 15:57:32 +02:00
Jenkins
f56a0ae2ec Merge "Fix devstack python detection if python3 is missing" 2017-09-14 23:42:06 +00:00
Sean Dague
4324f4ecd9 Fix devstack python detection if python3 is missing
On platforms without python3 installed devstack fails the python
version check. This does it more gracefully.

Change-Id: I4d79a41eb2d66852ab1a1aa8bd383f8e3e89cdc0
2017-09-14 13:13:15 -06:00
Ihar Hrachyshka
387aadd14e Install neutron rpms/debs when lib/neutron is used
lib/neutron service prefixes are neutron-* not q-*. We should install
those packages either way.

The patch moves files/*/neutron into files/*/neutron-common so that we
can correctly match */neutron against service specific dependency files
(f.e. */neutron-agent) and load the common packages if any neutron-*
service is present.

Change-Id: I57b36f2ed3f33737223a35d9ed734bb414f31e0b
2017-09-14 12:37:27 -06:00
James E. Blair
35a0c573b3 Update gate clone error message for Zuul v3
Change-Id: I351de28aa5d5f25187953f1a4502445bc2ce76cc
2017-09-10 15:38:21 -07:00
Jenkins
5b42dcef5d Merge "Get default python versions from interpreter" 2017-09-10 16:59:05 +00:00
Dinesh Bhor
ef60f2b718 Enable reloadable config in services
uwsgi services:
[1] By default uwsgi is set to exit on reload this breaks config reloading
    of the service [1][2]. It needs to be set to 'false'.
[2] Requires to add 'systemctl reload' command support by adding ExecReload
    in unit file.

Non uwsgi services:
[1] Non uwsgi services only requires to add ExecReload in unit file.

There was a similar patch submitted by Matthew Treinish [3] but it was
already set to workflow +1(not merged as having dependency on other patch)
and it was having some issues as specified in comment and missing reload
functionality for other services.

[1] https://etherpad.openstack.org/p/uwsgi-issues
[2] http://uwsgi-docs.readthedocs.io/en/latest/Options.html#exit-on-reload
[3] https://review.openstack.org/#/c/490904/2

Change-Id: I78f5e9d4574671c74a52af64724946feb41c2d7a
2017-09-08 12:49:00 +05:30
Jenkins
1ad91a7d4b Merge "Remove screen support from devstack completely" 2017-09-07 13:38:03 +00:00
Matthew Treinish
477a962861 Enable graceful shutdown for services
1] Process using uwsgi:
   uwsgi services doesn't support for graceful shutting down [1].
   It requires some changes in unit files [2] including adding below
   graceful shutdown hook and changing KillSignal:
      --hook-master-start "unix_signal:15 gracefully_kill_them_all
   All the steps and changes required are specified in etherpad [1].

2] Non uwsgi services needs below changes:
   In [service] section:
   a. Add KillMode = process
   b. Add TimeoutStopSec = infinity

NOTE:
Creating unit file for services other than uwsgi is handled by the
'write_user_unit_file' function [3]. This function is common for all
the services so this patch adds the above mentioned parameters for
services using ServiceLauncher also though they don't require.

Added a new stackrc variable WORKER_TIMEOUT which is required to add
graceful shutdown support to uwsgi services. It will be set as a value
to 'worker-reload-mercy' [4] in uwsgi file of service. The default
value set to this variable is 90.

[1] https://etherpad.openstack.org/p/uwsgi-issues
[2] https://www.freedesktop.org/software/systemd/man/systemd.kill.html
[3] 2967ca3dfd/functions-common (L1439-L1461)
[4] http://uwsgi-docs.readthedocs.io/en/latest/Options.html#worker-reload-mercy

Co-Authored-By: Dinesh Bhor <dinesh.bhor@nttdata.com>
Change-Id: Ia95291325ce4858b47102dd49504250183f339ab
2017-09-06 11:55:04 +05:30
Sean Dague
cdba1b3716 Remove screen support from devstack completely
This tears out the alternative path of using screen, so that we only
use systemd enabled paths. This simplifies the number of ways that
devstack can be run, and provides a much more reliable process
launcher than the screen based approach.

Change-Id: I8c27182f60b0f5310b3a8bf5feb02beb7ffbb26a
2017-09-01 15:08:17 -04:00
Ian Wienand
6d213dfda7 Get default python versions from interpreter
Query the python2/python3 interpreter for it's version to fill in
PYTHON3_VERSION and PYTHON2_VERSION defaults.  This means on a
python3.6 platform such as Fedora 26, we don't need to override the
default.

Change-Id: Id826f275b99b9f397b95e817941019fc503daa1d
2017-08-25 11:26:37 +10:00
Omer Anson
5158486124 Add function is_plugin_enabled
Add a function which tests if a plugin has been enabled with
enable_plugin. This is helpful if two co-ordinating projects want to run
specific setup in devstack in one only if the other is enabled.

Change-Id: Ibf113755595b19d028374cdc1c86e19b5170be4f
2017-08-24 17:47:37 +03:00
Jenkins
2967ca3dfd Merge "Actually check if roles are set" 2017-06-26 23:26:27 +00:00
Jenkins
2b06b1a072 Merge "Change restart Always to always" 2017-06-26 06:43:04 +00:00
Jenkins
3ed001821b Merge "Revert "systemd: Always create the systemd unit files"" 2017-06-23 23:41:16 +00:00
YAMAMOTO Takashi
c087c71834 Revert "systemd: Always create the systemd unit files"
This reverts commit 15b0a5f1eb849a30bcea9a1e437e9a88ae2c6f92.

The change had an assumption that a service
is properly configured even when it isn't enabled.
The assumption is not true.

Change-Id: Ib5a8ffe63eaec15bc29bfdd133db7169507bab82
Closes-Bug: #1698129
2017-06-23 15:08:38 +00:00
Jenkins
418bbddc81 Merge "Provide timings for OSC during devstack run" 2017-06-15 19:36:32 +00:00
Sean Dague
85cf2933cc Provide timings for OSC during devstack run
The OSC number remain high, and it's useful to understand how much
time we spend making OSC calls, especially to surface it relative to
other items. The way we embed this in our code makes it hard to
instrument.

This patch creates a wrapper function for OSC which collects the timings
then aliases `openstack` to that function. This means any invocations of
the openstack utility goes through our function while devstack is
running. Because this is an alias it only affects the stack.sh shell and
any subshells.

This also moves the time tracking infrastructure to count in ms,
instead of s, because some of these operations are close enough to a
second that rounding early is losing way to many significant
digits. We divide by 1000 before reporting to the user.

Change-Id: Ic5f1844ce732d447ee980b3c9fdc417f72482609
2017-06-14 16:43:49 -07:00
Clark Boylan
3e9b562d0b Actually check if roles are set
In the helper functions to check if roles are set and if not add the
role and return the id we weren't actually checking if the role was set.
The reason for this was we grepped for name values while outputing only
uuid values with OSC. Fix for this is straightforward, we just add the
--role argument to OSC which will filter for us then we don't have to
use a grep on the wrong value type.

Change-Id: I2691b347d2a6273100deb4a1750ab353a8e49673
2017-06-14 15:35:29 -07:00
Markus Zoeller
15b0a5f1eb systemd: Always create the systemd unit files
Commit 5edae54 introduced the usage of systemd in Devstack. This allowed
the transition away from 'screen'. Systemd needs "user unit files" to
describe the services. Currently, those unit files get only created when
an openstack service (n-cpu, c-sch, g-api, ...) is in the list of enabled
services (`ENABLED_SERVICES`). This means, when Devstack is fully stacked,
there is no way to start the systemd unit of an openstack service which
is *not* in that list.

This commit changes that behavior, and creates the systemd unit files
independently of the list ENABLED_SERVICES. This means, when Devstack
is fully stacked, I can start a systemd unit of an openstack service which
wasn't in the ENABLED_SERVICES list. This allows more flexible lifecycle
management of openstack services in the gate, which is useful for tests
which test components which are not in the "default configuration" (e.g.
the "nova-serialproxy" service).

The `clean.sh` script purges all traces of systemd user unit files created
by devstack.

Change-Id: I0f7e1ee8723f4de47cbc56b727182f90a2b32bfb
2017-06-14 14:29:39 +10:00
gong yong sheng
07b3bc24a3 Change restart Always to always
Change-Id: I1cb00cc012eda72ff50e958ba1fb04daeac69e26
Closes-bug: #1695822
2017-06-05 14:04:00 +08:00
Jenkins
8085df74a8 Merge "Use the proper keystone endpoints in clouds.yaml" 2017-05-26 19:24:51 +00:00
Jenkins
3420019d9d Merge "remove some unused *_PROTOCOL from export" 2017-05-16 14:46:50 +00:00
Jenkins
37a6b0b2d7 Merge "Set SyslogIdentifier for uwsgi processes" 2017-05-05 19:50:54 +00:00
Sean Dague
f41bf2a92f Set SyslogIdentifier for uwsgi processes
Now that we aren't using native journal messages by default, the
syslog identifier of all the uwsgi processes is the same. We should be
more explicit with those.

Change-Id: Id5406d02407b022d4016517c2e18890973876d88
2017-05-05 07:54:26 -04:00
Sean Dague
148d58c351 Make devstack fail early for common systemd pitfalls
There are a couple of issues that have ended up being hit by devstack
plugin authors which we can detect and error in a much nicer way
instead of them having a cryptic systemd failure.

Change-Id: I45e4ac363aeefb4503015f9e1b57c58f79b58f40
2017-05-04 13:08:25 -04:00
Sean Dague
803acffcf9 Make ./clean.sh work in more situations
When transitioning between different wsgi modes, or service modes, we
should really safely stop and cleanup things that are started in any
service mode, which makes it easier to ensure that we don't leave
things around from past runs.

Change-Id: I33acbee39e1a2da2bfd79a5dd54b84a12a778be1
2017-05-02 06:20:22 -04:00
Sean Dague
c006bbdeb2 make USE_SCREEN=False imply USE_SYSTEMD=True
In order to start making the transition in the gate make
USE_SCREEN=False also mean USE_SYSTEMD=True. We'll never actually
declare USE_SYSTEMD=True in the gate (as that doesn't exist for stable
branches), but this will let us roll over the existing transition.

We also have to install systemd-python 234 because we are recording
exception info in the journal, and all versions before that had a bug
in processing that.

Remove the somewhat pointless screen following journalctl commands. We
really don't want or need those, and they tend to build up over time.

Depends-On: I24513f5cbac2c34cf0130bf812ff2df6ad76657c

Change-Id: I6af6d1857effaf662a9d72bd394864934eacbe70
2017-04-28 13:21:22 +00:00
Monty Taylor
e43f60ba2a
Use the proper keystone endpoints in clouds.yaml
KEYSTONE_SERVICE_API is the keystone endpoint and it is what we should
use. The admin url should DIAF - but until it does, it CERTAINLY should
not be the thing we put into clouds.yaml.

Change-Id: If8196a04f852f633e0b7548793f68c92376aa6da
2017-04-27 19:32:29 -05:00
Sean Dague
1b245cef7f Cleanup duplicate get_or_add_user_domain_role
It turns out that we ended up with duplicate versions of this function
merging on top of each other within 3 days, and gerrit didn't catch
it. Boo gerrit. Boo bash.

Change-Id: Ic6aa2f9bafdec906de2bc51d5929beeec48a6a40
2017-04-20 00:19:20 +00:00
Sean Dague
0effa1a6cb remove some unused *_PROTOCOL from export
Things like SERVICE_PROTOCOL and KEYSTONE_AUTH_PROTOCOL shouldn't
really be exported in openrc as they encourage using them directly to
build up keystone urls instead of actually using the OS_AUTH_URL.

Remove them.

Change-Id: I4b7cc680f7f14dae29b706a227be540c9e212cad
2017-04-18 15:16:37 -04:00
Jenkins
602e2e93b6 Merge "Handle uwsgi on systemd properly" 2017-04-07 22:37:24 +00:00
Jenkins
03fbc0d71b Merge "Do not use libvirt-bin package anymore" 2017-04-07 00:37:36 +00:00
Matthew Treinish
9c5ffd8d13
Handle uwsgi on systemd properly
uwsgi is a different service type under systemd and shouldn't be run as
a standard oneshot type. The uwsgi docs outline a good pattern for
writing systemd unit files:

http://uwsgi-docs.readthedocs.io/en/latest/Systemd.html

This commit takes those suggestions and creates a separate path for
writing uwsgi unit files.

Change-Id: I9b541b86781afdded311dba058cedd783e1a0dfa
2017-04-03 12:38:29 -04:00
Sean Dague
5edae54855 initial work to enable systemd service running
During the PTG there was a discussion that the screen developer
workflow wasn't nearly as useful as it once was. There were now too
many services to see them all on one screen, and one of the most
common service restart scenarios was not restarting one service, but a
bunch to get code to take effect.

This implements a 3rd way of running services instead of direct
forking via bash, or running under screen, which is running as systemd
units.

Logging is adjusted because it's redundant to log datetime in oslo.log
when journald has that.

Swift needed to have services launched by absolute path to work.

This is disabled by default, but with instructions on using it. The
long term intent is to make this the way to run devstack, which would
be the same between both the gate and local use.

Some changes were also needed to run_process to pass the run User
in. A hack around the keystone uwsgi launcher was done at the same
time to remove a run_process feature that only keystone uwsgi uses.

Change-Id: I836bf27c4cfdc449628aa7641fb96a5489d5d4e7
2017-03-28 07:19:15 -04:00