Commit Graph

48 Commits

Author SHA1 Message Date
Chris Dent
2f27a0ed3c Replace screen_it() with run_process() throughout
run_process will use screen if USE_SCREEN=True (the default),
otherwise it will simply start the requested service. Therefore
wherever screen_it used, run_process can be instead.

Where stop_screen was found it has been replaced with stop_process.

A tail_log function has been added which will tail a logfile in a
screen if USE_SCREEN is True.

lib/template has been updated to reflect the use of the new
functions.

When using sg the quoting in run_process gets very complicated.
To get around this run_process and the functions it calls accepts
an optional third argument. If set it is a group to be used with sg.

Change-Id: Ia3843818014f7c6c7526ef3aa9676bbddb8a85ca
2014-09-11 18:59:39 +01:00
Dean Troyer
3159a821c2 Run processes without screen
This introduces new run_process() and screen_service() functions and sets the
groundwork to change how DevStack starts services.  screen_service() is simply a
direct call to the screen portion of the old screen_it() function and is intended
to run commands that only need to run under screen, such as log file watchers.

run_process() is a replacement for screen_it() (which remains until all of the
services are updated).  The usage is similar but requires updates to every current
screen_it() call to remove everything that requires the command to be interpreted
by a shell.

The old run_process() and _run_process() functions are still present as
old_run_process() and _old_run_process() to support the deprecated screen_it()
function.  These will all go away in the future once all services have been
confirmed to have been changed over.

There is a similar new set of stop process functions stop_process() and
screen_stop_service().  The old screen_stop() will also remain for the deprecation
period.

As an initial test/demostration this review also includes the changes for
lib/cinder to demonstrate what is required for every service.

I included the scripts I used to test this; tests/fake-service.sh and
tests/run-process.sh are quite rough around the edges and may bite.  They should
mature into productive members of the testing ecosystem someday.

Change-Id: I03322bf0208353ebd267811735c66f13a516637b
2014-09-09 13:54:01 +01:00
Jenkins
ba842f5374 Merge "Keystone access log in key-access screen" 2014-09-08 16:23:20 +00:00
Jenkins
17b6938125 Merge "install test-reqs when TESTONLY packages are installed" 2014-09-06 13:28:13 +00:00
Brant Knudson
966463c802 Keystone access log in key-access screen
When Keystone was run in Apache Httpd the access log wasn't in a
screen. A new screen called key-access is now started with
Keystone's access log.

Change-Id: Ie1f4d4b679e1b9c87f0cea37deb25f7b6636fe01
Closes-Bug: #1359995
2014-09-04 18:12:37 -05:00
Attila Fazekas
f71b500bea Faster nova fixed key generation
Using bc 64 times in loop is too verbose and slow,
replacing the echo/bc loop with hexdump and urandom.

The hexdump approach is 75 times faster and
does not floods the debug logs.

Using the common function for generating,
this kind of string with lib/heat and by the read_password.

Change-Id: If6a86dfaf0c21e2635c6de0a7b96a8ed7ec5b507
2014-08-27 09:21:13 +02:00
Sean Dague
f3f4b0ac78 install test-reqs when TESTONLY packages are installed
INSTALL_TESTONLY_PACKAGES in devstack only impacts .deb/.rpm files
even though most of our test requirements are actually specified
in python. This will do an explicit install of test-requirements.txt
if pip_install is called with args that look like a directory,
and test-requirements.txt exists in that directory.

Change-Id: Id21e282ddc945c819c9c8c4d724658e28bfde785
2014-08-05 17:01:16 -04:00
Ian Wienand
bdc90c5f02 Work around pip installed pkgs on Rackspace image
The upstream rackspace image has a bunch of pip installed packages as
cloud-init was installed via pip due to a lack of available system
packages.  This can break further system package installs, such as
markdown, which fails with

---
 Error unpacking rpm package python-markdown-2.4.1-1.el7.noarch
  error: unpacking of archive failed on file
   /usr/lib/python2.7/site-packages/Markdown-2.4.1-py2.7.egg-info: cpio: rename
---

Because that is a directory for the pip-installed package, and a file
in the RPM

Remove all pip installed packages on rackspace images before we start
to work around this.  I have filed an upstream issue with Rackspace
(ticket-id 140804-ord-0000134) and the issue is being worked on.

Change-Id: Id12d175143ed3b8e024d057d65fa67505c08042a
2014-08-05 14:45:02 +10:00
Jenkins
d91cef35c6 Merge "Resolve circular dependency when tracking dependencies." 2014-08-02 02:09:09 +00:00
Jenkins
fb2bf8cb9d Merge "Clean up local variable usage - misc functions" 2014-08-02 00:36:17 +00:00
Robbie Harwood (frozencemetery)
1229a0879f Resolve circular dependency when tracking dependencies.
When TRACK_DEPENDS=True, there is a circular dependency wherein virtualenv is
used to install virtualenv before a virtualenv has been established.
TRACK_DEPENDS does not work in any use case without this fix.

Change-Id: I7a6652ba091cb95dac3871b6c71edcd762a7be62
Closes-Bug: 1208867
2014-07-31 13:58:46 -04:00
Dean Troyer
d5dfa4c5c7 Clean up local variable usage - misc functions
A catch-all for a bunch of smaller functions

Change-Id: I3f97a514f9964ef36bff2f24a2f0a98614871a9f
2014-07-30 09:26:21 -05:00
Dean Troyer
50cda69f3c Clean up local variable usage - git functions
Cleans up the git-related functions in functions-common

Change-Id: I5f1851c0473e92c61b1e8af60e7ef32c3019f719
2014-07-25 13:58:36 -05:00
Jenkins
703f17e536 Merge "Fixed NO_UPDATE_REPOS variable usage" 2014-07-24 04:25:16 +00:00
Gael Chamoulaud
6dd8a8bee4 Users in service group should not have email addresses
Change-Id: Ieed9dffce5cf1e735e482dd3494ac1e103b50955
Closes-Bug: 1185201
Signed-off-by: Gael Chamoulaud <gchamoul@redhat.com>
2014-07-22 17:29:04 +02:00
anju Tiwari
6c639c9dd7 Added Oracle Linux distribution support
Enabled devstack to support oracle Linux Server

Change-Id: I1749cd3c7756a9903d2a0b0ab19606f87a4937d4
2014-07-18 09:59:59 +10:00
Paul Linchpiner
9e17974a62 Fixed NO_UPDATE_REPOS variable usage
Change-Id: I50a2fd73f30321e97c99947b62fb1729870a2c14
2014-07-13 22:23:00 -07:00
Bartosz Górski
0abde393c5 Adds support for multi-region
Change-Id: Ib85fe7cb375692b04aca4c46f61ba7e1fbfa501b
Implements: blueprint multi-region
2014-07-01 14:58:35 +00:00
Sean Dague
ea22a4fdba Revert "Build retry loop for screen sessions"
This reverts commit 0afa912e99.

This possibly made things worse, though it times in with the
trusty add, so it's hard to tell. Revert to see if grenade gets
better.

Change-Id: Ic399957fc9d4a7da28b030cdf895df061b2567c8
Related-Bug: #1331274
2014-06-27 15:22:28 -04:00
Sean Dague
0afa912e99 Build retry loop for screen sessions
There is a timing window where we might lose the commands being
stuffed into screen because bash is spawning. In those cases, loop
around and try building screen sessions again.

Change-Id: I49247de06bbd59424cb10fb9a8db145907be5138
Related-Bug: #1331274
2014-06-19 12:07:43 -04:00
Sean Dague
43d8f1cddd double the screen sleep timeout
HP Cloud 1.1 is quite slow per core. So much so that grenade is
often failing to start services because bash in a screen session
isn't ready in 1.5s. Double this to 3s to try to decrease failures
in the gate.

We should instead do this via some readiness check to decrease
races.

Change-Id: I2b066a687916742e966190f00b5b06a795d4a014
2014-06-16 09:39:19 -04:00
Monty Taylor
5cc6d2cd0c Update again and retry a failed package install
On ubuntu, if we run into an error installing the package, it can be
because of transient upstream errors with repo sync state. Although
retrying the update won't fix it all of the time, it's low cost enough
to be worth a try before we totally give up.

Related-bug: 1286818
Change-Id: I522ac0d0bd8f82dc98f386c89f66c2b743efa525
2014-06-06 17:34:28 +00:00
Yves-Gwenael Bourhis
d79a8acbb7 Fixed pip with http(s)_proxy
pip failed to insall packages with the http_proxy and https_proxy localrc
environment variables. indeed, the case of the variables was uppercase
and these variables are normally lowercase.

Change-Id: I73054aafd353ccf53986f0ec05426f6a2c4e2240
Closes-Bug: 1307492
2014-04-14 14:49:07 +02:00
Sean Dague
099e5e3f81 don't use pip -e install for libraries
libraries in openstack shouldn't be installed editable, as it
causes all manner of issues (especially complicated by the use
of namespace packages). Install these globally as part of the
devstack installation process.

Change-Id: I11acb169e74069be0618e57496ff342f9e788493
2014-04-01 08:48:00 -04:00
Adam Gandelman
539ec43882 Parse Ironic packages files/{apts, rpms}/ironic
When VIRT_DRIVER=ironic, make sure devstack picks up packages listed in its
respective packages file.  These were previously missed unless "ironic" was
explictly added to the enabled services.  This ensures required packages
are installed for any of the ir-* services.

Change-Id: I3d70009819a3a6933220cabd5a951a20c7b48849
2014-03-18 19:00:26 -07:00
Sean Dague
64bd01652e make git_clone safer
the ensures that if the function returns early, we return to a
sane directory, and not hang out somewhere that a future git
call might modify a directory in a weird way. This is especially
important in the case of stable branches where were are hopping
between stable for servers and master for clients.

Change-Id: Ib8ebbc23b1813bc1bfb31d0a079f1b882135bd39
2014-03-12 13:04:22 -04:00
YAMAMOTO Takashi
3b1f2e4e88 Fix inverted conditionals in setup_develop
This fixes regressions introduced by:
    Change-Id: Ic97e68348f46245b271567893b447fcedbd7bd6e
    ("Handle non-zero exit code from git diff")

Change-Id: I053a292c287f3035eef37db2264eda06a170f9bc
Closes-Bug: 1287513
2014-03-11 07:34:17 -04:00
Ian Wienand
b27f16d716 Detect missing packages with yum
yum -y doesn't report an error when packages are missing (see [1] for
upstream discussion).  Thus we run the output of yum through a small
awk script looking for missing packages output.

The one change required for RHEL is that python-wsgiref is included in
the distro python, so doesn't need a separate package.

[1] https://bugzilla.redhat.com/show_bug.cgi?id=965567

Change-Id: I9908ff4edbf2b0d961d25837a08a34e1417bbb02
2014-03-10 14:10:56 +11:00
Jenkins
5894db323f Merge "Move some comments of variables to right place" 2014-03-01 13:11:14 +00:00
Jenkins
3208d7ad43 Merge "Handle non-zero exit code from git diff" 2014-02-28 17:30:35 +00:00
Jenkins
2cf03d2469 Merge "make service_check fatal" 2014-02-28 07:33:42 +00:00
Jenkins
dad6e4114d Merge "Enforce function declaration format in bash8" 2014-02-28 06:32:20 +00:00
Masayuki Igawa
d20f632a70 Move some comments of variables to right place
setup_develop*() in functions has been moved to functions-common. But
some comments about the variables are still left.
This commit moves it to the right place.

Change-Id: Ic360454f1ee72f51c9979d0468dee0913e9b32e4
2014-02-28 10:35:49 +09:00
Jenkins
a340d68b1b Merge "Fix comments about System Functions" 2014-02-27 22:38:33 +00:00
Sean Dague
1237922b65 make service_check fatal
if we fail service check, we should do so in a fatal way, because
something is not right. This will be very useful in grenade.

Change-Id: I18811b0d8e6d06f364685c366cdc8f5dda3b8f7e
2014-02-27 17:16:46 -05:00
Ian Wienand
aee18c749b Enforce function declaration format in bash8
Check that function calls look like ^function foo {$ in bash8, and fix
all existing failures of that check.  Add a note to HACKING.rst

Change-Id: Ic19eecb39e0b20273d1bcd551a42fe400d54e938
2014-02-28 07:59:03 +11:00
Dean Troyer
83b6c99b50 Handle non-zero exit code from git diff
The check for a changed repo in setup_develop() 'git diff --exit-code'
returns a status of 1 when the repo has changes; trap that so errexit
does not abort the script.

Bug-Id: 1285780
Change-Id: Ic97e68348f46245b271567893b447fcedbd7bd6e
2014-02-27 12:41:32 -06:00
Jenkins
0ed4af02da Merge "enable -o errexit" 2014-02-27 14:57:43 +00:00
Jenkins
003a6a3e10 Merge "Move setup_develop() to common" 2014-02-27 12:32:54 +00:00
Sean Dague
09bd7c8fd5 enable -o errexit
devstack should run under -o errexit to ensure that we fail early
when something has gone wrong, otherwise determination of the root
failure location is often quite challenging.

this clears all the normal use cases for devstack, there could be
tests which now die early, which we're going to have to check for
later.

Change-Id: Ibd828c4f4fd95a60d3918d3d7ae90e10649479ab
2014-02-27 06:30:37 -05:00
Jenkins
cf7237c505 Merge "Add GIT_TIMEOUT variable to watch git operations" 2014-02-27 04:31:40 +00:00
Dean Troyer
a25a6f6d80 Unbuffer log output
* Force-flush log output so we don't lose log output in certain error cases.
* Slow down exit paths: add sleep to die(), wait until last moment to
  kill child processes (including the awk log output filter)

Change-Id: I1620fd33b89b237d9c2bb6206f3de2c81719f676
2014-02-26 13:17:36 -06:00
Sean Dague
45917cc4d9 xtrace less
we are xtrace happy, however that's just generating bulk in log
files that are mostly ignorable. For the basically bullet proof
functions we should not xtrace.

Change-Id: Iab4e6d270c1546e0db2a06395cefcdf7f7929c3c
2014-02-25 06:25:14 -05:00
Dean Troyer
af616d9341 Move setup_develop() to common
It's in the wrong place for current Grenade

Change-Id: Ia670198332af5945a56d708cd83d9239df0c2287
2014-02-24 10:38:18 -06:00
Ian Wienand
d53ad0b07d Add GIT_TIMEOUT variable to watch git operations
During my CI testing of each devstack change I can often see git get
itself stuck and hang indefinitely.  I'm not sure if it's transient
network issues, or issues at the remote end (seen with both github.com
and git.openstack.org) but it hits fairly frequently.  Retrying the
command usually gets it going again.  Searching for "git hanging" and
similar shows its not entirely uncommon...

This adds a watchdog timeout for remote git operations based on a new
environment variable GIT_TIMEOUT.  It will retry 3 times before giving
up.  The wrapper is applied to the main remote git calls.

Change-Id: I5b0114ca26b7ac2f25993264f761cba9ec8c09e1
2014-02-24 09:35:19 +11:00
Masayuki Igawa
f6368d3eac Fix comments about System Functions
This commit fixes comments about "System Functions".
 * Add a missing comment about System Functions in the header
 * Fix singular to plural like others

Change-Id: I3feb94cd11a6683ca80093574d60fdf7420e3af2
2014-02-20 13:34:28 +09:00
Dean Troyer
abc7b1d765 Backport Grenade updates
Backport changes made in Grenade's copy of functions since the last sync:
* d0654b9,i 4c7726e - get_release_name_from_branch()
* 7907766 - edits to install_package()

Change-Id: I0714c0b1072f1360c3c08fe24225e65e2a550fad
2014-02-12 12:10:32 -06:00
Dean Troyer
dff49a242e Split functions
Move shared and non-DevStack-specific functions to `functions-common`.  This is
a code move only with some updated comments.  The functions are now
sorted alphabetically within function groups, eg. all git-related functions
are grouped together.  The groups are listed at the top of the file.

'functions' sources 'functions-common' so no additional changes are required
for backward-compatability.

All functions shared with Grenade have also been moved.

functions-common was created from commit e0ed8ea038

Change-Id: I73bf7134fd6a60ec1ea44a5bfab08b0569b60ded
2014-02-12 12:10:11 -06:00