58 Commits

Author SHA1 Message Date
Spyros Trigazis
88ccd47c88 Keep old behavior of setuptools for editable installs
In the 25.0.0 release [1] of setuptools during any install
operation the package in not overwritten. If a package is
installed from another requirement via pip and then it is
installed again from git, it is not updated causing
check_libs_from_git to fail.

[1] https://setuptools.readthedocs.io/en/latest/history.html#v25-0-0

Change-Id: Ibaa1d4157816ea649f4452756fbde25951347001
Closes-Bug: #1605998
2016-07-25 14:31:51 +02:00
Sean Dague
bb35715cfe add local.conf modifying functions
This adds a set of local.conf modifying functions which make it easier
for consuming projects like devstack-gate to programatically add
elements to local.conf structured files.

Change-Id: I3427968c2bd43aba12b3619acc27f73c74f0dabb
Co-Authored-By: fumihiko kakuma <kakuma@valinux.co.jp>
2016-07-01 12:50:18 +00:00
Marc Koderer
46f8cb7f20 Add option to download all libs from git
For client debugging that invokes multiple libs it can be useful
to have all libs directly in git and not listing all of them
in LIBS_FROM_GIT.

TrivialFix

Change-Id: Ie631cc4045231ebbe8177d2d113e47e4bf83f61c
2016-05-17 08:57:39 +02:00
YAMAMOTO Takashi
c8c1c615f2 Normalize path for upper-constraints
It seems pip distinguishes paths with .. or extra / for constraints.
For example, the following directories are considered different.

    /path/to/dir
    /path/to//dir
    /path/to/dir/subdir/..

This commit tries to normalize the given directory name to avoid
"Could not satisfy constraints for 'xxxx': installation from path
or url cannot be constrained to a version" error due to directory
name mismatch.

Reference: https://github.com/pypa/pip/pull/3582
Closes-Bug: #1542545
Change-Id: Iae9d58c27d3b10bca16e4a471507c4d5c16439a0
2016-03-22 21:23:23 +09:00
Ian Wienand
7ae97298c5 Minor fixes for bashate trunk
We don't have a new bashate release yet, but this fixes some minor
issues when used with bashate trunk.

The only two things triggered are stricter checking of assignment in
local declartions (catches quotes) and one instance of evaluating
arithmetic result in tools/xen/functions.

Therefore, hopefully, this change has no effect!

Change-Id: I3fd5d55236e90c2f2c25243af87adaf0c1b763e2
2016-02-16 14:50:53 +11:00
Jenkins
b615691576 Merge "XenAPI: add support to use local.conf" 2016-01-07 04:51:19 +00:00
Jenkins
c615993fe1 Merge "Support extra dependencies when setup_develop" 2016-01-06 20:17:12 +00:00
Jenkins
8ba3cf0132 Merge "Preserve result returned by apt-get and pip" 2016-01-05 18:56:27 +00:00
Brant Knudson
0842b8147f Support extra dependencies when setup_develop
Recent pip supports using [extras] to install extra dependencies
from the project setup.cfg. Add support so that projects can take
advantage of it.

For example, if devstack is configured to use ldap, install the
extra ldap dependencies using:

 setup_develop $KEYSTONE_DIR ldap

Partial-Bug: 1479962
Change-Id: Ic13d95b99aaa4d3854b2723343e90f5de6b98aa2
2016-01-05 07:41:19 -05:00
Federico Ressi
e208d060e1 Preserve result returned by apt-get and pip
Change I855ffe9c7a75e9943106af0f70cf715c34ae25c5 and
I368fec44858bd97fc6a314fb20eed2b10932cbb1 added timing
infrastructure which hides the return value of
the main commands.  Restore the prior behaviour.

Change-Id: I4a536abefa7824ac6d42e7bbbf1079d9cc055984
Closes-Bug: #1518545
2016-01-05 08:21:05 +00:00
vsaienko
135bd484a9 Introduce iniget_sections function
This function returns list of sections from specified INI file.
It is needed for I7938f26015d953efdf720774b5ad2cee4068ac90.
'iniget_sections' is needed for automatic node enrollment to
Ironic from INI syntax file.

Change-Id: I52a69ad3bc8fc528e7f20e6b03a98f327acaaa74
Implements: blueprint ironic-ml2-integration
2015-12-29 09:42:35 +00:00
Huan Xie
cc6af3fc94 XenAPI: add support to use local.conf
XenServer install with devstack doesn't support local.conf, this fix
is to add support for using local.conf and backward-compatibility of
localrc

Change-Id: Ie494e01f8f1ecb8720e14392ef3f12d5a5a01dcd
Closes-Bug: #1528520
2015-12-28 08:56:14 +00:00
Jenkins
1f7631dce6 Merge "Enable optional Python 3 support" 2015-12-09 18:03:58 +00:00
Jenkins
563c68f3cd Merge "Fix path setup in add_sudo_secure_path" 2015-12-06 13:34:19 +00:00
Doug Hellmann
ddc3839bdc Enable optional Python 3 support
Add USE_PYTHON3 and PYTHON3_VERSION variables to allow services to use
python 3 if they indicate support in their python package metadata.

Tested in Heat here -> I837c2fba682ab430d50e9f43913f2fed20325a7a.
Project config change to add a dedicated job to Heat is here -> I0837e62d6ccc66397a5e409f0961edd4be31f467

Change-Id: I079e18b58b214bf8362945c253d6d894ca8b1a6b
2015-12-01 14:52:35 -05:00
Dmitry Guryanov
e0ac37c257 Fix path setup in add_sudo_secure_path
There are two bugs in add_sudo_secure_path.

Firstly we don't properly check if the file exists, so always append
the new line.  This will overwrite any existing changes.

Secondly the logic for checking if the path exists is inverted, so we
miss adding paths when we should.  This particularly causes failures
when installing with virtualenv's since the paths are inside the
virtualenv, rather than the standard system locations.

Change-Id: I646fe0c68958470d464fe4f3d81d5c17dd6f2ab6
Closes-bug: #1521241
2015-12-01 13:43:01 +11:00
Atsushi SAKAI
5509ed5797 Fix comment typos in inc/ and tests/ directories
valu3 => value3
enviromnet => environment
direcotry => directory
virualenv => virtualenv
editiable => editable
envirnment => environment

Change-Id: I97fb2d44a37b16d02d4fbdb08bfa33414349f651
2015-11-30 20:25:17 +09:00
Ian Wienand
523f488036 Namespace XTRACE commands
I noticed this when debugging some grenade issues failures.

An include of grenade/functions stores the current value of XTRACE
(on) and disables xtrace for the rest of the import.

We then include devstack's "functions" library, which now overwrites
the stored value of XTRACE the current state; i.e. disabled.

When it finishes it restores the prior state (disabled), and then
grenade restores the same value of XTRACE (disabled).

The result is that xtrace is incorrectly disabled until the next time
it just happens to be turned on.

The solution is to name-space the store of the current-value of xtrace
so when we finish sourcing a file, we always restore the tracing value
to what it was when we entered.

Some files had already discovered this.  In general there is
inconsistency around the setting of the variable, and a lot of obvious
copy-paste.  This brings consistency across all files by using
_XTRACE_* prefixes for the sotre/restore of tracing values.

Change-Id: Iba7739eada5711d9c269cb4127fa712e9f961695
2015-11-27 15:36:04 +11:00
Jenkins
73de4a42d9 Merge "Provide an error message on bogus config file spec" 2015-11-04 15:39:03 +00:00
Thomas Morin
85f42f698c Provide an error message on bogus config file spec
If local.conf specifies a config file addition in the following way...

[[post-config|$MY_CONF_FILE]]
[xyz]
foo=bar

...and $MY_CONF_FILE points to a file whose directory is not writable by
the user running the script, then stack.sh aborts with the following
obscure message:

  2015-09-01 08:20:08.113 | touch: setting times of '/': Permission denied

This patch modifies inc/meta-config to provide a useful error message,
such as:

  2015-09-01 08:20:08.114 | could not create config file / ($MY_CONF_FILE)

This patch also modifies inc/meta-config so that it provides an error
message if $MY_CONF_FILE is empty (instead of silently ignoring this local.conf
statement):

  2015-09-01 09:38:53.406 | bogus config file specification: $MY_CONF_FILE
  is undefined

Change-Id: I9b78407420318548561012a8672762bc7fdd6db6
Closes-Bug: 1490881
2015-11-02 15:35:01 +01:00
Jenkins
d111182f21 Merge "Don't mix declaration and set of locals" 2015-10-14 02:43:52 +00:00
Sean Dague
cb658fab15 add pip install timing
Change-Id: I368fec44858bd97fc6a314fb20eed2b10932cbb1
2015-10-12 23:10:09 +00:00
Ian Wienand
ada886dd43 Don't mix declaration and set of locals
Ia0957b47187c3dcadd46154b17022c4213781112 proposes to have bashate
find instances of setting a local value.  The issue is that "local"
always returns 0, thus hiding any failure in the commands running to
set the variable.

This is an automated replacement of such instances

Depends-On: I676c805e8f0401f75cc5367eee83b3d880cdef81
Change-Id: I9c8912a8fd596535589b207d7fc553b9d951d3fe
2015-10-07 17:03:32 +11:00
Ian Wienand
433a9b10dd Don't set xtrace directly in local call
Ia0957b47187c3dcadd46154b17022c4213781112 detects setting local
variables with subshell commands.

Although this is a particuarly benign case, it trips the test.  Rather
than putting in an ignore for this, we can easily change it to make
the test pass.  This seems better than putting in special work-arounds
to bashate, etc.

Change-Id: I37c3967c0f2d780a636a7d26cda83755085c5c69
2015-10-07 16:54:29 +11:00
Sirushti Murugesan
713fd2f6c6 Additionally install test-requirements with pip_install
When moving to Python 3, we also need to install test-requriements
to allow projects to install any python 3 test dependencies they
might be missing otherwise.

Change-Id: I2d19aa2f7ec8de869a82aa7764ab72cc8693101f
2015-09-30 15:24:59 +05:30
Sean Dague
c71973eb04 check LIBS_FROM_GIT
after the glance_store vs. upper-constraints bug, it's probably worth
actually enforcing and sanity checking that devstack is doing what
it's being asked of with LIBS_FROM_GIT. This will hopefully reduce
user generated error.

This *might* not work with the current oslo naming, we'll have to test
and normalize that.

Change-Id: Iffef2007f99a0e932b68c4c897ebbfb748cac2b4
2015-09-11 07:01:02 -04:00
Jenkins
9f52d648fb Merge "Add file creation test" 2015-08-29 00:38:16 +00:00
Jenkins
a30a98099b Merge "Add -sudo option to ini setting options" 2015-08-27 22:15:30 +00:00
Clark Boylan
05aa3846a0 Just use constraints everywhere
Having behavior on your laptop diverge from behavior in the gate is
confusing. Just use constraints on every devstack run to be consistent.
Users of devstack can edit the requirements repo in order to change
these constraints locally if necessary.

Change-Id: I843208e2e982eb04931b76f5cb4bd219fbcd70de
2015-08-18 09:39:04 -07:00
Ian Wienand
cede787482 Add file creation test
Ensure that iniadd is creating files that don't exist, as it has
historically done.

Change-Id: I2798996f3d46ff1dce410b815a87395f1bf729f9
2015-07-23 06:46:24 +10:00
Ian Wienand
f44a024f02 Add -sudo option to ini setting options
Add a -sudo option to allow these functions to operate on root-owned
files.  Test-case is updated, but not enabled by default as we can't
expect test-runner to have sudo access.

Change-Id: I134c3397314c7d9395996eb6c825ecb7e7fdfc69
2015-07-23 06:46:24 +10:00
Ian Wienand
92884ede5d ini-config : always reenable xtrace on return
These return paths doesn't renable xtrace, so output mysteriously goes
missing until the next time it is enabled.

Change-Id: I3a8018dfa9397c07534970c39fba8dc10afcbe41
2015-07-23 06:46:24 +10:00
Jenkins
2d25c9290d Merge "Fixed detection of a project in projects.txt" 2015-07-06 18:22:33 +00:00
Robert Collins
7c838616c9 Fix library runs.
Libraries were resulting in two edit-constraints runs:
 - one for the library, which adds a non-editable file path
 - then one for the editable servers, but that fails becuse
   pkg-resources couldn't parse the prior entry.

This is fixed in two parts: the dependent patch which supports parsing
file urls that have egg names, and this patch which changes from a
file path to a file url with an egg name.

Change-Id: I0f07858e96ea3baf46f8a453e253b9ed29c7f7e2
2015-07-04 09:55:56 +12:00
Robert Collins
635a5ba992 constraints file support for devstack.
Constraints files allow a global view of dependencies for devstack
without the side effect that requirements files have of installing
everything everytime. This is part of the cross project
requirements-management spec.

Change-Id: If089d30146629e6cf817edd634e5c2b80f1366dd
2015-07-02 11:37:22 +12:00
Ihar Hrachyshka
2ba4a7214c Fixed detection of a project in projects.txt
Before the fix, requirements soft-update was used for projects that are
in the file.

Change-Id: I095d42521f54b45a6b13837e2f8375fa04532faa
Closes-Bug: #1469067
2015-07-01 13:42:14 +00:00
Sean Dague
7af8a1b9b3 only soft enforce requirements not in projects.txt
We're adding the ability to have devstack plugins, which should be
much more free to require new things not in global requirements. Our
old thinking of locking down all the requirements doesn't really work
in a plugin model.

Instead, if the project is in projects.txt, continue with the old
behavior. If it is not, do a soft update (update all the requirements
we know about, leave the ones we don't). This was previously the SOFT
requirements update mode, but now it will just be the default.

Change-Id: Ic0c6e01a6d7613d712ac9e7e4a378cc3a8ce75e6
2015-06-24 05:55:02 -04:00
Robert Collins
40f3e33f73 Put requirements repo in a venv.
This is to prevent any possible contamination of test results from its
presence.

Change-Id: I5a929854745650cc6a182ffc4d15c50caabdd727
2015-06-19 08:04:00 +12:00
Davanum Srinivas
8afbaa1c80 Support for running Nova with oslo.rootwrap daemon
Nova is being enhanced to use rootwrap as a daemon. For this effort,
we need an additional entry for nova-rootwrap-daemon in the
sudoers.d/ directory.

Needed by:
I57dc2efa39b86fa1fa20730ad70d056e87617c96

Change-Id: I80c7b9dd8e9e0f940aa4e54a95b241dfc40d3574
2015-05-15 04:26:04 +00:00
Ian Wienand
c678241308 Infer rootwrap arguments from project
We can infer the binary and configuration paths just from the project
name and expanding this to the known *_DIR & *_BIN_DIR variables.  A
similar thing is done for policyd settings

Change-Id: I7c6a9fa106948ae5cbcf52555ade6154623798f1
2015-05-15 14:01:20 +10:00
Ian Wienand
fa3e841286 Create config file in merge_config_file
Change If132a94e53545d9134859aa508da7b9819ede2f8 introduced a small
regression; it added an "inidelete" which looks in the config file to
delete rows.

However, at least for the test-case, the config file isn't created
yet.  The end result is that the test fails but we don't notice.

 2015-04-17 00:55:03.169 | merge_config_file test-multiline: sed: can't read test-multiline.conf: No such file or directory
 2015-04-17 00:55:03.195 | OK

So fix this up by creating the config-file if it isn't there.

Also, add "-e" to the test file so we catch things like this in the
future.

Change-Id: I43a4ecc247f19cccf51d5931dfb687adbd23d6b1
2015-04-17 13:23:25 +10:00
Amrith Kumar
9860876f5d perform install_infra sooner in stack.sh
The install_infra() call needs to be done earlier since
pip_install_gr() depends on it. Also the fact that python module names
are supposed to be lower case but some use camel case is a problem
(for example with XenAPI).

Change-Id: I7012d77134fa0d9c746d87e837934e7dcb337e5c
Closes-Bug: #1441820
2015-04-08 16:28:34 -04:00
Sean Dague
60996b1b60 introduce pip_install_gr
This creates a new pip_install_gr that installs from global
requirements allowed versions. Now that stable branches are getting
capped all of devstack needs to be fixed to do things like this.

Change-Id: I8fd0ef2bfc544ca2576fab09d3018f760b8848fe
2015-04-08 10:28:17 -04:00
Jenkins
9e63b59a8e Merge "Add inc/rootwrap" 2015-04-03 16:25:00 +00:00
Jenkins
d7c874bca6 Merge "Move back isset to the functions-common" 2015-03-30 15:05:59 +00:00
Dean Troyer
32d6bc6ad1 Add inc/rootwrap
Rootwrap shouldn't be a unique snowflake.  Plus the binaries tend
to be called assuming PATH will find them.  Not so with venvs
so we need to work around that brokenness.

Configure Cinder and Nova to use configure_rootwrap().

Change-Id: I8ee1f66014875caf20a2d14ff6ef3672673ba85a
2015-03-29 22:29:28 -05:00
Dean Troyer
41d6f858be Clean up additional INSTALL_TESTONLY_PACKAGES bits
The original removal is in https://review.openstack.org/#/c/167669/

Change-Id: I3c59f040523d2cd1453465e80280955218880634
2015-03-26 15:36:23 -05:00
Chris Dent
ebdd9ac5b4 Provide an option to force pip --upgrade
Make it possible for someone to config

  PIP_UPGRADE=True

in local.conf and thus force pip_install calls to upgrade. In
automated testing this is probably a bad idea, but in manual testing
or situations where devstack is being used to spin up proof of
concepts having the option to use the latest and greatest Python
modules is a useful way of exploring the health of the ecosystem.

To help with visibility of the setting, and section has been added
in configuration.rst near other similar settings.

Change-Id: I484c954f1e1f05ed02c0b08e8e4a9c18558c05ef
2015-03-26 14:27:54 +00:00
Attila Fazekas
1bd79596c3 Move back isset to the functions-common
isset function was moved to config file related functions by accident,
this change also simplfies the isset in a bash >=4.2 way.

All supported distro has at least bash 4.2. (RHEL6 used 4.1)

Change-Id: Id644b46ff9cdbe18cde46e96aa72764e1c8653ac
2015-03-26 14:00:02 +01:00
Sean Dague
eeb7bda510 eliminate TEST_ONLY differentiation
devstack is a development and test environment, but by default we were
only installing the runtime dependencies. We should install all the
testing required packages as well.

Change-Id: I7c95927b9daad15766aac9d1276b10ca62efb24c
2015-03-25 11:55:32 -04:00