103 Commits

Author SHA1 Message Date
zhubingbing
89392f4f6a Add gnocchi ansible role
Partially-Implements: blueprint ansible-gnocchi

Change-Id: I8dd0460bd21ac0a233fab0142ec7b6079459bdc2
2016-09-06 13:17:47 +00:00
Sean Mooney
538dbac24a adds bifrost ssh key generation
- This change extend the genpwd.py
  command to generate an ssh key pair
  bifrost.
- This change bifrost config and bootstrap
  task to install the generated keys.
- This change updates the bifrost guide to
  discribe how to provide your own key.

Change-Id: I05243f58843d9195cace253dff5628fae89c78e8
Implements: blueprint bifrost-support
2016-08-31 13:12:20 +00:00
Jenkins
1fd2d434b1 Merge "Fix bandit gate jobs" 2016-08-28 15:51:00 +00:00
Paul Bourke
fc30d583f9 Fix bandit gate jobs
* Inspected each error and fixed / added nosec where appropriate.
* build-swift-ring.py which was throwing sec errors is no longer used so
  removed it.
* Removed the dev/ directory from being checked.

Closes-Bug: #1617713
Change-Id: I25664cabca4137e5c9f499c1af3f5ce78b86fb56
2016-08-28 08:52:44 +00:00
Shaun Smekel
1c68ae389b Add full support for fernet
This addresses the ansible aspects of fernet key bootstrapping as
well as distributed key rotation.

- Bootstrapping is handled in the same way as keystone bootstrap.
- A new keystone-fernet and keystone-ssh container is created to allow
  the nodes to communicate with each other (taken from nova-ssh).
- The keystone-fernet is a keystone container with crontab installed.
  This will handle key rotations through keystone-manage and trigger
  an rsync to push new tokens to other nodes.
- Key rotation is setup to be balanced across the keystone nodes using
  a round-robbin style. This ensures that any node failures will not
  stop the keys from rotating. This is configured by a desired token
  expiration time which then determines the cron scheduling for each
  node as well as the number of fernet tokens in rotation.
- Ability for recovered node to resync with the cluster. When a node
  starts it will run sanity checks to ensure that its fernet tokens
  are not stale. If they are it will rsync with other nodes to ensure
  its tokens are up to date.

The Docker component is implemented in:
  https://review.openstack.org/#/c/349366

Change-Id: I15052c25a1d1149d364236f10ced2e2346119738
Implements: blueprint keystone-fernet-token
2016-08-25 20:08:22 +10:00
Sean Mooney
3e8f9986d1 add baremetal role to install kolla deps.
Change-Id: Ie70db1b18a73528b1194e4fbcf53d09fed20f4a0
Implements: blueprint kolla-host
2016-08-10 16:03:08 +00:00
Lu lei
684b483fa8 Using sys.exit(main()) instead of main()
As we known, Exceptions are raised by the sys.exit() function. When they
are not handled, no stack traceback is printed in the Python interpreter.

In this patch we have known main() got return values(eg:0, 1), but
it can't specifies exit status when terminate the main thread by
unusing sys.exit(). So when using sys.exit(main()) instead of main()
may be more readable and reasonable.
TrivialFix.

Change-Id: I184289d28b92a7e345907247c045535a0c42c974
2016-08-02 09:32:28 +08:00
Ryan Hallisey
d360d4e5af Change the search path to look locally
In order to use tools/build.py, we need to search
locally for imports.

Closes-bug: #1592030
Change-Id: Idfa651c1268f93366de9f4e3fa80c33be42c71c3
2016-06-13 07:05:26 -04:00
Jeffrey Zhang
c31688be36 Make build.py importable lib
First step to break it down is to make it
importable lib rather than self-contained script.

Change-Id: Ibbc924c2849cc79006046f62f31a1909ce959066
Partially-implements: bp build-refactor
2016-06-07 16:40:16 +00:00
Jeffrey Zhang
9585c1be9f Remove the deprecated kolla-build section
TrivialFix

Change-Id: I3ee10efc093ed394c3af483a229dcd9dc732ec9a
2016-06-07 09:04:57 +08:00
Joshua Harlow
7c6990ccec Use an image object, recorder object and status constants
Instead of using raw dicts and passing data around via
dictionaries (which makes it really hard to figure out
what is in those dictionaries at any point) prefer to
use objects. That way people can actually understand what
the object is supposed to be, vs guessing and/or having to
decipher its usage.

The same goes for raw string constants, prefer using
named constants instead.

Closes-Bug: #1586475

Change-Id: Ide179dc6593c50696d47a2d3d4cd000f343855d4
2016-06-02 11:21:24 -07:00
Joshua Harlow
e78ae9bc61 Use a task model and dumb workers
Instead of having smart workers that need to know
about the full logic of how they operate and what
they will do which makes it hard to reuse parts of
those workers switch the model so that workers are
dumb and they just run tasks which themselves can
request additional tasks to run when succesful.

This makes it more consistent when tasks are to
be retried and also makes it easier to know which
task is running and when.

Less specialized workers == good.

Closes-Bug: #1586474

Change-Id: Ie7b7303b8bf2a80b3e26b6d4ffc27787f8c794d6
2016-06-02 11:21:09 -07:00
Jenkins
3a6a748aba Merge "Don't always unpack image build results tuple, as it may be None" 2016-06-01 08:35:53 +00:00
Jenkins
2bed574032 Merge "Pin the base distro release version" 2016-06-01 05:13:10 +00:00
Mark Goddard
b3e8ac8371 Don't always unpack image build results tuple, as it may be None
Change-Id: I21b1327da1de1bd3cc3018ba37715b0a19c1dbef
Closes-Bug: #1587501
2016-05-31 12:19:17 -05:00
Jeffrey Zhang
4044a3475e Pin the base distro release version
Pin the base distro release version in the master branch(Newton).
Only one distro version is supported. The supported versions are:

* CentOS: 7
* RedHat: 7
* OracleLinux: 7
* Debian: 8
* Ubuntu: 14.04

NOTE: Kolla will move to Ubuntu 16.04 in the Newton cycle

DocImpact
TrivialFix

Change-Id: I32c0edbee4de14c8a0cc7c961d566e84d70f19a4
2016-05-31 01:41:48 +00:00
Will Miller
18ae114e75 Fix kolla-build return codes
Add a main wrapper function to return 1 if any images failed to build;
otherwise, return 0. Rename the old main() to run_build(), which still
returns image statuses, to avoid interfering with unit tests.

Change-Id: I5817cc9cebb4f42dc13e0535223e5dc8931a3aab
Co-Authored-By: Mark Goddard <mgoddard@cray.com>
Closes-Bug: #1578404
2016-05-27 06:39:52 -05:00
Jenkins
857d8db60c Merge "Implement non-default signal handler for SIGINT" 2016-05-26 06:38:14 +00:00
Swapnil Kulkarni (coolsvap)
e1b5b149f2 Make passwords.yml file generation configurable
partially implements blueprint multiple-clouds

Change-Id: I676c4245e6f058ffbed345970ee78d1750dd0f2f
2016-05-21 01:51:17 +00:00
Jenkins
cf3e68213e Merge "Catch all the exception in the push thread" 2016-05-10 21:02:47 +00:00
Jeffrey Zhang
070bf25835 Support a custom base image
Add a new option(base-image) to support custom base image
In default, the base-image is equal to base. When using custom
image, please use command line like:

    ./tool/build.py --base-image my-image --base centos --base-tag 0.1

DocImpact

Closes-Bug: #1573544
Change-Id: If4b9c57a2b68f6f06d3e981f30b2173064e02193
2016-05-09 07:29:19 +00:00
Jeff Peeler
75492cf82b Implement non-default signal handler for SIGINT
Pressing Ctrl-C will stop a build, but does not clean up the /tmp
directory of dockerfiles. Implement a signal handler such that clean up
does occur.

Python apparently ignores signals while calling join on a queue, so
that's why the small modification was needed there.

Change-Id: I43faf67726c154b6640d3596aea61ffc59871999
Closes-Bug: #1579242
2016-05-06 17:37:42 -04:00
Michal Rostecki
6654cb1682 Add images from profile as matched only if no regex is used
Change-Id: I053e54c54883318dfc7daa64a58276487805de66
Closes-Bug: 1578955
2016-05-06 10:19:21 +02:00
Jeff Peeler
696f0b9a06 Generate image dependencies on CLI
Using the new --list-dependencies flag, one can query dependencies for a
given filter. For example:

./build.py --list-dependencies heat swift
{'base': [{'openstack-base': [{'heat-base': ['heat-engine',
                                             'heat-api-cfn',
                                             'heat-api']},
                              {'swift-base': ['swift-object',
                                              'swift-proxy-server',
                                              'swift-container',
                                              'swift-rsyncd',
                                              'swift-account']}]}]}

Also added --list-images to list all available images.

Change-Id: I1797e32e32705182a763f53329eeb5c4a361abec
Implements: blueprint images-dependency-tree-cli
2016-05-05 00:03:31 -04:00
Travis McPeak
cc33c103d4 Adding a few #nosec tags to prepare for Bandit usage
This commit adds a few #nosec tags for non-issues.  I've examined
the code and determined that these don't represent security issues
so we should add a #nosec tag that tells Bandit not to find these
issues in the future.

Closes-Bug: #1577498
Change-Id: Ic37216c08442c700c64118c78cfb46e6cedd237c
2016-05-02 20:58:57 -07:00
Jeffrey Zhang
e043fab3a0 Catch all the exception in the push thread
Change-Id: I1ba48cd25e529bdd846d3ded1f5923f312e5d17d
Closes-Bug: #1574931
2016-04-28 00:59:14 +00:00
Michal (inc0) Jastrzebski
e2fa75fce6 Override default latest tag on ubuntu
After 16.04 got released we can't build our images as we pull latest
(16.04 now) and our sources lists are built for 14.04. We pin ubuntu to
14.04 here.

Change-Id: Ib29a1c607c3a725cdaf2098edd1c4b80f06a5e38
Closes-Bug: #1575884
2016-04-27 20:35:25 +00:00
Paul Bourke
cfd806aeff Log the SHA of the Git ref for source builds
When using named refs in source builds, e.g. "reference = master", one
can't easily tell from the build output what SHA master actually pointed
to at that point and time. Update the debug output to include this.

Closes-Bug: #1566316

Change-Id: I95c1cc316ab7332fa30fa43e207a5a5512b76708
2016-04-05 13:31:03 +00:00
Jeffrey Zhang
42420830f6 Implement nova-ssh container
Add a nova-ssh container to handle the `nova migrate` and
`nova resize` case, in which the nova will use ssh to copy
files between machines.

Change-Id: Ie6675943f3aeabfbba8589d308d55b9c89d732db
Closes-Bug: #1562141
2016-04-03 07:21:17 +00:00
SamYaple
f03e06e09b Add generate_passwords.py to generate passwords
As with all tools, this is a first pass at the generation. Perhaps we
even want to move this into kolla/kolla/cmd and be generated with tox
itself in the future.

This tool, when run, will only populate empty fields that have no
values meaning that it is safe to run repeatedly on the same file.

Of note, there is no way to preserve comments in the file after it has
been processed by the yaml parser in python. Comments and sections
will remain in the passwords.yml template for additional documentation
if the user wishes to populate the file themselves.

Use SystemRandom and clean up the docs a bit to not use pronouns.

Co-Authored-By: Steven Dake <stdake@cisco.com>

Closes-Bug: #1559266
Change-Id: I2932d592df8871f1b7811059206d0b4d0553a687
2016-03-21 17:02:23 +00:00
SamYaple
279594ab64 Add local directory or file for source build
This change allows using a local directory or tarball to build from
sources by specifying the 'local' type in your kolla-build.conf:

    [keystone-base]
    type=local
    location=/home/martin/src/keystone

    [neutron-server-plugin-networking-cisco]
    type=local
    location=/tmp/networking-cisco.tar.gz

Closes-Bug: #1509625
Co-Authored-By: Martin André <m.andre@redhat.com>
Change-Id: I3f6f68b0f44efdac653d186d2625591ad7cb3b2a
2016-03-09 18:36:13 +09:00
Swapnil Kulkarni (coolsvap)
5669f7e0ad Handle kolla-build exception
kolla-build currently does not provide information
if docker service is not running. Added exception
handling.

Change-Id: I1f167804b8b4d2ad9220281954ff7ea7162a9817
Closes-Bug:#1553912
2016-03-08 09:45:04 +05:30
Vikram Hosakote
320c1f9272 Add timeout to requests.get() in kolla/cmd/build.py
When kolla-build is running, if there are network issues or if the
source's location (like http://tarballs.openstack.org) fails to
respond due to high number of concurrent requests, kolla-build just
hangs/blocks indefinitely.

This patch set resolves this issue by adding a timeout of 120
seconds for requests.get() in kolla/cmd/build.py, adds a unit test
for it in kolla/tests/test_build.py and also the "timeout" argument
in kolla/common/config.py.

Change-Id: I7c8745a20b9bd1c3f5d6a55c72a794f16fd7e513
Closes-Bug: #1548614
2016-02-28 00:54:47 +00:00
Jeffrey Zhang
da0f0cf1e6 fix the custom profile raise exception issue
Closes-Bug: #1536272
Change-Id: I6c91e522923eade16ba23711e6014e6b4b8cd3e0
2016-02-15 15:25:15 +08:00
Paul Bourke
477fc18bd0 Fix image plugin functionality for oslo.config
The new oslo.config style parsing for kolla-build.conf stopped plugins
from working. This patch fixes it and adds a unit test.

Co-Authored-By: Jeffrey Zhang <zhang.lei.fly@gmail.com>
Closes-Bug: 1534556
Change-Id: I135797c733ae0cae040a009c364073769b19e4eb
2016-02-12 06:00:20 +00:00
Jenkins
16630340c7 Merge "Ubuntu custom apt repos" 2016-02-05 08:28:53 +00:00
Allen Gao
db53ad1ddf Move the install base type check to kolla-build
The install type is converted in kolla-build, so it will never
fail in dockerfile, move the check to kolla-build just above
the install type converting.

TrivialFix

Co-Authored-By: Jeffrey Zhang <jeffrey.zhang@99cloud.net>
Change-Id: I1500d3b47e909f94ea9f68c5245297733f63a70b
2016-02-04 16:48:12 +08:00
SamYaple
9dae768e69 Ubuntu custom apt repos
Change-Id: I3afb8a29c4ae2369f29e0cc94d75ff8a3c42594b
Implements: blueprint custom-repos
2016-02-03 15:56:54 +00:00
Jenkins
87355c231a Merge "Pick up proxy buildargs from environment" 2016-01-26 11:50:04 +00:00
Jeffrey Zhang
f7f2194967 Import modules rather than objects
Base on the OpenStack Style Guidelines[0]: Do not import objects, only
modules

[0] http://docs.openstack.org/developer/hacking/index.html#imports

TrivialFix

Change-Id: Id1bc5e4b24fd561f1a9f6167af341b00a343a718
2016-01-20 20:46:12 +08:00
Paul Bourke
b82111d589 Pick up proxy buildargs from environment
We currently support any buildarg via --build-args. This patch picks up
the default supported proxy buildargs if set in the user's environment,
to allow for more transparent proxy support.

The --build-args will take precendence.

DocImpact
Implements blueprint: better-proxy-support

Change-Id: I084e5d1cc8f9993d65167878f9983ad58a68e86e
2016-01-18 11:53:25 +00:00
Jeffrey Zhang
7093d37f18 Expose the docker build_arg to build.py
With this implement, we can add variables at building stage.
For example, add HTTP_PROXY and NO_PROXY when needed like below.

    build.py --build-args \
        HTTP_PROXY:http://127.0.0.1:8080,NO_PROXY:127.0.0.1

More info about build_arg, pls check[0]

[0] https://docs.docker.com/engine/reference/commandline/build/#set-build-time-variables-build-arg

DocImpact

Implements: bp add-buildargs
Change-Id: I29ed7f536670fef59d697603dc562a69d90743c9
2016-01-18 11:42:00 +00:00
ting.wang
defd70cef9 Use dict.items() for adding Python 3 compatibility
Replacing dict.iteritems()/.itervalues() with
six.iteritems(dict)/six.itervalues(dict) was preferred in the past,
but there was a discussion suggesting to avoid six for this.

ref:
http://lists.openstack.org/pipermail/openstack-dev/2015-June/066391.html

Partially implements: blueprint python3
Change-Id: Ie931bc02f558f6db3677b44f645eb3bdf78ccae8
2016-01-16 13:44:16 +08:00
Jenkins
459d564573 Merge "Fix the package source don't register issue" 2016-01-08 03:21:49 +00:00
Jeffrey Zhang
9b19db099d Fix the package source don't register issue
Closes-Bug: #1531848
Change-Id: I3c6ceb9f1f367b8b19ab1b1d06c67817ab93cabb
2016-01-07 21:06:22 +08:00
Jenkins
90e178b57e Merge "Log information about successful push only if it really is" 2016-01-07 08:41:39 +00:00
SamYaple
ea5fa4e248 Fix broken local and global building
Two things broke in the related patches. This patch aims to fix that.

1) tools/build.py was no longer usable as it imported kolla from the
global pip install source. If that didnt exist (because kolla wasn't
installed with pip) then it breaks on import.

2) When kolla was installed globally it looked up the path that
matched the installation of docker-py instead of looking up a path
that had the stuff we wanted (kolla images directory)

Change-Id: I94f1856547ce54d506de72926b08c966a36ac608
Related-Id: I05f10d13e7ba1e2b985c2944aec71ce55630442b
Related-Bug: #1530256
Closes-Bug: #1531569
2016-01-07 03:08:14 +00:00
Michal Rostecki
7137212ab2 Log information about successful push only if it really is
The try-except-finally block in push thread was catching only
ConnectionError and logging information about successful push
even if some other error occured during push.

TrivialFix

Change-Id: I94f5ed67237d291701469a65f76146881cd67a35
2016-01-06 20:29:55 +01:00
Jenkins
ecfc955a7e Merge "Fix the kolla to find the docker image folder in virtualenv" 2016-01-06 14:53:58 +00:00
Jeffrey Zhang
f2bded7f94 Gen the image dependency in the Graphviz dot format
By using

    ./tools/build.py --save-dependency a.dot

to gen the Graphviz dot dependency file. Later, you can use the `dot`
to draw the picture.

    dot -Tjpg -o a.jpg a.dot

Implements: blueprint images-dependency-tree
Depends-On: I8e07a1b69fab5f1c587470bfd2104aaba93f0050
Change-Id: If00f4f3fb9d0b10a07ab2abb7ffb1cd9d64902f2
2016-01-06 16:33:05 +08:00