46 Commits

Author SHA1 Message Date
Jenkins
d08db850ec Merge "use oslo.config instead of raw argparse.ArgumentParser" 2015-12-31 07:35:05 +00:00
Jeffrey Zhang
3b3d6df0c2 use oslo.config instead of raw argparse.ArgumentParser
* Use oslo.config library
* kolla-build section is deprecated in favor of DEFAULT section
* Remove the deprecated `--template` option
* `--no-cache` is deprecated in favor of `--cache` and `--nocache`
  options, which are using oslo BoolOpt inverse feature[0]

[0] https://github.com/openstack/oslo.config/blob/master/oslo_config/cfg.py#L1074

Closes-Bug: #1528542
Change-Id: I62c5ca3c8e057917d2798256e9d9efc8f6578f60
2015-12-30 18:53:05 +08:00
Jeffrey Zhang
991be99c89 Make the logging more safe
TrivialFix

Change-Id: I72c541c1846e503486b402e88d5b29364920fc83
2015-12-30 15:11:43 +08:00
Jenkins
c5c5c6e444 Merge "Push the image as soon as it is built successfully" 2015-12-28 07:34:47 +00:00
Jeffrey Zhang
bdd92da6ae Push the image as soon as it is built successfully
Closes-Bug: #1529327
Change-Id: Iad6c1454f42d3410c82c722b60d0d305b9eec129
2015-12-28 01:11:46 +08:00
SamYaple
73ac472e4e Fix logging params
Something was missed during the logging update

TrivialFix

Change-Id: Ic611b7a0e2a53725b1a3a76aba56faeca5f68dfa
2015-12-25 03:26:27 +00:00
David Moreau Simard
e8ad7488f6 Make the yum repositories configurable in the base image
- Removed hardcoded yum repository configuration in favor of
  commands dynamically generated based on repo-url and repo-file
  arguments. We maintain a sane default set of repositories.
- Added generic rpm_setup_config parameter to add support for
  installing .rpm or .repo files before building containers.

Co-Authored-By: Ryan Hallisey <rhallise@redhat.com>
Implements: blueprint custom-repos
Change-Id: I1b3a7647a9e7239de3cd162cb6f464f05632bde1
2015-12-24 02:08:52 +00:00
Michal Rostecki
d7095790a2 Use c-style string interpolation for log messages in kolla-build
TrivialFix

Change-Id: I4b01402f8feb029563a90d4941c8d829a05794e0
2015-12-23 16:36:13 +01:00
Michal Jastrzebski
9cd603dfde Add kolla version to base container
We want to record kolla version of running containers to be able to
detect whether or not we need to perform certain downtime-causing
actions during upgrade.

Change-Id: Ie113029da98303e6809d56edbf6d8de37be128d7
Implements: blueprint record-version
2015-12-17 17:55:04 +00:00
David Moreau Simard
70223bb59d Alphabetically sort long list of parameters/arguments
TrivialFix
Change-Id: I39ad214d6a8e1fbdba799076d6a5371cc47824a1
2015-12-15 10:26:51 -05:00
Michal Jastrzebski
3f2a4f2c06 Record version on container build
This changes default behaviour of build.py to instad of putting latest
tag on it, it puts current kolla version as found in setup.cfg

Change-Id: I4d6e9a0159c6a5598abd58072594df4204427308
Partially-Implements: blueprint upgrade-nova
Partially-Implements: blueprint record-version
2015-12-11 11:02:46 -06:00
Martin André
06794fdc6b Catch exception when directory creation fails
The build script now creates now creates a plugins directory for source
builds to store plugins required for the container, and creates a tar
of all plugins that is extracted at runtime.

However, the creation of the 'plugins' directory is not handled
correctly and generate an exception in the running thread if the
directory exists. This is the case for example if the build script
retries a failing image.

This commit fixes the timeout issues we've experienced with the gate.

Change-Id: Ic5d4fd1ddf71f01c547130518e311fded911c445
Closes-Bug: 1524897
2015-12-11 20:07:02 +09:00
Jenkins
b121d3cf3b Merge "Allow fetching plugins as part of the build" 2015-12-08 10:17:46 +00:00
Artur Zarzycki
dc2ad7d134 Set docker api version to auto
Change-Id: I25d957a2368fa4e52b5fa9253d7ea5ec4046d830
Closes-Bug: #1521949
2015-12-03 11:28:21 +01:00
Paul Bourke
2581e3099c Allow fetching plugins as part of the build
This change allows additional archives to be fetched as part of the
build and added to images.

An example use case is the installation of plugins for neutron. For
example, to add the networking-arista plugin to neutron-server, one
would do the following in kolla-build.conf:

[neutron-server-plugin-networking-arista]
type = url
location = http://tarballs.openstack.org/networking-arista/networking-arista-master.tar.gz

The format for plugin sections is <image-name>-plugin-<plugin-name>

The required Dockerfile additions will be added in the next patchset.

Background to this change:
https://review.openstack.org/#/c/207545/3/docker_templates/neutron/neutron-server/Dockerfile.j2

Related: blueprint neutron-third-party-plugins
DocImpact
Change-Id: Idec2ca9fd06163c7d820e8c4bc9ebca8d9673856
2015-11-30 10:00:53 +00:00
Swapnil Kulkarni (coolsvap)
59a1c8b9fb Remove hardcoded MAINTAINER in Dockerfiles
Added new option in kolla-build.conf

Change-Id: I45fe51966bcb59ea19d112281ba3d5a1ba091a56
Closes-Bug:#1514304
2015-11-23 11:03:47 +00:00
Hui Kang
e88e9bd29c Add error handler for unknown build type
Change-Id: Ia5d9ad1ffa119c2dd3e74272df3fb6d78501ec09
closes-Bug: #1513175
2015-11-22 00:10:00 -05:00
Michal Rostecki
eb0d0cdc9e Add Python 3.x support
Implements: blueprint python3

Change-Id: I1e02568907aa38dd0efcb428235004ce47d73af8
2015-11-16 20:36:47 +01:00
Martin André
b2530cdd44 Fix --retries option to kolla-build
Off by one error made the --retries option control the number of tries
rather than the number of retries.

Closes-Bug: #1514730
backport: Liberty

Change-Id: I976a8bb9e489d226f44926a6562d4d2af5de099c
2015-11-10 17:30:55 +09:00
Jenkins
c2191b101f Merge "Make RHEL build properly" 2015-11-06 15:55:43 +00:00
Jenkins
b178ed2021 Merge "Few improvements in Kolla tools. Added Ubuntu support." 2015-11-06 12:22:35 +00:00
Kirill Proskurin
396014f8d1 Few improvements in Kolla tools. Added Ubuntu support.
pip install default prefix in Ubuntu is /usr/local, and Kolla tools scripts
didnt respect that. So I added few OS checks in this scripts.

I improve config path check in build.py. Added more verbose error if we can't
find config directory.

Change-Id: Ide521ed205b0dc1fc27e237a9a8f4da0168e664f
Closes-Bug: #1512302
2015-11-06 14:03:30 +03:00
Steven Dake
5eb15d2e4a Make RHEL build properly
build.py -b rhel -t [rdo|rhos|source|binary]

The last patch for this didn't quite fix the problem properly as
it only permitted RHOS builds.

backport: liberty

Change-Id: I27eed202560adce450c07d043cc224e7a6c6bbf6
Closes-Bug: #1513088
2015-11-06 02:33:21 -05:00
Jeffrey Zhang
27c3f6ff48 Use the absoluate path
Use the absoluate path rather than that with `..`. This will be
helpfull for end-user to see where is the folder/file.

Closes-Bug: #1513726
Change-Id: I7169952d874ddf14469605444044de0163b033d3
2015-11-06 15:10:41 +08:00
Ryan Hallisey
0340bc4043 Fix RHEL builds
Register with RHEL on the host machine and use yum to setup
the repos in the container.

Change-Id: I38aaf43fffaf7a235e69b330d5d9f0f1be31fe83
Backport: Liberty
Closes-Bug: #1513088
2015-11-04 09:32:08 -05:00
Paul Bourke
592b3bd066 Add build profiles to build.py
Add ability to define "profiles" in kolla-build.conf, which are
predefined sets of images to build at once.

The supplied profiles match what Steven defined in the
restructure-template bp, though others can easily be added by the user.

Multiple profiles can be supplied at once, as well as supplemented by
the existing regex mechanism, e.g. to build profiles infra, main, and
also ironic, one can use:

tools/build.py --profile infra --profile main ironic

Change-Id: I0c6f450152cb23dcfc58e0969669fcedf03fab01
Implements: bp restructure-template
Doc-Impact
2015-10-19 13:31:30 +01:00
Kuo-tung Kao
52b7253811 add "registry" flag to "tools/build.py"
With registry flag,
build image using `tools/build.py --registry 172.22.2.81:4000`
the image name looks like `172.22.2.81:4000/kollaglue/data`
When user use kolla-ansible,
user set `docker_registry` to `172.22.2.81:4000` in `/etc/kolla/globals.yml`.

Build image using `tools/build.py -n abcd`
the image name looks like `abcd/data`
When user use kolla-ansible,
user set `docker_namespace:` to `abcd` in `/etc/kolla/globals.yml`.

build image using `tools/build.py -n abcd --registry 172.22.2.81:4000`
the image name looks like `abcd/data`
When user use kolla-ansible,
user set `docker_namespace:` to `abcd` in `/etc/kolla/globals.yml`.
user set `docker_registry` to `172.22.2.81:4000` in `/etc/kolla/globals.yml`.

With the feature, it will reduce user confusing and mistaking.

Change-Id: I18ac7a3ccec032888e35f5e9a79fc190760cc8a0
Closes-Bug: #1505056
2015-10-17 10:08:03 +08:00
Kuo-tung Kao
9bacf1bdbd add "--template-only" flag for tools/build.py
Add "--template-only" flag for tools/build.py
to generate dockerfile for read easily.

Change-Id: Ic4cf43d925b5f2d3dae809c11f3f95c51b4bbce2
Closes-Bug: #1501671
2015-10-05 14:57:29 +08:00
Steven Dake
0e99b69de4 Implement a install_type and install_metatype
This prepares for the RHEL OSP implementation by making the build
tool convert all binary-* into an install_type of binary and * into
an install_metatype variable substitution inside the Dockerfiles.
Further binary-* is substituted as install_name to enable proper
building only.

Change-Id: Ib681b29176eb79a3cab12ec824313fdecb6e7a5f
Partially-Implements: blueprint rhel-based-image-support
2015-09-28 03:16:48 -07:00
Martin André
6c5aeeee41 Consolidate kolla-build configuration in one file
The source location definition previously in build.ini was merged in
etc/kolla/kolla-build.conf since it is really configuration for the
kolla-build tool.

Change-Id: I7531af0029bdfeabbaefb8a6aa97b70bb12c021a
Closes-Bug: #1496702
2015-09-21 21:20:48 +09:00
Martin André
7a4a9eef20 Implement configuration file for kolla-build
Kolla build script configuration now lives in
/etc/kolla/kolla-build.conf.

This allows overriding configuration options without the need to pass
arguments on the command line.

Command line arguments still get precedence.

Closes-Bug: #1496185
Change-Id: I69482a11437087f19d7f76b379dca85b1f622268
2015-09-17 17:21:31 +09:00
Vladislav Belogrudov
34cbe82e6e Correct name of source archive if reference contains '/'
If we have reference like stable/kilo name of source archive
will contain only last part of reference. We should replace '/'
with '-' to have full name. This is useful for correct
identification of source code afterwards.

Change-Id: I1ff1015e3f7926110d0fcb59504e58a5e80a75f9
Closes-Bug: #1492279
2015-09-07 12:13:59 +03:00
Vladislav Belogrudov
c22cf33268 Allow arbitrary source archive name to be imported into Dockerfile
Currently we cannot import source archives with names different
than expected by hardcoded line in Dockerfiles. This worked well
for Openstack services' tarballs where we expected SERVICE-* root
folder after extraction or kanaka-noVNC for nova-novncproxy docker.
The latter fails if one tries to clone or get tarball under other
names. This fix allows any archive (tar,tgz,zip) or repo name to be
imported into dockerfile.

Change-Id: I869a6a19afaf0e93925572746c22b7589b6600c9
Closes-Bug: #1491415
2015-09-03 16:49:50 +03:00
Sam Yaple
45f625a73e Remove dest_filename from build.ini
We can, and should, figure out the filename dynamically rather than
hardcode that value in build.ini since it is not actually a
configurable paramater.

Change-Id: I496d6555e9fa356ab09e62063fd707f43ed08121
Closes-Bug: #1490386
2015-08-31 04:09:59 +00:00
Sam Yaple
cbd42ca6e9 Move docker_templates to docker dir
Updated build.py to reflect this change.
Deprecate --template option and make it a noop.

Change-Id: I7cd98d1ee684a4c64984a49597159868152683b2
Partially-Implements: blueprint remove-docker-dir
2015-08-28 13:33:50 +00:00
Vladislav Belogrudov
630f437576 Add custom footer section to final image Dockefiles
This patch allows to add instructions to final images
like setting runtime environment variables or deleting
build time HTTP proxies

Change-Id: I23ad325e0898d77804a22038ddc618e1e49afd88
Partially-Implements: blueprint add-proxy-to-dockerfiles
2015-08-28 13:17:41 +03:00
Sam Yaple
59dc528909 Make logging more better
build.py logging has never done what people think it should do, this
patch aims to fix that. All logs related to an image will now be
printed with the image name prefixed to the message.

To acheive this we have to print out the messages as we get them, even
in parallel building mode. If you wish to view the logs for an single
image, a grep would be advised to filter the logs.

Change-Id: I15ae361892f06a47afafe73750ddd8628697cfea
Implements: blueprint build-script
2015-08-27 18:08:35 +00:00
Jenkins
3bb6ad27cb Merge "Fixes matching issue build.py" 2015-08-27 13:35:28 +00:00
Sam Yaple
73ceb2af32 Fixes matching issue build.py
And issue would occur when the namespace contained a partial amount
of the base distro's name causing the base and data images to assume
they had parent images in Kolla instead of setting thier parent to
None

Change-Id: I3b39a3e232c3faa77ee78d1ebb26b7da17bd4df4
Closes-Bug: #1489021
2015-08-27 12:35:19 +00:00
Sam Yaple
5fe9c059e0 Fixes any() in an if statement build-py
Change-Id: I8aa7d7b876491a79eabda878b95729e3468921cd
Partially-Implements: blueprint build-script
2015-08-27 11:26:07 +00:00
Sam Yaple
aada29b4db Remove tiered building in build.py
Many tiers have been shed over this issue and frankly enough is enough
on this tiering thing, it's tiering us apart! Now we use threading a
bit better as well.
No more tiers! Puns very much intended.

Additionally, this refactors the function names to make inc0 happier
and be consistent.

We also modify the test_build.py to catch the new statuses introduced
by the regex patch. Then we remove the Ubuntu Source from the Docker
folder test as that will never ever be functional.

Also fixed regex to properly match true regex expressions.

Co-Authored-By: Michal Jastrzebski (inc0) <michal.jastrzebski@intel.com>
Change-Id: I650fd6af76eddb809756762222e66aefd6fc1dca
Partially-Implements: blueprint build-script
2015-08-27 08:56:49 +00:00
Vladislav Belogrudov
68850dee15 Add ability to make tarballs from Git clones
This feature allows to specify added tarballs source
by git repository and reference (tag/branch).

Change-Id: Ie7e8a5a5852421a9d99e6371f22ad3638fe077b6
Partially-Implements: blueprint build-script
2015-08-26 14:33:44 +03:00
rthallisey
2b00ba4119 Fail faster when docker isn't running
In order to make build.py more friendly, have it fail faster
when the script is unable to connect to Docker because it's not
running.

Change-Id: I6d480e601ea2511df9396fcf4e89c06c06d4c21c
Closes-Bug: #1488133
2015-08-25 08:15:23 -04:00
Sam Yaple
71f07484ea Fix regression in build-script
The regex patch introduced a regression where the summary was not
working due to not copying the images lists, but rather referencing it

This only happens when you do not use a regex to build, but instead
build all images

Change-Id: Ife0ef2d459248aa73af877313140b5392c05673c
Partially-Implements: blueprint build-script
2015-08-25 07:44:33 +00:00
Suhail Syed
bd8d795124 Modified build.py to run from installed location
Modified build.py to run from installed location or developer
environment. To run from development environment, run the
command:

sudo pip install -e .

Additionally, remove a TODO that has been done

Co-Authored-By: Sam Yaple <sam@yaple.net>
Change-Id: I922696ba439da2c9747e65cddcba47203e54d220
Implements: blueprint python-package
2015-08-25 06:46:15 +00:00
Suhail Syed
d3872aa9ee Make Kolla installable via PIP
Install Ansible code, etc file examples, scripts, an open
rc example, a tool to initialize the OpenStack deployment,
and other various useful bits.

Start the versioning at 0.1.0 as recommended by ttx.

Change-Id: I35fa4042040957e7725c7fc3b146009e0135499a
Partially-Implements: blueprint python-package
2015-08-24 16:08:49 -07:00