From 4f9131e692042b9b1d9ed8e4f3f264014e57bc43 Mon Sep 17 00:00:00 2001
From: Gregory Haynes <greg@greghaynes.net>
Date: Wed, 11 Feb 2015 23:07:47 -0800
Subject: [PATCH] Split out README into separate files
This makes the docs site a lot more manageable and begins moving us in
the direction of separate user and developer docs.
Change-Id: I1650ef9d5be1733b8bc118d99090143cb5b06102
---
doc/source/caches.rst | 38 ++++
doc/source/components.rst | 29 +++
doc/source/copyright.rst | 19 ++
doc/source/design.rst | 51 +++++
doc/source/developing_elements.rst | 288 +++++++++++++++++++++++++++++
doc/source/elements.rst | 11 ++
doc/source/index.rst | 30 ++-
doc/source/install_types.rst | 45 +++++
doc/source/installation.rst | 24 +++
doc/source/invocation.rst | 19 ++
10 files changed, 545 insertions(+), 9 deletions(-)
create mode 100644 doc/source/caches.rst
create mode 100644 doc/source/components.rst
create mode 100644 doc/source/copyright.rst
create mode 100644 doc/source/design.rst
create mode 100644 doc/source/developing_elements.rst
create mode 100644 doc/source/elements.rst
create mode 100644 doc/source/install_types.rst
create mode 100644 doc/source/installation.rst
create mode 100644 doc/source/invocation.rst
diff --git a/doc/source/caches.rst b/doc/source/caches.rst
new file mode 100644
index 000000000..601e31dc7
--- /dev/null
+++ b/doc/source/caches.rst
@@ -0,0 +1,38 @@
+Caches and offline mode
+=======================
+
+Since retrieving and transforming operating system image files, git
+repositories, Python or Ruby packages, and so on can be a significant overhead,
+we cache many of the inputs to the build process in ~/.cache/image-create/. The
+writing an element documention describes the interface within
+disk-image-builder for caching. When invoking disk-image-builder the --offline
+option will instruct disk-image-builder to not refresh cached resources.
+
+Note that we don't maintain operating system package caches, instead depending
+on your local infrastructure (e.g. Squid cache, or an APT or Yum proxy) to
+facilitate caching of that layer, so you need to arrange independently for
+offline mode.
+
+Base images
+-----------
+
+These are cached by the standard elements - fedora, redhat, ubuntu,
+debian and opensuse.
+
+source-repositories
+-------------------
+
+Git repositories and tarballs obtained via the source-repositories element will
+be cached.
+
+C and C++ compilation
+---------------------
+
+Ccache is configured by the base element. Any compilation that honours ccache
+will be cached.
+
+PyPI
+----
+
+The pypi element will bind mount a PyPI mirror from the cache dir and configure
+pip and easy-install to use it.
diff --git a/doc/source/components.rst b/doc/source/components.rst
new file mode 100644
index 000000000..4c5c89674
--- /dev/null
+++ b/doc/source/components.rst
@@ -0,0 +1,29 @@
+Componenets
+===========
+
+* disk-image-create [-a i386|amd64|armhf] -o filename {element} [{element} ...]
+ Create an image of element {element}, optionally mixing in other elements.
+ Element dependencies are automatically included. Support for other
+ architectures depends on your environment being able to run binaries of that
+ platform. For instance, to enable armhf on Ubuntu install the qemu-user-static
+ package. The default output format from disk-image-create is qcow2. To instead
+ output a tarball pass in "-t tar". This tarball could then be used as an image
+ for a linux container(see docs/docker.md).
+
+* ramdisk-image-create -o filename {element} [{element} ...] : Create a kernel+
+ ramdisk pair for running maintenance on bare metal machines (deployment,
+ inventory, burnin etc).
+
+ To generate kernel+ramdisk pair for use with nova-baremetal, use
+ ramdisk-image-create -o deploy.ramdisk deploy-baremetal
+
+ To generate kernel+ramdisk pair for use with ironic, use
+ ramdisk-image-create -o deploy.ramdisk deploy-ironic
+
+* disk-image-get-kernel filename : **DEPRECATED** Extract the appropriate
+ kernel and ramdisk to use when doing PXE boot using filename as the image
+ for a machine. Consider using the `baremetal` element, rather than this tool.
+
+* elements can be found in the top level elements directory.
+
+* element-info : Extract information about elements.
diff --git a/doc/source/copyright.rst b/doc/source/copyright.rst
new file mode 100644
index 000000000..b9782a765
--- /dev/null
+++ b/doc/source/copyright.rst
@@ -0,0 +1,19 @@
+Copyright
+=========
+
+Copyright 2012 Hewlett-Packard Development Company, L.P.
+Copyright (c) 2012 NTT DOCOMO, INC.
+
+All Rights Reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may
+not use this file except in compliance with the License. You may obtain
+a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+License for the specific language governing permissions and limitations
+under the License.
diff --git a/doc/source/design.rst b/doc/source/design.rst
new file mode 100644
index 000000000..e767eadc9
--- /dev/null
+++ b/doc/source/design.rst
@@ -0,0 +1,51 @@
+Design
+======
+
+Images are built using a chroot and bind mounted /proc /sys and /dev. The goal
+of the image building process is to produce blank slate machines that have all
+the necessary bits to fulfill a specific purpose in the running of an OpenStack
+cloud: e.g. a nova-compute node. Images produce either a filesystem image with
+a label of cloudimg-rootfs, or can be customised to produce whole disk images
+(but will still contain a filesystem labelled cloudimg-rootfs). Once the file
+system tree is assembled a loopback device with filesystem (or partition table
+and file system) is created and the tree copied into it. The file system
+created is an ext4 filesystem just large enough to hold the file system tree
+and can be resized up to 1PB in size.
+
+An element is a particular set of code that alters how the image is built, or
+runs within the chroot to prepare the image. E.g. the local-config element
+copies in the http proxy and ssh keys of the user running the image build
+process into the image, whereas the vm element makes the image build a regular
+VM image with partition table and installed grub boot sector. The mellanox
+element adds support for mellanox infiniband hardware to both the deploy
+ramdisk and the built images.
+
+Images must specify a base distribution image element. Currently base
+distribution elements exist for fedora, rhel, ubuntu, debian and
+opensuse. Other distributions may be added in future, the
+infrastructure deliberately makes few assumptions about the exact
+operating system in use. The base image has opensshd running (a new
+key generated on first boot) and accepts keys via the cloud metadata
+service, loading them into the distribution specific default user
+account.
+
+The goal of a built image is to have any global configuration ready to roll,
+but nothing that ties it to a specific cloud instance: images should be able to
+be dropped into a test cloud and validated, and then deployed into a production
+cloud (usually via bare metal nova) for production use. As such, the image
+contents can be modelled as three distinct portions:
+
+- global content: the actual code, kernel, always-applicable config (like
+ disabling password authentication to sshd).
+- metadata / config management provided configuration: user ssh keys, network
+ address and routes, configuration management server location and public key,
+ credentials to access other servers in the cloud. These are typically
+ refreshed on every boot.
+- persistent state: sshd server key, database contents, swift storage areas,
+ nova instance disk images, disk image cache. These would typically be stored
+ on a dedicated partition and not overwritten when re-deploying the image.
+
+The goal of the image building tools is to create machine images that contain
+the correct global content and are ready for 'last-mile' configuration by the
+nova metadata API, after which a configuration management system can take over
+(until the next deploy, when it all starts over from scratch).
diff --git a/doc/source/developing_elements.rst b/doc/source/developing_elements.rst
new file mode 100644
index 000000000..86f30a993
--- /dev/null
+++ b/doc/source/developing_elements.rst
@@ -0,0 +1,288 @@
+Developing Elements
+===================
+
+Conform to the following conventions:
+
+* Use the environment for overridable defaults, prefixing environment variable
+ names with "DIB\_". For example: DIB\_MYDEFAULT=${DIB\_MYDEFAULT:-default}
+ If you do not use the DIB\_ prefix you may find that your overrides are
+ discarded as the build environment is sanitised.
+
+* Consider that your element co-exists with many others and try to guard
+ against undefined behaviours. Some examples:
+
+ * Two elements use the source-repositories element, but use the same filename
+ for the source-repositories config file. Files such as these (and indeed the
+ scripts in the various .d directories listed below) should be named such
+ that they are unique. If they are not unique, when the combined tree is
+ created by disk-image-builder for injecting into the build environment, one
+ of the files will be overwritten.
+
+ * Two elements copy different scripts into /usr/local/bin with the same name.
+ If they both use set -e and cp -n then the conflict will be caught and cause
+ the build to fail.
+
+* If your element mounts anything into the image build tree ($TMP\_BUILD\_DIR)
+ then it will be automatically unmounted when the build tree is unmounted -
+ and not remounted into the filesystem image - if the mount point is needed
+ again, your element will need to remount it at that point.
+
+Phase Subdirectories
+^^^^^^^^^^^^^^^^^^^^
+
+Make as many of the following subdirectories as you need, depending on what
+part of the process you need to customise. The subdirectories are executed in
+the order given here. Scripts within the subdirectories should be named with a
+two-digit numeric prefix, and are executed in numeric order.
+
+* root.d: Create or adapt the initial root filesystem content. This is where
+ alternative distribution support is added, or customisations such as
+ building on an existing image.
+
+ Only one element can use this at a time unless particular care is taken not
+ to blindly overwrite but instead to adapt the context extracted by other
+ elements.
+
+ * runs: outside chroot
+ * inputs: $ARCH=i386|amd64|armhf $TARGET\_ROOT=/path/to/target/workarea
+
+* extra-data.d: pull in extra data from the host environment that hooks may
+ need during image creation. This should copy any data (such as SSH keys,
+ http proxy settings and the like) somewhere under $TMP\_HOOKS\_PATH.
+
+ * runs: outside chroot
+ * inputs: $TMP\_HOOKS\_PATH
+ * outputs: None
+
+* pre-install.d: Run code in the chroot before customisation or packages are
+ installed. A good place to add apt repositories.
+
+ * runs: in chroot
+
+* install.d: Runs after pre-install.d in the chroot. This is a good place to
+ install packages, chain into configuration management tools or do other
+ image specific operations.
+
+ * runs: in chroot
+
+* post-install.d: Run code in the chroot. This is a good place to perform
+ tasks you want to handle after the OS/application install but before the
+ first boot of the image. Some examples of use would be: Run chkconfig
+ to disable unneeded services and clean the cache left by the package
+ manager to reduce the size of the image.
+
+ * runs: in chroot
+
+* block-device.d: customise the block device that the image will be made on
+ (e.g. to make partitions). Runs after the target tree has been fully
+ populated but before the cleanup hook runs.
+
+ * runs: outside chroot
+ * inputs: $IMAGE\_BLOCK\_DEVICE={path} $TARGET\_ROOT={path}
+ * outputs: $IMAGE\_BLOCK\_DEVICE={path}
+
+* finalise.d: Perform final tuning of the root filesystem. Runs in a chroot
+ after the root filesystem content has been copied into the mounted
+ filesystem: this is an appropriate place to reset SELinux metadata, install
+ grub bootloaders and so on. Because this happens inside the final image, it
+ is important to limit operations here to only those necessary to affect the
+ filesystem metadata and image itself. For most operations, post-install.d
+ is preferred.
+
+ * runs: in chroot
+
+* cleanup.d: Perform cleanup of the root filesystem content. For
+ instance, temporary settings to use the image build environment HTTP proxy
+ are removed here in the dpkg element.
+
+ * runs: outside chroot
+ * inputs: $ARCH=i386|amd64|armhf $TARGET\_ROOT=/path/to/target/workarea
+
+Other Subdirectories
+^^^^^^^^^^^^^^^^^^^^
+
+Elements may have other subdirectories that are processed by specific elements
+rather than the diskimage-builder tools themselves.
+
+One example of this is the ``bin`` directory. The ``rpm-distro``, ``dpkg`` and
+``opensuse`` elements install all files found in the ``bin`` directory into
+``/usr/local/bin`` within the image as executable files.
+
+Environment Variables
+^^^^^^^^^^^^^^^^^^^^^
+
+To set environment variables for other hooks, add a file to environment.d.
+This directory contains bash script snippets that are sourced before running
+scripts in each phase.
+
+DIB exposes an internal IMAGE\_ELEMENT variable which provides elements access
+to the full set of elements that are included in the image build. This can
+be used to process local in-element files across all the elements
+(pkg-map for example).
+
+Dependencies
+^^^^^^^^^^^^
+
+Each element can use the following files to define or affect dependencies:
+
+* element-deps: a plain text, newline separated list of elements which will
+ be added to the list of elements built into the image at image creation time.
+
+* element-provides: A plain text, newline separated list of elements which
+ are provided by this element. These elements will be excluded from elements
+ built into the image at image creation time. For example if element A depends
+ on element B and element C includes element B in its "element-provides"
+ file and A and C are included when building an image, then B is not used.
+
+
+
+Ramdisk Elements
+^^^^^^^^^^^^^^^^
+
+Ramdisk elements support the following files in their element directories:
+
+* binary-deps.d : text files listing executables required to be fed into the
+ ramdisk. These need to be present in $PATH in the build chroot (i.e. need to
+ be installed by your elements as described above).
+
+* init.d : POSIX shell script fragments that will be appended to the default
+ script executed as the ramdisk is booted (/init).
+
+* ramdisk-install.d : called to copy files into the ramdisk. The variable
+ TMP\_MOUNT\_PATH points to the root of the tree that will be packed into
+ the ramdisk.
+
+* udev.d : udev rules files that will be copied into the ramdisk.
+
+Element coding standard
+^^^^^^^^^^^^^^^^^^^^^^^
+
+- lines should not include trailing whitespace.
+- there should be no hard tabs in the file.
+- indents are 4 spaces, and all indentation should be some multiple of
+ them.
+- `do` and `then` keywords should be on the same line as the if, while or
+ for conditions.
+
+Global image-build variables
+----------------------------
+
+* DIB\_OFFLINE : this is always set. When not empty, any operations that
+ perform remote data access should avoid it if possible. If not possible
+ the operation should still be attempted as the user may have an external
+ cache able to keep the operation functional.
+
+* DIB\_IMAGE\_ROOT\_FS\_UUID : this contains the UUID of the root fs, when
+ diskimage-builder is building a disk image. This works only for ext
+ filesystems.
+
+Structure of an element
+-----------------------
+
+The above-mentioned global content can be further broken down in a way that
+encourages composition of elements and reusability of their components. One
+possible approach to this would be to label elements as either a "driver",
+"service", or "config" element. Below are some examples.
+
+- Driver-specific elements should only contain the necessary bits for that
+ driver:
+
+ elements/
+ driver-mellanox/
+ init - modprobe line
+ install.d/
+ 10-mlx - package installation
+
+- An element that installs and configures Nova might be a bit more complex,
+ containing several scripts across several phases:
+
+ elements/
+ service-nova/
+ source-repository-nova - register a source repository
+ pre-install.d/
+ 50-my-ppa - add a PPA
+ install.d/
+ 10-user - common Nova user accts
+ 50-my-pack - install packages from my PPA
+ 60-nova - install nova and some dependencies
+
+- In the general case, configuration should probably be handled either by the
+ meta-data service (eg, o-r-c) or via normal CM tools
+ (eg, salt). That being said, it may occasionally be desirable to create a
+ set of elements which express a distinct configuration of the same software
+ components.
+
+In this way, depending on the hardware and in which availability zone it is
+to be deployed, an image would be composed of:
+
+ * zero or more driver-elements
+ * one or more service-elements
+ * zero or more config-elements
+
+It should be noted that this is merely a naming convention to assist in
+managing elements. Diskimage-builder is not, and should not be, functionally
+dependent upon specific element names.
+
+diskimage-builder has the ability to retrieve source code for an element and
+place it into a directory on the target image during the extra-data phase. The
+default location/branch can then be overridden by the process running
+diskimage-builder, making it possible to use the same element to track more
+then one branch of a git repository or to get source for a local cache. See
+elements/source-repositories/README.md for more information.
+
+Debugging elements
+------------------
+
+The build-time environment and command line arguments are captured by the
+'base' element and written to /etc/dib\_environment and /etc/dib\_arguments
+inside the image.
+
+Export 'break' to drop to a shell during the image build. Break points can be
+set either before or after any of the hook points by exporting
+"break=[before|after]-hook-name". Multiple break points can be specified as a
+comma-delimited string. Some examples:
+
+* break=before-block-device-size will break before the block device size hooks
+ are called.
+
+* break=before-pre-install will break before the pre-install hooks.
+
+* break=after-error will break after an error during a in target hookpoint.
+
+Images are built such that the Linux kernel is instructed not to switch into
+graphical consoles (i.e. it will not activate KMS). This maximises
+compatibility with remote console interception hardware, such as HP's iLO.
+However, you will typicallly only see kernel messages on the console - init
+daemons (e.g. upstart) will usually be instructed to output to a serial
+console so nova's console-log command can function. There is an element in the
+tripleo-image-elements repository called "remove-serial-console" which will
+force all boot messages to appear on the main console.
+
+Ramdisk images can be debugged at run-time by passing "troubleshoot" as a
+kernel command line argument, or by pressing "t" when an error is reached. This
+will spawn a shell on the console (this can be extremely useful when network
+interfaces or disks are not detected correctly).
+
+Testing Elements
+----------------
+
+Elements can be tested using python. To create a test:
+
+* Create a directory called 'tests' in the element directory.
+
+* Create an empty file called '\_\_init\_\_.py' to make it into a python
+ package.
+
+* Create your test files as 'test\_whatever.py', using regular python test
+ code.
+
+To run all the tests use testr - `testr run`. To run just some tests provide
+one or more regex filters - tests matching any of them are run -
+`testr run apt-proxy`.
+
+Third party elements
+--------------------
+
+Pending implementation. The idea is to have a search path for elements.
+
+
diff --git a/doc/source/elements.rst b/doc/source/elements.rst
new file mode 100644
index 000000000..16e1947a0
--- /dev/null
+++ b/doc/source/elements.rst
@@ -0,0 +1,11 @@
+Elements
+========
+
+Elements are found in the subdirectory elements. Each element is in a directory
+named after the element itself. Elements *should* have a README.rst in the root
+of the element directory describing what it is for.
+
+.. toctree::
+ :glob:
+
+ elements/*/*
diff --git a/doc/source/index.rst b/doc/source/index.rst
index d28c4c35e..2d04ddf8e 100644
--- a/doc/source/index.rst
+++ b/doc/source/index.rst
@@ -1,14 +1,26 @@
-.. diskimage-builder documentation master file, created by
- sphinx-quickstart on Sat Feb 7 02:01:37 2015.
- You can adapt this file completely to your liking, but it should at least
- contain the root `toctree` directive.
+Diskimage-builder Documentation
+===============================
-.. include:: ../../README.rst
+Diskimage-builder is a tool for building disk images, file system images and
+ramdisk images.
-Elements
-========
+Why?
+----
+
+Automation: While users and operators can manually script or put together
+ramdisks and disk images, mature automation makes customisation and testing
+easier.
.. toctree::
- :glob:
+ :maxdepth: 2
+
+ design
+ components
+ installation
+ invocation
+ caches
+ install_types
+ developing_elements
+ elements
+ copyright
- elements/*/*
diff --git a/doc/source/install_types.rst b/doc/source/install_types.rst
new file mode 100644
index 000000000..02b2ade71
--- /dev/null
+++ b/doc/source/install_types.rst
@@ -0,0 +1,45 @@
+Install Types
+=============
+
+Install types permit elements to be installed from different sources, such as
+git repositories, distribution packages, or pip. The default install type
+is 'source' but it can be modified on the disk-image-create command line
+via the --install-type option. For example you can set:
+
+ --install-type=package
+
+to enable package installs by default. Alternately, you can also
+set DIB\_DEFAULT\_INSTALLTYPE.
+
+Many elements expose different install types. The different implementations
+live under `<install-dir-prefix>-<install-type>-install` directories under an
+element's install.d. The base element enables the chosen install type by
+symlinking the correct hook scripts under install.d directly.
+`<install-dir-prefix>` can be a string of alphanumeric and '-' characters, but
+typically corresponds to the element name.
+
+For example, the nova element would provide:
+
+ nova/install.d/nova-package-install/74-nova
+ nova/install.d/nova-source-install/74-nova
+
+The following symlink would be created for the package install type:
+
+ install.d/74-nova -> nova-package-install/74-nova
+
+Or, for the source install type:
+
+ install.d/74-nova -> nova-source-install/74-nova
+
+All other scripts that exist under install.d for an element will be executed as
+normal. This allows common install code to live in a script under install.d.
+
+To set the install type for an element define an environment variable
+`DIB_INSTALLTYPE_<install_dir_prefx>`. Note that if you used `-` characters in
+your install directory prefix, those need to be replaced with `_` in the
+environment variable.
+
+For example, to enable the package install type for the set of nova elements
+that use `nova` as the install type prefix, define the following:
+
+ export DIB_INSTALLTYPE_nova=package
diff --git a/doc/source/installation.rst b/doc/source/installation.rst
new file mode 100644
index 000000000..363886cb6
--- /dev/null
+++ b/doc/source/installation.rst
@@ -0,0 +1,24 @@
+Installation
+============
+
+Diskimage-builder is run directly out of the source repository.
+
+Requirements
+------------
+
+If you have 4GB of available physical RAM (As reported by
+/proc/meminfo MemTotal), or more, diskimage-builder will create a tmpfs mount
+to build the image in. This will improve image build time by building in RAM.
+This can be disabled completely by passing --no-tmpfs to disk-image-create.
+ramdisk-image-create builds a regular image and then within that does ramdisk
+creation. If tmpfs is not used, you will need enough room in /tmp to store two
+uncompressed cloud images. If you do have tmpfs, you will still need /tmp space
+for one uncompressed cloud image and about 20% of that for working files.
+
+Installation
+------------
+
+* Clone the repository locally, then add bin to your path.
+
+* Make sure you have qemu-img (qemu-utils package on Ubuntu/Debian,
+ qemu on Fedora/RHEL/openSUSE) and kpartx installed.
diff --git a/doc/source/invocation.rst b/doc/source/invocation.rst
new file mode 100644
index 000000000..4609ef0ba
--- /dev/null
+++ b/doc/source/invocation.rst
@@ -0,0 +1,19 @@
+Invocation
+==========
+
+The scripts can generally just be run. Options can be set on the command line
+or by exporting variables to override those present in lib/img-defaults. -h to
+get help.
+The image building scripts expect to be able to invoke commands with sudo, so if you
+want them to run non-interactively, you should either run them as root, with
+sudo -E, or allow your build user to run any sudo command without password.
+
+Using the variable ELEMENTS\_PATH will allow to specify multiple elements locations.
+It's a colon (:) separated path list, and it will work in a first path/element found,
+first served approach. The included elements tree is used when no path is supplied,
+and is added to the end of the path if a path is supplied.
+
+By default, the image building scripts will not overwrite existing disk images,
+allowing you to compare the newly built image with the existing one. To change
+that behaviour, set the variable OVERWRITE\_OLD\_IMAGE to any value that isn't
+0.