openstack/diskimage-builder
Code Issues Proposed changes

Prettify 'Developing Elements' documentation

Browse Source 
Augment the developing_elements.rst by taking advantage of Sphinx
markup. Most of the doc used to be in /README.md and thus did not
leverage on Sphinx.

Use inline codeblock to denote variables, files, command: ``foo``
Phase Subdirectories:
  List phase names in the preliminary introduction
  Get rid of lists in favor of definitions
  Highlight whether the phase runs in or outside the chroot
  Input parameters are now lists

Use definition lists in Dependencies and Ramdisk sections.

Link to elements README when they exist.

Testing Element: split into two subsections: 'shell' and 'python'.

Use "sourcecode:: sh" for the couple examples at the very top and very
bottom of the document.

Change-Id: I2421f76ec452cac243ccb2208f88c7d320ffedd3
This commit is contained in:
Antoine Musso 2015-09-16 15:44:49 +02:00
parent 6afe6577ac
commit e8cecab45c
1 changed files with 187 additions and 120 deletions

307
doc/source/developer/developing_elements.rst

@ -6,8 +6,13 @@ Developing Elements
Conform to the following conventions: Conform to the following conventions:
* Use the environment for overridable defaults, prefixing environment variable * Use the environment for overridable defaults, prefixing environment variable
names with "DIB\_". For example: DIB\_MYDEFAULT=${DIB\_MYDEFAULT:-default} names with ``DIB_``. For example:
If you do not use the DIB\_ prefix you may find that your overrides are
.. sourcecode:: sh
DIB_MYDEFAULT=${DIB_MYDEFAULT:-default}
If you do not use the ``DIB`` prefix you may find that your overrides are
discarded as the build environment is sanitised. discarded as the build environment is sanitised.
* Consider that your element co-exists with many others and try to guard * Consider that your element co-exists with many others and try to guard
@ -15,27 +20,29 @@ Conform to the following conventions:
* Two elements use the source-repositories element, but use the same filename * Two elements use the source-repositories element, but use the same filename
for the source-repositories config file. Files such as these (and indeed the for the source-repositories config file. Files such as these (and indeed the
scripts in the various .d directories listed below) should be named such scripts in the various .d directories :ref:`listed below
that they are unique. If they are not unique, when the combined tree is <phase-subdirectories>`) should be named such that they are unique. If they
created by disk-image-builder for injecting into the build environment, one are not unique, when the combined tree is created by disk-image-builder for
of the files will be overwritten. injecting into the build environment, one of the files will be overwritten.
* Two elements copy different scripts into /usr/local/bin with the same name. * Two elements copy different scripts into ``/usr/local/bin`` with the same
If they both use set -e and cp -n then the conflict will be caught and cause name. If they both use ``set -e`` and ``cp -n`` then the conflict will be
the build to fail. caught and cause the build to fail.
* If your element mounts anything into the image build tree ($TMP\_BUILD\_DIR) * If your element mounts anything into the image build tree (``$TMP_BUILD_DIR``)
then it will be automatically unmounted when the build tree is unmounted - then it will be automatically unmounted when the build tree is unmounted - and
and not remounted into the filesystem image - if the mount point is needed not remounted into the filesystem image - if the mount point is needed again,
again, your element will need to remount it at that point. your element will need to remount it at that point.
* If caching is required, elements should use a location under * If caching is required, elements should use a location under
$DIB\_IMAGE\_CACHE. ``$DIB_IMAGE_CACHE``.
* Elements should allow for remote data to be cached. When * Elements should allow for remote data to be cached. When ``$DIB_OFFLINE`` is
$DIB\_OFFLINE is set, this cached data should be used if set, this cached data should be used if possible.
possible. See the *Global image-build variables* section of this See the :ref:`dev-global-image-build-variables` section of this document for
document for more information. more information.
.. _phase-subdirectories:
Phase Subdirectories Phase Subdirectories
^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^
@ -43,12 +50,26 @@ Phase Subdirectories
Make as many of the following subdirectories as you need, depending on what Make as many of the following subdirectories as you need, depending on what
part of the process you need to customise. The subdirectories are executed in part of the process you need to customise. The subdirectories are executed in
the order given here. Scripts within the subdirectories should be named with a the order given here. Scripts within the subdirectories should be named with a
two-digit numeric prefix, and are executed in numeric order. Only files which two-digit numeric prefix, and are executed in numeric order.
are marked executable (+x) will be run, so other files can be stored in these
directories if needed. As a convention, we try to only store executable scripts
in the phase subdirectories and store data files elsewhere in the element.
* root.d: Create or adapt the initial root filesystem content. This is where Only files which are marked executable (+x) will be run, so other files can be
stored in these directories if needed. As a convention, we try to only store
executable scripts in the phase subdirectories and store data files elsewhere in
the element.
The phases are:
#. ``root.d``
#. ``extra-data.d``
#. ``pre-install.d``
#. ``install.d``
#. ``post-install.d``
#. ``block-device.d``
#. ``finalise.d``
#. ``cleanup.d``
``root.d``
Create or adapt the initial root filesystem content. This is where
alternative distribution support is added, or customisations such as alternative distribution support is added, or customisations such as
building on an existing image. building on an existing image.
@ -56,60 +77,81 @@ in the phase subdirectories and store data files elsewhere in the element.
to blindly overwrite but instead to adapt the context extracted by other to blindly overwrite but instead to adapt the context extracted by other
elements. elements.
* runs: outside chroot * runs: **outside chroot**
* inputs: $ARCH=i386|amd64|armhf $TARGET\_ROOT=/path/to/target/workarea * inputs:
* extra-data.d: pull in extra data from the host environment that hooks may * ``$ARCH=i386|amd64|armhf``
* ``$TARGET_ROOT=/path/to/target/workarea``
``extra-data.d``
Pull in extra data from the host environment that hooks may
need during image creation. This should copy any data (such as SSH keys, need during image creation. This should copy any data (such as SSH keys,
http proxy settings and the like) somewhere under $TMP\_HOOKS\_PATH. http proxy settings and the like) somewhere under ``$TMP_HOOKS_PATH``.
* runs: outside chroot * runs: **outside chroot**
* inputs: $TMP\_HOOKS\_PATH * inputs: ``$TMP_HOOKS_PATH``
* outputs: None * outputs: None
* pre-install.d: Run code in the chroot before customisation or packages are ``pre-install.d``
installed. A good place to add apt repositories. Run code in the chroot before customisation or packages are installed. A good
place to add apt repositories.
* runs: in chroot * runs: **in chroot**
* install.d: Runs after pre-install.d in the chroot. This is a good place to ``install.d``
install packages, chain into configuration management tools or do other Runs after ``pre-install.d`` in the chroot. This is a good place to
image specific operations. install packages, chain into configuration management tools or do other image
specific operations.
* runs: in chroot * runs: **in chroot**
* post-install.d: Run code in the chroot. This is a good place to perform ``post-install.d``
tasks you want to handle after the OS/application install but before the Run code in the chroot. This is a good place to perform tasks you want to
first boot of the image. Some examples of use would be: Run chkconfig handle after the OS/application install but before the first boot of the
to disable unneeded services and clean the cache left by the package image. Some examples of use would be:
manager to reduce the size of the image.
* runs: in chroot Run ``chkconfig`` to disable unneeded services
* block-device.d: customise the block device that the image will be made on Clean the cache left by the package manager to reduce the size of the image.
(e.g. to make partitions). Runs after the target tree has been fully
populated but before the cleanup hook runs.
* runs: outside chroot * runs: **in chroot**
* inputs: $IMAGE\_BLOCK\_DEVICE={path} $TARGET\_ROOT={path}
* outputs: $IMAGE\_BLOCK\_DEVICE={path}
* finalise.d: Perform final tuning of the root filesystem. Runs in a chroot ``block-device.d``
after the root filesystem content has been copied into the mounted Customise the block device that the image will be made on (for example to
filesystem: this is an appropriate place to reset SELinux metadata, install make partitions). Runs after the target tree has been fully populated but
grub bootloaders and so on. Because this happens inside the final image, it before the ``cleanup.d`` phase runs.
is important to limit operations here to only those necessary to affect the
filesystem metadata and image itself. For most operations, post-install.d
is preferred.
* runs: in chroot * runs: **outside chroot**
* inputs:
* cleanup.d: Perform cleanup of the root filesystem content. For * ``$IMAGE_BLOCK_DEVICE={path}``
instance, temporary settings to use the image build environment HTTP proxy * ``$TARGET_ROOT={path}``
are removed here in the dpkg element.
* outputs: ``$IMAGE_BLOCK_DEVICE={path}``
``finalise.d``
Perform final tuning of the root filesystem. Runs in a chroot after the root
filesystem content has been copied into the mounted filesystem: this is an
appropriate place to reset SELinux metadata, install grub bootloaders and so
on.
Because this happens inside the final image, it is important to limit
operations here to only those necessary to affect the filesystem metadata and
image itself. For most operations, ``post-install.d`` is preferred.
* runs: **in chroot**
``cleanup.d``
Perform cleanup of the root filesystem content. For instance, temporary
settings to use the image build environment HTTP proxy are removed here in
the dpkg element.
* runs: outside chroot
* inputs:
* ``$ARCH=i386|amd64|armhf``
* ``$TARGET_ROOT=/path/to/target/workarea``
* runs: outside chroot
* inputs: $ARCH=i386|amd64|armhf $TARGET\_ROOT=/path/to/target/workarea
Other Subdirectories Other Subdirectories
^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^
@ -117,35 +159,42 @@ Other Subdirectories
Elements may have other subdirectories that are processed by specific elements Elements may have other subdirectories that are processed by specific elements
rather than the diskimage-builder tools themselves. rather than the diskimage-builder tools themselves.
One example of this is the ``bin`` directory. The ``rpm-distro``, ``dpkg`` and One example of this is the ``bin`` directory. The `rpm-distro`,
``opensuse`` elements install all files found in the ``bin`` directory into :doc:`../elements/dpkg/README` and :doc:`../elements/opensuse/README` elements
``/usr/local/bin`` within the image as executable files. install all files found in the ``bin`` directory into ``/usr/local/bin`` within
the image as executable files.
Environment Variables Environment Variables
^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^
To set environment variables for other hooks, add a file to environment.d. To set environment variables for other hooks, add a file to your element
``environment.d``.
This directory contains bash script snippets that are sourced before running This directory contains bash script snippets that are sourced before running
scripts in each phase. scripts in each phase.
DIB exposes an internal IMAGE\_ELEMENT variable which provides elements access DIB exposes an internal ``$IMAGE_ELEMENT`` variable which provides elements
to the full set of elements that are included in the image build. This can access to the full set of elements that are included in the image build. This
be used to process local in-element files across all the elements can be used to process local in-element files across all the elements
(pkg-map for example). (``pkg-map`` for example).
Dependencies Dependencies
^^^^^^^^^^^^ ^^^^^^^^^^^^
Each element can use the following files to define or affect dependencies: Each element can use the following files to define or affect dependencies:
* element-deps: a plain text, newline separated list of elements which will ``element-deps``
be added to the list of elements built into the image at image creation time. A plain text, newline separated list of elements which will be added to the
list of elements built into the image at image creation time.
* element-provides: A plain text, newline separated list of elements which ``element-provides``
are provided by this element. These elements will be excluded from elements A plain text, newline separated list of elements which are provided by this
built into the image at image creation time. For example if element A depends element. These elements will be excluded from elements built into the image
on element B and element C includes element B in its "element-provides" at image creation time.
file and A and C are included when building an image, then B is not used.
For example if element A depends on element B and element C includes element B
in its ``element-provides`` file and A and C are included when building an
image, then B is not used.
Operating system elements Operating system elements
^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^
@ -166,18 +215,21 @@ Ramdisk Elements
Ramdisk elements support the following files in their element directories: Ramdisk elements support the following files in their element directories:
* binary-deps.d : text files listing executables required to be fed into the ``binary-deps.d``
ramdisk. These need to be present in $PATH in the build chroot (i.e. need to Text files listing executables required to be fed into the ramdisk. These
be installed by your elements as described above). need to be present in ``$PATH`` in the build chroot (i.e. need to be installed
by your elements as described above).
* init.d : POSIX shell script fragments that will be appended to the default ``init.d``
script executed as the ramdisk is booted (/init). POSIX shell script fragments that will be appended to the default script
executed as the ramdisk is booted (``/init``).
* ramdisk-install.d : called to copy files into the ramdisk. The variable ``ramdisk-install.d``
TMP\_MOUNT\_PATH points to the root of the tree that will be packed into Called to copy files into the ramdisk. The variable ``$TMP_MOUNT_PATH`` points
the ramdisk. to the root of the tree that will be packed into the ramdisk.
* udev.d : udev rules files that will be copied into the ramdisk. ``udev.d``
``udev`` rules files that will be copied into the ramdisk.
Element coding standard Element coding standard
^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^
@ -189,20 +241,24 @@ Element coding standard
- `do` and `then` keywords should be on the same line as the if, while or - `do` and `then` keywords should be on the same line as the if, while or
for conditions. for conditions.
.. _dev-global-image-build-variables:
Global image-build variables Global image-build variables
---------------------------- ----------------------------
* DIB\_OFFLINE : this is always set. When not empty, any operations that ``DIB_OFFLINE``
perform remote data access should avoid it if possible. If not possible This is always set. When not empty, any operations that perform remote data
the operation should still be attempted as the user may have an external access should avoid it if possible. If not possible the operation should still
cache able to keep the operation functional. be attempted as the user may have an external cache able to keep the operation
functional.
* DIB\_IMAGE\_ROOT\_FS\_UUID : this contains the UUID of the root fs, when ``DIB_IMAGE_ROOT_FS_UUID``
diskimage-builder is building a disk image. This works only for ext This contains the UUID of the root filesystem, when diskimage-builder is
filesystems. building a disk image. This works only for ext filesystems.
* DIB\_IMAGE\_CACHE : path to where cached inputs to the build process ``DIB_IMAGE_CACHE``
are stored. Defaults to ~/.cache/image_create. Path to where cached inputs to the build process are stored. Defaults to
``~/.cache/image_create``.
Structure of an element Structure of an element
----------------------- -----------------------
@ -256,26 +312,26 @@ place it into a directory on the target image during the extra-data phase. The
default location/branch can then be overridden by the process running default location/branch can then be overridden by the process running
diskimage-builder, making it possible to use the same element to track more diskimage-builder, making it possible to use the same element to track more
then one branch of a git repository or to get source for a local cache. See then one branch of a git repository or to get source for a local cache. See
elements/source-repositories/README.md for more information. :doc:`../elements/source-repositories/README` for more information.
Debugging elements Debugging elements
------------------ ------------------
The build-time environment and command line arguments are captured by the The build-time environment and command line arguments are captured by the
'base' element and written to /etc/dib\_environment and /etc/dib\_arguments :doc:`../elements/base/README` element and written to ``/etc/dib_environment``
inside the image. and ``/etc/dib_arguments`` inside the image.
Export 'break' to drop to a shell during the image build. Break points can be Export ``break`` to drop to a shell during the image build. Break points can be
set either before or after any of the hook points by exporting set either before or after any of the hook points by exporting
"break=[before|after]-hook-name". Multiple break points can be specified as a "break=[before|after]-hook-name". Multiple break points can be specified as a
comma-delimited string. Some examples: comma-delimited string. Some examples:
* break=before-block-device-size will break before the block device size hooks * ``break=before-block-device-size`` will break before the block device size
are called. hooks are called.
* break=before-pre-install will break before the pre-install hooks. * ``break=before-pre-install`` will break before the pre-install hooks.
* break=after-error will break after an error during a in target hookpoint. * ``break=after-error`` will break after an error during an in target hookpoint.
Images are built such that the Linux kernel is instructed not to switch into Images are built such that the Linux kernel is instructed not to switch into
graphical consoles (i.e. it will not activate KMS). This maximises graphical consoles (i.e. it will not activate KMS). This maximises
@ -286,7 +342,7 @@ console so nova's console-log command can function. There is an element in the
tripleo-image-elements repository called "remove-serial-console" which will tripleo-image-elements repository called "remove-serial-console" which will
force all boot messages to appear on the main console. force all boot messages to appear on the main console.
Ramdisk images can be debugged at run-time by passing "troubleshoot" as a Ramdisk images can be debugged at run-time by passing ``troubleshoot`` as a
kernel command line argument, or by pressing "t" when an error is reached. This kernel command line argument, or by pressing "t" when an error is reached. This
will spawn a shell on the console (this can be extremely useful when network will spawn a shell on the console (this can be extremely useful when network
interfaces or disks are not detected correctly). interfaces or disks are not detected correctly).
@ -294,42 +350,53 @@ interfaces or disks are not detected correctly).
Testing Elements Testing Elements
---------------- ----------------
An element can have functional tests encapsulated inside the element itself. In An element can have functional tests encapsulated inside the element itself. The
order to create a test case, follow these steps: tests can be written either as shell or python unit tests.
* Create a directory called 'test-elements' inside your element. shell
"""""
In order to create a test case, follow these steps:
* Create a directory called ``test-elements`` inside your element.
* Inside the test-elements directory, create a directory with the name of your * Inside the test-elements directory, create a directory with the name of your
test case. The test case directory should have the same structure as an test case. The test case directory should have the same structure as an
element. element. For example::
i.e. elements/apt-sources/test-elements/test-case-1
elements/apt-sources/test-elements/test-case-1
* Assert state during each of the element build phases you would like to test. * Assert state during each of the element build phases you would like to test.
You can exit 1 to indicate a failure. You can exit 1 to indicate a failure.
* To exit early and indicate a success, touch a file /tmp/dib-test-should-fail * To exit early and indicate a success, touch a file
in the image chroot, then exit 1. ``/tmp/dib-test-should-fail`` in the image chroot, then exit 1.
python
""""""
Additionally, elements can be tested using python unittests. To create a Additionally, elements can be tested using python unittests. To create a
a python test: a python test:
* Create a directory called 'tests' in the element directory. * Create a directory called ``tests`` in the element directory.
* Create an empty file called '\_\_init\_\_.py' to make it into a python * Create an empty file called ``__init__.py`` to make it into a python
package. package.
* Create your test files as 'test\_whatever.py', using regular python test * Create your test files as ``test\whatever.py``, using regular python test
code. code.
To run all the tests use testr - `testr run`. To run just some tests provide To run all the tests use testr - ``testr run``. To run just some tests provide
one or more regex filters - tests matching any of them are run - one or more regex filters - tests matching any of them are run -
`testr run apt-proxy`. ``testr run apt-proxy``.
Third party elements Third party elements
-------------------- --------------------
Additional elements can be incorporated by setting ELEMENTS_PATH, for example Additional elements can be incorporated by setting ``ELEMENTS_PATH``, for
if one were building tripleo-images, the variable would be set like:: example if one were building tripleo-images, the variable would be set like:
export ELEMENTS_PATH=tripleo-image-elements/elements .. sourcecode:: sh
disk-image-create rhel7 cinder-api
export ELEMENTS_PATH=tripleo-image-elements/elements
disk-image-create rhel7 cinder-api