infra-specs/specs/jenkins-job-builder_2.0.0-api-changes.rst

::

  Copyright (c) 2015 Wayne Warren

  This work is licensed under a Creative Commons Attribution 3.0
  Unported License.
  http://creativecommons.org/licenses/by/3.0/legalcode

===============================
JJB 2.0.0 API Changes
===============================

https://storyboard.openstack.org/#!/story/2000258

The goal of this spec is to define a series of backwards-incompatible API
changes that will necessitate a 2.x release of Jenkins Job Builder. The
high-level goals of these changes are:

* to ensure that the various classes are useful outside of JJB itself where
  appropriate
* to clearly define public interfaces for various classes and document the
  purpose/role of these classes in the API
* to eliminate unuseful class relationships in favor of tiered delegation of
  program behavior
* to reduce the importance of the YamlParser as the only source of the data
  structures necessary for XML generation


Problem Description
===================

Jenkins Job Builder has history of organic, piecemeal development wherein
patches tend to be highly focused on improving one or another aspect of the
program. For the most part this is understandable, expected even--the modular
implementation of XML generation on a per-Jenkins-plugin basis lends itself well
to this style of development. However when it comes to changes to the core
abstractions, in particular the Builder and YamlParser clases, many ad hoc
design decisions have led to tightly bound implementations that make the overall
API difficult if not impossible to reuse as a library.

It's true that yes, we can import jenkins_jobs.Builder or
jenkins_jobs.parser.YamlParser to use them programmatically outside of JJB
proper, but it is difficult to imagine anyone actually doing this given the
current constructors for these classes as well as their relationships--Builder
objects actually own references to YamlParser objects when there are no clear
benefits to this organization save for expediency during JJB development.

The YamlParser itself actually does more than "parse" YAML, it is also
responsible for generating XmlJob objects from the yaml data it handles. This
means that all the lovely XML building code that makes JJB truly special cannot
reliably be accessed using JJB as a library since it depends on implementation
details of the YamlParser (specifically the xml_jobs attribute).

In addition to the tight binding between classes that make them practically
unusable, there is an almost impossibly tight binding between code in
jenkins_jobs.execute() and jenkins_jobs.Builder, which depends on code in
jenkins_jobs.execute() to munge various command line options and configuration
data, ie to obtain user, password, plugins_list, cache settings, etc. To top it
off, jenkins_jobs.Builder ends up taking the "config" object *anyway*, which
should make us question to what purpose jenkins_jobs.execute() is even handling
various pieces of configuration logic at all--arguably, this logic ought to live
in a separate "Config" class.

The problems described up to this point indicate a need to clean up the API, but
one issue that really deserves to be driven home is the essential character of
the YamlParser to everything that JJB does. Because of the tight bindings
between classes described above, understanding the tricky behavior of the
YamlParser is absolutely necessary to construct anything more simple than a
collection of "job" objects or even a simple set of "project"s containing lists
of "job-template"s.

This is primarily problematic because if JJB is going to offer a programmatic
API, it should be possible to use it to do something more interesting than the
exact same thing that is being done in jenkins_jobs.execute(). For example, it
should be possible to define one's own set of abstraction classes in Python that
can be used to generate the data used by jenkins_jobs.Builder to POST jobs to a
Jenkins server.

JJB plugin modules all take a "parser" argument, but rarely if ever is this
parser argument used in these modules. This parser is intended to be a reference
to a YamlParser instance but the benefit of this relationship does not really
outweigh its cost in terms of tight binding between plugin module and the means
used to obtain JJB data structures actually necessary for generating XML; its
primary use case most of the time is to provide access to the ModuleRegitry
object owned by the YamlParser. It would make more sense to replace this with an
explicit reference to the ModuleRegistry itself.

A Side Note on Macros
---------------------

A problematic JJB feature when it comes to cleaning up the API is the "macro".
The current implementation leads to an unnecessarily tight binding between
the YamlParser and the ModuleRegistry because the ModuleRegistry and XML
generation code require the YamlParser's xml_jobs attribute in order to
determine whether or not a component is a macro. This tight binding is
problematic because it means we need to mock the YamlParser's xml_jobs data
structure in order to use any other means of producing JJB data structures than
the YamlParser itself. As of right now, it is fine to mock YamlParser's xml_jobs
data structure with an empty dictionary but some detail of the way it is
accessed in the ModuleRegistry or XML generation code could easily change that
would cause unforeseen problems when attempting to avoid using the YamlParser.

Proposed Changes
================

Modularize Command Line Parsing
-------------------------------

Since so much of JJB's API depends on JJB's configuration file and command line
options, it makes sense to modularize these in such a way that they can easily
be used by third party scripts/tools. This will be done using a dedicated
"Config" class.

This will allow constructors for other JJB classes such as the Builder and
YamlParser to be simplified--rather than taking both a "config" object and a
number of other parameters they will instead take a single "config" parameter.

Redefine Programmatic APIs
--------------------------

The following classes will need their APIs redesigned to make them usable by
third parties:

* YamlParser
* ModuleRegistry
* Builder

These rewrites should be accompanied by docstrings that explicitly define the
purpose of these classes so that these purposes can be considered in the context
of future patches against JJB that attempt to make foundational changes to their
behavior.

YamlParser
''''''''''

The current YamlParser exposes all of its methods as "public" methods even when
there is no obvious or implicit value in doing so. The goal for rewriting its
API will be to provide explicit "public" methods that other classes can call to
get information from it.

ModuleRegistry
''''''''''''''

The new ModuleRegistry will serve a dual role; in addition to registering JJB
modules for use during XML generation it will also provide an API for modules to
use to access information other than the given JJB data. For example, it should
provide access to Jenkins plugin version information to allow modules to
implement plugin version dependent XML.


Add XmlBuilder class
--------------------

The XmlBuilder class should define a method that takes as input a dictionary
where:

* keys are job names, and
* values are all the key/value pairs necessary to produce an XmlJob for the
  named job

This method should produce as output a list of XmlJobs that can then be passed
to a Builder method either to produce test output or to update jobs on the
configured Jenkins instance.

Alternate names for this class might be JenkinsXmlBuilder,
JenkinsJobConfigBuilder.

Replace references to YamlParser with references to ModuleRegistry in modules
-----------------------------------------------------------------------------

This addresses the need to decouple plugin modules from the YamlParser while
also allowing us to access components of the ModuleRegistry when necessary (ie,
accessing Jenkins plugins_info to generate different XML for different versions
of plugins).

Rewrite JJB Macro Implementation
--------------------------------

The goal of this rewrite should be primarily to remove the need for referencing
the YamlParser's raw yaml data structure while generating XML. This way the
proposed XmlBuilder class does not have to be concerned with any particular
details of the YamlParser--reducing the risk that implementation of one of these
classes has unforeseen side effects on the other's behavior.

One method that has been proposed is to implement a secondary module registry
that specifically handles macros.

Implementation
==============

Assignee(s)
-----------

Primary assignee:
  Wayne Warren, wayne+launchpad@sdf.org

Gerrit Topic
------------

Use Gerrit topic "jjb-2.0.0-api" for all patches related to this spec.

.. code-block:: bash

    git-review -t jjb-2.0.0-api

Work Items
----------

Development Process
'''''''''''''''''''

All work for this spec will target a feature branch of JJB called "2.0.x" which
will contain all backwards-incompatible changes targeted for this release,
including any work covered by other specs targeting the same release.

Once the work is complete, we will release Jenkins Job Builder 2.0.0.

Code Changes
''''''''''''

* Break command line parsing function into dynamically-loaded, modular
  components for consumption outside of the primary JJB command line tool.
* Create function that specifically handles pre-Builder processing
  of command line options and configuration settings; this will be necessary to
  make the Builder class usable as an API to be consumed externally.
  (Alternatively, consider ways to clean up the Builder constructor in such a
  way that doesn't need to take the full "config" object)
* Separate XML generation code from YamlParser, move into new class "XmlBuilder"
  that takes a ModuleRegistry object in its constructor and contains various
  methods that take one or more JJB job dictionaries and output a corresponding
  number of XmlJob objects.
* Separate YamlParser from YamlInterpreter:
  * YamlParser will be responsible for loading a given list of YAML files
  * YamlInterpreter will be responsible for applying JJB's DSL semantics to the
  loaded lists of dictionaries.
* Refactor Builder methods and their calling contexts to simplify the interface;
  the "update_jobs" method, for instance, should simply take a list of XmlJobs
  to update.

Repositories
------------

None

Servers
-------

None

DNS Entries
-----------

None

Documentation
-------------

Documentation changes may be necessary as this work is primarily intended to
make existing JJB API usable outside of the jenkins_jobs.cmd module.

Security
--------

None

Testing
-------

Backwards Compatibility
'''''''''''''''''''''''

No existing test scenarios will be changed to support this spec; however, there
will need to be changes to test fixture base classes to support the new API
changes.

We will also monitor openstack-infra's JJB output to ensure that it does not
change because of this spec.

New Tests
'''''''''

Unit tests will explicitly validate the behavior of the following:

* the Builder class' "public" methods
* the YamlParser class' "public" methods
* the XmlBuilder class' "public" methods
* the Config class' "public" methods

Dependencies
============

Python Support
--------------

We intend to continue supporting the same Python versions supported by JJB 1.x:

* Python 2.7
* Python 3.4