docs: Add developer overview documentation

This patchset adds developer overview documentation for providing
a high-level introduction to Deckhand, including its architecture,
modules, test utilities, Helm utilities, and other errata. This
work is based off this Drydock patchset:

  https://review.openstack.org/#/c/571298/

Change-Id: Ic3382d4e04edf02a65184651d272fe9cd1db56a4

REVIEWING.rst

@@ -0,0 +1,38 @@
+Reviewing Deckhand Code
+=======================
+To start read the `OpenStack Common Review Checklist
+<https://docs.openstack.org/infra/manual/developers.html#peer-review>`_
+
+
+Unit Tests
+----------
+For any change that adds new functionality to either common functionality or
+fixes a bug unit tests are required. This is to ensure we don't introduce
+future regressions and to test conditions which we may not hit in the gate
+runs.
+
+
+Functional Tests
+----------------
+For any change that adds major new functionality functional tests are required.
+This is to ensure that the Deckhand API follows the contract it promises.
+In addition, functional tests are run against the Deckhand container, which
+uses an image built from the latest source code to validate the integrity
+of the image.
+
+
+Deprecated Code
+---------------
+Deprecated code should go through a deprecation cycle -- long enough for other
+Airship projects to modify their code base to reference new code. Features,
+APIs or configuration options are marked deprecated in the code. Appropriate
+warnings will be sent to the end user, operator or library user.
+
+
+When to approve
+---------------
+* Every patch needs two +2s before being approved.
+* Its OK to hold off on an approval until a subject matter expert reviews it.
+* If a patch has already been approved but requires a trivial rebase to merge,
+  you do not have to wait for a second +2, since the patch has already had
+  two +2s.

doc/requirements.txt

@@ -4,6 +4,7 @@
 sphinx>=1.6.2 # BSD
 sphinx_rtd_theme==0.2.4
 reno>=2.5.0 # Apache-2.0
+plantuml
 
 # NOTE(fmontei): The requirement below is only included because readthedocs
 # depends on it to work, since it only takes in one requirements file.

doc/source/HACKING.rst

@@ -13,7 +13,10 @@
     See the License for the specific language governing permissions and
     limitations under the License.
 
-=======
-Hacking
-=======
+.. _coding-guide:
+
+=====================
+Deckhand Coding Guide
+=====================
+
 .. include:: ../../HACKING.rst

doc/source/REVIEWING.rst

@@ -0,0 +1,5 @@
+=======================
+Reviewing Deckhand Code
+=======================
+
+.. include:: ../../REVIEWING.rst

doc/source/developer-overview.rst

@@ -0,0 +1,141 @@
+..
+      Copyright 2018 AT&T Intellectual Property.
+      All Rights Reserved.
+
+      Licensed under the Apache License, Version 2.0 (the "License"); you may
+      not use this file except in compliance with the License. You may obtain
+      a copy of the License at
+
+          http://www.apache.org/licenses/LICENSE-2.0
+
+      Unless required by applicable law or agreed to in writing, software
+      distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+      WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+      License for the specific language governing permissions and limitations
+      under the License.
+
+==============================
+Developer Overview of Deckhand
+==============================
+
+The core objective of Deckhand is to provide storage, rendering, validation and
+version control for declarative YAML documents. Deckhand ingests raw,
+Airship-formatted documents and outputs fully rendered documents to other
+Airship components.
+
+Architecture
+============
+
+.. image:: images/architecture.png
+   :alt: High level architecture of Deckhand
+
+From a high-level perspective, Deckhand consists of a RESTful API, a document
+rendering engine, and a PostgreSQL relational database for document storage.
+Deckhand ingests Airship-formatted documents, validates them, and stores them
+in its database for future processing. On demand, Deckhand will fully render
+the documents, after which they can be consumed by the other Airship
+components.
+
+Deckhand uses Barbican to securely storage sensitive document data.
+
+`Pegleg <http://airship-pegleg.readthedocs.io/en/latest/>`_ in effect provides
+Deckhand with a CLI, which facilitates communication with Deckhand.
+
+.. image:: images/architecture-pegleg.png
+   :alt: High level architecture of Deckhand
+
+Components
+==========
+
+control
+-------
+
+The ``control`` module is simply the RESTful API. It is based on the
+`Falcon Framework <https://falconframework.org/>`_ and utilizes
+`oslo.policy <https://docs.openstack.org/oslo.policy/latest/>`_
+for RBAC enforcement of the API endpoints. The normal deployment of Deckhand
+uses `uWSGI <http://uwsgi-docs.readthedocs.io/en/latest/>`_ and PasteDeploy
+to build a pipeline that includes Keystone Middleware for authentication
+and role decoration of the request.
+
+The ``control`` module is also responsible for communicating with
+`Barbican <https://docs.openstack.org/barbican/latest/>`_, which it uses to
+store and retrieve document :ref:`secrets <encryption>`, which it passes to the
+``engine`` module for :ref:`rendering`.
+
+engine
+------
+
+The ``engine`` module is the interface responsible for all
+:ref:`rendering`. Rendering consists of applying a series of algorithms to the
+documents, including: topological sorting, :ref:`layering`,
+:ref:`substitution`, and :ref:`replacement`.
+
+db
+--
+
+The ``db`` module is responsible for implementing the database tables needed
+to store all Airship documents. The module also realizes version control and
+:ref:`revision-diffing` functionality.
+
+client
+------
+
+The API client library provides an interface for other services to communicate
+with Deckhand's API. Requires
+`Keystone <https://docs.openstack.org/keystone/latest/>`_ authentication to
+use.
+
+Developer Workflow
+==================
+
+Because Airship is a container-centric platform, the developer workflow heavily
+utilizes containers for testing and publishing. It also requires Deckhand to
+produce multiple artifacts that are related, but separate: the Python package,
+the Docker image and the Helm chart. The code is published via the
+Docker image artifact.
+
+Deckhand strives to conform to the
+`Airship coding conventions <http://airshipit.readthedocs.io/en/latest/conventions.html>`_.
+
+Python
+------
+
+The Deckhand code base lives under ``/deckhand``. Deckhand supports py27
+and py35 interpreters. Once OpenStack deprecates usage of py27, Deckhand
+will as well.
+
+See :ref:`coding-guide` for more information on contribution guidelines.
+
+Docker
+------
+
+The Deckhand Dockerfile is located in ``/images/deckhand`` along with any
+artifacts built specifically to enable the container image. Make targets are
+used for generating and testing the artifacts.
+
+* ``make images`` - Build the Deckhand Docker image.
+
+Helm
+----
+
+The Deckhand Helm chart is located in ``/charts/deckhand``. Local testing
+currently only supports linting and previewing the rendered artifacts.
+Richer functional chart testing is a TODO.
+
+* ``make charts`` - Pull down dependencies for the Deckhand charts and package
+  everything into a ``.tgz`` file.
+* ``make helm_lint`` - Lint the Helm charts.
+* ``make dry-run`` - Render the chart and output the Kubernetes manifest YAML
+  documents.
+
+Testing
+=======
+
+All Deckhand tests are nested under ``/deckhand/tests``.
+
+Deckhand comes equipped with a number of
+`tox <https://tox.readthedocs.io/en/latest/>`_ targets for running unit and
+functional tests. See :ref:`development-utilities` for a list of commands.
+
+See :ref:`testing` for more information on testing guidelines.

doc/source/diagrams/architecture-pegleg.uml

@@ -0,0 +1,39 @@
+' PlantUML file to generate the architecture component diagram
+@startuml
+
+actor "DE"
+entity "HTTP"
+entity "SQL"
+entity "WSGI"
+
+frame "Deckhand" {
+  [Control] ..> [Engine]
+}
+
+frame "Pegleg" {
+  [Pegleg CLI] ..> [Pegleg Engine]
+}
+
+frame "OpenStack Components" {
+  [Barbican] ..> [Secret Store Back-end]
+  [Keystone]
+}
+
+database "PostgreSQL" {
+  SQL - [deckhand_db]
+}
+
+HTTP - [uWSGI]
+[uWSGI] --> [Keystone Middleware]
+[Keystone Middleware] --> WSGI
+WSGI - [Control]
+[Engine] --> [SQL]
+[Control] --> [Barbican]
+[Pegleg Engine] --> [Engine]
+DE -> [Pegleg CLI]
+
+legend right
+  DE: Deployment Engineer
+endlegend
+
+@enduml

doc/source/diagrams/architecture.uml

@@ -0,0 +1,28 @@
+' PlantUML file to generate the architecture component diagram
+@startuml
+
+entity "HTTP"
+entity "SQL"
+entity "WSGI"
+
+frame "Deckhand" {
+  [Control] ..> [Engine]
+}
+
+frame "OpenStack Components" {
+  [Barbican] ..> [Secret Store Back-end]
+  [Keystone]
+}
+
+database "PostgreSQL" {
+  SQL - [deckhand_db]
+}
+
+HTTP - [uWSGI]
+[uWSGI] --> [Keystone Middleware]
+[Keystone Middleware] --> WSGI
+WSGI - [Control]
+[Engine] --> [SQL]
+[Control] --> [Barbican]
+
+@enduml

doc/source/getting-started.rst

@@ -206,6 +206,8 @@ After, from the command line, execute:
         -v $CONF_DIR:/etc/deckhand \
         quay.io/attcomdev/deckhand:latest server
 
+.. _development-utilities:
+
 Development Utilities
 ---------------------
 
@@ -244,6 +246,9 @@ deployment, execute (respectively)::
 
 .. _Bandit: https://github.com/openstack/bandit
 
+For additional commands, reference the ``tox.ini`` file for a list of all
+the jobs.
+
 Database Model Updates
 ----------------------
 

doc/source/index.rst

@@ -60,6 +60,8 @@ Developer's Guide
    :maxdepth: 2
 
    HACKING
+   REVIEWING
+   developer-overview
    policy-enforcement
    testing
 

doc/source/revision-history.rst

@@ -29,6 +29,8 @@ documents.
 Revisions can be thought of as commits in a linear git history, thus looking
 at a revision includes all content from previous revisions.
 
+.. _revision-diffing:
+
 Revision Diffing
 ----------------
 

doc/source/testing.rst

@@ -13,11 +13,13 @@
   See the License for the specific language governing permissions and
   limitations under the License.
 
+.. _testing:
+
 =======
 Testing
 =======
 
-.. warning::
+.. note::
 
   Deckhand has only been tested against a Ubuntu 16.04 environment. The guide
   below assumes the user is using Ubuntu.
@@ -100,11 +102,11 @@ Overview
 Deckhand uses `gabbi <https://github.com/cdent/gabbi>`_ as its functional
 testing framework. Functional tests can be executed via::
 
-    $ tox -e functional
+    $ tox -e functional-dev
 
 You can also run a subset of tests via a regex::
 
-    $ tox -e functional -- gabbi.suitemaker.test_gabbi_document-crud-success-multi-bucket
+    $ tox -e functional-dev -- gabbi.suitemaker.test_gabbi_document-crud-success-multi-bucket
 
 The command executes ``tools/functional-tests.sh`` which:
 
@@ -142,7 +144,7 @@ testing. To test Deckhand against a containerized image, run, for example:
 ::
 
   export DECKHAND_IMAGE=quay.io/attcomdev/deckhand:latest
-  tox -e functional
+  tox -e functional-dev
 
 Which will result in the following script output:
 

tools/build-docs.sh

@@ -0,0 +1,11 @@
+#!/usr/bin/env bash
+
+# Builds documentation and generates documentation diagrams from .uml
+# files. Must be run from root project directory.
+
+set -ex
+rm -rf doc/build
+rm -rf releasenotes/build
+sphinx-build -W -b html doc/source doc/build/html
+python -m plantuml doc/source/diagrams/*.uml
+mv doc/source/diagrams/*.png doc/source/images

tools/whitespace-linter.sh

@@ -6,6 +6,7 @@ RES=$(find . \
   -not -path "*/*.egg-info/*" \
   -not -path "*/releasenotes/build/*" \
   -not -path "*/doc/build/*" \
+  -not -path "*/doc/source/images/*" \
   -not -name "*.tgz" \
   -type f -exec egrep -l " +$" {} \;)
 

tox.ini

@@ -118,13 +118,10 @@ ignore = H405
 exclude =  .venv,.git,.tox,dist,*lib/python*,*egg,build,releasenotes,doc,alembic/versions
 
 [testenv:docs]
-deps = -r{toxinidir}/doc/requirements.txt
+deps =
+  -r{toxinidir}/doc/requirements.txt
 commands =
-  rm -rf doc/build
-  rm -rf releasenotes/build
-  sphinx-build -W -b html doc/source doc/build/html
-whitelist_externals =
-  rm
+  {toxinidir}/tools/build-docs.sh
 
 [testenv:releasenotes]
 deps = -r{toxinidir}/doc/requirements.txt