airship/spyglass
Code Issues Proposed changes

Spyglass Docs Update

Browse Source 
Changed paths for all examples, removed references to Tugboat
and Formation

Change-Id: Ibeb80a1c239169bd4f4ecb124e86671891bef99a
This commit is contained in:
Ryan Schroder 2019-11-21 10:47:55 -06:00
parent 9d983e4874
commit 161528365d
6 changed files with 105 additions and 79 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
.zuul.yaml
View File

@ -84,6 +84,7 @@
timeout: 1800
run: tools/gate/playbooks/docker-image-build.yaml
nodeset: spyglass-single-node
voting: false
vars:
publish: false
distro: opensuse_15
@ -132,6 +133,7 @@
nodeset: spyglass-single-node
secrets:
- airship_spyglass_quay_creds
voting: false
vars:
publish: true
distro: opensuse_15

33
README.rst
View File

@ -6,27 +6,26 @@ Spyglass is the data extractor tool which can interface with
different input data sources to generate site manifest YAML files.
The data sources will provide all the configuration data needed
for a site deployment. These site manifest YAML files generated
by spyglass will be saved in a Git repository, from where Pegleg
by Spyglass will be saved in a Git repository, from where Pegleg
can access and aggregate them. This aggregated file can then be
fed to shipyard for site deployment / updates.
fed to Shipyard for site deployment/updates.
Spyglass follows plugin model to support multiple input data sources.
Current supported plugins are formation-plugin and Tugboat. Formation
is a rest API based service which will be the source of information
related to hardware, networking, site data. Formation plugin will
interact with Formation API to gather necessary configuration.
Similarly Tugboat accepts engineering spec which is in the form of
spreadsheet and an index file to read spreadsheet as inputs and
generates the site level manifests.
As an optional step it can generate an intermediary yaml which contain
all the information that will be rendered to generate Airship site
manifests. This optional step will help the deployment engineer to
modify any data if required.
Spyglass follows a plugin model to support multiple input data sources.
The currently supported plugin is the Spyglass Excel plugin
(`spyglass-plugin-xls`_).
The Spyglass Excel plugin accepts an engineering spec in the form of a
spreadsheet and an index file to read the spreadsheet as inputs and
generates site level manifests. As an optional step, it can generate an
intermediary YAML which contains all the information that will be rendered to
generate Airship site manifests. This optional step will help the deployment
engineer modify any data if required.
Getting Started
---------------
For more detailed installation and setup information, please refer to the
`Getting Started`_ guide.
For more detailed information, please refer to the `Index`_ to easily navigate
the Getting Started, Developer Quickstart, and Command Line Interface guides.
.. _`Getting Started`: ./doc/source/getting_started.rst
.. _spyglass-plugin-xls: https://opendev.org/airship/spyglass-plugin-xls
.. _`Index`: ./doc/source/index.rst

89
doc/source/cli.rst
View File

@ -28,19 +28,19 @@ The Spyglass CLI is used in conjunction with the script ``tools/spyglass.sh``.
CLI Options
===========
**-v / --verbose** (Optional). False by default.
**-v / \\-\\-verbose** (Optional). False by default.
Enable debug logging.
Excel Plugin
************
Commands available under the excel plugin package.
Commands available under the Excel plugin package.
Generate Intermediary
---------------------
Generates an intermediary file from passed excel data.
Generates an intermediary file from passed Excel data.
.. code-block:: bash
@ -52,42 +52,42 @@ Generates an intermediary file from passed excel data.
Options
^^^^^^^
**-d / --intermediary-dir** (Optional).
**-d / \\-\\-intermediary-dir** (Optional).
Path where the intermediary file will be created. Must be a writeable
directory.
**-x / --excel-file** (Required for "tugboat" plugin).
**-x / \\-\\-excel-file** (Required for Excel plugin).
Path to the engineering excel file. Multiple files can be included, provided
Path to the engineering Excel file. Multiple files can be included, provided
they follow the same specification. Must be readable file(s) in a Microsoft
Excel supported format (.xls, .xslx, etc...).
**-e / --excel-spec** (Required for "tugboat" plugin).
**-e / \\-\\-excel-spec** (Required for Excel plugin).
Path to the specification YAML that defines the content of the provided
engineering excel files. Must be a readable file in YAML format.
engineering Excel files. Must be a readable file in YAML format.
**-c / --site-configuration** (Optional).
**-c / \\-\\-site-configuration** (Optional).
Path to site specific configuration YAML. Must be a readable file.
**--intermediary-schema** (Optional).
**\\-\\-intermediary-schema** (Optional).
Path to the intermediary schema to be used for validation.
**--no-validation** (Optional).
**\\-\\-no-validation** (Optional).
Skips validation on generated intermediary data.
**-s / --site-name** (Optional).
**-s / \\-\\-site-name** (Optional).
Name of the site for which the intermediary is generated.
Generate Manifests
------------------
Generates manifests from intermediary file created from passed excel data.
Generates manifests from intermediary file created from passed Excel data.
Intermediary data is always generated, but will not be saved unless specified.
.. code-block:: bash
@ -99,48 +99,48 @@ Intermediary data is always generated, but will not be saved unless specified.
Options
^^^^^^^
**-i / --generate-intermediary** (Optional). False by default.
**-i / \\-\\-generate-intermediary** (Optional). False by default.
Saves the intermediary file used to make the manifests created by the command.
**-d / --intermediary-dir** (Optional).
**-d / \\-\\-intermediary-dir** (Optional).
Path where the intermediary file will be created. Must be a writeable
directory.
**-x / --excel-file** (Required for "tugboat" plugin).
**-x / \\-\\-excel-file** (Required for Excel plugin).
Path to the engineering excel file. Multiple files can be included, provided
Path to the engineering Excel file. Multiple files can be included, provided
they follow the same specification. Must be readable file(s) in a Microsoft
Excel supported format (.xls, .xslx, etc...).
**-e / --excel-spec** (Required for "tugboat" plugin).
**-e / \\-\\-excel-spec** (Required for Excel plugin).
Path to the specification YAML that defines the content of the provided
engineering excel files. Must be a readable file in YAML format.
engineering Excel files. Must be a readable file in YAML format.
**-c / --site-configuration** (Optional).
**-c / \\-\\-site-configuration** (Optional).
Path to site specific configuration YAML. Must be a readable file.
**--intermediary-schema** (Optional).
**\\-\\-intermediary-schema** (Optional).
Path to the intermediary schema to be used for validation.
**--no-validation** (Optional).
**\\-\\-no-validation** (Optional).
Skips validation on generated intermediary data.
**-s / --site-name** (Optional).
**-s / \\-\\-site-name** (Optional).
Name of the site for which the intermediary is generated.
**-t / --template-dir** (Required).
**-t / \\-\\-template-dir** (Required).
Path to the Jinja2 template files that will be used to generate manifest files.
Must be a readable directory with Jinja2 files using the .j2 extension.
**-m / --manifest-dir** (Optional).
**-m / \\-\\-manifest-dir** (Optional).
Path where generated manifest files should be written. Must be a writeable
directory.
@ -169,17 +169,17 @@ manifests.
Options
^^^^^^^
**-t / --template-dir** (Required).
**-t / \\-\\-template-dir** (Required).
Path to the Jinja2 template files that will be used to generate manifest files.
Must be a readable directory with Jinja2 files using the .j2 extension.
**-m / --manifest-dir** (Optional).
**-m / \\-\\-manifest-dir** (Optional).
Path where generated manifest files should be written. Must be a writeable
directory.
**--force** (Optional).
**\\-\\-force** (Optional).
Forces manifests to be written, regardless of undefined data.
@ -195,11 +195,11 @@ Validates pegleg documents against their schema.
Options
^^^^^^^
**-d / --document-path**
**-d / \\-\\-document-path**
Path to the document(s) to validate.
**-p / --schema-path**
**-p / \\-\\-schema-path**
Path to a schema or directory of schema files used to validate documents in
document path.
@ -220,27 +220,33 @@ Generating intermediary and manifests
.. code-block:: bash
spyglass excel documents -i -x SiteDesignSpec_v0.1.xlsx \
-e excel_spec_upstream.yaml -c site_config.yaml \
-s airship-seaworthy -t <j2 template dir>
spyglass excel documents -i \
-x ../spyglass-plugin-xls/spyglass_plugin_xls/examples/SiteDesignSpec_v0.1.xlsx \
-e ../spyglass-plugin-xls/spyglass_plugin_xls/examples/excel_spec.yaml \
-c spyglass/examples/site_config.yaml \
-s airship-seaworthy -t spyglass/examples/templates/
Generating intermediary without manifests
-----------------------------------------
.. code-block:: bash
spyglass excel intermediary -x SiteDesignSpec_v0.1.xlsx \
-e excel_spec_upstream.yaml -c site_config.yaml \
-s airship-seaworthy
spyglass excel intermediary \
-x ../spyglass-plugin-xls/spyglass_plugin_xls/examples/SiteDesignSpec_v0.1.xlsx \
-e ../spyglass-plugin-xls/spyglass_plugin_xls/examples/excel_spec.yaml \
-c spyglass/examples/site_config.yaml \
-s airship-seaworthy
Generating manifests without intermediary
-----------------------------------------
.. code-block:: bash
spyglass excel documents -x SiteDesignSpec_v0.1.xlsx \
-e excel_spec_upstream.yaml -c site_config.yaml \
-s airship-seaworthy -t <j2 template dir>
spyglass excel documents \
-x ../spyglass-plugin-xls/spyglass_plugin_xls/examples/SiteDesignSpec_v0.1.xlsx \
-e ../spyglass-plugin-xls/spyglass_plugin_xls/examples/excel_spec.yaml \
-c spyglass/examples/site_config.yaml \
-s airship-seaworthy -t spyglass/examples/templates/
Generating manifests using intermediary
***************************************
@ -249,8 +255,9 @@ Generating manifests using intermediary
spyglass mi <intermediary.yaml> -t <j2 template dir>
Where sample 'excel_spec_upstream.yaml', 'SiteDesignSpec_v0.1.xlsx'
'site_config.yaml' and J2 templates can be found under 'spyglass/examples'
Where sample `excel_spec.yaml` and `SiteDesignSpec_v0.1.xlsx` can be found in
spyglass-plugin-xls in the `spyglass_plugin_xls/examples` folder. The Jinja2
templates and `site_config.yaml` can be found in the `spyglass/examples`
folder.
Validate Documents

34
doc/source/developer_quickstart.rst
View File

@ -18,52 +18,56 @@
Developer Quickstart Guide
==========================
To run your first spyglass job, follow these steps from inside the
airship-spyglass directory.
1. Clone the Spyglass directory. (Perform the following steps from inside the
spyglass directory)
1. Install external dependencies if not already installed.
.. code-block:: console
git clone https://opendev.org/airship/spyglass.git
2. Install external dependencies if not already installed.
.. code-block:: console
sudo apt install -y python3-pip
sudo apt install -y tox
2. Install Pipenv.
3. Install Pipenv.
.. code-block:: console
.. code-block:: console
pip3 install pipenv
pip3 install pipenv
2. Set up an environment with Pipenv
4. Set up an environment with Pipenv
.. code-block:: console
pipenv install
3. Enter the Pipenv environment.
5. Enter the Pipenv environment.
.. code-block:: console
pipenv shell
4. Install spyglass in the tox environment.
6. Install spyglass in the tox environment.
.. code-block:: console
pip3 install .
5. Run spyglass on the example files to generate an intermediate document.
7. Run spyglass on the example files to generate an intermediate document.
.. code-block:: console
mkdir intermediate
spyglass excel documents -s airship-seaworthy -d intermediate -i \
--excel-spec spyglass/examples/excel_spec.yaml \
--excel-file spyglass/examples/SiteDesignSpec_v0.1.xlsx \
--site-configuration spyglass/examples/site_config.yaml \
--template-dir spyglass/examples/templates/
--excel-spec ../spyglass-plugin-xls/spyglass_plugin_xls/examples/excel_spec.yaml \
--excel-file ../spyglass-plugin-xls/spyglass_plugin_xls/examples/SiteDesignSpec_v0.1.xlsx \
--site-configuration spyglass/examples/site_config.yaml \
--template-dir spyglass/examples/templates/
6. Run spyglass on the intermediate document to generate manifests.
8. Run spyglass on the intermediate document to generate manifests.
.. code-block:: console

14
doc/source/getting_started.rst
View File

@ -25,10 +25,10 @@ Spyglass is a data extraction tool which can interface with
different input data sources to generate site manifest YAML files.
The data sources will provide all the configuration data needed
for a site deployment. These site manifest YAML files generated
by spyglass will be saved in a Git repository, from where Pegleg
by Spyglass will be saved in a Git repository, from where Pegleg
can access and aggregate them. This aggregated file can then be
fed to Shipyard for site deployment / updates.
Reference: https://airshipit.readthedocs.io/projects/specs/en/latest/specs/approved/data_config_generator.html
Reference: `airship-specs`_
Architecture
------------
@ -39,7 +39,7 @@ Architecture
| | | +-------+ |
| | +------>| |Generic| |
+-----------+ | | | |Object | |
|Tugboat(Xl)| I | | | +-------+ |
|Excel | I | | | +-------+ |
|Plugin | N | | | | |
+-----------+ T | | | | |
| E | | | +------+ |
@ -65,7 +65,7 @@ Architecture
Supported Features
------------------
1. Spyglass XLS Plugin: https://opendev.org/airship/spyglass-plugin-xls
1. Spyglass Excel Plugin: https://opendev.org/airship/spyglass-plugin-xls
Future Work
-----------
@ -76,7 +76,7 @@ managed by considering a mapping of j2 templates with schemas and site type.
List of Generated Site Manifests:
---------------------------------
The spyglass uses the plugin data source to generate the following site
The Spyglass uses the plugin data source to generate the following site
manifests:
- site-definition.yaml
@ -126,8 +126,10 @@ Before using Spyglass you must:
git clone https://opendev.org/airship/spyglass.git
2. Install the required packages in spyglass:
2. Install the required packages in Spyglass:
.. code-block:: console
pip3 install pipenv && pipenv install
.. _airship-specs: https://airshipit.readthedocs.io/projects/specs/en/latest/specs/1.x/approved/data_config_generator.html

12
doc/source/index.rst
View File

@ -28,9 +28,21 @@ by spyglass will be saved in a Git repository, from where Pegleg
can access and aggregate them. This aggregated file can then be
fed to Shipyard for site deployment / updates.
Spyglass follows a plugin model to support multiple input data sources.
The currently supported plugin is the Spyglass Excel plugin
(`spyglass-plugin-xls`_).
The Spyglass Excel plugin accepts an engineering spec in the form of a
spreadsheet and an index file to read the spreadsheet as inputs and
generates site level manifests. As an optional step, it can generate an
intermediary YAML which contains all the information that will be rendered to
generate Airship site manifests. This optional step will help the deployment
engineer modify any data if required.
.. toctree::
:maxdepth: 2
getting_started
developer_quickstart
cli
.. _spyglass-plugin-xls: https://opendev.org/airship/spyglass-plugin-xls