Sebastian Marcet 81e6a8ced0 Updated Search widget

Migrated from 3rd party search service swiftype to open source SOLR due to cost increase.

Change-Id: I57a2b56430c9cf088964c3c48a21144127a01e8c

2018-07-20 01:02:52 -03:00

14 KiB

Raw Blame History

Template generator details

The portion of the content on docs.openstack.org that is not managed as document sets built by Sphinx is handled via a custom template rendering tool in tools/www-generator.py within the openstack-manuals git repository.

The script reads YAML data files in www/project-data to determine which projects exist in a given series and how they should be displayed on the list of installation, configuration, and other guides.

After the script loads the data about the external projects, it scans www/ for HTML template files. It uses Jinja2 to convert the templates to complete HTML pages, which it writes to the publish-docs output directory.

The Jinja2 documentation includes a guide for template designers that covers the syntax of templates, inheritance between templates, and the various filters and other features available when writing templates.

The template files

The template files in the openstack-manuals repo are all under the www directory. They are organized in the same structure used to publish to the site, so the path to a published URL corresponds directly to a path to the template file that produces it.

Here are a few representative files under www/:

austin
- index.html -- landing page for $series docs (one per series)
de
- index.html -- list of guides translated into $LANG
errorpage.html
mitaka -- newer series have more complex page organizations; each directory is unique
- admin
  - index.html
- api
  - index.html
- de
  - index.html -- landing page for $series docs translated into $LANG
- index.html
- language-bindings.html
- projects.html
- user
  - index.html
pike
- admin
  - index.html
- api
  - index.html
- configuration
  - index.html
- deploy
  - index.html
- index.html
- install
  - index.html
- language-bindings.html
- projects.html
- user
  - index.html
project-data -- YAML files with data about projects in each $series
- latest.yaml
- mitaka.yaml
- newton.yaml
- ocata.yaml
- pike.yaml
- README.rst
- schema.yaml
redirect-tests.txt -- input file for whereto to test .htaccess; see http://files.openstack.org/docs-404s/ for a list of recent 404 errors
static -- contains files that are not templates (CSS, JS, sitemap, etc.)
templates -- contains reused templates (base pages, partial pages, etc.)
- api_guides.tmpl
- base.tmpl
- contributor_guides.tmpl
- css.tmpl
- default.tmpl
- dropdown_releases_and_languages.tmpl
- footer.tmpl
- google_analytics.tmpl
- header.tmpl
- indexbase.tmpl
- navigation.tmpl
- ops_and_admin_guides.tmpl
- project_guides.tmpl
- script_footer.tmpl
- script_search.tmpl
- series_status.tmpl
- os_search_install.tmpl
- os_search_mobile.tmpl
- os_search.tmpl
- training_guides.tmpl
- user_guides.tmpl

Defining release series

The set of release series and their individual status and other metadata is embedded in the template generator script in the SERIES_INFO data structure. The structure is a dictionary mapping the name of the release series to a SeriesInfo structure holding the metadata.

For each release series, the generator needs to know:

date

The date value should be a string containing the month name and 4 digit year.

status

The 'status' field should be one of:

obsolete: the release existed, but we have no more artifacts for it
EOL: the release is closed but we have docs for it
maintained: the release still has an open branch
development: the current release being developed

release-www-page-updates has some additional information about how the status values are updated at the end of a release cycle.

Project data file format

The projects associated with each release series are listed in a separate YAML file in the www/project-data directory. Each file is named for the series (austin.yaml, bexar.yaml, etc.) except for the series currently under development which is always kept in latest.yaml.

The schema for the project data files is defined in www/project-data/schema.yaml.

Each file should contain an array or list of entries. Each entry must define the name, service, and type properties.

The name should be the base name of a git repository.

The deliverable-name should be the name of the deliverable as defined in openstack/governance/reference/projects.yaml. This value only needs to be set if the deliverable name does not match the project name (such as glance_store and glance-store).

The service string should be taken from the governance repository definition of the project.

The type must be one of the values listed below:

service: A REST API service.
cloud-client: A library for talking to a cloud.
service-client: A library for talking to a service.
library: Another type of library.
tool: A command line tool or other project that is used with, or used to build, OpenStack.
networking: A plugin for the networking service.
baremetal: A subproject for the bare metal project, Ironic.
deployment: A tool for deploying OpenStack.
other: A project that does run in a cloud but does not provide a REST API.

An entry can also optionally define service_type, which must match the value associated with the name in the service-types-authority repository.

Entries with type set to client should include a description field with a short description, such as "keystone client".

Entries may optionally set flags to indicate that the repository includes particular types of documentation in an expected location, to include a link to that documentation on the templated landing pages.

has_install_guide: produces a link to docs.o.o/{{name}}/latest/install/
has_api_guide: produces a link to developer.o.o/api-guide/{{service_type}}/
has_api_ref: produces a link to developer.o.o/api-ref/{{service_type}}/
has_config_ref: produces a link to docs.o.o/{{name}}/latest/configuration/
has_in_tree_api_docs: produces a link to docs.o.o/{{name}}/latest/api/
has_admin_guide: produces a link to docs.o.o/{{name}}/latest/admin/
has_in_tree_htaccess: enables full redirects to old paths, not just to the top of /{{name}}/latest/
has_deployment_guide: produces a link to docs.o.o/project-deploy-guide/{{name}}/{{series}}/

Note

The documentation associated with the flags must exist before the flags are set.

Template variables

The template generator uses the input data to set several variables visible within the template. This allows us to reuse the same template to generate content for multiple pages of the same style, filling in different data.

By convention, all of the variables defined in the template generator use all uppercase names. This makes it easy to differentiate the generator variables from variables defined within templates (such as loop contexts).

TEMPLATE_FILE

The name of the template file being rendered, with the www prefix removed. For example, pike/index.html.

PROJECT_DATA

All of the project data loaded from the data files in a dictionary mapping the series name to the parsed data file. Most template pages will assign a local variable using PROJECT_DATA[SERIES] to extract the correct subset of the data.

TOPDIR

The relative path to the top of the build output. This is useful for building paths between output pages in a way that allows those pages to move around later.

SCRIPTDIR

The relative path to the location of the JavaScript directory in the build output. This is useful for building links to JavaScript files.

CSSDIR

The relative path to the location of the directory containing the CSS files in the build output. This is useful for building links to CSS files.

IMAGEDIR

The relative path to the location of the directory containing image files in the build output. This is useful for building links to images.

SERIES

A string containing the name of the series usable in URLs. For the series under development, this is "latest". For other series, it is the series name in lower case.

This value is derived from the path to the template file. If the file is under a directory that matches one of the known series names, that value is used to set SERIES.

SERIES_TITLE

A string containing the name of the series usable in text visible to the reader. It is always the actual name of the series in "title case" (the first letter of each word is uppercase). For example, "Pike".

This value is derived from the path to the template file. If the file is under a directory that matches one of the known series names, that value is used to set SERIES.

ALL_SERIES

A list of all of the series names for all OpenStack releases, in order of release.

This list is derived from the keys of the SERIES_INFO dictionary defined in the template generator.

PAST_SERIES

A list of the series names for OpenStack releases with a status other than "development".

This list is derived from the values in the SERIES_INFO dictionary defined in the template generator.

RELEASED_SERIES

A string containing the lowercase name of the most recent series to be released. For example, during the Pike series this value was "ocata".

This value is derived from the values in the SERIES_INFO dictionary defined in the template generator.

SERIES_IN_DEVELOPMENT

A string containing the lowercase name of the series under active development. For example, during the Pike series this value was "pike".

This value is derived from the values in the SERIES_INFO dictionary defined in the template generator.

SERIES_INFO

The SeriesInfo object associated with the current series. This provides access to the date and status values for the series.

This value is taken from the SERIES_INFO dictionary defined in the template generator.

REGULAR_REPOS

A list of all of the names of regular repositories for official OpenStack projects. Here "regular" differentiates the repositories from infrastructure team repositories, which have their documentation published to a different location and therefore need some different URLs for redirects in the .htaccess template. See INFRA_REPOS.

This value is derived from data published from the governance repository.

INFRA_REPOS

A list of all of the names of repositories for the infrastructure team. See REGULAR_REPOS.

This value is derived from data published from the governance repository.

Common tasks

How would I change a page?

Look for the TEMPLATE_FILE value in the page source to find which file produces the page.

The source for https://docs.openstack.org/pike/ shows:
```

```
Modify the file or one of the other templates from which it inherits.

www/pike/index.html has a base template of www/templates/indexbase.tmpl which contains:
```
{% include "templates/series_status.tmpl" %}
```
and that directive pulls in www/templates/series_status.tmpl.

How would I add a new project?

Modify www/project-data/latest.yaml to add the new stanza.

Flags for having various types of docs default to off; only list the ones that should be turned on. Set the type for the project to ensure it shows up in the correct list(s).

How would I add a new flag to the project metadata?

Update the schema to allow the flag by changing www/project-data/schema.yaml.
Update the documentation team contributor guide to explain the flag's use by modifying doc/doc-contribu-guide/source/doc-tools/template-generator.rst.
Update www/project-data/latest.yaml to set the flag for some project(s).
Update/create the template that will use the flag.

How would I add a new page?

Copy an existing template file to the new name under www/ and then modify it.

How does the final release process work?

See ../release/taskdetail.

Testing the build

There are two commands useful for testing the build locally:

$ tox -e checkbuild

and

$ tools/test.sh

The test script supports a few options to make it more effective.

--skip-links: Skip link checks
--series SERIES: series to update/test
--check-all-links: Check for links with flags set false.

To test template rendering without waiting for link checks:

$ ./tools/test.sh --skip-links

To test project links only for items listed in latest.yaml:

$ ./tools/test.sh --series latest

To produce a list of the unset flags for latest.yaml that could be set (the pages linked do exist):

$ ./tools/test.sh --check-all-links --series latest

14 KiB Raw Blame History