starlingx/app-gen-tool
Code Issues Proposed changes
b57e56f487
app-gen-tool/README.md
Reed, Joshua b57e56f487 Remove tempo-vulture example and keep only adminer. 
The adminer example is an APACHE-2.0 license which
mathes this repo and the tempo-vulture one does not.
Therefore, upating this repo to keep similar licenses.

Test Plan:
PASS - Make sure pytests keep working after update.
PASS - Make sure all examples continue to work after
       changing them.

Story: 2010937
Task: 49379

Change-Id: Iba24c9e356886dc8c300c092fb42590f53ae4bc7
Signed-off-by: Reed, Joshua <Joshua.Reed@windriver.com>
2024-01-04 09:47:26 -07:00

470 lines
17 KiB
Markdown
Raw Blame History

# StarlingX Application Generation Tool
The purpose of this tool is to generate a StarlingX App from a workload/app
in an easy way without the complete StarlingX build environment. The tool is ulimately a python
package, but is also wrapped as a debian package because this tool is included in StarlingX. After
a full install users may use the generator to help develop apps directly on platform.
Below you will find the steps to deploy an application as a **StarlingX App**.
- [StarlingX Application Generation Tool](#starlingx-application-generation-tool)
- [Why deploy an application as a StarlingX application?](#why-deploy-an-application-as-a-starlingx-application)
- [Tools requirements](#tools-requirements)
- [Prerequisites](#prerequisites)
- [Generate the StarlingX Application package](#generate-the-starlingx-application-package)
- [App manifest configuration](#app-manifest-configuration)
- [Metadata File Configuration](#metadata-file-configuration)
- [App Setup configuration](#app-setup-configuration)
- [Run the StarlingX App Generator](#run-the-stx-app-generator)
- [FluxCD Packaging](#fluxcd-packaging)
- [FluxCD Manifest](#fluxcd-manifest)
- [Plugins](#plugins)
- [Metadata](#metadata)
- [Tarballs](#tarballs)
- [Customizing the application](#customizing-the-application)
- [FluxCD Manifest](#fluxcd-manifest)
- [FluxCD](#fluxcd)
- [Plugins](#plugins-1)
- [Other files](#other-files)
## Why deploy an application as a StarlingX application?
It's important to understand that any user workload can be deployed in many
ways to the Kubernetes cluster(s) that StarlingX manages:
- with the most common Kubernetes package manager, [Helm](https://helm.sh/);
- with [Flux](https://fluxcd.io/), to enjoy all the benefits that come with it
; and finally
- as a StarlingX Application, which benefits from tight integration with the
[StarlingX system](https://opendev.org/starlingx/config).
Deploying applications offers a number of benefits in a deployed environment:
1. Applications are managed throughout an upgrade from one version of STX to another.
2. Appliations are managed throughout an upgrade of Kubernetes.
3. Deployment of applications is managed automatically in a distributed cloud environment.
```TODO Elaborate on additional vantages of deploying an app as a StarlingX app```
## Software Requirements
- Helm version 3+
- Python version 3.8+
- Python packages tracked in ./stx-app-generator/stx-app-generator/requirements.txt
- For Testing, Python packages tracked in ./test-requirements.txt
## Prerequisites
As the StarlingX Platform manages a distributed Kubernetes cluster, for an
application to be deployed as a StarlingX App it needs to be designed so it can
run on [Kubernetes](https://kubernetes.io/).
Additionally, it needs to provide a [Helm Chart](https://helm.sh/)
which will be managed via [FluxCD](https://fluxcd.io/) by StarlingX itself.
## Standalone Installation
If one desires to install the python package in an environment other than StarlingX, the use one of
the following two options.
### Option 1 - Install in same folder
```shell
git clone https://opendev.org/starlingx/app-gen-tool.git
cd ./app-gen-tool/
python -m venv venv
source ./venv/bin/activate
pip install -e ./stx-app-generator/stx-app-generator
```
### Option 2 - Install From Url
```shell
python -m venv venv
source ./venv/bin/activate
pip install git+https://opendev.org/starlingx/app-gen-tool.git#subdirectory=./stx-app-generator/stx-app-generator
```
## Generate the StarlingX Application package
This is what you'll find in the `app-gen-tool` repository:
```shell.
.
├── LICENSE
├── README.md
├── app_manifest.yaml
├── bandit.yaml
├── debian_build_layer.cfg
├── debian_iso_image.inc
├── debian_pkg_dirs
├── debian_stable_wheels.inc
├── example
│ ├── app-test-example.yaml
│ ├── helm-adminer
│ │ └── adminer-0.2.1.tgz
│ └── poc-starlingx-messages
│ ├── Chart.yaml
│ ├── templates
│ │ ├── deployment.yaml
│ │ └── service.yaml
│ └── values.yaml
├── output
│ └── app-test
│ ├── charts
│ ├── fluxcd-manifests
│ │ ├── adminer
│ │ │ ├── adminer-static-overrides.yaml
│ │ │ ├── adminer-system-overrides.yaml
│ │ │ ├── helmrelease.yaml
│ │ │ └── kustomization.yaml
│ │ ├── base
│ │ │ ├── helmrepository.yaml
│ │ │ ├── kustomization.yaml
│ │ │ └── namespace.yaml
│ │ └── kustomization.yaml
│ ├── metadata.yaml
│ └── plugins
│ ├── __init__.py
│ ├── k8sapp_app_test
│ │ ├── __init__.py
│ │ ├── common
│ │ │ ├── __init__.py
│ │ │ └── constants.py
│ │ ├── helm
│ │ │ ├── __init__.py
│ │ │ └── adminer.py
│ │ ├── kustomize
│ │ │ ├── __init__.py
│ │ │ └── kustomize_app_test.py
│ │ └── lifecycle
│ │ ├── __init__.py
│ │ └── lifecycle_app_test.py
│ ├── setup.cfg
│ └── setup.py
├── playbooks
│ └── app-gen-tool-tox-coverage
│ └── pre.yaml
├── stx-app-generator
│ ├── debian
│ │ ├── deb_folder
│ │ │ ├── changelog
│ │ │ ├── control
│ │ │ ├── copyright
│ │ │ ├── rules
│ │ │ ├── source
│ │ │ │ ├── format
│ │ │ │ └── options
│ │ │ ├── stx-app-generator-wheels.install
│ │ │ ├── stx-app-generator.dirs
│ │ │ └── stx-app-generator.install
│ │ └── meta_data.yaml
│ └── stx-app-generator
│ ├── app_gen_tool
│ │ ├── __init__.py
│ │ ├── application.py
│ │ ├── cmd
│ │ │ ├── __init__.py
│ │ │ └── generator.py
│ │ ├── common.py
│ │ ├── constants.py
│ │ ├── fluxcd.py
│ │ ├── generator.py
│ │ ├── templates_fluxcd
│ │ │ ├── base
│ │ │ │ ├── helmrepository.template
│ │ │ │ ├── kustomization.template
│ │ │ │ └── namespace.template
│ │ │ ├── fluxcd-manifest
│ │ │ │ ├── helmrelease.template
│ │ │ │ └── kustomization.template
│ │ │ └── kustomization.template
│ │ └── templates_plugins
│ │ ├── common.template
│ │ ├── helm.template
│ │ ├── kustomize.template
│ │ └── lifecycle.template
│ ├── requirements.txt
│ ├── setup.cfg
│ ├── setup.py
│ └── tests
│ ├── __init__.py
│ ├── conftest.py
│ └── unit
│ ├── __init__.py
│ ├── resources
│ │ ├── adminer-0.2.1.tgz
│ │ └── app-test.yaml
│ ├── test_application_class.py
│ └── test_common_functions.py
├── test-requirements.txt
└── tox.ini
```
The `app_manifest.yaml` is the most important configuration step since it
specifies everything necessary to build the StarlingX application.
The app manifest configuration can be divided into three parts, which will
have their own dedicated section below:
- [App manifest configuration](#app-manifest-configuration)
- [Metadata file configuration](#metadata-file-configuration)
- [App Setup Configuration](#app-setup-configuration)
### App manifest configuration
In this stage the section **appManifestFile-config** from the
`app_manifest.yaml` will be configured.
Below you will find a brief explanation of every one of the required fields
which will help you fill them out for you application:
- **appName** field: desired StarlingX application name, referenced throughout
the whole system.
- **appVersion** field: the version of the application that the generated
package will represent.
- **namespace** field: desired StarlingX application namespace (note that this
namespace is not the same as the Kubernetes namespace).
- **chart**
- **name** field: your Helm chart name as it is in the chart metadata.
- **version** field: your chart version as it is in the chart metadata.
- **path** field: relative path to the Helm chart directory, Helm repo or
Helm package file.
> _NOTE_: Currently only Helm charts in directories have been tested.
- **chartGroup** field: chartgroup in which the helm-chart belong
- **chartGroup**
- **name**: Name of the chartgroup
- **description**: description of chart set
- **sequenced**: enables sequenced chart deployment in a group
- **chart_names**: a list of the names of the charts from your application.
- **manifest**:
- **name**: manifest name
- **releasePrefix**: appends to the front of all charts released by the manifest in order to manage releases throughout their lifecycle
Note that the minimum required fields that will need to be filled in order
for the StarlingX App Generator to work properly will depend on the intended
type of packaging.
### Metadata File Configuration
In this stage the section **metadataFile-config** from the
`app_manifest.yaml` will be configured.
This section's objective is to configure the generation/creation of a
`metadata.yaml` file, which is the core metadata file for a StarlingX App
package.
This `metadata.yaml` file is very flexible, hence the **metadataFile-config**
section is also very flexible. Other values may be passed in order to enable
advanced features within the StarlingX platform. For a better understanding of
each attribute in this section please refer to
[this link](https://wiki.openstack.org/wiki/StarlingX/Containers/StarlingXAppsInternals#metadata.yaml).
### App Setup configuration
In this stage the section **setupFile-config** from the `app_manifest.yaml`
will be configured.
Below you will find a brief explanation of every one of the required fields
which will help you fill them out for you application:
- **metadata** section:
- **author/author-email/url fields**: authorship information.
- **classifier** section: an array of additional information.
This section is related to the `setup-cfg` file that will be generated. For
more advanced use cases you may want to refer to [the documentation](https://setuptools.pypa.io/en/latest/userguide/declarative_config.html).
## Run the StarlingX App Generator
One must install via pip install method described in the Installation section.
To get command line options:
```shell
stx-app-generator -h
```
Recommend reviewing the '-h' output for a full list of options.
Here is an example.
```shell
stx-app-generator -i app_manifest.yaml -o ./output
```
With the command above, the StarlingX App Generator will create a set of files
and package everything in the chosen StarlingX format, and then finally place
the file in an output folder.
## Supported Packaging Methods
The following sections explain in high-level the most important parts of the
package.
### Flux Packaging
#### FluxCD Manifest
The generator will first create the FluxCD Manifest following the structure below:
```shell
fluxcd-manifests/
├── base
│ ├── helmrepository.yaml
│ ├── kustomization.yaml
│ └── namespace.yaml
├── kustomization.yaml
└── CHART-NAME
├── helmrelease.yaml
├── kustomization.yaml
├── CHART-NAME-static-overrides.yaml
└── CHART-NAME-system-overrides.yaml
```
For every Helm chart configured in the `app_manifest.yaml` file, a folder with
the name of the chart will be created.
> **_NOTE_**: The `CHART-NAME-static-overrides.yaml` file will be empty.
#### Plugins
After the creation of the FluxCD Manifest, the generator will also create a set
of plugins with an empty implementation.
The Structure of the plugins created will be:
```shell
plugins/
├── k8sapp_APP_NAME
│ ├── common
│ │ ├── __init__.py
│ │ └── constants.py
│ ├── helm
│ │ ├── __init__.py
│ │ └── CHART_NAME.py
│ ├── kustomize
│ │ ├── __init__.py
│ │ └── kustomize_APP_NAME.py
│ └── lifecycle
│ ├── __init__.py
│ └── lifecycle_APP_NAME.py
├── __init__.py
├── setup.cfg
└── setup.py
```
The `setup.cfg` file will be created according to the
[`setupFile-config`](#app-setup-configuration) section in the `app_manifest.yaml`.
#### Metadata
In the third step of the execution the `metadata.yaml` file will be generated
with the information given in [`metadataFile-config`](#metadata-file-configuration)
section in the `app_manifest.yaml`.
#### Tarballs
After the main files have been created, the generator will start packaging
everything.
Firstly it will package every helm-chart, that was given in the
`app_manifest.yaml` file, into a `.tgz` file, saving these files into a folder
named `charts`.
The generator, then, will package the plugins with the [wheel](https://peps.python.org/pep-0491/)
format.
Lastly, creates a checksum sha256 signature file for the output tarball and
the output tarball itself, which will be called
```
<APPNAME>-<APPVERSION>.tgz
```
The structure of the app inside the tarball will be the following:
```shell
APPNAME-APPVERSION.tgz/
├── charts/
├── fluxcd-manifests/
├── plugins/
├── checksum.sha256
└── metadata.yaml
```
> **Warning:**
> At this point, the generated package is a working StarlingX App, however it
> contains empty templates for some files. The following sections will describe
> how to further enhance your StarlingX App.
If you wish to add customization for the particularities of your application,
it is important to modify some of the generated files.
In order to allow such customization, the generator provides additional
functions to modify specific files in the package.
```shell
stx-app-generator -i app_manifest.yaml [-o ./output] [--overwrite] [--no-package]|[--package-only]
```
Where:
- `-i/--input`: path to the `app_manifest.yaml` configuration file.
- `-o/--output`: output folder. Defaults to a new folder with the app name in
the current directory.
- `--overwrite`: deletes existing output folder before starting.
- `--no-package`: only creates the FluxCD manifest, plugins and the
metadata file, without compressing them in a tarball.
- `--package-only`: create the plugins wheels, sha256 file, helm-chart tarball
and package the entire application into a tarball.
This means that, in order to be able to make additional configuration, one must:
- first run the App Generator with `--no-package`;
- then do the changes (described in the following sections);
- finally, run the App Generator again with `--package-only`.
### Flux Manifest
#### FluxCD
> _NOTE_: this section needs improvement.
Most of the generated manifest won't need any modification, but for every
Helm chart in the `app_manifest.yaml`, a static-overrides file will be created.
The static-overrides file contains all information that is not to be
overwritten inside the values.yaml of the Helm chart.
#### Plugins
The StarlingX App Generator will create 3 main plugins: the Helm,
the Kustomize and the Lifecycle plugins.
- The `helm/APP_NAME.py` file is responsible for the overriding methods that will
be used to create the Helm overrides for the StarlingX App.
- The `kustomize_APP_NAME.py` is a plugin that is used to make changes to the
top-level `kustomization` resource list based on the platform mode.
- The `lifecycle_APP_NAME.py` is responsible for performing lifecycle actions on the
application using the lifecycle hooks of the StarlingX Platform.
The files created by the generator will have an empty implementation and is up
to the developer to implement everything that is necessary for the application
to run as intended.
The `sysinv` folder in the [StarlingX config repository](https://opendev.org/starlingx/config/src/branch/master/sysinv/sysinv/sysinv/sysinv)
contains a multitude of functions and variables that may be helpful in the
development of application plugins.
#### Other files
For the customization of the application the modifications above, in the FluxCD
and the plugins, should be enough for the application to run as expected in the
StarlingX platform.
With that in mind, it is recommended to check if the `metadata` and the `setup.cfg`
have been created as they should. Particularly, the `setup.cfg` may need careful
attention if the modifications on the plugin file should be reflected in it.
View Git Blame Copy Permalink