starlingx/app-gen-tool
Code Issues Proposed changes

Merge "Update app generator tool"

Browse Source 
Signed-off-by: Scott Little <scott.little@windriver.com>
Change-Id: I5931bbf1ceb466616d79922578d68e5acaa1afef
This commit is contained in:
Zuul 2023-10-20 16:52:45 +00:00 committed by Scott Little
commit 3a8341b0af
16 changed files with 1339 additions and 186 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

388
README.md
View File

@ -1,60 +1,368 @@
# StarlingX Application Generation Tool
The purpose of this tool is to generate StarlingX user applications in an easy
way without stx build environment and armada manifest schema knowledge.
The purpose of this tool is to generate a StarlingX App from a workload/app
in an easy way without the complete StarlingX build environment.
## Pre-requisite
Below you will find the steps to deploy an application as a **StarlingX App**.
1. Helm2 installed
2. python3.5+
3. pyyaml>=5.0.0 package
- [StarlingX Application Generation Tool](#starlingx-application-generation-tool)
- [Why deploy an application as a StarlingX application?](#why-deploy-an-application-as-a-starlingx-application)
- [Tools requirements](#tools-requirements)
- [Prerequisites](#prerequisites)
- [Generate the StarlingX Application package](#generate-the-starlingx-application-package)
- [App manifest configuration](#app-manifest-configuration)
- [Metadata File Configuration](#metadata-file-configuration)
- [App Setup configuration](#app-setup-configuration)
- [Run the StarlingX App Generator](#run-the-starlingx-app-generator)
- [Flux Packaging](#flux-packaging)
- [FluxCD Manifest](#fluxcd-manifest)
- [Plugins](#plugins)
- [Metadata](#metadata)
- [Tarballs](#tarballs)
- [Armada Packaging](#armada-packaging)
- [Armada Manifest](#armada-manifest)
- [Metadata](#metadata-1)
- [Tarballs](#tarballs-1)
- [Customizing the application](#customizing-the-application)
- [Flux Manifest](#flux-manifest)
- [FluxCD](#fluxcd)
- [Plugins](#plugins-1)
- [Other files](#other-files)
- [Armada Manifest](#armada-manifest-1)
`$ pip3 install pyyaml==5.1.2`
## Why deploy an application as a StarlingX application?
## 3 Steps to create a starlingx user app
It's important to understand that any user workload can be deployed in many
ways to the Kubernetes cluster(s) that StarlingX manages:
#### 1. Prepare a helm chart(s)
- with the most common Kubernetes package manager, [Helm](https://helm.sh/);
- with [Flux](https://fluxcd.io/), to enjoy all the benefits that come with it
; and finally
- as a StarlingX Application, which benefits from tight integration with the
[StarlingX system](https://opendev.org/starlingx/config).
##### What is helm and helm chart?
```TODO Elaborate on the vantages of deploying an app as a StarlingX app```
Helm is a Kubernetes package and operations manager. A Helm chart can contain
any number of Kubernetes objects, all of which are deployed as part of the
chart.
## Tools requirements
A list of official Helm Charts locates [here](https://github.com/helm/charts)
- Helm version 2+
- Python version 3.8+
- `pyyaml` version 6.0+
##### How to develop a helm chart?
## Prerequisites
Refer to official [helm doc](https://helm.sh/docs/)
As the StarlingX Platform manages a distributed Kubernetes cluster, for an
application to be deployed as a StarlingX App it needs to be designed so it can
run on [Kubernetes](https://kubernetes.io/).
#### 2. Create an app manifest
Additionally, it needs to provide a [Helm Chart](https://helm.sh/)
which will be managed via [Flux](https://fluxcd.io/) by StarlingX itself.
A few essential fields needed to create the app, simplest one could be:
## Generate the StarlingX Application package
Clone the app-generator repository.
```shell
git clone https://opendev.org/starlingx/tools.git
cd tools/app-gen-tool/
```
This is what you'll find in the `app-gen-tool` folder of the repository:
```shell
.
├── app-gen.py
├── app_manifest.yaml
├── bin
│   └── fetch_chart_info.sh
├── README.md
├── template
│   ├── armada-chartgroup.template
│   ├── armada-chart.template
│   └── armada-manifest.template
├── templates_flux
│   ├── base
│   │   ├── helmrepository.template
│   │   ├── kustomization.template
│   │   └── namespace.template
│   ├── fluxcd-manifest
│   │   ├── helmrelease.template
│   │   └── kustomization.template
│   └── kustomization.template
└── templates_plugins
├── common.template
├── helm.template
├── kustomize.template
└── lifecycle.template
```
appName: stx-app
namespace: stx-app
version: 1.0-1
chart:
- name: chart1
path: /path/to/chart1
chartGroup:
- name: chartgroup1
description: "This is the first chartgroup"
sequenced: true
chart_group:
- chart1
manifest:
name: stx-app-manifest
releasePrefix: myprefix
chart_groups:
- chartgroup1
The `app_manifest.yaml` is the most important configuration step since it
specifies everything necessary to build the StarlingX application.
The app manifest configuration can be divided into three parts, which will
have their own dedicated section below:
- [App manifest configuration](#app-manifest-configuration)
- [Metadata file configuration](#metadata-file-configuration)
- [App Setup Configuration](#app-setup-configuration)
### App manifest configuration
In this stage the section **appManifestFile-config** from the
`app_manifest.yaml` will be configured.
Below you will find a brief explanation of every one of the required fields
which will help you fill them out for you application:
- **appName** field: desired StarlingX application name, referenced throughout
the whole system.
- **appVersion** field: the version of the application that the generated
package will represent.
- **namespace** field: desired StarlingX application namespace (note that this
namespace is not the same as the Kubernetes namespace).
- **chart**
- **name** field: your Helm chart name as it is in the chart metadata.
- **version** field: your chart version as it is in the chart metadata.
- **path** field: relative path to the Helm chart directory, Helm repo or
Helm package file.
> _NOTE_: Currently only Helm charts in directories have been tested.
- **chartGroup** field: chartgroup in which the helm-chart belong
- **chartGroup**
- **name**: Name of the chartgroup
- **description**: description of chart set
- **sequenced**: enables sequenced chart deployment in a group
- **chart_names**: a list of the names of the charts from your application.
- **manifest**:
- **name**: manifest name
- **releasePrefix**: appends to the front of all charts released by the manifest in order to manage releases throughout their lifecycle
Note that the minimum required fields that will need to be filled in order
for the StarlingX App Generator to work properly will depend on the intended
type of packaging.
>_NOTE_: The two other sections bellow ([Metadata file configuration](#metadata-file-configuration)
and [App Setup Configuration](#app-setup-configuration)) will only be necessary
if you intend to package your application utilizing FluxCD.
### Metadata File Configuration
In this stage the section **metadataFile-config** from the
`app_manifest.yaml` will be configured.
This section's objective is to configure the generation/creation of a
`metadata.yaml` file, which is the core metadata file for a StarlingX App
package.
This `metadata.yaml` file is very flexible, hence the **metadataFile-config**
section is also very flexible. Other values may be passed in order to enable
advanced features within the StarlingX platform. For a better understanding of
each attribute in this section please refer to
[this link](https://wiki.openstack.org/wiki/StarlingX/Containers/StarlingXAppsInternals#metadata.yaml).
### App Setup configuration
In this stage the section **setupFile-config** from the `app_manifest.yaml`
will be configured.
Below you will find a brief explanation of every one of the required fields
which will help you fill them out for you application:
- **metadata** section:
- **author/author-email/url fields**: authorship information.
- **classifier** section: an array of additional information.
This section is related to the `setup-cfg` file that will be generated. For
more advanced use cases you may want to refer to [the documentation](https://setuptools.pypa.io/en/latest/userguide/declarative_config.html).
## Run the StarlingX App Generator
```shell
python3 app-gen.py -i app_manifest.yaml -t <armada/flux/both>
```
For more details, please refer to example.yaml
#### 3. Run app-gen.py
With the command above, the StarlingX App Generator will create a set of files
and package everything in the chosed StarlingX format.
`$ python3 app-gen.py -i app_manifest.yaml [-o ./output] [--overwrite]`
The following sections explain in high-level the most important parts of the
package.
The application will be generated automatically along with the tarball located
in the folder of your application name.
### Flux Packaging
#### FluxCD Manifest
The generator will first create the FluxCD Manifest following the structure below:
```shell
fluxcd-manifests/
├── base
│ ├── helmrepository.yaml
│ ├── kustomization.yaml
│ └── namespace.yaml
├── kustomization.yaml
└── CHART-NAME
├── helmrelease.yaml
├── kustomization.yaml
├── CHART-NAME-static-overrides.yaml
└── CHART-NAME-system-overrides.yaml
```
For every Helm chart configured in the `app_manifest.yaml` file, a folder with
the name of the chart will be created.
> **_NOTE_**: The `CHART-NAME-static-overrides.yaml` file will be empty.
#### Plugins
After the creation of the FluxCD Manifest, the generator will also create a set
of plugins with an empty implementation.
The Structure of the plugins created will be:
```shell
plugins/
├── k8sapp_APP_NAME
│ ├── common
│ │ ├── __init__.py
│ │ └── constants.py
│ ├── helm
│ │ ├── __init__.py
│ │ └── CHART_NAME.py
│ ├── kustomize
│ │ ├── __init__.py
│ │ └── kustomize_APP_NAME.py
│ └── lifecycle
│ ├── __init__.py
│ └── lifecycle_APP_NAME.py
├── __init__.py
├── setup.cfg
└── setup.py
```
The `setup.cfg` file will be created according to the
[`setupFile-config`](#app-setup-configuration) section in the `app_manifest.yaml`.
#### Metadata
In the third step of the execution the `metadata.yaml` file will be generated
with the information given in [`metadataFile-config`](#metadata-file-configuration)
section in the `app_manifest.yaml`.
#### Tarballs
After the main files have been created, the generator will start packaging
everything.
Firstly it will package every helm-chart, that was given in the
`app_manifest.yaml` file, into a `.tgz` file, saving these files into a folder
named `charts`.
>_NOTE_: For the Armada packaging, the creation of the helm-charts tarball will
be before the creation of the manifest.
The generator, then, will package the plugins with the [wheel](https://peps.python.org/pep-0491/)
format.
Lastly, creates a checksum sha256 signature file for the output tarball and
the output tarball itself, which will be called
```
<APPNAME>-<APPVERSION>.tgz
```
The structure of the app inside the tarball will be the following:
```shell
APPNAME-APPVERSION.tgz/
├── charts/
├── fluxcd-manifests/
├── plugins/
├── checksum.sha256
└── metadata.yaml
```
> **Warning:**
> At this point, the generated package is a working StarlingX App, however it
> contains empty templates for some files. The following sections will describe
> how to further enhance your StarlingX App.
### Armada Packaging
#### Armada Manifest
TODO: Check about adding this link
https://opendev.org/airship/armada/src/commit/2b714888c490a9f7c5a11383eb18b7226d1b1dc8/docs/source/operations/guide-build-armada-yaml.rst
#### Metadata
#TODO
#### Tarballs
#TODO
## Customizing the application
If you wish to add customization for the particularities of your application,
it is important to modify some of the generated files.
In order to allow such customization, the generator provides additional
functions to modify specific files in the package.
```shell
python3 app-gen.py -i app_manifest.yaml -t <armada/flux/both> [-o ./output] [--overwrite] [--no-package]|[--package-only]
```
Where:
- `-i/--input`: path to the `app_manifest.yaml` configuration file.
- `-t/--type`: type of packaging, needs to choose between armada, flux or both.
- `-o/--output`: output folder. Defaults to a new folder with the app name in
the current directory.
- `--overwrite`: deletes existing output folder before starting.
- `--no-package`: only creates the FluxCD manifest, plugins and the
metadata file, without compressing them in a tarball.
- `--package-only`: create the plugins wheels, sha256 file, helm-chart tarball
and package the entire application into a tarball.
This means that, in order to be able to make additional configuration, one must:
- first run the App Generator with `--no-package`;
- then do the changes (described in the following sections);
- finally, run the App Generator again with `--package-only`.
### Flux Manifest
#### FluxCD
> _NOTE_: this section needs improvement.
Most of the generated manifest won't need any modification, but for every
Helm chart in the `app_manifest.yaml`, a static-overrides file will be created.
The static-overrides file contains all information that is not to be
overwritten inside the values.yaml of the Helm chart.
#### Plugins
The StarlingX App Generator will create 3 main plugins: the Helm,
the Kustomize and the Lifecycle plugins.
- The `helm/APP_NAME.py` file is responsible for the overriding methods that will
be used to create the Helm overrides for the StarlingX App.
- The `kustomize_APP_NAME.py` is a plugin that is used to make changes to the
top-level `kustomization` resource list based on the platform mode.
- The `lifecycle_APP_NAME.py` is responsible for performing lifecycle actions on the
application using the lifecycle hooks of the StarlingX Platform.
The files created by the generator will have an empty implementation and is up
to the developer to implement everything that is necessary for the application
to run as intended.
The `sysinv` folder in the [StarlingX config repository](https://opendev.org/starlingx/config/src/branch/master/sysinv/sysinv/sysinv/sysinv)
contains a multitude of functions and variables that may be helpful in the
development of application plugins.
#### Other files
For the customization of the application the modifications above, in the FluxCD
and the plugins, should be enough for the application to run as expected in the
StarlingX platform.
With that in mind, it is recommended to check if the `metadata` and the `setup.cfg`
have been created as they should. Particularly, the `setup.cfg` may need careful
attention if the modifications on the plugin file should be reflected in it.
### Armada Manifest

858
app-gen.py
View File

File diff suppressed because it is too large Load Diff

85
app_manifest.yaml Normal file
View File

@ -0,0 +1,85 @@
---
## App Manifest Configuration
appManifestFile-config:
appName: # required
appVersion: # required
namespace: # required
chart:
- name: # required
version: # required
path: # required
chartGroup: # required
# add more if you have more than one chart
chartGroup:
- name: # required
description: # required for Armada
sequenced: # required for Armada <true/false>
chart_names:
- # required
- # optional
# add more if you have more than one chartgroup for your Armada app
manifest:
name: # required for Armada
releasePrefix: # required for Armada
## For Armada packaging the sections bellow are not necessary.
#################################################
## App Metadata Configuration
# for further details about possible configurations on this file, please
# visit the link: https://wiki.openstack.org/wiki/StarlingX/Containers/StarlingXAppsInternals#metadata.yaml
metadataFile-config:
# the following configurations are optional
# uncomment and configure properly the ones you need for your application metadata
# upgrades:
# auto_update: <true/false/yes/no>
# update_failure_no_rollback: <true/false/yes/no>
# from_versions:
# - <version.1>
# - <version.2>
# # ...
# supported_k8s_version:
# minimum: <version>
# maximum: <version>
# supported_releases:
# <release>:
# - <patch.1>
# - <patch.2>
# # ...
# repo: <helm repo>
# disabled_charts:
# - <chart name>
# - <chart name>
# # ...
# maintain_attributes: <true|false>
# maintain_user_overrides: <true|false>
# behavior:
# platform_managed_app: <true/false/yes/no>
# desired_state: <uploaded/applied>
# evaluate_reapply: <string>
# after:
# - <app_name.1>
# - <app_name.2>
# - ...
# triggers:
# - type: <key in APP_EVALUATE_REAPPLY_TRIGGER_TO_METADATA_MAP>
# - type: <key in APP_EVALUATE_REAPPLY_TRIGGER_TO_METADATA_MAP>
# - ...
# filters:
# - <field_name.1>: <value_name.1>
# - <field_name.2>: <value_name.2>
# - ...
# filter_field: <field_name>
# apply_progress_adjust: <0/1/2/.../100>
#################################################
## App Setup Configuration
# if you wish to see a setup.cfg example, please see the link
# https://opendev.org/starlingx/app-dell-storage/src/branch/master/python3-k8sapp-dell-storage/k8sapp_dell_storage/setup.cfg
setupFile-config:
metadata:
author: # required
author-email: # required
url: # required
classifier: # required
- # required

0
template/armada-chart.template → template_armada/armada-chart.template
View File

3
template/armada-chartgroup.template → template_armada/armada-chartgroup.template
View File

@ -6,5 +6,6 @@ metadata:
data:
description: $DESCRIPTION%This is a chartgroup$
sequenced: $SEQUENCED%false$
@CHART_GROUP|2@
chart_group:
@CHART_NAMES|4@

3
template/armada-manifest.template → template_armada/armada-manifest.template
View File

@ -5,4 +5,5 @@ metadata:
name: $NAME$
data:
release_prefix: $RELEASE_PREFIX$
@CHART_GROUPS|2@
chart_groups:
@CHART_GROUPS|4@

7
templates_flux/base/helmrepository.template Normal file
View File

@ -0,0 +1,7 @@
apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: HelmRepository
metadata:
name: stx-platform
spec:
url: http://192.168.206.1:8080/helm_charts/stx-platform
interval: 60m

3
templates_flux/base/kustomization.template Normal file
View File

@ -0,0 +1,3 @@
resources:
- helmrepository.yaml
- namespace.yaml

4
templates_flux/base/namespace.template Normal file
View File

@ -0,0 +1,4 @@
apiVersion: v1
kind: Namespace
metadata:
name: $NAMESPACE$

30
templates_flux/fluxcd-manifest/helmrelease.template Normal file
View File

@ -0,0 +1,30 @@
apiVersion: "helm.toolkit.fluxcd.io/v2beta1"
kind: HelmRelease
metadata:
name: $NAME$
labels:
chart_group: $CHART_GROUP$
spec:
releaseName: $NAME$
chart:
spec:
chart: $NAME$
version: $VERSION$
sourceRef:
kind: HelmRepository
name: stx-platform
interval: 5m
timeout: 30m
test:
enable: false
install:
disableHooks: false
upgrade:
disableHooks: false
valuesFrom:
- kind: Secret
name: $NAME$-static-overrides
valuesKey: $NAME$-static-overrides.yaml
- kind: Secret
name: $NAME$-system-overrides
valuesKey: $NAME$-system-overrides.yaml

12
templates_flux/fluxcd-manifest/kustomization.template Normal file
View File

@ -0,0 +1,12 @@
namespace: $NAMESPACE$
resources:
- helmrelease.yaml
secretGenerator:
- name: $NAME$-static-overrides
files:
- $NAME$-static-overrides.yaml
- name: $NAME$-system-overrides
files:
- $NAME$-system-overrides.yaml
generatorOptions:
disableNameSuffixHash: true

6
templates_flux/kustomization.template Normal file
View File

@ -0,0 +1,6 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: $NAMESPACE$
resources:
- base
- @CHART_NAMES|2@

7
templates_plugins/common.template Normal file
View File

@ -0,0 +1,7 @@
HELM_NS = "{namespace}"
HELM_APP = "{name}"
HELM_CHART = "{name}"
FLUXCD_HELMRELEASE = "{name}"

30
templates_plugins/helm.template Normal file
View File

@ -0,0 +1,30 @@
from sysinv.helm import base
from sysinv.common import exception
from {appname}.common import constants
class {name}Helm(base.FluxCDBaseHelm):
"""Class to encapsulate helm operations for the app chart"""
SUPPORTED_NAMESPACES = base.BaseHelm.SUPPORTED_NAMESPACES + \
[constants.HELM_NS]
SUPPORTED_APP_NAMESPACES = {{constants.HELM_APP: SUPPORTED_NAMESPACES,
}}
CHART = constants.HELM_CHART
HELM_RELEASE = constants.FLUXCD_HELMRELEASE
def get_namespaces(self):
return self.SUPPORTED_NAMESPACES
def get_overrides(self, namespace=None):
overrides = {{}}
if namespace in self.SUPPORTED_NAMESPACES:
return overrides[namespace]
elif namespace:
raise exception.InvalidHelmNamespace(chart=self.CHART,
namespace=namespace)
else:
return overrides

18
templates_plugins/kustomize.template Normal file
View File

@ -0,0 +1,18 @@
from {appname}.common import constants
from sysinv.helm import kustomize_base as base
class {appnameStriped}FluxCDKustomizeOperator(base.FluxCDKustomizeOperator):
APP = constants.HELM_APP
def platform_mode_kustomize_updates(self, dbapi, mode):
""" Update the top-level kustomization resource list
Make changes to the top-level kustomization resource list based on the
platform mode
:param dbapi: DB api object
:param mode: mode to control when to update the resource list
"""
pass

23
templates_plugins/lifecycle.template Normal file
View File

@ -0,0 +1,23 @@
""" System inventory App lifecycle operator."""
from oslo_log import log as logging
from sysinv.common import constants
from sysinv.helm import lifecycle_base as base
from sysinv.helm import lifecycle_utils as lifecycle_utils
LOG = logging.getLogger(__name__)
class {appnameStriped}AppLifecycleOperator(base.AppLifecycleOperator):
def app_lifecycle_actions(self, context, conductor_obj, app_op, app, hook_info):
""" Perform lifecycle actions for an operation
:param context: request context
:param conductor_obj: conductor object
:param app_op: AppOperator object
:param app: AppOperator.Application object
:param hook_info: LifecycleHookInfo object
"""
super({appnameStriped}AppLifecycleOperator, self).app_lifecycle_actions(context, conductor_obj, app_op, app, hook_info)