diff --git a/docs/conf.py b/docs/conf.py index 7d36536..5082de7 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -60,6 +60,9 @@ # So we can use the ::: syntax myst_enable_extensions = ["colon_fence"] +# Generate anchors for headings (levels 1-3) so pages can link to sections. +myst_heading_anchors = 3 + # If true, Sphinx will warn about all references where the target cannot # be found. nitpicky = True diff --git a/docs/explanations.md b/docs/explanations.md index c2b70f1..47f6d3b 100644 --- a/docs/explanations.md +++ b/docs/explanations.md @@ -5,6 +5,7 @@ Explanations of how it works and why it works that way. ```{toctree} :maxdepth: 1 +explanations/configuration-model explanations/deployment-steps explanations/deployment-area explanations/snapshots-and-compare diff --git a/docs/explanations/configuration-model.md b/docs/explanations/configuration-model.md new file mode 100644 index 0000000..afeb792 --- /dev/null +++ b/docs/explanations/configuration-model.md @@ -0,0 +1,98 @@ +# The configuration model + +This page is for anyone writing or editing deployment configuration. It explains how the +files you author are structured — the objects they map to and how they nest. For every +field in exhaustive detail see the [schema reference](../schemas.md); for the meaning of +individual terms, the [glossary](../glossary.md). + +## What you author + +You author two kinds of file: a single `settings.yaml`, and one `/.yaml` +per Module version. The [schema reference](../schemas.md) covers which schema validates +which and how to point your editor at them. + +They are structured as follows: + +```text +settings.yaml + └── default_versions + +/.yaml one file per Module version + ├── deprecated: false lifecycle flag + └── module + ├── name / version / description + ├── env_vars variables set on load + ├── dependencies other Modules to load first + └── applications one or more, each of: + ├── apptainer container image + entrypoints + ├── shell a bash script + └── binary a downloaded, hash-checked executable +``` + +## A Module + +A Module is the unit an end user loads. Alongside its `name` and `version` it carries: + +| Field | Purpose | +|-------|---------| +| `description` | Shown by `module whatis `. | +| `env_vars` | Environment variables set when the Module is loaded. | +| `dependencies` | Other Modules loaded first, optionally version-pinned. | +| `applications` | One or more applications providing the executables (below). | +| `allow_updates` | Permit in-place changes to this version — see [the guard rails](deprecation-lifecycle.md#the-guard-rails). | +| `exclude_from_defaults` | Keep this version out of automatic default selection — see [default versions](default-versions.md#excluding-a-version-from-the-automatic-default). | + +`load_script` and `unload_script` run extra commands when the Module is loaded and +unloaded respectively, injected raw into the generated Modulefile. They are an advanced +escape hatch — check with a `deploy-tools` admin before using them. + +Each per-version file also carries a `deprecated` flag; see +[the release lifecycle](deprecation-lifecycle.md). + +## The three application types + +Every entry under `applications` sets `app_type` to select one of three kinds; a single +Module can mix them. + +| `app_type` | Provides | Key fields | +|------------|----------|------------| +| `apptainer` | Commands that run inside a container image | `container`, `entrypoints`, `global_options` | +| `shell` | A single executable running a bash script | `name`, `script` | +| `binary` | A downloaded executable added to the path | `name`, `url`, `hash`, `hash_type` | + +The demo `example-module-apps` Module combines an Apptainer app with a Shell app: + +```{literalinclude} ../../src/deploy_tools/demo_configuration/example-module-apps/0.1.yaml +:language: yaml +:lines: 3- +``` + +**Apptainer** uses one container image with one or more `entrypoints`, each mapping an +executable name to a command run inside the container. Options — container `mounts`, +`command_args`, `apptainer_args`, `host_binaries` — can be set per entrypoint or shared +across all of them via `global_options`. + +**Shell** exposes a single executable (`name`) that runs the `script` list as bash. + +**Binary** downloads `url`, checks it against `hash` using `hash_type` (`sha256`, +`sha512`, `md5`, or `none` to skip the check), and adds the executable to the path as +`name`. The demo `argocd` Module uses one: + +```{literalinclude} ../../src/deploy_tools/demo_configuration/argocd/v2.14.10.yaml +:language: yaml +:lines: 3- +``` + +## Settings + +`settings.yaml` holds deployment-wide settings. It currently has a single field, +`default_versions`, mapping a Module name to the version handed to `module load ` +when no version is given: + +```{literalinclude} ../../src/deploy_tools/demo_configuration/settings.yaml +:language: yaml +:lines: 3- +``` + +How that choice is resolved — and how to keep a version out of automatic selection — is +covered in [default version resolution](default-versions.md). diff --git a/docs/explanations/deployment-area.md b/docs/explanations/deployment-area.md index 0f11755..74be178 100644 --- a/docs/explanations/deployment-area.md +++ b/docs/explanations/deployment-area.md @@ -37,7 +37,8 @@ corresponding `modulefile`. Users put only the *modulefiles* directories on thei Deprecation is therefore cheap and reversible: the built files never move, only the symlink moves between `modulefiles/` and `deprecated/modulefiles/`. See -[the release lifecycle](deprecation-lifecycle.md) for the full set of transitions. +[the release lifecycle](deprecation-lifecycle.md#configuration-is-declarative) for +the full set of transitions. ## The build area diff --git a/docs/explanations/deployment-steps.md b/docs/explanations/deployment-steps.md index 349a5a4..dd77ab9 100644 --- a/docs/explanations/deployment-steps.md +++ b/docs/explanations/deployment-steps.md @@ -7,7 +7,7 @@ a command of their own. For the commands themselves see the [CLI reference](../c | Step | Description | Run by | |------|-------------|--------| | Compare | Compare the current deployment snapshot against the modulefiles and built modules that actually exist, confirming the [deployment area](deployment-area.md) is healthy. | `compare` | -| Validate | Diff the new configuration against the current snapshot to determine the set of actions to take, and check those actions are permitted by the [release lifecycle](deprecation-lifecycle.md). | `validate`, `sync` | +| Validate | Diff the new configuration against the current snapshot to determine the set of actions to take, and check those actions are permitted by the [lifecycle guard rails](deprecation-lifecycle.md#the-guard-rails). | `validate`, `sync` | | Build | Generate entrypoint scripts, configuration files and environment variables for each changed Module, writing them to the build area. | `sync` (`validate --test-build`) | | Deploy | Move the built Modules from the build area into the Modules Area, link each modulefile into the live or deprecated tree according to its status, and update default versions. | `sync` | diff --git a/docs/explanations/deprecation-lifecycle.md b/docs/explanations/deprecation-lifecycle.md index 39a6b95..b8a200e 100644 --- a/docs/explanations/deprecation-lifecycle.md +++ b/docs/explanations/deprecation-lifecycle.md @@ -9,8 +9,8 @@ deleted — is expressed by adding or removing a Release in configuration and to You never tell `deploy-tools` to "deprecate" or "remove" something directly. You describe the set of Releases you want, and the tool compares that against the -[snapshot](snapshots-and-compare.md) of the last `sync` to derive the actions needed. Each -Release falls into one of these cases: +[snapshot](snapshots-and-compare.md#what-the-snapshot-is) of the last `sync` to derive +the actions needed. Each Release falls into one of these cases: | Transition | Detected when… | Effect on the deployment area | |------------|----------------|-------------------------------| diff --git a/docs/glossary.md b/docs/glossary.md index e53d2cf..d12cd21 100644 --- a/docs/glossary.md +++ b/docs/glossary.md @@ -9,7 +9,7 @@ for the on-disk layout these terms refer to. | Environment Modules | A [standard package for Linux](https://modules.readthedocs.io/en/latest/) that provides the commands for loading and unloading 'Environment Modules' (each defined by a Modulefile). Note that while we are using this system, our definition of Module is separate. If we are referring to an Environment Module, we will use the full name. | | Modulefile | Used by the Environment Modules package to specify all details of an Environment Module. This can include executables to add to the path, environment variables to set, etc. | | Module | A set of files that can be used to provide applications on your path, provide configuration, and set environment variables. We do this using the Environment Modules system by providing a Modulefile with the relevant configuration. | -| Application | Each Module can be configured with multiple Applications, each one providing one or more executables. There are 3 types of Application: `Apptainer` (an executable container image), `Shell` (a Bash script) and `Binary` (an executable downloaded from a URL and verified against a hash). | +| Application | Each Module can be configured with multiple Applications, each one providing one or more executables. There are 3 types of Application: `Apptainer` (an executable container image), `Shell` (a Bash script) and `Binary` (an executable downloaded from a URL and verified against a hash). See [the configuration model](explanations/configuration-model.md). | | Release (noun) | A Module, including version, alongside its [lifecycle](explanations/deprecation-lifecycle.md) (i.e. deprecation) status. | | Deployment | The declared configuration for a Deployment Area: all Releases (deprecated or not) to be maintained there, plus global settings. Written to the `deployment.yaml` snapshot by `sync`. The act of deploying is written lowercase. | | Deployment Step | Refers to one of the primary steps that make up the deployment process. See [the deployment process](explanations/deployment-steps.md) for a breakdown. | diff --git a/docs/how-to.md b/docs/how-to.md index 649f7b0..c6149a6 100644 --- a/docs/how-to.md +++ b/docs/how-to.md @@ -5,6 +5,7 @@ Practical step-by-step guides for the more experienced user. ```{toctree} :maxdepth: 1 +how-to/write-module-configuration how-to/ci-pipeline how-to/run-container how-to/vscode-tasks diff --git a/docs/how-to/ci-pipeline.md b/docs/how-to/ci-pipeline.md index 55893b9..e484c82 100644 --- a/docs/how-to/ci-pipeline.md +++ b/docs/how-to/ci-pipeline.md @@ -15,9 +15,9 @@ When someone opens a change (before it is merged), run, without altering the are - `deploy-tools compare ` — confirm the area still matches its last snapshot, so the change is being checked against a known-healthy baseline. - `deploy-tools validate ` — confirm the new configuration is valid and its - [lifecycle transitions](../explanations/deprecation-lifecycle.md) are permitted. Add - `--test-build` to build every changed Module in a temporary directory, catching build - failures before merge. + [lifecycle transitions](../explanations/deprecation-lifecycle.md#the-guard-rails) are + permitted. Add `--test-build` to build every changed Module in a temporary directory, + catching build failures before merge. This gives reviewers a green light that the change is deployable without changing anything on the filesystem. diff --git a/docs/how-to/regenerate-schemas.md b/docs/how-to/regenerate-schemas.md index f707561..9d67888 100644 --- a/docs/how-to/regenerate-schemas.md +++ b/docs/how-to/regenerate-schemas.md @@ -12,7 +12,7 @@ schemas and commit again. If you bypass the hooks (for example with `git commit --no-verify`), regenerate the schemas manually. The simplest way is the **Generate Schema** VSCode task (see -[the VSCode tasks guide](vscode-tasks.md)), which writes to the correct location. +[the VSCode tasks guide](vscode-tasks.md#running-a-task)), which writes to the correct location. Equivalently, run the CLI, pointing it at that folder: ```bash diff --git a/docs/how-to/run-container.md b/docs/how-to/run-container.md index bb1ec23..3bbf881 100644 --- a/docs/how-to/run-container.md +++ b/docs/how-to/run-container.md @@ -12,3 +12,6 @@ $ docker run ghcr.io/diamondlightsource/deploy-tools:latest --version ``` To get a released version, use a numbered release instead of `latest`. + +In practice this image is what a CI pipeline runs to drive `deploy-tools` against a shared +deployment area — see [drive deploy-tools from CI](ci-pipeline.md). diff --git a/docs/how-to/write-module-configuration.md b/docs/how-to/write-module-configuration.md new file mode 100644 index 0000000..77f75e0 --- /dev/null +++ b/docs/how-to/write-module-configuration.md @@ -0,0 +1,78 @@ +# Write a Module configuration + +To add or change a Module you edit YAML in the configuration folder. This guide covers the +mechanics. For the shape of the files see +[the configuration model](../explanations/configuration-model.md); for every field, the +[schema reference](../schemas.md). + +## Add a new Module version + +1. Create the file at `//.yaml`. The folder name is the + Module `name` and the filename is the `version`, so `phoebus/0.1.yaml` defines version + `0.1` of `phoebus`. The `name` and `version` inside the file must match the path. + +2. Add the schema line as the first line so your editor validates as you type (see the + [schema reference](../schemas.md) for details, including how to associate schemas + repository-wide instead of per file): + + ```yaml + # yaml-language-server: $schema=https://raw.githubusercontent.com/DiamondLightSource/deploy-tools/main/src/deploy_tools/models/schemas/release.json + ``` + +3. Define the Module. Most Modules provide one or more applications; the smallest useful + one is a single shell script: + + ```yaml + module: + name: my-module + version: "1.0" + description: What this Module provides + applications: + - app_type: shell + name: hello + script: + - echo "hello from my-module" + ``` + + Swap the application for an `apptainer` or `binary` one as needed — see + [the three application types](../explanations/configuration-model.md#the-three-application-types). + + A Module doesn't have to provide an application: it can instead just set environment + variables or pull in other Modules as + [dependencies](../explanations/configuration-model.md#a-module). Give such a Module an + empty `applications: []`. + +## Set the default version + +`module load ` with no version loads the default. If you don't choose one the +highest version is picked automatically; to pin a specific version, add it to +`settings.yaml`: + +```yaml +default_versions: + my-module: "1.0" +``` + +To keep a version out of automatic selection — an alpha or release candidate, say — while +still allowing an explicit `module load /`, set +`exclude_from_defaults: true` on that Module. + +See [default version resolution](../explanations/default-versions.md) for how the +automatic choice is made. + +## Get your change deployed + +You don't run `deploy-tools` yourself. Open a merge request in the configuration +repository; CI validates the change and, once it is merged, deploys it (see +[drive deploy-tools from CI](ci-pipeline.md)). The `yaml-language-server` schema line +catches structural mistakes in your editor as you type, before you open the request. + +## Change or retire a version + +- **Update an existing version in place.** Rejected by default so published versions stay + stable; prefer publishing a new version. If you must, set `allow_updates: true` on the + Module — see [the guard rails](../explanations/deprecation-lifecycle.md#the-guard-rails). +- **Retire a version.** Set `deprecated: true` to steer users away from it. Deleting it + outright has to wait until after it is deprecated (unless the Module has + `allow_updates: true`). See + [the guard rails](../explanations/deprecation-lifecycle.md#the-guard-rails). diff --git a/docs/schemas.md b/docs/schemas.md index 05893d6..78e89e6 100644 --- a/docs/schemas.md +++ b/docs/schemas.md @@ -28,6 +28,12 @@ This requires an editor with a YAML language server — e.g. VS Code with the or any [LSP](https://microsoft.github.io/language-server-protocol/)-capable editor running [`yaml-language-server`](https://github.com/redhat-developer/yaml-language-server). +Alternatively, VS Code's [`yaml.schemas`](https://github.com/redhat-developer/vscode-yaml#associating-schemas) +setting (committed to `.vscode/settings.json`) maps schemas to file paths once for the +whole repository, avoiding the per-file comment. This documentation uses the comment +because it is self-contained and editor-agnostic, but a real configuration repository may +reasonably prefer the repository-level setting. + ```{note} The bundled `demo_configuration` instead points at the locally generated schemas via an absolute workspace path (e.g.