From 9ae97d6bafab2869f50d83776bb09650da936ef7 Mon Sep 17 00:00:00 2001 From: Martin Gaughran Date: Thu, 30 Jul 2026 16:57:59 +0000 Subject: [PATCH 01/18] Add explanation doc for the configuration structure and contents --- docs/explanations.md | 1 + docs/explanations/configuration-model.md | 98 ++++++++++++++++++++++++ docs/glossary.md | 2 +- 3 files changed, 100 insertions(+), 1 deletion(-) create mode 100644 docs/explanations/configuration-model.md diff --git a/docs/explanations.md b/docs/explanations.md index c2b70f1..47f6d3b 100644 --- a/docs/explanations.md +++ b/docs/explanations.md @@ -5,6 +5,7 @@ Explanations of how it works and why it works that way. ```{toctree} :maxdepth: 1 +explanations/configuration-model explanations/deployment-steps explanations/deployment-area explanations/snapshots-and-compare diff --git a/docs/explanations/configuration-model.md b/docs/explanations/configuration-model.md new file mode 100644 index 0000000..670d2e8 --- /dev/null +++ b/docs/explanations/configuration-model.md @@ -0,0 +1,98 @@ +# The configuration model + +This page is for anyone writing or editing deployment configuration. It explains how the +files you author are structured — the objects they map to and how they nest. For every +field in exhaustive detail see the [schema reference](../schemas.md); for the meaning of +individual terms, the [glossary](../glossary.md). + +## What you author + +You author two kinds of file: a single `settings.yaml`, and one `/.yaml` +per Module version. The [schema reference](../schemas.md) covers which schema validates +which and how to point your editor at them. + +They are structured as follows: + +```text +settings.yaml + └── default_versions + +/.yaml one file per Module version + ├── deprecated: false lifecycle flag + └── module + ├── name / version / description + ├── env_vars variables set on load + ├── dependencies other Modules to load first + └── applications one or more, each of: + ├── apptainer container image + entrypoints + ├── shell a bash script + └── binary a downloaded, hash-checked executable +``` + +## A Module + +A Module is the unit an end user loads. Alongside its `name` and `version` it carries: + +| Field | Purpose | +|-------|---------| +| `description` | Shown by `module whatis `. | +| `env_vars` | Environment variables set when the Module is loaded. | +| `dependencies` | Other Modules loaded first, optionally version-pinned. | +| `applications` | One or more applications providing the executables (below). | +| `allow_updates` | Permit in-place changes to this version — see [the release lifecycle](deprecation-lifecycle.md). | +| `exclude_from_defaults` | Keep this version out of automatic default selection — see [default versions](default-versions.md). | + +`load_script` and `unload_script` run extra commands when the Module is loaded and +unloaded respectively, injected raw into the generated Modulefile. They are an advanced +escape hatch — check with a `deploy-tools` admin before using them. + +Each per-version file also carries a `deprecated` flag; see +[the release lifecycle](deprecation-lifecycle.md). + +## The three application types + +Every entry under `applications` sets `app_type` to select one of three kinds; a single +Module can mix them. + +| `app_type` | Provides | Key fields | +|------------|----------|------------| +| `apptainer` | Commands that run inside a container image | `container`, `entrypoints`, `global_options` | +| `shell` | A single executable running a bash script | `name`, `script` | +| `binary` | A downloaded executable added to the path | `name`, `url`, `hash`, `hash_type` | + +The demo `example-module-apps` Module combines an Apptainer app with a Shell app: + +```{literalinclude} ../../src/deploy_tools/demo_configuration/example-module-apps/0.1.yaml +:language: yaml +:lines: 3- +``` + +**Apptainer** uses one container image with one or more `entrypoints`, each mapping an +executable name to a command run inside the container. Options — container `mounts`, +`command_args`, `apptainer_args`, `host_binaries` — can be set per entrypoint or shared +across all of them via `global_options`. + +**Shell** exposes a single executable (`name`) that runs the `script` list as bash. + +**Binary** downloads `url`, checks it against `hash` using `hash_type` (`sha256`, +`sha512`, `md5`, or `none` to skip the check), and adds the executable to the path as +`name`. The demo `argocd` Module uses one: + +```{literalinclude} ../../src/deploy_tools/demo_configuration/argocd/v2.14.10.yaml +:language: yaml +:lines: 3- +``` + +## Settings + +`settings.yaml` holds deployment-wide settings. It currently has a single field, +`default_versions`, mapping a Module name to the version handed to `module load ` +when no version is given: + +```{literalinclude} ../../src/deploy_tools/demo_configuration/settings.yaml +:language: yaml +:lines: 3- +``` + +How that choice is resolved — and how to keep a version out of automatic selection — is +covered in [default version resolution](default-versions.md). diff --git a/docs/glossary.md b/docs/glossary.md index e53d2cf..d12cd21 100644 --- a/docs/glossary.md +++ b/docs/glossary.md @@ -9,7 +9,7 @@ for the on-disk layout these terms refer to. | Environment Modules | A [standard package for Linux](https://modules.readthedocs.io/en/latest/) that provides the commands for loading and unloading 'Environment Modules' (each defined by a Modulefile). Note that while we are using this system, our definition of Module is separate. If we are referring to an Environment Module, we will use the full name. | | Modulefile | Used by the Environment Modules package to specify all details of an Environment Module. This can include executables to add to the path, environment variables to set, etc. | | Module | A set of files that can be used to provide applications on your path, provide configuration, and set environment variables. We do this using the Environment Modules system by providing a Modulefile with the relevant configuration. | -| Application | Each Module can be configured with multiple Applications, each one providing one or more executables. There are 3 types of Application: `Apptainer` (an executable container image), `Shell` (a Bash script) and `Binary` (an executable downloaded from a URL and verified against a hash). | +| Application | Each Module can be configured with multiple Applications, each one providing one or more executables. There are 3 types of Application: `Apptainer` (an executable container image), `Shell` (a Bash script) and `Binary` (an executable downloaded from a URL and verified against a hash). See [the configuration model](explanations/configuration-model.md). | | Release (noun) | A Module, including version, alongside its [lifecycle](explanations/deprecation-lifecycle.md) (i.e. deprecation) status. | | Deployment | The declared configuration for a Deployment Area: all Releases (deprecated or not) to be maintained there, plus global settings. Written to the `deployment.yaml` snapshot by `sync`. The act of deploying is written lowercase. | | Deployment Step | Refers to one of the primary steps that make up the deployment process. See [the deployment process](explanations/deployment-steps.md) for a breakdown. | From 82b721cf9d6741fbf629b71ad93de2f706211102 Mon Sep 17 00:00:00 2001 From: Martin Gaughran Date: Thu, 30 Jul 2026 18:35:50 +0000 Subject: [PATCH 02/18] Add docs on writing Module configuration --- docs/conf.py | 3 + docs/how-to.md | 1 + docs/how-to/write-module-configuration.md | 77 +++++++++++++++++++++++ 3 files changed, 81 insertions(+) create mode 100644 docs/how-to/write-module-configuration.md diff --git a/docs/conf.py b/docs/conf.py index 7d36536..5082de7 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -60,6 +60,9 @@ # So we can use the ::: syntax myst_enable_extensions = ["colon_fence"] +# Generate anchors for headings (levels 1-3) so pages can link to sections. +myst_heading_anchors = 3 + # If true, Sphinx will warn about all references where the target cannot # be found. nitpicky = True diff --git a/docs/how-to.md b/docs/how-to.md index 649f7b0..c6149a6 100644 --- a/docs/how-to.md +++ b/docs/how-to.md @@ -5,6 +5,7 @@ Practical step-by-step guides for the more experienced user. ```{toctree} :maxdepth: 1 +how-to/write-module-configuration how-to/ci-pipeline how-to/run-container how-to/vscode-tasks diff --git a/docs/how-to/write-module-configuration.md b/docs/how-to/write-module-configuration.md new file mode 100644 index 0000000..707ecf6 --- /dev/null +++ b/docs/how-to/write-module-configuration.md @@ -0,0 +1,77 @@ +# Write a Module configuration + +To add or change a Module you edit YAML in the configuration folder. This guide covers the +mechanics. For the shape of the files see +[the configuration model](../explanations/configuration-model.md); for every field, the +[schema reference](../schemas.md). + +## Add a new Module version + +1. Create the file at `//.yaml`. The folder name is the + Module `name` and the filename is the `version`, so `phoebus/0.1.yaml` defines version + `0.1` of `phoebus`. The `name` and `version` inside the file must match the path. + +2. Add the schema line as the first line so your editor validates as you type (see the + [schema reference](../schemas.md) for details): + + ```yaml + # yaml-language-server: $schema=https://raw.githubusercontent.com/DiamondLightSource/deploy-tools/main/src/deploy_tools/models/schemas/release.json + ``` + +3. Define the Module. Most Modules provide one or more applications; the smallest useful + one is a single shell script: + + ```yaml + module: + name: my-module + version: "1.0" + description: What this Module provides + applications: + - app_type: shell + name: hello + script: + - echo "hello from my-module" + ``` + + Swap the application for an `apptainer` or `binary` one as needed — see + [the three application types](../explanations/configuration-model.md#the-three-application-types). + + A Module doesn't have to provide an application: it can instead just set environment + variables or pull in other Modules as + [dependencies](../explanations/configuration-model.md#a-module). Give such a Module an + empty `applications: []`. + +## Set the default version + +`module load ` with no version loads the default. If you don't choose one the +highest version is picked automatically; to pin a specific version, add it to +`settings.yaml`: + +```yaml +default_versions: + my-module: "1.0" +``` + +To keep a version out of automatic selection — an alpha or release candidate, say — while +still allowing an explicit `module load /`, set +`exclude_from_defaults: true` on that Module. + +See [default version resolution](../explanations/default-versions.md) for how the +automatic choice is made. + +## Get your change deployed + +You don't run `deploy-tools` yourself. Open a merge request in the configuration +repository; CI validates the change and, once it is merged, deploys it (see +[drive deploy-tools from CI](ci-pipeline.md)). The `yaml-language-server` schema line +catches structural mistakes in your editor as you type, before you open the request. + +## Change or retire a version + +- **Update an existing version in place.** Rejected by default so published versions stay + stable; prefer publishing a new version. If you must, set `allow_updates: true` on the + Module — see [the release lifecycle](../explanations/deprecation-lifecycle.md). +- **Retire a version.** Set `deprecated: true` to steer users away from it. Deleting it + outright has to wait until after it is deprecated (unless the Module has + `allow_updates: true`). See + [the release lifecycle](../explanations/deprecation-lifecycle.md). From d6042cf4af3a32af32829d509ee66f3633a99f65 Mon Sep 17 00:00:00 2001 From: Martin Gaughran Date: Thu, 30 Jul 2026 18:49:06 +0000 Subject: [PATCH 03/18] Clarify use of container with deploy-tools --- docs/how-to/run-container.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/how-to/run-container.md b/docs/how-to/run-container.md index bb1ec23..3bbf881 100644 --- a/docs/how-to/run-container.md +++ b/docs/how-to/run-container.md @@ -12,3 +12,6 @@ $ docker run ghcr.io/diamondlightsource/deploy-tools:latest --version ``` To get a released version, use a numbered release instead of `latest`. + +In practice this image is what a CI pipeline runs to drive `deploy-tools` against a shared +deployment area — see [drive deploy-tools from CI](ci-pipeline.md). From 95c62ee7d40ba679c7af54a33dc63af5578b7ea6 Mon Sep 17 00:00:00 2001 From: Martin Gaughran Date: Thu, 30 Jul 2026 19:32:09 +0000 Subject: [PATCH 04/18] Mention possibility of associating schemas to config globally --- docs/how-to/write-module-configuration.md | 3 ++- docs/schemas.md | 6 ++++++ 2 files changed, 8 insertions(+), 1 deletion(-) diff --git a/docs/how-to/write-module-configuration.md b/docs/how-to/write-module-configuration.md index 707ecf6..db04f5a 100644 --- a/docs/how-to/write-module-configuration.md +++ b/docs/how-to/write-module-configuration.md @@ -12,7 +12,8 @@ mechanics. For the shape of the files see `0.1` of `phoebus`. The `name` and `version` inside the file must match the path. 2. Add the schema line as the first line so your editor validates as you type (see the - [schema reference](../schemas.md) for details): + [schema reference](../schemas.md) for details, including how to associate schemas + repository-wide instead of per file): ```yaml # yaml-language-server: $schema=https://raw.githubusercontent.com/DiamondLightSource/deploy-tools/main/src/deploy_tools/models/schemas/release.json diff --git a/docs/schemas.md b/docs/schemas.md index 05893d6..78e89e6 100644 --- a/docs/schemas.md +++ b/docs/schemas.md @@ -28,6 +28,12 @@ This requires an editor with a YAML language server — e.g. VS Code with the or any [LSP](https://microsoft.github.io/language-server-protocol/)-capable editor running [`yaml-language-server`](https://github.com/redhat-developer/yaml-language-server). +Alternatively, VS Code's [`yaml.schemas`](https://github.com/redhat-developer/vscode-yaml#associating-schemas) +setting (committed to `.vscode/settings.json`) maps schemas to file paths once for the +whole repository, avoiding the per-file comment. This documentation uses the comment +because it is self-contained and editor-agnostic, but a real configuration repository may +reasonably prefer the repository-level setting. + ```{note} The bundled `demo_configuration` instead points at the locally generated schemas via an absolute workspace path (e.g. From 1e7809158092f7782862f4d9946eb39685c61ce2 Mon Sep 17 00:00:00 2001 From: Martin Gaughran Date: Thu, 30 Jul 2026 19:25:00 +0000 Subject: [PATCH 05/18] Use more specific links between docs pages --- docs/explanations/configuration-model.md | 4 ++-- docs/explanations/deployment-area.md | 3 ++- docs/explanations/deployment-steps.md | 2 +- docs/explanations/deprecation-lifecycle.md | 4 ++-- docs/how-to/ci-pipeline.md | 6 +++--- docs/how-to/regenerate-schemas.md | 2 +- docs/how-to/write-module-configuration.md | 4 ++-- 7 files changed, 13 insertions(+), 12 deletions(-) diff --git a/docs/explanations/configuration-model.md b/docs/explanations/configuration-model.md index 670d2e8..afeb792 100644 --- a/docs/explanations/configuration-model.md +++ b/docs/explanations/configuration-model.md @@ -39,8 +39,8 @@ A Module is the unit an end user loads. Alongside its `name` and `version` it ca | `env_vars` | Environment variables set when the Module is loaded. | | `dependencies` | Other Modules loaded first, optionally version-pinned. | | `applications` | One or more applications providing the executables (below). | -| `allow_updates` | Permit in-place changes to this version — see [the release lifecycle](deprecation-lifecycle.md). | -| `exclude_from_defaults` | Keep this version out of automatic default selection — see [default versions](default-versions.md). | +| `allow_updates` | Permit in-place changes to this version — see [the guard rails](deprecation-lifecycle.md#the-guard-rails). | +| `exclude_from_defaults` | Keep this version out of automatic default selection — see [default versions](default-versions.md#excluding-a-version-from-the-automatic-default). | `load_script` and `unload_script` run extra commands when the Module is loaded and unloaded respectively, injected raw into the generated Modulefile. They are an advanced diff --git a/docs/explanations/deployment-area.md b/docs/explanations/deployment-area.md index 0f11755..74be178 100644 --- a/docs/explanations/deployment-area.md +++ b/docs/explanations/deployment-area.md @@ -37,7 +37,8 @@ corresponding `modulefile`. Users put only the *modulefiles* directories on thei Deprecation is therefore cheap and reversible: the built files never move, only the symlink moves between `modulefiles/` and `deprecated/modulefiles/`. See -[the release lifecycle](deprecation-lifecycle.md) for the full set of transitions. +[the release lifecycle](deprecation-lifecycle.md#configuration-is-declarative) for +the full set of transitions. ## The build area diff --git a/docs/explanations/deployment-steps.md b/docs/explanations/deployment-steps.md index 349a5a4..dd77ab9 100644 --- a/docs/explanations/deployment-steps.md +++ b/docs/explanations/deployment-steps.md @@ -7,7 +7,7 @@ a command of their own. For the commands themselves see the [CLI reference](../c | Step | Description | Run by | |------|-------------|--------| | Compare | Compare the current deployment snapshot against the modulefiles and built modules that actually exist, confirming the [deployment area](deployment-area.md) is healthy. | `compare` | -| Validate | Diff the new configuration against the current snapshot to determine the set of actions to take, and check those actions are permitted by the [release lifecycle](deprecation-lifecycle.md). | `validate`, `sync` | +| Validate | Diff the new configuration against the current snapshot to determine the set of actions to take, and check those actions are permitted by the [lifecycle guard rails](deprecation-lifecycle.md#the-guard-rails). | `validate`, `sync` | | Build | Generate entrypoint scripts, configuration files and environment variables for each changed Module, writing them to the build area. | `sync` (`validate --test-build`) | | Deploy | Move the built Modules from the build area into the Modules Area, link each modulefile into the live or deprecated tree according to its status, and update default versions. | `sync` | diff --git a/docs/explanations/deprecation-lifecycle.md b/docs/explanations/deprecation-lifecycle.md index 39a6b95..b8a200e 100644 --- a/docs/explanations/deprecation-lifecycle.md +++ b/docs/explanations/deprecation-lifecycle.md @@ -9,8 +9,8 @@ deleted — is expressed by adding or removing a Release in configuration and to You never tell `deploy-tools` to "deprecate" or "remove" something directly. You describe the set of Releases you want, and the tool compares that against the -[snapshot](snapshots-and-compare.md) of the last `sync` to derive the actions needed. Each -Release falls into one of these cases: +[snapshot](snapshots-and-compare.md#what-the-snapshot-is) of the last `sync` to derive +the actions needed. Each Release falls into one of these cases: | Transition | Detected when… | Effect on the deployment area | |------------|----------------|-------------------------------| diff --git a/docs/how-to/ci-pipeline.md b/docs/how-to/ci-pipeline.md index 55893b9..e484c82 100644 --- a/docs/how-to/ci-pipeline.md +++ b/docs/how-to/ci-pipeline.md @@ -15,9 +15,9 @@ When someone opens a change (before it is merged), run, without altering the are - `deploy-tools compare ` — confirm the area still matches its last snapshot, so the change is being checked against a known-healthy baseline. - `deploy-tools validate ` — confirm the new configuration is valid and its - [lifecycle transitions](../explanations/deprecation-lifecycle.md) are permitted. Add - `--test-build` to build every changed Module in a temporary directory, catching build - failures before merge. + [lifecycle transitions](../explanations/deprecation-lifecycle.md#the-guard-rails) are + permitted. Add `--test-build` to build every changed Module in a temporary directory, + catching build failures before merge. This gives reviewers a green light that the change is deployable without changing anything on the filesystem. diff --git a/docs/how-to/regenerate-schemas.md b/docs/how-to/regenerate-schemas.md index f707561..9d67888 100644 --- a/docs/how-to/regenerate-schemas.md +++ b/docs/how-to/regenerate-schemas.md @@ -12,7 +12,7 @@ schemas and commit again. If you bypass the hooks (for example with `git commit --no-verify`), regenerate the schemas manually. The simplest way is the **Generate Schema** VSCode task (see -[the VSCode tasks guide](vscode-tasks.md)), which writes to the correct location. +[the VSCode tasks guide](vscode-tasks.md#running-a-task)), which writes to the correct location. Equivalently, run the CLI, pointing it at that folder: ```bash diff --git a/docs/how-to/write-module-configuration.md b/docs/how-to/write-module-configuration.md index db04f5a..77f75e0 100644 --- a/docs/how-to/write-module-configuration.md +++ b/docs/how-to/write-module-configuration.md @@ -71,8 +71,8 @@ catches structural mistakes in your editor as you type, before you open the requ - **Update an existing version in place.** Rejected by default so published versions stay stable; prefer publishing a new version. If you must, set `allow_updates: true` on the - Module — see [the release lifecycle](../explanations/deprecation-lifecycle.md). + Module — see [the guard rails](../explanations/deprecation-lifecycle.md#the-guard-rails). - **Retire a version.** Set `deprecated: true` to steer users away from it. Deleting it outright has to wait until after it is deprecated (unless the Module has `allow_updates: true`). See - [the release lifecycle](../explanations/deprecation-lifecycle.md). + [the guard rails](../explanations/deprecation-lifecycle.md#the-guard-rails). From ae7ce946affe42bc6846ff4e0dd2cbdc265de4c8 Mon Sep 17 00:00:00 2001 From: Martin Gaughran Date: Mon, 24 Aug 2026 13:41:46 +0000 Subject: [PATCH 06/18] Improve organisation of configuration-model diagram Fixed indentation and clearly labelled meaning of application types. --- docs/explanations/configuration-model.md | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-) diff --git a/docs/explanations/configuration-model.md b/docs/explanations/configuration-model.md index afeb792..359ffa8 100644 --- a/docs/explanations/configuration-model.md +++ b/docs/explanations/configuration-model.md @@ -20,15 +20,20 @@ settings.yaml /.yaml one file per Module version ├── deprecated: false lifecycle flag └── module - ├── name / version / description + ├── name Module name + ├── version Module version + ├── description shown by `module whatis` ├── env_vars variables set on load ├── dependencies other Modules to load first - └── applications one or more, each of: - ├── apptainer container image + entrypoints - ├── shell a bash script - └── binary a downloaded, hash-checked executable + └── applications one or more, each with an app_type of: + ├── apptainer container image + entrypoints + ├── shell a bash script + └── binary a downloaded, hash-checked executable ``` +Those last three are the values `app_type` can take rather than fields of their own — +see [the three application types](#the-three-application-types) below. + ## A Module A Module is the unit an end user loads. Alongside its `name` and `version` it carries: From 1395dfa47d934d760a273ba1f0b932a2141f8e53 Mon Sep 17 00:00:00 2001 From: Martin Gaughran Date: Mon, 24 Aug 2026 13:45:34 +0000 Subject: [PATCH 07/18] Include all fields in configuration-model table and improve wording --- docs/explanations/configuration-model.md | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/docs/explanations/configuration-model.md b/docs/explanations/configuration-model.md index 359ffa8..42a1a24 100644 --- a/docs/explanations/configuration-model.md +++ b/docs/explanations/configuration-model.md @@ -36,20 +36,24 @@ see [the three application types](#the-three-application-types) below. ## A Module -A Module is the unit an end user loads. Alongside its `name` and `version` it carries: +A Module is the unit an end user loads. It carries: | Field | Purpose | |-------|---------| +| `name` | The name an end user loads it by. | +| `version` | The version an end user loads it by. | | `description` | Shown by `module whatis `. | | `env_vars` | Environment variables set when the Module is loaded. | | `dependencies` | Other Modules loaded first, optionally version-pinned. | | `applications` | One or more applications providing the executables (below). | +| `load_script` | Extra commands run when the Module is loaded — see below. | +| `unload_script` | Extra commands run when the Module is unloaded — see below. | | `allow_updates` | Permit in-place changes to this version — see [the guard rails](deprecation-lifecycle.md#the-guard-rails). | | `exclude_from_defaults` | Keep this version out of automatic default selection — see [default versions](default-versions.md#excluding-a-version-from-the-automatic-default). | -`load_script` and `unload_script` run extra commands when the Module is loaded and -unloaded respectively, injected raw into the generated Modulefile. They are an advanced -escape hatch — check with a `deploy-tools` admin before using them. +`load_script` and `unload_script` are injected raw into the generated Modulefile. They +are for advanced cases the other fields cannot cover — check with a `deploy-tools` admin +before using them. Each per-version file also carries a `deprecated` flag; see [the release lifecycle](deprecation-lifecycle.md). From a959723be310700759679cc646198ed9c46de745 Mon Sep 17 00:00:00 2001 From: Martin Gaughran Date: Mon, 24 Aug 2026 14:01:39 +0000 Subject: [PATCH 08/18] Break out fields in configuration-model.md to separate tables For consistency with rest of documentation. --- docs/explanations/configuration-model.md | 86 ++++++++++++++++++++---- 1 file changed, 73 insertions(+), 13 deletions(-) diff --git a/docs/explanations/configuration-model.md b/docs/explanations/configuration-model.md index 42a1a24..561986c 100644 --- a/docs/explanations/configuration-model.md +++ b/docs/explanations/configuration-model.md @@ -51,6 +51,20 @@ A Module is the unit an end user loads. It carries: | `allow_updates` | Permit in-place changes to this version — see [the guard rails](deprecation-lifecycle.md#the-guard-rails). | | `exclude_from_defaults` | Keep this version out of automatic default selection — see [default versions](default-versions.md#excluding-a-version-from-the-automatic-default). | +Each entry under `env_vars` is a name/value pair: + +| Field | Purpose | +|-------|---------| +| `name` | The variable to set. | +| `value` | The value to set it to. | + +Each entry under `dependencies` names another Module: + +| Field | Purpose | +|-------|---------| +| `name` | The Module to load first. | +| `version` | The version to pin to. If omitted, that Module's default version is resolved at load time. | + `load_script` and `unload_script` are injected raw into the generated Modulefile. They are for advanced cases the other fields cannot cover — check with a `deploy-tools` admin before using them. @@ -63,11 +77,11 @@ Each per-version file also carries a `deprecated` flag; see Every entry under `applications` sets `app_type` to select one of three kinds; a single Module can mix them. -| `app_type` | Provides | Key fields | -|------------|----------|------------| -| `apptainer` | Commands that run inside a container image | `container`, `entrypoints`, `global_options` | -| `shell` | A single executable running a bash script | `name`, `script` | -| `binary` | A downloaded executable added to the path | `name`, `url`, `hash`, `hash_type` | +| `app_type` | Provides | +|------------|----------| +| `apptainer` | Commands that run inside a container image | +| `shell` | A single executable running a bash script | +| `binary` | A downloaded executable added to the path | The demo `example-module-apps` Module combines an Apptainer app with a Shell app: @@ -76,16 +90,62 @@ The demo `example-module-apps` Module combines an Apptainer app with a Shell app :lines: 3- ``` -**Apptainer** uses one container image with one or more `entrypoints`, each mapping an -executable name to a command run inside the container. Options — container `mounts`, -`command_args`, `apptainer_args`, `host_binaries` — can be set per entrypoint or shared -across all of them via `global_options`. +### Apptainer + +One container image with one or more entrypoints, each mapping an executable name to a +command run inside the container. + +| Field | Purpose | +|-------|---------| +| `container` | The image to use (below). | +| `entrypoints` | The executables provided (below). | +| `global_options` | Options applied to every entrypoint. | + +`container` splits the image reference into `path:version`: + +| Field | Purpose | +|-------|---------| +| `path` | The image URL, excluding the version or tag. `docker`, `shub`, `oras` and `https` schemes are accepted. | +| `version` | The image version or tag. | + +Each entry under `entrypoints` is one executable: -**Shell** exposes a single executable (`name`) that runs the `script` list as bash. +| Field | Purpose | +|-------|---------| +| `name` | The executable provided. | +| `command` | The command to run inside the container. Defaults to `name`. | +| `options` | Options applied to this entrypoint only. | + +Both `options` and `global_options` take the same fields: + +| Field | Purpose | +|-------|---------| +| `apptainer_args` | Arguments passed to Apptainer when launching the container. | +| `command_args` | Arguments passed to the command being run. | +| `mounts` | Mount points as `host_path[:container_path[:opts]]`, where `opts` is `ro` or `rw` (default `rw`). | +| `host_binaries` | Host binaries, found on the current `PATH`, to mount into the container at `/usr/bin/`. | + +### Shell + +A single executable running a bash script. + +| Field | Purpose | +|-------|---------| +| `name` | The executable provided. | +| `script` | The lines of bash it runs. | + +### Binary + +An executable downloaded, hash-checked and added to the path. + +| Field | Purpose | +|-------|---------| +| `name` | The executable provided. | +| `url` | Where the binary is downloaded from. | +| `hash` | The expected hash of the download. | +| `hash_type` | `sha256`, `sha512`, `md5`, or `none` to skip the check. | -**Binary** downloads `url`, checks it against `hash` using `hash_type` (`sha256`, -`sha512`, `md5`, or `none` to skip the check), and adds the executable to the path as -`name`. The demo `argocd` Module uses one: +The demo `argocd` Module uses one: ```{literalinclude} ../../src/deploy_tools/demo_configuration/argocd/v2.14.10.yaml :language: yaml From b645411444b1a8a354a7c006f6bd0f1501172a98 Mon Sep 17 00:00:00 2001 From: Martin Gaughran Date: Mon, 24 Aug 2026 14:21:14 +0000 Subject: [PATCH 09/18] Move configuration-model.md into the reference section Since this now contains an exhaustive overview of the fields required. --- docs/explanations.md | 1 - docs/glossary.md | 2 +- docs/how-to/write-module-configuration.md | 12 ++++++------ docs/reference.md | 1 + .../configuration-model.md | 18 ++++++++++-------- docs/schemas.md | 9 +++++++-- 6 files changed, 25 insertions(+), 18 deletions(-) rename docs/{explanations => reference}/configuration-model.md (88%) diff --git a/docs/explanations.md b/docs/explanations.md index 47f6d3b..c2b70f1 100644 --- a/docs/explanations.md +++ b/docs/explanations.md @@ -5,7 +5,6 @@ Explanations of how it works and why it works that way. ```{toctree} :maxdepth: 1 -explanations/configuration-model explanations/deployment-steps explanations/deployment-area explanations/snapshots-and-compare diff --git a/docs/glossary.md b/docs/glossary.md index d12cd21..a9a3360 100644 --- a/docs/glossary.md +++ b/docs/glossary.md @@ -9,7 +9,7 @@ for the on-disk layout these terms refer to. | Environment Modules | A [standard package for Linux](https://modules.readthedocs.io/en/latest/) that provides the commands for loading and unloading 'Environment Modules' (each defined by a Modulefile). Note that while we are using this system, our definition of Module is separate. If we are referring to an Environment Module, we will use the full name. | | Modulefile | Used by the Environment Modules package to specify all details of an Environment Module. This can include executables to add to the path, environment variables to set, etc. | | Module | A set of files that can be used to provide applications on your path, provide configuration, and set environment variables. We do this using the Environment Modules system by providing a Modulefile with the relevant configuration. | -| Application | Each Module can be configured with multiple Applications, each one providing one or more executables. There are 3 types of Application: `Apptainer` (an executable container image), `Shell` (a Bash script) and `Binary` (an executable downloaded from a URL and verified against a hash). See [the configuration model](explanations/configuration-model.md). | +| Application | Each Module can be configured with multiple Applications, each one providing one or more executables. There are 3 types of Application: `Apptainer` (an executable container image), `Shell` (a Bash script) and `Binary` (an executable downloaded from a URL and verified against a hash). See [the configuration model](reference/configuration-model.md). | | Release (noun) | A Module, including version, alongside its [lifecycle](explanations/deprecation-lifecycle.md) (i.e. deprecation) status. | | Deployment | The declared configuration for a Deployment Area: all Releases (deprecated or not) to be maintained there, plus global settings. Written to the `deployment.yaml` snapshot by `sync`. The act of deploying is written lowercase. | | Deployment Step | Refers to one of the primary steps that make up the deployment process. See [the deployment process](explanations/deployment-steps.md) for a breakdown. | diff --git a/docs/how-to/write-module-configuration.md b/docs/how-to/write-module-configuration.md index 77f75e0..8ff6dbe 100644 --- a/docs/how-to/write-module-configuration.md +++ b/docs/how-to/write-module-configuration.md @@ -1,9 +1,9 @@ # Write a Module configuration -To add or change a Module you edit YAML in the configuration folder. This guide covers the -mechanics. For the shape of the files see -[the configuration model](../explanations/configuration-model.md); for every field, the -[schema reference](../schemas.md). +To add or change a Module, you edit YAML in the configuration folder. This guide covers +the mechanics. For the shape of the files and every field they take, see +[the configuration model](../reference/configuration-model.md); to point your editor at +the matching schema, the [schema reference](../schemas.md). ## Add a new Module version @@ -35,11 +35,11 @@ mechanics. For the shape of the files see ``` Swap the application for an `apptainer` or `binary` one as needed — see - [the three application types](../explanations/configuration-model.md#the-three-application-types). + [the three application types](../reference/configuration-model.md#the-three-application-types). A Module doesn't have to provide an application: it can instead just set environment variables or pull in other Modules as - [dependencies](../explanations/configuration-model.md#a-module). Give such a Module an + [dependencies](../reference/configuration-model.md#a-module). Give such a Module an empty `applications: []`. ## Set the default version diff --git a/docs/reference.md b/docs/reference.md index da6115c..f54cbef 100644 --- a/docs/reference.md +++ b/docs/reference.md @@ -6,6 +6,7 @@ Technical reference material including APIs and release notes. :maxdepth: 1 :glob: +Configuration CLI API <_api/deploy_tools> Schemas diff --git a/docs/explanations/configuration-model.md b/docs/reference/configuration-model.md similarity index 88% rename from docs/explanations/configuration-model.md rename to docs/reference/configuration-model.md index 561986c..c6f02c8 100644 --- a/docs/explanations/configuration-model.md +++ b/docs/reference/configuration-model.md @@ -1,9 +1,11 @@ # The configuration model -This page is for anyone writing or editing deployment configuration. It explains how the -files you author are structured — the objects they map to and how they nest. For every -field in exhaustive detail see the [schema reference](../schemas.md); for the meaning of -individual terms, the [glossary](../glossary.md). +This page is for anyone writing or editing deployment configuration. It documents every +field of the files you author and how they nest. The field tables are curated by hand +from the Pydantic models in `src/deploy_tools/models`, so a change to those models needs +a change here too; the [schema reference](../schemas.md) renders the same fields +mechanically from the generated JSON schemas. For the meaning of individual terms, see +the [glossary](../glossary.md). ## What you author @@ -48,8 +50,8 @@ A Module is the unit an end user loads. It carries: | `applications` | One or more applications providing the executables (below). | | `load_script` | Extra commands run when the Module is loaded — see below. | | `unload_script` | Extra commands run when the Module is unloaded — see below. | -| `allow_updates` | Permit in-place changes to this version — see [the guard rails](deprecation-lifecycle.md#the-guard-rails). | -| `exclude_from_defaults` | Keep this version out of automatic default selection — see [default versions](default-versions.md#excluding-a-version-from-the-automatic-default). | +| `allow_updates` | Permit in-place changes to this version — see [the guard rails](../explanations/deprecation-lifecycle.md#the-guard-rails). | +| `exclude_from_defaults` | Keep this version out of automatic default selection — see [default versions](../explanations/default-versions.md#excluding-a-version-from-the-automatic-default). | Each entry under `env_vars` is a name/value pair: @@ -70,7 +72,7 @@ are for advanced cases the other fields cannot cover — check with a `deploy-to before using them. Each per-version file also carries a `deprecated` flag; see -[the release lifecycle](deprecation-lifecycle.md). +[the release lifecycle](../explanations/deprecation-lifecycle.md). ## The three application types @@ -164,4 +166,4 @@ when no version is given: ``` How that choice is resolved — and how to keep a version out of automatic selection — is -covered in [default version resolution](default-versions.md). +covered in [default version resolution](../explanations/default-versions.md). diff --git a/docs/schemas.md b/docs/schemas.md index 78e89e6..ae0d7b7 100644 --- a/docs/schemas.md +++ b/docs/schemas.md @@ -2,8 +2,13 @@ These JSON schema files are generated from the Pydantic models in `src/deploy_tools/models`. The YAML language server and other tooling use them to validate -deployment configuration files. This section links to per-schema reference pages that -render each schema and list its properties. +deployment configuration files. This page covers which file each schema validates and how +to point your editor at it. + +To look a field up, use [the configuration model](reference/configuration-model.md) — it +documents the same fields in a form meant for reading. The +[schema pages](#schema-pages) below render each schema mechanically, so they are the +exhaustive detail rather than the place to start. ## Which file uses which schema From 07960571eb4d5a66d3cb6e01a74f6b0c30e7328b Mon Sep 17 00:00:00 2001 From: Martin Gaughran Date: Mon, 24 Aug 2026 14:40:35 +0000 Subject: [PATCH 10/18] Rename configuration-model.md to configuration.md --- docs/glossary.md | 2 +- docs/how-to/write-module-configuration.md | 6 +++--- docs/reference.md | 2 +- docs/reference/{configuration-model.md => configuration.md} | 2 +- docs/schemas.md | 2 +- 5 files changed, 7 insertions(+), 7 deletions(-) rename docs/reference/{configuration-model.md => configuration.md} (99%) diff --git a/docs/glossary.md b/docs/glossary.md index a9a3360..07b0e20 100644 --- a/docs/glossary.md +++ b/docs/glossary.md @@ -9,7 +9,7 @@ for the on-disk layout these terms refer to. | Environment Modules | A [standard package for Linux](https://modules.readthedocs.io/en/latest/) that provides the commands for loading and unloading 'Environment Modules' (each defined by a Modulefile). Note that while we are using this system, our definition of Module is separate. If we are referring to an Environment Module, we will use the full name. | | Modulefile | Used by the Environment Modules package to specify all details of an Environment Module. This can include executables to add to the path, environment variables to set, etc. | | Module | A set of files that can be used to provide applications on your path, provide configuration, and set environment variables. We do this using the Environment Modules system by providing a Modulefile with the relevant configuration. | -| Application | Each Module can be configured with multiple Applications, each one providing one or more executables. There are 3 types of Application: `Apptainer` (an executable container image), `Shell` (a Bash script) and `Binary` (an executable downloaded from a URL and verified against a hash). See [the configuration model](reference/configuration-model.md). | +| Application | Each Module can be configured with multiple Applications, each one providing one or more executables. There are 3 types of Application: `Apptainer` (an executable container image), `Shell` (a Bash script) and `Binary` (an executable downloaded from a URL and verified against a hash). See [the configuration reference](reference/configuration.md). | | Release (noun) | A Module, including version, alongside its [lifecycle](explanations/deprecation-lifecycle.md) (i.e. deprecation) status. | | Deployment | The declared configuration for a Deployment Area: all Releases (deprecated or not) to be maintained there, plus global settings. Written to the `deployment.yaml` snapshot by `sync`. The act of deploying is written lowercase. | | Deployment Step | Refers to one of the primary steps that make up the deployment process. See [the deployment process](explanations/deployment-steps.md) for a breakdown. | diff --git a/docs/how-to/write-module-configuration.md b/docs/how-to/write-module-configuration.md index 8ff6dbe..0eb96d7 100644 --- a/docs/how-to/write-module-configuration.md +++ b/docs/how-to/write-module-configuration.md @@ -2,7 +2,7 @@ To add or change a Module, you edit YAML in the configuration folder. This guide covers the mechanics. For the shape of the files and every field they take, see -[the configuration model](../reference/configuration-model.md); to point your editor at +[the configuration reference](../reference/configuration.md); to point your editor at the matching schema, the [schema reference](../schemas.md). ## Add a new Module version @@ -35,11 +35,11 @@ the matching schema, the [schema reference](../schemas.md). ``` Swap the application for an `apptainer` or `binary` one as needed — see - [the three application types](../reference/configuration-model.md#the-three-application-types). + [the three application types](../reference/configuration.md#the-three-application-types). A Module doesn't have to provide an application: it can instead just set environment variables or pull in other Modules as - [dependencies](../reference/configuration-model.md#a-module). Give such a Module an + [dependencies](../reference/configuration.md#a-module). Give such a Module an empty `applications: []`. ## Set the default version diff --git a/docs/reference.md b/docs/reference.md index f54cbef..b26e7b1 100644 --- a/docs/reference.md +++ b/docs/reference.md @@ -6,7 +6,7 @@ Technical reference material including APIs and release notes. :maxdepth: 1 :glob: -Configuration +reference/configuration CLI API <_api/deploy_tools> Schemas diff --git a/docs/reference/configuration-model.md b/docs/reference/configuration.md similarity index 99% rename from docs/reference/configuration-model.md rename to docs/reference/configuration.md index c6f02c8..150da7e 100644 --- a/docs/reference/configuration-model.md +++ b/docs/reference/configuration.md @@ -1,4 +1,4 @@ -# The configuration model +# Configuration This page is for anyone writing or editing deployment configuration. It documents every field of the files you author and how they nest. The field tables are curated by hand diff --git a/docs/schemas.md b/docs/schemas.md index ae0d7b7..7252efc 100644 --- a/docs/schemas.md +++ b/docs/schemas.md @@ -5,7 +5,7 @@ These JSON schema files are generated from the Pydantic models in deployment configuration files. This page covers which file each schema validates and how to point your editor at it. -To look a field up, use [the configuration model](reference/configuration-model.md) — it +To look a field up, use [the configuration reference](reference/configuration.md) — it documents the same fields in a form meant for reading. The [schema pages](#schema-pages) below render each schema mechanically, so they are the exhaustive detail rather than the place to start. From aab06d368775903b8368afe4f46e629ef4eb77bb Mon Sep 17 00:00:00 2001 From: Martin Gaughran Date: Mon, 24 Aug 2026 15:01:52 +0000 Subject: [PATCH 11/18] Add bolded highlights to improve distinction between tables in docs --- docs/reference/configuration.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/reference/configuration.md b/docs/reference/configuration.md index 150da7e..b945a78 100644 --- a/docs/reference/configuration.md +++ b/docs/reference/configuration.md @@ -53,14 +53,14 @@ A Module is the unit an end user loads. It carries: | `allow_updates` | Permit in-place changes to this version — see [the guard rails](../explanations/deprecation-lifecycle.md#the-guard-rails). | | `exclude_from_defaults` | Keep this version out of automatic default selection — see [default versions](../explanations/default-versions.md#excluding-a-version-from-the-automatic-default). | -Each entry under `env_vars` is a name/value pair: +**`env_vars`** — each entry is a name/value pair: | Field | Purpose | |-------|---------| | `name` | The variable to set. | | `value` | The value to set it to. | -Each entry under `dependencies` names another Module: +**`dependencies`** — each entry names another Module: | Field | Purpose | |-------|---------| @@ -103,14 +103,14 @@ command run inside the container. | `entrypoints` | The executables provided (below). | | `global_options` | Options applied to every entrypoint. | -`container` splits the image reference into `path:version`: +**`container`** — splits the image reference into `path:version`: | Field | Purpose | |-------|---------| | `path` | The image URL, excluding the version or tag. `docker`, `shub`, `oras` and `https` schemes are accepted. | | `version` | The image version or tag. | -Each entry under `entrypoints` is one executable: +**`entrypoints`** — each entry is one executable: | Field | Purpose | |-------|---------| @@ -118,7 +118,7 @@ Each entry under `entrypoints` is one executable: | `command` | The command to run inside the container. Defaults to `name`. | | `options` | Options applied to this entrypoint only. | -Both `options` and `global_options` take the same fields: +**`options` and `global_options`** — both take the same fields: | Field | Purpose | |-------|---------| From 5eeac17c08bdca1775012479845a088763e2e33d Mon Sep 17 00:00:00 2001 From: Martin Gaughran Date: Mon, 24 Aug 2026 15:05:57 +0000 Subject: [PATCH 12/18] Indicate requirements for yaml-language-server in how-to doc --- docs/how-to/write-module-configuration.md | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/docs/how-to/write-module-configuration.md b/docs/how-to/write-module-configuration.md index 0eb96d7..47ecb49 100644 --- a/docs/how-to/write-module-configuration.md +++ b/docs/how-to/write-module-configuration.md @@ -11,14 +11,17 @@ the matching schema, the [schema reference](../schemas.md). Module `name` and the filename is the `version`, so `phoebus/0.1.yaml` defines version `0.1` of `phoebus`. The `name` and `version` inside the file must match the path. -2. Add the schema line as the first line so your editor validates as you type (see the - [schema reference](../schemas.md) for details, including how to associate schemas - repository-wide instead of per file): +2. Add the schema line as the first line so your editor validates as you type: ```yaml # yaml-language-server: $schema=https://raw.githubusercontent.com/DiamondLightSource/deploy-tools/main/src/deploy_tools/models/schemas/release.json ``` + The line is read by [`yaml-language-server`](https://github.com/redhat-developer/yaml-language-server), + so it takes effect in VS Code with the Red Hat YAML extension or any other editor + running that language server; elsewhere it is an inert comment. See the + [schema reference](../schemas.md) for details. + 3. Define the Module. Most Modules provide one or more applications; the smallest useful one is a single shell script: From 22b27bdba536e86b5fd5e4f19ee57c16affaf7eb Mon Sep 17 00:00:00 2001 From: Martin Gaughran Date: Mon, 24 Aug 2026 15:11:39 +0000 Subject: [PATCH 13/18] Add small note on schema linking for settings.yaml --- docs/how-to/write-module-configuration.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/how-to/write-module-configuration.md b/docs/how-to/write-module-configuration.md index 47ecb49..ca9fab7 100644 --- a/docs/how-to/write-module-configuration.md +++ b/docs/how-to/write-module-configuration.md @@ -56,6 +56,9 @@ default_versions: my-module: "1.0" ``` +`settings.yaml` can take a schema line of its own, as +[above](#add-a-new-module-version), pointing at `deployment-settings.json`. + To keep a version out of automatic selection — an alpha or release candidate, say — while still allowing an explicit `module load /`, set `exclude_from_defaults: true` on that Module. From f74aa393784bfc0b894b8230aa32c9ad200eb33a Mon Sep 17 00:00:00 2001 From: Martin Gaughran Date: Mon, 24 Aug 2026 15:26:36 +0000 Subject: [PATCH 14/18] Clarify that using CI pipeline for deploy-tools is only a recommendation --- README.md | 8 ++++---- docs/explanations/deployment-steps.md | 2 +- docs/explanations/snapshots-and-compare.md | 4 ++-- docs/how-to/ci-pipeline.md | 21 ++++++++++++--------- docs/how-to/run-container.md | 4 ++-- docs/how-to/write-module-configuration.md | 11 +++++++---- docs/tutorials/your-first-deployment.md | 11 ++++++----- 7 files changed, 34 insertions(+), 27 deletions(-) diff --git a/README.md b/README.md index 472a7bb..87c7ad9 100644 --- a/README.md +++ b/README.md @@ -24,9 +24,9 @@ The demo_configuration folder in this repository can be passed as the config_fol the deploy-tools commands. The deployment_root needs to be a writeable location for all files to get deployed under. -In normal use these commands are not run by hand: they act on a shared deployment area and -belong in a CI pipeline, gated by change review. Running them manually against the demo -configuration, as below, is just the quickest way to see what each does — the +These commands act on a shared deployment area, so we recommend running them from a CI +pipeline gated by change review rather than by hand a requirement. Running them manually +against the demo configuration, as below, is the quickest way to see what each does — the [documentation](https://diamondlightsource.github.io/deploy-tools) has a hands-on tutorial and a guide to driving them from CI. @@ -46,7 +46,7 @@ deploy-tools validate $deployment_root $config_folder deploy-tools sync $deployment_root $config_folder # Compare the current deployment snapshot against what is actually deployed in the -# deployment area. CI/CD should run this before a deploy to confirm a healthy state. +# deployment area. Run this before a deploy to confirm a healthy state. deploy-tools compare $deployment_root ``` diff --git a/docs/explanations/deployment-steps.md b/docs/explanations/deployment-steps.md index dd77ab9..07d9354 100644 --- a/docs/explanations/deployment-steps.md +++ b/docs/explanations/deployment-steps.md @@ -23,4 +23,4 @@ place, or nothing changes. read-only checks run before it. See [snapshots and the compare safety net](snapshots-and-compare.md) for how the snapshot ties them together, and [drive deploy-tools from CI](../how-to/ci-pipeline.md) for the -order a pipeline runs them in. +order to run them in. diff --git a/docs/explanations/snapshots-and-compare.md b/docs/explanations/snapshots-and-compare.md index 7c864e3..ddfbf5b 100644 --- a/docs/explanations/snapshots-and-compare.md +++ b/docs/explanations/snapshots-and-compare.md @@ -37,7 +37,7 @@ will not be detected. The risk of corruption is avoided at build time instead: a is built on the same filesystem as the deployment area and published by a single atomic rename, so a partial or failed build (including `.sif` files) is never moved into place. -This is why CI should run `compare` *before* every `sync`: it confirms the area is in the +This is why `compare` should be run *before* every `sync`: it confirms the area is in the healthy state the last `sync` claimed to leave it in. ## Recovery is manual @@ -55,7 +55,7 @@ Two facilities help here: often easier to rollback the configuration to a previous state rather than fix the latest deployment. - `compare --from-scratch` asserts only that the deployment root exists and is *empty* — - this is the check to run in CI before the very first `sync`, when no snapshot exists yet. + this is the check to run before the very first `sync`, when no snapshot exists yet. The git repository in the deployment area exists only to give `compare --use-ref` this reference point. It deliberately excludes the build area and Apptainer images, and is diff --git a/docs/how-to/ci-pipeline.md b/docs/how-to/ci-pipeline.md index e484c82..c7fd1e5 100644 --- a/docs/how-to/ci-pipeline.md +++ b/docs/how-to/ci-pipeline.md @@ -1,12 +1,15 @@ # Drive deploy-tools from CI -In normal use, nobody should run `sync`, `validate` or `compare` by hand. Those commands -touch a shared deployment area, so they belong to a CI pipeline in the configuration -repository, gated by change review. End users only edit configuration and open a change; -merging it deploys. +`sync`, `validate` and `compare` touch a shared deployment area, so we recommend driving +them from a CI pipeline in the configuration repository, gated by change review: end users +only edit configuration and open a change, and merging it deploys. `deploy-tools` does not +require this — an administrator can run the commands by hand — but a pipeline gives you +review, repeatability and one place to serialise runs. -This guide describes what that pipeline must do. It is deliberately independent of any -particular CI system — translate the responsibilities below into your own. +This guide describes the responsibilities such a pipeline has. It is deliberately +independent of any particular CI system — translate them into your own. If you run the +commands by hand, the same ordering and the [one at a time](#run-one-at-a-time) rule still +apply. ## On a proposed change @@ -32,15 +35,15 @@ When the change is merged to the main branch, deploy it: ## Run one at a time -`deploy-tools` has no locking of its own. The pipeline must ensure only one run +`deploy-tools` has no locking of its own. Whatever drives it must ensure only one run touches the area at a time — two concurrent `sync`s, or a `sync` racing a `compare`, can corrupt the area or report false drift. Serialise the relevant jobs (and, if possible, restrict them to a single runner). ## Manual operations -Some tasks fall outside the automatic flows and are best run from a manually-triggered -pipeline, exposing the relevant options as parameters: +Some tasks fall outside the automatic flows and are run by hand, or from a +manually-triggered pipeline exposing the relevant options as parameters: - **The first deployment.** A brand-new area has no snapshot to compare against, so it is run manually rather than triggered by a merge. Use `--from-scratch`, which assumes the diff --git a/docs/how-to/run-container.md b/docs/how-to/run-container.md index 3bbf881..a1b8a7b 100644 --- a/docs/how-to/run-container.md +++ b/docs/how-to/run-container.md @@ -13,5 +13,5 @@ $ docker run ghcr.io/diamondlightsource/deploy-tools:latest --version To get a released version, use a numbered release instead of `latest`. -In practice this image is what a CI pipeline runs to drive `deploy-tools` against a shared -deployment area — see [drive deploy-tools from CI](ci-pipeline.md). +This image is a convenient way to drive `deploy-tools` against a shared deployment area, +typically from a CI pipeline — see [drive deploy-tools from CI](ci-pipeline.md). diff --git a/docs/how-to/write-module-configuration.md b/docs/how-to/write-module-configuration.md index ca9fab7..8c33de4 100644 --- a/docs/how-to/write-module-configuration.md +++ b/docs/how-to/write-module-configuration.md @@ -68,10 +68,13 @@ automatic choice is made. ## Get your change deployed -You don't run `deploy-tools` yourself. Open a merge request in the configuration -repository; CI validates the change and, once it is merged, deploys it (see -[drive deploy-tools from CI](ci-pipeline.md)). The `yaml-language-server` schema line -catches structural mistakes in your editor as you type, before you open the request. +How your change reaches the deployment area depends on how your site runs `deploy-tools`. +The recommended setup is a CI pipeline in the configuration repository: you open a merge +request, CI validates the change, and merging it deploys (see +[drive deploy-tools from CI](ci-pipeline.md)). CI is not a requirement — an administrator +can run the same `validate` and `sync` commands by hand instead. Either way, the +`yaml-language-server` schema line catches structural mistakes in your editor as you type, +before anyone else looks at the change. ## Change or retire a version diff --git a/docs/tutorials/your-first-deployment.md b/docs/tutorials/your-first-deployment.md index 18c360e..1299366 100644 --- a/docs/tutorials/your-first-deployment.md +++ b/docs/tutorials/your-first-deployment.md @@ -4,9 +4,10 @@ This tutorial takes you from a set of configuration files to a Module you can lo using the demo configuration that ships in the repository. It is a hands-on learning exercise: you run each command yourself, in a throwaway directory. -Real deployments don't work this way — a CI pipeline runs these commands against a shared -area (see [drive deploy-tools from CI](../how-to/ci-pipeline.md)) — but running them by -hand once is the quickest way to see what each does. +A real deployment targets a shared area, where we recommend driving these commands from +a CI pipeline instead (see +[drive deploy-tools from CI](../how-to/ci-pipeline.md)) — but running them by hand once +is the quickest way to see what each does. ## Before you start @@ -145,8 +146,8 @@ dls-pmac-control` loads `0.1` unless you ask for `dls-pmac-control/0.2` explicit You took a configuration folder, checked the area was clean, previewed the changes, and deployed Modules you could load and run. Those are the same `compare`, `validate` and -`sync` commands a real pipeline runs, though it spreads them across separate stages (see -[drive deploy-tools from CI](../how-to/ci-pipeline.md)). +`sync` commands a real deployment runs; a CI pipeline spreads them across separate stages +(see [drive deploy-tools from CI](../how-to/ci-pipeline.md)). From here: From 3acc69b501272cc4282af9742767271b3a3742dd Mon Sep 17 00:00:00 2001 From: Martin Gaughran Date: Wed, 26 Aug 2026 16:17:05 +0000 Subject: [PATCH 15/18] Rename "per-version file" to "Release file" --- docs/glossary.md | 1 + docs/how-to/write-module-configuration.md | 13 +++++++------ docs/reference/configuration.md | 10 +++++----- docs/schemas.md | 2 +- docs/tutorials/your-first-deployment.md | 6 +++--- 5 files changed, 17 insertions(+), 15 deletions(-) diff --git a/docs/glossary.md b/docs/glossary.md index 07b0e20..261a6a3 100644 --- a/docs/glossary.md +++ b/docs/glossary.md @@ -11,6 +11,7 @@ for the on-disk layout these terms refer to. | Module | A set of files that can be used to provide applications on your path, provide configuration, and set environment variables. We do this using the Environment Modules system by providing a Modulefile with the relevant configuration. | | Application | Each Module can be configured with multiple Applications, each one providing one or more executables. There are 3 types of Application: `Apptainer` (an executable container image), `Shell` (a Bash script) and `Binary` (an executable downloaded from a URL and verified against a hash). See [the configuration reference](reference/configuration.md). | | Release (noun) | A Module, including version, alongside its [lifecycle](explanations/deprecation-lifecycle.md) (i.e. deprecation) status. | +| Release file | The `/.yaml` file you author to define one Release. See [the configuration reference](reference/configuration.md). | | Deployment | The declared configuration for a Deployment Area: all Releases (deprecated or not) to be maintained there, plus global settings. Written to the `deployment.yaml` snapshot by `sync`. The act of deploying is written lowercase. | | Deployment Step | Refers to one of the primary steps that make up the deployment process. See [the deployment process](explanations/deployment-steps.md) for a breakdown. | | End User | Refers to anybody who is intending to make use of a deployed Module. This can include the people modifying configuration themselves. | diff --git a/docs/how-to/write-module-configuration.md b/docs/how-to/write-module-configuration.md index 8c33de4..5063119 100644 --- a/docs/how-to/write-module-configuration.md +++ b/docs/how-to/write-module-configuration.md @@ -7,9 +7,10 @@ the matching schema, the [schema reference](../schemas.md). ## Add a new Module version -1. Create the file at `//.yaml`. The folder name is the - Module `name` and the filename is the `version`, so `phoebus/0.1.yaml` defines version - `0.1` of `phoebus`. The `name` and `version` inside the file must match the path. +1. Create the Release file at `//.yaml`. The folder name + is the Module `name` and the filename is the `version`, so `phoebus/0.1.yaml` defines + version `0.1` of `phoebus`. The `name` and `version` inside the file must match the + path. 2. Add the schema line as the first line so your editor validates as you type: @@ -81,7 +82,7 @@ before anyone else looks at the change. - **Update an existing version in place.** Rejected by default so published versions stay stable; prefer publishing a new version. If you must, set `allow_updates: true` on the Module — see [the guard rails](../explanations/deprecation-lifecycle.md#the-guard-rails). -- **Retire a version.** Set `deprecated: true` to steer users away from it. Deleting it - outright has to wait until after it is deprecated (unless the Module has - `allow_updates: true`). See +- **Retire a version.** Set `deprecated: true` in the Release file to steer users away + from it. Deleting it outright has to wait until after it is deprecated (unless the + Module has `allow_updates: true`). See [the guard rails](../explanations/deprecation-lifecycle.md#the-guard-rails). diff --git a/docs/reference/configuration.md b/docs/reference/configuration.md index b945a78..c7c03eb 100644 --- a/docs/reference/configuration.md +++ b/docs/reference/configuration.md @@ -9,9 +9,9 @@ the [glossary](../glossary.md). ## What you author -You author two kinds of file: a single `settings.yaml`, and one `/.yaml` -per Module version. The [schema reference](../schemas.md) covers which schema validates -which and how to point your editor at them. +You author two kinds of file: a single `settings.yaml`, and one Release file, +`/.yaml`, per Module version. The [schema reference](../schemas.md) covers +which schema validates which and how to point your editor at them. They are structured as follows: @@ -19,7 +19,7 @@ They are structured as follows: settings.yaml └── default_versions -/.yaml one file per Module version +/.yaml one Release file per Module version ├── deprecated: false lifecycle flag └── module ├── name Module name @@ -71,7 +71,7 @@ A Module is the unit an end user loads. It carries: are for advanced cases the other fields cannot cover — check with a `deploy-tools` admin before using them. -Each per-version file also carries a `deprecated` flag; see +Each Release file also carries a `deprecated` flag; see [the release lifecycle](../explanations/deprecation-lifecycle.md). ## The three application types diff --git a/docs/schemas.md b/docs/schemas.md index 7252efc..0cf9a9d 100644 --- a/docs/schemas.md +++ b/docs/schemas.md @@ -19,7 +19,7 @@ You author two kinds of configuration file, each validated against a different s | `settings.yaml` (one per config folder) | `deployment-settings.json` | `DeploymentSettings` | | `/.yaml` (one per Module version) | `release.json` | `Release` | -Per-version files sit in a folder named after the Module, so the path is +Release files sit in a folder named after the Module, so the path is `//.yaml` (folder = Module `name`, filename = `version`). Add a `yaml-language-server` comment as the first line of each file, pointing at the matching schema, so your editor validates it as you type: diff --git a/docs/tutorials/your-first-deployment.md b/docs/tutorials/your-first-deployment.md index 1299366..f7230d9 100644 --- a/docs/tutorials/your-first-deployment.md +++ b/docs/tutorials/your-first-deployment.md @@ -28,9 +28,9 @@ $ git clone https://github.com/DiamondLightSource/deploy-tools.git $ export CONFIG=$PWD/deploy-tools/src/deploy_tools/demo_configuration ``` -That folder contains a `settings.yaml` and one folder per Module, each holding a -`.yaml` file. Have a look — `example-module-apps/0.1.yaml`, for example, defines -a Module with a containerised app and a couple of small shell entrypoints. +That folder contains a `settings.yaml` and one folder per Module, each holding a Release +file named `.yaml`. Have a look — `example-module-apps/0.1.yaml`, for example, +defines a Module with a containerised app and a couple of small shell entrypoints. ```{note} The `# yaml-language-server: $schema=…` line at the top of each file only drives editor From 57bda27669bded3d866a4b35e41920d19d836ad3 Mon Sep 17 00:00:00 2001 From: Martin Gaughran Date: Wed, 26 Aug 2026 16:20:45 +0000 Subject: [PATCH 16/18] Use separate table to indicate Release-level fields --- docs/reference/configuration.md | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/docs/reference/configuration.md b/docs/reference/configuration.md index c7c03eb..c2571ac 100644 --- a/docs/reference/configuration.md +++ b/docs/reference/configuration.md @@ -36,6 +36,15 @@ settings.yaml Those last three are the values `app_type` can take rather than fields of their own — see [the three application types](#the-three-application-types) below. +## A Release file + +Each Release file has two fields: + +| Field | Purpose | +|-------|---------| +| `module` | The Module being released (below). | +| `deprecated` | Whether this version is deprecated. Defaults to `false` — see [the release lifecycle](../explanations/deprecation-lifecycle.md). | + ## A Module A Module is the unit an end user loads. It carries: @@ -71,9 +80,6 @@ A Module is the unit an end user loads. It carries: are for advanced cases the other fields cannot cover — check with a `deploy-tools` admin before using them. -Each Release file also carries a `deprecated` flag; see -[the release lifecycle](../explanations/deprecation-lifecycle.md). - ## The three application types Every entry under `applications` sets `app_type` to select one of three kinds; a single From 05afedf85c9540e6d800dbfeed6bec4edc536f43 Mon Sep 17 00:00:00 2001 From: Martin Gaughran Date: Wed, 26 Aug 2026 16:33:17 +0000 Subject: [PATCH 17/18] Clarify docs regarding our recommended CI pipeline --- docs/tutorials/your-first-deployment.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/tutorials/your-first-deployment.md b/docs/tutorials/your-first-deployment.md index f7230d9..a502cdc 100644 --- a/docs/tutorials/your-first-deployment.md +++ b/docs/tutorials/your-first-deployment.md @@ -146,8 +146,8 @@ dls-pmac-control` loads `0.1` unless you ask for `dls-pmac-control/0.2` explicit You took a configuration folder, checked the area was clean, previewed the changes, and deployed Modules you could load and run. Those are the same `compare`, `validate` and -`sync` commands a real deployment runs; a CI pipeline spreads them across separate stages -(see [drive deploy-tools from CI](../how-to/ci-pipeline.md)). +`sync` commands a real deployment runs; our recommended pipeline spreads them across +separate stages (see [drive deploy-tools from CI](../how-to/ci-pipeline.md)). From here: From 947ef50e9cc916d3cb3ea8b65ae8ce1a0b8493d1 Mon Sep 17 00:00:00 2001 From: Martin Gaughran Date: Wed, 26 Aug 2026 16:35:12 +0000 Subject: [PATCH 18/18] Remove outdated text (typo) in README --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 87c7ad9..041617b 100644 --- a/README.md +++ b/README.md @@ -25,8 +25,8 @@ the deploy-tools commands. The deployment_root needs to be a writeable location files to get deployed under. These commands act on a shared deployment area, so we recommend running them from a CI -pipeline gated by change review rather than by hand a requirement. Running them manually -against the demo configuration, as below, is the quickest way to see what each does — the +pipeline gated by change review rather than by hand. Running them manually against the demo +configuration, as below, is the quickest way to see what each does — the [documentation](https://diamondlightsource.github.io/deploy-tools) has a hands-on tutorial and a guide to driving them from CI.