Hla 1142 module creation docs

docs/conf.py

@@ -60,6 +60,9 @@
 # So we can use the ::: syntax
 myst_enable_extensions = ["colon_fence"]
 
+# Generate anchors for headings (levels 1-3) so pages can link to sections.
+myst_heading_anchors = 3
+
 # If true, Sphinx will warn about all references where the target cannot
 # be found.
 nitpicky = True

docs/explanations.md

@@ -5,6 +5,7 @@ Explanations of how it works and why it works that way.
 ```{toctree}
 :maxdepth: 1
 
+explanations/configuration-model
 explanations/deployment-steps
 explanations/deployment-area
 explanations/snapshots-and-compare

docs/explanations/configuration-model.md

@@ -0,0 +1,98 @@
+# The configuration model
+
+This page is for anyone writing or editing deployment configuration. It explains how the
+files you author are structured — the objects they map to and how they nest. For every
+field in exhaustive detail see the [schema reference](../schemas.md); for the meaning of
+individual terms, the [glossary](../glossary.md).
+
+## What you author
+
+You author two kinds of file: a single `settings.yaml`, and one `<name>/<version>.yaml`
+per Module version. The [schema reference](../schemas.md) covers which schema validates
+which and how to point your editor at them.
+
+They are structured as follows:
+
+```text
+settings.yaml
+  └── default_versions
+
+<name>/<version>.yaml               one file per Module version
+  ├── deprecated: false             lifecycle flag
+  └── module
+      ├── name / version / description
+      ├── env_vars                  variables set on load
+      ├── dependencies              other Modules to load first
+      └── applications              one or more, each of:
+          ├── apptainer  container image + entrypoints
+          ├── shell      a bash script
+          └── binary     a downloaded, hash-checked executable
+```
+
+## A Module
+
+A Module is the unit an end user loads. Alongside its `name` and `version` it carries:
+
+| Field | Purpose |
+|-------|---------|
+| `description` | Shown by `module whatis <name>`. |
+| `env_vars` | Environment variables set when the Module is loaded. |
+| `dependencies` | Other Modules loaded first, optionally version-pinned. |
+| `applications` | One or more applications providing the executables (below). |
+| `allow_updates` | Permit in-place changes to this version — see [the guard rails](deprecation-lifecycle.md#the-guard-rails). |
+| `exclude_from_defaults` | Keep this version out of automatic default selection — see [default versions](default-versions.md#excluding-a-version-from-the-automatic-default). |
+
+`load_script` and `unload_script` run extra commands when the Module is loaded and
+unloaded respectively, injected raw into the generated Modulefile. They are an advanced
+escape hatch — check with a `deploy-tools` admin before using them.
+
+Each per-version file also carries a `deprecated` flag; see
+[the release lifecycle](deprecation-lifecycle.md).
+
+## The three application types
+
+Every entry under `applications` sets `app_type` to select one of three kinds; a single
+Module can mix them.
+
+| `app_type` | Provides | Key fields |
+|------------|----------|------------|
+| `apptainer` | Commands that run inside a container image | `container`, `entrypoints`, `global_options` |
+| `shell` | A single executable running a bash script | `name`, `script` |
+| `binary` | A downloaded executable added to the path | `name`, `url`, `hash`, `hash_type` |
+
+The demo `example-module-apps` Module combines an Apptainer app with a Shell app:
+
+```{literalinclude} ../../src/deploy_tools/demo_configuration/example-module-apps/0.1.yaml
+:language: yaml
+:lines: 3-
+```
+
+**Apptainer** uses one container image with one or more `entrypoints`, each mapping an
+executable name to a command run inside the container. Options — container `mounts`,
+`command_args`, `apptainer_args`, `host_binaries` — can be set per entrypoint or shared
+across all of them via `global_options`.
+
+**Shell** exposes a single executable (`name`) that runs the `script` list as bash.
+
+**Binary** downloads `url`, checks it against `hash` using `hash_type` (`sha256`,
+`sha512`, `md5`, or `none` to skip the check), and adds the executable to the path as
+`name`. The demo `argocd` Module uses one:
+
+```{literalinclude} ../../src/deploy_tools/demo_configuration/argocd/v2.14.10.yaml
+:language: yaml
+:lines: 3-
+```
+
+## Settings
+
+`settings.yaml` holds deployment-wide settings. It currently has a single field,
+`default_versions`, mapping a Module name to the version handed to `module load <name>`
+when no version is given:
+
+```{literalinclude} ../../src/deploy_tools/demo_configuration/settings.yaml
+:language: yaml
+:lines: 3-
+```
+
+How that choice is resolved — and how to keep a version out of automatic selection — is
+covered in [default version resolution](default-versions.md).

docs/explanations/deployment-area.md

@@ -37,7 +37,8 @@ corresponding `modulefile`. Users put only the *modulefiles* directories on thei
 
 Deprecation is therefore cheap and reversible: the built files never move, only the symlink
 moves between `modulefiles/` and `deprecated/modulefiles/`. See
-[the release lifecycle](deprecation-lifecycle.md) for the full set of transitions.
+[the release lifecycle](deprecation-lifecycle.md#configuration-is-declarative) for
+the full set of transitions.
 
 ## The build area
 

docs/explanations/deployment-steps.md

@@ -7,7 +7,7 @@ a command of their own. For the commands themselves see the [CLI reference](../c
 | Step | Description | Run by |
 |------|-------------|--------|
 | Compare | Compare the current deployment snapshot against the modulefiles and built modules that actually exist, confirming the [deployment area](deployment-area.md) is healthy. | `compare` |
-| Validate | Diff the new configuration against the current snapshot to determine the set of actions to take, and check those actions are permitted by the [release lifecycle](deprecation-lifecycle.md). | `validate`, `sync` |
+| Validate | Diff the new configuration against the current snapshot to determine the set of actions to take, and check those actions are permitted by the [lifecycle guard rails](deprecation-lifecycle.md#the-guard-rails). | `validate`, `sync` |
 | Build | Generate entrypoint scripts, configuration files and environment variables for each changed Module, writing them to the build area. | `sync` (`validate --test-build`) |
 | Deploy | Move the built Modules from the build area into the Modules Area, link each modulefile into the live or deprecated tree according to its status, and update default versions. | `sync` |
 

docs/explanations/deprecation-lifecycle.md

@@ -9,8 +9,8 @@ deleted — is expressed by adding or removing a Release in configuration and to
 
 You never tell `deploy-tools` to "deprecate" or "remove" something directly. You describe
 the set of Releases you want, and the tool compares that against the
-[snapshot](snapshots-and-compare.md) of the last `sync` to derive the actions needed. Each
-Release falls into one of these cases:
+[snapshot](snapshots-and-compare.md#what-the-snapshot-is) of the last `sync` to derive
+the actions needed. Each Release falls into one of these cases:
 
 | Transition | Detected when… | Effect on the deployment area |
 |------------|----------------|-------------------------------|

docs/glossary.md

@@ -9,7 +9,7 @@ for the on-disk layout these terms refer to.
 | Environment Modules | A [standard package for Linux](https://modules.readthedocs.io/en/latest/) that provides the commands for loading and unloading 'Environment Modules' (each defined by a Modulefile). Note that while we are using this system, our definition of Module is separate. If we are referring to an Environment Module, we will use the full name. |
 | Modulefile | Used by the Environment Modules package to specify all details of an Environment Module. This can include executables to add to the path, environment variables to set, etc. |
 | Module | A set of files that can be used to provide applications on your path, provide configuration, and set environment variables. We do this using the Environment Modules system by providing a Modulefile with the relevant configuration. |
-| Application | Each Module can be configured with multiple Applications, each one providing one or more executables. There are 3 types of Application: `Apptainer` (an executable container image), `Shell` (a Bash script) and `Binary` (an executable downloaded from a URL and verified against a hash). |
+| Application | Each Module can be configured with multiple Applications, each one providing one or more executables. There are 3 types of Application: `Apptainer` (an executable container image), `Shell` (a Bash script) and `Binary` (an executable downloaded from a URL and verified against a hash). See [the configuration model](explanations/configuration-model.md). |
 | Release (noun) | A Module, including version, alongside its [lifecycle](explanations/deprecation-lifecycle.md) (i.e. deprecation) status. |
 | Deployment | The declared configuration for a Deployment Area: all Releases (deprecated or not) to be maintained there, plus global settings. Written to the `deployment.yaml` snapshot by `sync`. The act of deploying is written lowercase. |
 | Deployment Step | Refers to one of the primary steps that make up the deployment process. See [the deployment process](explanations/deployment-steps.md) for a breakdown. |

docs/how-to.md

@@ -5,6 +5,7 @@ Practical step-by-step guides for the more experienced user.
 ```{toctree}
 :maxdepth: 1
 
+how-to/write-module-configuration
 how-to/ci-pipeline
 how-to/run-container
 how-to/vscode-tasks

docs/how-to/ci-pipeline.md

@@ -15,9 +15,9 @@ When someone opens a change (before it is merged), run, without altering the are
 - `deploy-tools compare <area>` — confirm the area still matches its last snapshot, so
   the change is being checked against a known-healthy baseline.
 - `deploy-tools validate <area> <config>` — confirm the new configuration is valid and its
-  [lifecycle transitions](../explanations/deprecation-lifecycle.md) are permitted. Add
-  `--test-build` to build every changed Module in a temporary directory, catching build
-  failures before merge.
+  [lifecycle transitions](../explanations/deprecation-lifecycle.md#the-guard-rails) are
+  permitted. Add `--test-build` to build every changed Module in a temporary directory,
+  catching build failures before merge.
 
 This gives reviewers a green light that the change is deployable without changing anything
 on the filesystem.

docs/how-to/regenerate-schemas.md

@@ -12,7 +12,7 @@ schemas and commit again.
 
 If you bypass the hooks (for example with `git commit --no-verify`), regenerate the
 schemas manually. The simplest way is the **Generate Schema** VSCode task (see
-[the VSCode tasks guide](vscode-tasks.md)), which writes to the correct location.
+[the VSCode tasks guide](vscode-tasks.md#running-a-task)), which writes to the correct location.
 Equivalently, run the CLI, pointing it at that folder:
 
 ```bash

docs/how-to/run-container.md

@@ -12,3 +12,6 @@ $ docker run ghcr.io/diamondlightsource/deploy-tools:latest --version
 ```
 
 To get a released version, use a numbered release instead of `latest`.
+
+In practice this image is what a CI pipeline runs to drive `deploy-tools` against a shared
+deployment area — see [drive deploy-tools from CI](ci-pipeline.md).

docs/how-to/write-module-configuration.md

@@ -0,0 +1,78 @@
+# Write a Module configuration
+
+To add or change a Module you edit YAML in the configuration folder. This guide covers the
+mechanics. For the shape of the files see
+[the configuration model](../explanations/configuration-model.md); for every field, the
+[schema reference](../schemas.md).
+
+## Add a new Module version
+
+1. Create the file at `<config folder>/<name>/<version>.yaml`. The folder name is the
+   Module `name` and the filename is the `version`, so `phoebus/0.1.yaml` defines version
+   `0.1` of `phoebus`. The `name` and `version` inside the file must match the path.
+
+2. Add the schema line as the first line so your editor validates as you type (see the
+   [schema reference](../schemas.md) for details, including how to associate schemas
+   repository-wide instead of per file):
+
+   ```yaml
+   # yaml-language-server: $schema=https://raw.githubusercontent.com/DiamondLightSource/deploy-tools/main/src/deploy_tools/models/schemas/release.json
+   ```
+
+3. Define the Module. Most Modules provide one or more applications; the smallest useful
+   one is a single shell script:
+
+   ```yaml
+   module:
+     name: my-module
+     version: "1.0"
+     description: What this Module provides
+     applications:
+       - app_type: shell
+         name: hello
+         script:
+           - echo "hello from my-module"
+   ```
+
+   Swap the application for an `apptainer` or `binary` one as needed — see
+   [the three application types](../explanations/configuration-model.md#the-three-application-types).
+
+   A Module doesn't have to provide an application: it can instead just set environment
+   variables or pull in other Modules as
+   [dependencies](../explanations/configuration-model.md#a-module). Give such a Module an
+   empty `applications: []`.
+
+## Set the default version
+
+`module load <name>` with no version loads the default. If you don't choose one the
+highest version is picked automatically; to pin a specific version, add it to
+`settings.yaml`:
+
+```yaml
+default_versions:
+  my-module: "1.0"
+```
+
+To keep a version out of automatic selection — an alpha or release candidate, say — while
+still allowing an explicit `module load <name>/<version>`, set
+`exclude_from_defaults: true` on that Module.
+
+See [default version resolution](../explanations/default-versions.md) for how the
+automatic choice is made.
+
+## Get your change deployed
+
+You don't run `deploy-tools` yourself. Open a merge request in the configuration
+repository; CI validates the change and, once it is merged, deploys it (see
+[drive deploy-tools from CI](ci-pipeline.md)). The `yaml-language-server` schema line
+catches structural mistakes in your editor as you type, before you open the request.
+
+## Change or retire a version
+
+- **Update an existing version in place.** Rejected by default so published versions stay
+  stable; prefer publishing a new version. If you must, set `allow_updates: true` on the
+  Module — see [the guard rails](../explanations/deprecation-lifecycle.md#the-guard-rails).
+- **Retire a version.** Set `deprecated: true` to steer users away from it. Deleting it
+  outright has to wait until after it is deprecated (unless the Module has
+  `allow_updates: true`). See
+  [the guard rails](../explanations/deprecation-lifecycle.md#the-guard-rails).

docs/schemas.md

@@ -28,6 +28,12 @@ This requires an editor with a YAML language server — e.g. VS Code with the
 or any [LSP](https://microsoft.github.io/language-server-protocol/)-capable editor
 running [`yaml-language-server`](https://github.com/redhat-developer/yaml-language-server).
 
+Alternatively, VS Code's [`yaml.schemas`](https://github.com/redhat-developer/vscode-yaml#associating-schemas)
+setting (committed to `.vscode/settings.json`) maps schemas to file paths once for the
+whole repository, avoiding the per-file comment. This documentation uses the comment
+because it is self-contained and editor-agnostic, but a real configuration repository may
+reasonably prefer the repository-level setting.
+
 ```{note}
 The bundled `demo_configuration` instead points at the locally generated schemas via an
 absolute workspace path (e.g.