Update "latest" tagged images on new tag

.github/workflows/build.yaml

@@ -37,6 +37,23 @@ jobs:
         id: build-and-push-image
         with:
           image-flavor: "${{ matrix.image-flavor }}"
+      - name: Push latest tag
+        if: startsWith(github.ref, 'refs/tags/')
+        run: |
+          if ! git merge-base --is-ancestor "${{ github.sha }}" origin/main; then
+            echo "Skipping latest push: tagged commit is not on main"
+            exit 0
+          fi
+          CURRENT="${{ github.ref_name }}"
+          LATEST="$(git tag --merged origin/main --sort=-version:refname | head -1)"
+          if [[ "${CURRENT}" != "${LATEST}" ]]; then
+            echo "Skipping latest push: ${CURRENT} is not the highest version tag on main (${LATEST} is)"
+            exit 0
+          fi
+          IMAGE="${{ steps.build-and-push-image.outputs.image-tag }}"
+          LATEST_IMAGE="quay.io/stackrox-io/apollo-ci:${{ matrix.image-flavor }}-latest"
+          docker tag "${IMAGE}" "${LATEST_IMAGE}"
+          docker push "${LATEST_IMAGE}"
       - name: Save image info
         run: |
           mkdir -p image-info
@@ -67,6 +84,23 @@ jobs:
         id: build-and-push-image
         with:
           image-flavor: "${{ matrix.image-flavor }}"
+      - name: Push latest tag
+        if: startsWith(github.ref, 'refs/tags/')
+        run: |
+          if ! git merge-base --is-ancestor "${{ github.sha }}" origin/main; then
+            echo "Skipping latest push: tagged commit is not on main"
+            exit 0
+          fi
+          CURRENT="${{ github.ref_name }}"
+          LATEST="$(git tag --merged origin/main --sort=-version:refname | head -1)"
+          if [[ "${CURRENT}" != "${LATEST}" ]]; then
+            echo "Skipping latest push: ${CURRENT} is not the highest version tag on main (${LATEST} is)"
+            exit 0
+          fi
+          IMAGE="${{ steps.build-and-push-image.outputs.image-tag }}"
+          LATEST_IMAGE="quay.io/stackrox-io/apollo-ci:${{ matrix.image-flavor }}-latest"
+          docker tag "${IMAGE}" "${LATEST_IMAGE}"
+          docker push "${LATEST_IMAGE}"
       - name: Save image info
         run: |
           mkdir -p image-info

.github/workflows/promote-stable.yaml

@@ -0,0 +1,31 @@
+name: Promote to stable
+
+on:
+  workflow_dispatch:
+    inputs:
+      version:
+        description: "Version to promote (e.g. 0.5.7). Defaults to 'latest'."
+        required: false
+        default: "latest"
+
+env:
+  QUAY_STACKROX_IO_RW_USERNAME: ${{ secrets.QUAY_STACKROX_IO_RW_USERNAME }}
+  QUAY_STACKROX_IO_RW_PASSWORD: ${{ secrets.QUAY_STACKROX_IO_RW_PASSWORD }}
+
+jobs:
+  promote-stable:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Log in to Quay
+        run: |
+          docker login -u "$QUAY_STACKROX_IO_RW_USERNAME" --password-stdin <<<"$QUAY_STACKROX_IO_RW_PASSWORD" quay.io
+      - name: Retag all flavors as stable
+        run: |
+          VERSION="${{ inputs.version }}"
+          VERSION="${VERSION:-latest}"
+          for flavor in scanner-build scanner-test stackrox-build stackrox-test stackrox-ui-test jenkins-plugin; do
+            SRC="quay.io/stackrox-io/apollo-ci:${flavor}-${VERSION}"
+            DST="quay.io/stackrox-io/apollo-ci:${flavor}-stable"
+            echo "Promoting ${SRC} → ${DST}"
+            docker buildx imagetools create --tag "${DST}" "${SRC}"
+          done

README.md

@@ -7,6 +7,79 @@ This repository holds the Dockerfiles for images used in StackRox CI & builds.
 [gha-badge]: https://github.com/stackrox/rox-ci-image/actions/workflows/build.yaml/badge.svg
 [gha-link]:  https://github.com/stackrox/rox-ci-image/actions/workflows/build.yaml
 
+## Image Tags and Release Process
+
+Each image flavor (e.g. `stackrox-test`, `scanner-test`) is pushed to
+`quay.io/stackrox-io/apollo-ci` with three types of tags:
+
+| Tag | Example | Updated when | Use in |
+|-----|---------|-------------|--------|
+| **versioned** | `stackrox-test-0.5.7` | Every merge to main (auto-tagged) | Release branch prow configs, pinned references |
+| **latest** | `stackrox-test-latest` | Every new version tag on main | Testing rox-ci-image version in openshift/release PRs with `/pj-rehearse` before promoting to stable. |
+| **stable** | `stackrox-test-stable` | Manual promotion via workflow | Master/nightly prow configs in openshift/release |
+
+### How it works
+
+1. **Merge to main** -- `tag.yaml` auto-creates a semver tag (e.g. `0.5.8`)
+2. **Tag push** -- `build.yaml` builds all images, pushes versioned tags, and
+   updates `latest` (only if the tag is the highest version on main)
+3. **Promote to stable** -- [run manually](https://github.com/stackrox/rox-ci-image/actions/workflows/promote-stable.yaml) when ready:
+   ```bash
+   gh workflow run promote-stable.yaml
+   # or with a specific version:
+   gh workflow run promote-stable.yaml -f version=0.5.8
+   ```
+   This does a server-side retag (no rebuild) of all image flavors from the
+   specified version (default: `latest`) to `stable`.
+
+### Updating prow jobs in openshift/release
+
+Prow [job configs](https://github.com/openshift/release/tree/main/ci-operator/config/stackrox/stackrox) in the `openshift/release` repository reference these images via
+`build_root.image_stream_tag`. The tags must first be mirrored in
+`core-services/image-mirroring/_config.yaml`.
+
+- **Master/nightly configs**: use `stable` tag -- periodically, automatically picks up
+  promoted versions without config changes.
+- **Release branch configs**: pin to a specific version (e.g. `scanner-test-0.5.7`)
+  for reproducibility.
+- **`latest` tag**: use only for testing PRs against openshift/release.
+  `latest` is a moving target and should not be used for required jobs --
+  it is intended only for validation before promoting to `stable`.
+
+### Mirroring new versions to openshift CI
+
+To mirror a new versioned tag for release branch use:
+
+1. Add an entry to `core-services/image-mirroring/_config.yaml` in openshift/release
+2. PR requires testplatform team review
+
+The `latest` and `stable` floating tags are mirrored once and do not need
+updates per version.
+
+## Step-by-step: Making a CI Image Change
+
+Example: you need to bump a dependency or add a tool to the CI image.
+
+1. **Make your change** on a branch and open a PR in this repo.
+2. **Merge to main** -- once approved, merge the PR. The `tag.yaml` workflow
+   auto-creates a semver tag (e.g. `0.5.8`).
+3. **Wait for the build** -- the `build.yaml` workflow builds all image flavors
+   and pushes both versioned and `latest` tags to quay.io.
+4. **Test in openshift/release** -- open a PR in `openshift/release` that
+   references the `latest` tag and run `/pj-rehearse` to validate affected
+   prow jobs. No config change is needed if the jobs already use `latest`.
+5. **Promote to stable** -- once rehearsals pass, [run the promote-stable
+   workflow](https://github.com/stackrox/rox-ci-image/actions/workflows/promote-stable.yaml)
+   (or `gh workflow run promote-stable.yaml`). This retags `latest` → `stable`.
+   Master/nightly prow jobs pick up the new image automatically.
+6. **Pin release branches** (if needed) -- for release branch configs, update
+   `openshift/release` to reference the specific versioned tag
+   (e.g. `stackrox-test-0.5.8`). This requires testplatform review.
+
+> **Note:** There is only one `latest` tag per flavor, so only one
+> rox-ci-image change can be tested via rehearsal at a time. Coordinate
+> with others if multiple changes are in flight.
+
 ## Updating the Go Version
 
 To bump the Go version across all Docker images in this repository, use the automated script: