Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 37 additions & 0 deletions pipeline/workflow/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -70,3 +70,40 @@ uv run python spanner_ingestion_test.py
2. **Database Cleanup:** Deletes existing test import records in the staging Spanner database.
3. **Workflow Triggering:** Connects to GCP and triggers the staging Cloud Workflows (`import-automation-workflow` and `spanner-ingestion-workflow`).
4. **Verification:** Polls the workflows for completion and queries the staging Spanner database to verify the ingested data is marked as `SUCCESS`.

---

## Promoting to Stable (Production Release)

By default, the automated CI/CD pipeline deploys the latest tested version from the `master` branch to the autopush environment using the `:latest` tag.

To promote a verified version to `stable` (so it can be used for stable production releases), use the provided promotion script. You can use the `--dry-run` (or `-d`) flag to perform a dry run and preview the commands without executing them. By default, the script targets the `datcom-ci` project, but you can override this by setting the `PROJECT_ID` environment variable.

> [!IMPORTANT]
> Promoting by **Git commit SHA** (Option 1) is the recommended and safest method. This ensures you promote the exact version you tested, avoiding race conditions if new commits were merged to `master` in the meantime.

### Option 1: Promote a specific version (SHA) - *Recommended*
To promote a specific build version (e.g., Git commit SHA) that has been verified in staging:
```bash
# Inside pipeline/workflow/
./promote.sh <commit-sha>
```
This will retag the images built with `<commit-sha>` as `:stable` in the Artifact Registry, and copy the corresponding Dataflow template to `ingestion-stable.json`.

To run the promotion in a different GCP project (e.g., your sandbox project):
```bash
# Inside pipeline/workflow/
PROJECT_ID=my-sandbox-project ./promote.sh <commit-sha>
```

### Option 2: Promote the current `latest` image - *Use with caution*
If you are certain that the current `:latest` tag in the registry points to the version you want to promote:
```bash
# Inside pipeline/workflow/
./promote.sh latest
```
> [!WARNING]
> Use this with caution. If new commits were merged to `master` after your testing started, the `:latest` tag might have already updated to a newer, untested version.

---
*Note: This script only handles version promotion (tagging). Deployment to a stable environment is handled separately (e.g., via manual `deploy-services.yaml` execution with the `stable` tag).*
2 changes: 1 addition & 1 deletion pipeline/workflow/cloudbuild.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ substitutions:
_AR_REPO_URL: 'us-docker.pkg.dev/datcom-ci/gcr.io'
_BQ_SPANNER_CONN_ID: 'projects/datcom-ci/locations/us-central1/connections/bq_spanner_conn_test'
_VERSION: '${SHORT_SHA}'
_PROD_TAG: 'stable'
_PROD_TAG: 'latest'

steps:

Expand Down
76 changes: 76 additions & 0 deletions pipeline/workflow/promote.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
#!/bin/bash
# Copyright 2026 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# Promote a specified version (SHA or tag) to 'stable' in the Artifact Registry.
#
# Usage: [PROJECT_ID=<gcp_project_id>] ./promote.sh <version> [--dry-run]

set -e

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

It is a best practice to use set -u (to exit on undefined variables) and set -o pipefail (to prevent errors in pipelines from being masked) along with set -e for more robust error handling in Bash scripts.

Suggested change
set -e
set -euo pipefail


VERSION=""
DRY_RUN=false
PROJECT_ID=${PROJECT_ID:-"datcom-ci"}

usage() {
echo "Usage: [PROJECT_ID=<gcp_project_id>] $0 <version> [--dry-run]"
echo " <version>: Version (image tag or SHA) to promote (required)"
echo " --dry-run, -d: Dry run (only print the commands that would be executed)"
echo " PROJECT_ID: The GCP project hosting the registry and running the build (default: datcom-ci)"
exit 1
}

# Parse arguments
while [[ "$#" -gt 0 ]]; do
case "$1" in
-d|--dry-run)
DRY_RUN=true
shift
;;
-h|--help)
usage
;;
*)
if [ -z "$VERSION" ]; then
VERSION="$1"
else
echo "Error: Unknown argument '$1'"
usage
fi
shift
;;
esac
done

if [ -z "$VERSION" ]; then
echo "Error: Version is required."
usage
fi

run_cmd() {
if [ "$DRY_RUN" = true ]; then
echo "[DRY RUN] Would run: $*"
else
"$@"
fi
}

# Promote to stable (runs in target project where the images are hosted)
echo "Promoting '$VERSION' to 'stable' in Artifact Registry (project: $PROJECT_ID)..."
run_cmd gcloud builds submit . \
--config=update-version.yaml \
--project="$PROJECT_ID" \
--substitutions=_VERSION="$VERSION",_PROD_TAG=stable

echo "Promotion complete."
1 change: 0 additions & 1 deletion pipeline/workflow/spanner_ingestion_test.py
Original file line number Diff line number Diff line change
Expand Up @@ -131,7 +131,6 @@ def main(argv):
cleanup_spanner(short_import_name)

# 1. Trigger Import Automation Workflow
job_name = "test-import"
import_config = {
"gcp_project_id": PROJECT_ID,
"gcs_project_id": PROJECT_ID,
Expand Down