feat(runway): resolve merge targets per queue and provision their checkouts

## Summary

### Why?

Three things stopped Runway from performing a real merge. It resolved one merger for the whole process, so a deployment could serve only one repository. Nothing created the checkout the git merger requires, which it declares must already exist. And the container had no `git` binary, while the merger demanded three absolute runtime paths — so enabling it failed at startup with `git runtime executable is required` rather than working.

### What?

Merge targets come from `MERGE_CONFIG_PATH`, a YAML file with a `defaults` block and per-queue overrides. Each git target is provisioned at startup — repository initialised, remote configured, credential written, target branch checked out — idempotently, so a restart against a persisted volume costs nothing and a rotated token takes effect. The existing `MERGE_*` environment variables still configure a single target when no file is given.

Two queues naming one checkout resolve to the *same* merger instance, which is what serializes them: a git merger locks the working tree it owns, and two instances over one tree would reset it out from under each other mid-merge. Naming one checkout for two different targets is rejected at startup.

The credential never enters the remote URL. The merger folds git's stderr into the errors it returns, so a URL-embedded token would be reprinted into logs and dead-letter payloads by any failed fetch. It is written as an HTTP `Authorization` header into a `0600` config fragment the repository includes — which keeps it off the command line too. SSH needs no code at all: the merger already passes `SSH_AUTH_SOCK` and `GIT_SSH_COMMAND` through its scrubbed environment.

The git runtime is derived from the installed git when unset — executable from `PATH`, exec path from `git --exec-path`, templates from the matching install prefix — so the pinning safeguard stops being a boot-time trap. `git` is installed in the image.

## Test Plan

✅ `bazel test //service/runway/server:go_default_test` — provisioning runs against real git (pinned, as the merger does): clone into an empty directory, idempotent re-run, drifted remote URL corrected, non-default target branch, credential written `0600` and absent from the remote URL, stale fragment removed, unset token rejected, and git-runtime derivation with and without the environment overrides.

# Conflicts:
#	service/runway/server/main.go

Author: behinddwalls

service/runway/README.md

@@ -15,7 +15,38 @@ These topic keys and their wire contracts are owned by the queue's producer side
 
 ### Merger backend
 
-The merge work is done by the [`merger`](../../runway/extension/merger) extension. By default the server wires the **noop** merger (always succeeds — for local dev and compose). Set `MERGE_CHECKOUT_PATH` to wire the real **git** merger, built from the `MERGE_*` / `GIT_*` environment (see Configuration); the git merger owns the checkout at that path and pushes to the configured remote/target.
+The merge work is done by the [`merger`](../../runway/extension/merger) extension, resolved **per queue** — so one Runway can serve several repositories by giving each queue its own merge target. By default every queue gets the **noop** merger (always succeeds — for local dev and compose). Point `MERGE_CONFIG_PATH` at a merge configuration file to wire real **git** merge targets, or set `MERGE_CHECKOUT_PATH` to configure a single one from the environment (see Configuration).
+
+Two queues naming the same checkout resolve to the *same* merger instance, which is what serializes them against each other: a git merger locks the working tree it owns, and two instances over one tree would reset it out from under each other mid-merge. Naming one checkout for two *different* targets is rejected at startup.
+
+### Merge configuration file
+
+`MERGE_CONFIG_PATH` names a YAML file with a `defaults` block and per-queue overrides. A queue's `merger` block replaces the default wholesale rather than merging field by field.
+
+```yaml
+defaults:
+  merger: {type: noop}            # noop | git
+queues:
+  - name: demo-queue
+    merger:
+      type: git
+      remoteUrl: https://github.com/uber/sq-sandbox.git
+      target: main
+      checkoutPath: /var/runway/checkouts/sq-sandbox
+      defaultStrategy: SQUASH_REBASE
+      updateHeadBranch: true
+      tokenEnv: GITHUB_TOKEN
+```
+
+The file holds **no secret**: `tokenEnv` names the environment variable carrying the credential, so the file stays committable and rotating the token needs no edit. Omitting `remoteUrl` means the checkout is provisioned by something else and is used as it stands.
+
+### Checkout provisioning
+
+A git merge target's checkout is created at startup if it is not already there: the repository is initialised, the remote configured, the credential written, and the target branch checked out. It is idempotent, so restarting against a persisted volume costs nothing and a rotated token takes effect.
+
+The credential never goes into the remote URL. The merger folds git's stderr into the errors it returns, so a URL-embedded token would be reprinted into logs and dead-letter payloads by any failed fetch. Instead it is written as an HTTP `Authorization` header into a `0600` config fragment inside `.git/`, which the repository config includes — keeping it out of the command line too.
+
+For SSH, leave `tokenEnv` unset and use an `ssh://` remote: the merger already passes `SSH_AUTH_SOCK` and `GIT_SSH_COMMAND` through to git, so an agent or a mounted deploy key works with no further configuration.
 
 Because Runway only consumes queues and serves `Ping`, it needs a **queue** database but no application/storage database.
 
@@ -40,14 +71,16 @@ The Runway controllers themselves live under [`runway/controller/`](../../runway
 | `QUEUE_MYSQL_DSN` | yes      | Queue database DSN                            | —                        |
 | `PORT`            | no       | gRPC listen address                           | `:8086`                  |
 | `HOSTNAME`        | no       | Subscriber name for the queue consumer        | `runway-<unix_ts>`       |
-| `MERGE_CHECKOUT_PATH` | no   | Absolute path to the git checkout the merger owns. When unset, the noop merger is used. | — (noop) |
+| `MERGE_CONFIG_PATH` | no | Path to the per-queue merge configuration file (see above). Takes precedence over the `MERGE_*` variables below, which configure a single target. | — |
+| `MERGE_CHECKOUT_PATH` | no   | Absolute path to the git checkout the merger owns. When unset (and no config file), the noop merger is used. | — (noop) |
 | `MERGE_REMOTE`    | no       | Git remote to fetch/push                       | `origin`                 |
 | `MERGE_TARGET`    | no       | Destination branch on the remote               | `main`                   |
 | `MERGE_DEFAULT_STRATEGY` | no | Strategy a `DEFAULT` step resolves to: `REBASE`, `SQUASH_REBASE`, `MERGE`, or `PROMOTE` | `REBASE` |
 | `MERGE_COMMITTER_NAME` | no  | Committer name for service-created commits      | `SubmitQueue Runway`     |
 | `MERGE_COMMITTER_EMAIL` | no | Committer email for service-created commits     | `runway@submitqueue.invalid` |
-| `GIT_EXECUTABLE` / `GIT_EXEC_PATH` / `GIT_TEMPLATE_DIR` | when `MERGE_CHECKOUT_PATH` set | Absolute paths to the pinned git runtime | — |
+| `GIT_EXECUTABLE` / `GIT_EXEC_PATH` / `GIT_TEMPLATE_DIR` | no | Absolute paths pinning the git runtime. Each is derived from the installed git when unset — the executable from `PATH`, the exec path from `git --exec-path`, the templates from the matching install prefix. | derived |
 | `MERGE_CHECK_STALENESS` | no | Verify each change's provider ref still points at the commit its URI names before applying | `true` |
+| `MERGE_UPDATE_HEAD_BRANCH` | no | After a committing merge, move each change's head branch to the commit it landed as, so the provider marks it merged. Only affects `REBASE`/`SQUASH_REBASE`. | `false` |
 | `MERGE_ALLOW_UNRELATED_HISTORIES` | no | Let a `MERGE` step integrate a change sharing no ancestry with the target (repository imports). Leave off unless the queue exists to perform imports. | `false` |
 | `MERGE_FETCH_REFSPECS` | no | Comma-separated extra refspecs fetched each cycle. Only needed for a remote that refuses to serve an unadvertised commit by SHA. | — |
 

service/runway/server/BUILD.bazel

@@ -1,4 +1,4 @@
-load("@rules_go//go:def.bzl", "go_binary", "go_cross_binary", "go_library")
+load("@rules_go//go:def.bzl", "go_binary", "go_cross_binary", "go_library", "go_test")
 
 exports_files(
     ["docker-compose.yml"],
@@ -7,7 +7,12 @@ exports_files(
 
 go_library(
     name = "go_default_library",
-    srcs = ["main.go"],
+    srcs = [
+        "checkout.go",
+        "config.go",
+        "gitruntime.go",
+        "main.go",
+    ],
     importpath = "github.com/uber/submitqueue/service/runway/server",
     visibility = ["//visibility:private"],
     deps = [
@@ -33,6 +38,7 @@ go_library(
         "//runway/extension/merger/noop:go_default_library",
         "@com_github_go_sql_driver_mysql//:go_default_library",
         "@com_github_uber_go_tally//:go_default_library",
+        "@in_gopkg_yaml_v3//:go_default_library",
         "@org_golang_google_grpc//:go_default_library",
         "@org_golang_google_grpc//reflection:go_default_library",
         "@org_uber_go_zap//:go_default_library",
@@ -66,3 +72,33 @@ filegroup(
     ],
     visibility = ["//test:__subpackages__"],
 )
+
+go_test(
+    name = "go_default_test",
+    srcs = [
+        "checkout_test.go",
+        "config_test.go",
+    ],
+    # Checkout provisioning runs real git, so the test uses the same pinned
+    # runtime the merger does rather than whatever git the host happens to have.
+    data = [
+        "@git",
+        "@git//:git_receive_pack",
+        "@git//:git_upload_archive",
+        "@git//:git_upload_pack",
+        "@git//:templates",
+        "@git//:templates/description",
+    ],
+    embed = [":go_default_library"],
+    env = {
+        "SUBMITQUEUE_TEST_GIT": "$(location @git//:git)",
+        "SUBMITQUEUE_TEST_GIT_TEMPLATE_DESCRIPTION": "$(location @git//:templates/description)",
+    },
+    deps = [
+        "//api/base/mergestrategy/protopb:go_default_library",
+        "//runway/extension/merger/git:go_default_library",
+        "@com_github_stretchr_testify//assert:go_default_library",
+        "@com_github_stretchr_testify//require:go_default_library",
+        "@org_uber_go_zap//zaptest:go_default_library",
+    ],
+)

service/runway/server/Dockerfile

@@ -1,6 +1,9 @@
 FROM debian:bookworm-slim
 
-RUN apt-get update && apt-get install -y ca-certificates && rm -rf /var/lib/apt/lists/* \
+# git is a runtime dependency, not a build one: the git merger shells out to it
+# to fetch, apply, and push. Without it the service still starts, but only the
+# noop merger works — a git merge target fails at startup instead.
+RUN apt-get update && apt-get install -y ca-certificates git && rm -rf /var/lib/apt/lists/* \
     && mkdir -p /app && chmod 0755 /app
 WORKDIR /app
 

service/runway/server/checkout.go

@@ -0,0 +1,232 @@
+// Copyright (c) 2025 Uber Technologies, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package main
+
+import (
+	"bytes"
+	"context"
+	"encoding/base64"
+	"fmt"
+	"os"
+	"os/exec"
+	"path/filepath"
+	"strings"
+
+	"go.uber.org/zap"
+
+	gitmerger "github.com/uber/submitqueue/runway/extension/merger/git"
+)
+
+// credentialFile is the name, inside the checkout's .git directory, of the
+// config fragment holding the remote's credential. It is included from the
+// repository config rather than written into it, so the secret lives in one
+// file this service owns and can chmod, and `git config --list` on the
+// repository does not print it back out.
+const credentialFile = "submitqueue-credentials.config"
+
+// provisionCheckout makes the working tree the git merger requires: a
+// repository at CheckoutPath with the configured remote, its credential in
+// place, and the target branch checked out.
+//
+// The merger deliberately does none of this — it declares that the checkout
+// must already exist, so that it never has to decide whether a surprising
+// working tree is one it should repair or one it should refuse. Creating it is
+// the wiring layer's job, and doing it here keeps the decision out of the merge
+// path.
+//
+// It is idempotent: an existing checkout has its remote URL and credential
+// brought back in line rather than being recreated, so a restart against a
+// persisted volume costs nothing and a rotated token takes effect.
+func provisionCheckout(ctx context.Context, logger *zap.SugaredLogger, runtime gitmerger.GitRuntime, cfg mergerConfig) error {
+	if err := os.MkdirAll(cfg.CheckoutPath, 0o755); err != nil {
+		return fmt.Errorf("create checkout dir %q: %w", cfg.CheckoutPath, err)
+	}
+
+	fresh, err := initRepository(ctx, runtime, cfg)
+	if err != nil {
+		return err
+	}
+
+	// Written before the first fetch, because a private remote cannot be read
+	// without it.
+	if err := writeCredential(cfg); err != nil {
+		return err
+	}
+	if err := configureRemote(ctx, runtime, cfg); err != nil {
+		return err
+	}
+
+	if _, err := runGit(ctx, runtime, cfg.CheckoutPath, "fetch", cfg.Remote); err != nil {
+		return fmt.Errorf("fetch %s from %s: %w", cfg.Remote, cfg.RemoteURL, err)
+	}
+	remoteRef := cfg.Remote + "/" + cfg.Target
+	if _, err := runGit(ctx, runtime, cfg.CheckoutPath, "checkout", "-B", cfg.Target, remoteRef); err != nil {
+		return fmt.Errorf("check out %s: %w", remoteRef, err)
+	}
+
+	// The merger points HOME here. Its own `git clean -fdx` may remove it
+	// again, which is harmless — nothing is stored there, and the scrubbed
+	// environment means git reads no configuration from it either way.
+	if err := os.MkdirAll(filepath.Join(cfg.CheckoutPath, ".submitqueue-git-home", "xdg"), 0o700); err != nil {
+		return fmt.Errorf("create git home in %q: %w", cfg.CheckoutPath, err)
+	}
+
+	logger.Infow("provisioned merge checkout",
+		"checkout", cfg.CheckoutPath,
+		"remote", cfg.Remote,
+		"remote_url", cfg.RemoteURL,
+		"target", cfg.Target,
+		"cloned", fresh,
+		"credential", cfg.needsHTTPCredential(),
+	)
+	return nil
+}
+
+// initRepository creates the repository if the path does not already hold one,
+// reporting whether it had to. An existing repository is left in place: it may
+// carry fetched objects worth keeping, and re-creating it would discard them.
+func initRepository(ctx context.Context, runtime gitmerger.GitRuntime, cfg mergerConfig) (bool, error) {
+	if info, err := os.Stat(filepath.Join(cfg.CheckoutPath, ".git")); err == nil && info.IsDir() {
+		return false, nil
+	}
+	if _, err := runGit(ctx, runtime, cfg.CheckoutPath, "init", "-b", cfg.Target); err != nil {
+		return false, fmt.Errorf("init repository at %q: %w", cfg.CheckoutPath, err)
+	}
+	return true, nil
+}
+
+// configureRemote points the configured remote name at the configured URL,
+// adding it when absent and correcting it when it has drifted.
+func configureRemote(ctx context.Context, runtime gitmerger.GitRuntime, cfg mergerConfig) error {
+	out, err := runGit(ctx, runtime, cfg.CheckoutPath, "remote")
+	if err != nil {
+		return fmt.Errorf("list remotes in %q: %w", cfg.CheckoutPath, err)
+	}
+	for _, name := range strings.Fields(string(out)) {
+		if name != cfg.Remote {
+			continue
+		}
+		if _, err := runGit(ctx, runtime, cfg.CheckoutPath, "remote", "set-url", cfg.Remote, cfg.RemoteURL); err != nil {
+			return fmt.Errorf("set url of remote %q: %w", cfg.Remote, err)
+		}
+		return nil
+	}
+	if _, err := runGit(ctx, runtime, cfg.CheckoutPath, "remote", "add", cfg.Remote, cfg.RemoteURL); err != nil {
+		return fmt.Errorf("add remote %q: %w", cfg.Remote, err)
+	}
+	return nil
+}
+
+// writeCredential stores the remote's token as an HTTP Authorization header in
+// a config fragment the repository includes.
+//
+// The token deliberately never enters the remote URL. The merger folds git's
+// stderr into the errors it returns, so a URL-embedded credential would be
+// reprinted into logs and dead-letter payloads by any failed fetch. It is kept
+// out of the command line for the same reason — a header written by this
+// process into a file it owns is visible to neither.
+func writeCredential(cfg mergerConfig) error {
+	path := filepath.Join(cfg.CheckoutPath, ".git", credentialFile)
+
+	if !cfg.needsHTTPCredential() {
+		// A local path or SSH remote authenticates by other means. Remove any
+		// fragment a previous configuration left, so a target that stops using
+		// a token stops carrying one.
+		if err := os.Remove(path); err != nil && !os.IsNotExist(err) {
+			return fmt.Errorf("remove stale credential %q: %w", path, err)
+		}
+		return nil
+	}
+
+	token, ok := cfg.token()
+	if !ok {
+		return fmt.Errorf("environment variable %q named by tokenEnv is not set", cfg.TokenEnv)
+	}
+	if token == "" {
+		return fmt.Errorf("environment variable %q named by tokenEnv is empty", cfg.TokenEnv)
+	}
+
+	basic := base64.StdEncoding.EncodeToString([]byte(cfg.TokenUser + ":" + token))
+	fragment := fmt.Sprintf("[http %q]\n\textraheader = Authorization: Basic %s\n", cfg.RemoteURL, basic)
+	if err := os.WriteFile(path, []byte(fragment), 0o600); err != nil {
+		return fmt.Errorf("write credential %q: %w", path, err)
+	}
+
+	// include.path resolves relative to the config file holding it, which is
+	// .git/config — so the bare filename lands beside it.
+	if err := setLocalConfig(cfg.CheckoutPath, "include.path", credentialFile); err != nil {
+		return err
+	}
+	return nil
+}
+
+// setLocalConfig sets a repository-local config key, replacing any previous
+// value for it. It edits .git/config through git itself rather than by hand so
+// the file stays git's to format.
+func setLocalConfig(checkoutPath, key, value string) error {
+	// Uses the ambient git rather than the pinned runtime: this touches only
+	// the repository's own config file, never the remote, and the pinned
+	// runtime exists to make *merge results* reproducible.
+	cmd := exec.Command("git", "config", "--local", "--replace-all", key, value)
+	cmd.Dir = checkoutPath
+	var stderr bytes.Buffer
+	cmd.Stderr = &stderr
+	if err := cmd.Run(); err != nil {
+		return fmt.Errorf("git config --local %s: %w: %s", key, err, strings.TrimSpace(stderr.String()))
+	}
+	return nil
+}
+
+// runGit invokes the pinned git in dir with an environment scrubbed of ambient
+// configuration but retaining what is needed to reach a remote — the same split
+// the merger draws, so provisioning and merging authenticate identically.
+func runGit(ctx context.Context, runtime gitmerger.GitRuntime, dir string, args ...string) ([]byte, error) {
+	full := append([]string{
+		"--exec-path=" + runtime.ExecPath,
+		"-c", "init.templateDir=" + runtime.TemplateDir,
+	}, args...)
+
+	cmd := exec.CommandContext(ctx, runtime.Executable, full...)
+	cmd.Dir = dir
+	cmd.Env = []string{
+		"HOME=" + filepath.Join(dir, ".submitqueue-git-home"),
+		"GIT_CONFIG_NOSYSTEM=1",
+		"GIT_CONFIG_GLOBAL=" + os.DevNull,
+		"GIT_TERMINAL_PROMPT=0",
+		"GIT_EXEC_PATH=" + runtime.ExecPath,
+		"GIT_TEMPLATE_DIR=" + runtime.TemplateDir,
+		"LC_ALL=C",
+		"LANG=C",
+	}
+	for _, name := range []string{
+		"PATH", "SSH_AUTH_SOCK", "SSH_AGENT_PID",
+		"GIT_SSH", "GIT_SSH_COMMAND", "GIT_SSH_VARIANT",
+		"GIT_SSL_CAINFO", "GIT_SSL_CAPATH", "SSL_CERT_DIR", "SSL_CERT_FILE",
+		"HTTP_PROXY", "HTTPS_PROXY", "NO_PROXY",
+		"http_proxy", "https_proxy", "no_proxy",
+	} {
+		if v, ok := os.LookupEnv(name); ok {
+			cmd.Env = append(cmd.Env, name+"="+v)
+		}
+	}
+
+	var stdout, stderr bytes.Buffer
+	cmd.Stdout = &stdout
+	cmd.Stderr = &stderr
+	if err := cmd.Run(); err != nil {
+		return nil, fmt.Errorf("%w: %s", err, strings.TrimSpace(stderr.String()))
+	}
+	return stdout.Bytes(), nil
+}