feat(run-ops): run-store routing seam + run-engine read seams

[d-cs]

What

Introduces the run-store routing seam and the run-engine read seams that let run lifecycle operations be dispatched to either the control-plane database or a separately-generated run-ops database, depending on where a run/batch resides.

• run-store (internal-packages/run-store): adds runOpsStore.ts and substantially expands PostgresRunStore.ts so the store can resolve residency and route reads/writes to the correct backing client. types.ts grows the routing/residency types; NoopRunStore.ts is removed.
• run-engine (internal-packages/run-engine): adds engine/controlPlaneResolver.ts and routes the per-system read paths (dequeue, enqueue, waitpoint, checkpoint, run-attempt, ttl, delayed-run, execution-snapshot, pending-version, debounce, batch) through the resolver/store instead of talking to a single Prisma client directly. engine/errors.ts, engine/types.ts, and engine/index.ts are extended to support injecting the store/resolver.

Three fixes are included on top of the seam work:

• c6cadd85f — routes read-your-writes to the owning store's writer, not its lagging replica, so an operation immediately reading back what it just wrote sees a consistent result.
• 05c912e05 — normalizes run-ops-generation Prisma errors to the control-plane error class at the store write boundary, so instanceof checks and the P2002 → 422 handling continue to work across the separately-generated run-ops Prisma client.
• 88d12907f — resolves NEW-resident batches in ApiBatchResultsPresenter by routing the batch read through the store, so a dedicated-DB batch resolves instead of returning 404.

The change is heavily test-first: the bulk of the diff is new unit/integration coverage for the store routing, residency, and each run-engine system's control-plane resolver path.

Why

PR4 of the run-ops split stack (PR1–PR3 land the ClickHouse test-container and earlier plumbing). This PR is the read-path foundation: it adds the seam and read-routing but leaves the write path to route through the same seam in a later PR. Behavior-changing where the three fixes above touch existing read-your-writes / error-normalization / batch-resolution paths; otherwise additive (new store module, new resolver, injectable dependencies with existing single-client behavior preserved when no dedicated store is configured).

Tests

Extensive new vitest coverage under run-store/src/*.test.ts (routing, residency, dual-schema select, cross-generation error normalization, read-after-write, idempotency dedup, mixed residency, waitpoint co-location) and run-engine/src/engine/**/*.test.ts (per-system controlPlaneResolver tests, injectability, block-edge residency, waitpoint read residency, trigger-create routing, lifecycle router). Testcontainers-backed; no mocks.

Notes

Draft, stacked on #4114 (runops/pr03-clickhouse-tc). Review that first; this diff is against it.

Server-change / changeset note to be added at stack-assembly time.

🤖 Generated with Claude Code

[changeset-bot]

⚠️ No Changeset found

Latest commit: d46aab2

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[coderabbitai]

Walkthrough

Changes

This PR introduces a RunStore/ControlPlaneResolver seam in RunEngine, separating run-scalar persistence (via RunStore, backed by PostgresRunStore) from control-plane data (organization/project/environment/worker) resolved via a new ControlPlaneResolver interface with a PassthroughControlPlaneResolver default. Batch, checkpoint, debounce, delayed-run, dequeue, enqueue, execution-snapshot, pending-version, run-attempt, TTL, and waitpoint systems are refactored to route reads/writes through these abstractions instead of direct Prisma joins/CTEs, enabling run data and control-plane data to reside on physically separate Postgres databases. A CrossSeamGuardHook allows the webapp to select waitpoint completion routing. NoopRunStore is removed, run-store gains a dependency on @internal/run-ops-database, and the webapp's ApiBatchResultsPresenter now resolves batches via runStore. Extensive new integration tests validate single-DB and heterogeneous two-database routing scenarios across all affected systems.

Sequence Diagram(s)

sequenceDiagram
  participant RunEngine
  participant WaitpointSystem
  participant CrossSeamGuard
  participant RunStore
  RunEngine->>WaitpointSystem: completeWaitpoint(waitpointId)
  WaitpointSystem->>CrossSeamGuard: consult(routeKind)
  CrossSeamGuard-->>WaitpointSystem: store decision
  WaitpointSystem->>RunStore: updateManyWaitpoints via resolved store
  RunStore-->>WaitpointSystem: COMPLETED waitpoint

Loading

Related PRs: None identified.

Suggested labels: area: run-engine, area: internal-packages

Suggested reviewers: matt-aitken, ericallam

Poem: 🐰 A rabbit hopped between two dens,
one for runs, one for their friends,
resolvers bridge the seam so wide,
stores now know just where to hide,
control-plane truths and run-scalar tales,
split clean across the database rails.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Description check	⚠️ Warning	The description does not follow the required template and is missing the Closes line plus Checklist, Testing, Changelog, and Screenshots sections.	Add the Closes # line and fill in the Checklist, Testing, Changelog, and Screenshots sections in the required template order.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title is concise and accurately summarizes the main run-store routing and run-engine seam changes.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch runops/pr04-store-engine

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[d-cs]

On the out-of-diff quit() shutdown comment: leaving this as-is. The shared try/catch predates this PR — this change only appends two new cleanup steps to the existing chain. Reworking shutdown into per-step isolation is orthogonal to the run-store routing seam and is better handled separately.

[pkg-pr-new]

Open in StackBlitz

@trigger.dev/build

npm i https://pkg.pr.new/@trigger.dev/build@d46aab2

trigger.dev

npm i https://pkg.pr.new/trigger.dev@d46aab2

@trigger.dev/core

npm i https://pkg.pr.new/@trigger.dev/core@d46aab2

@trigger.dev/python

npm i https://pkg.pr.new/@trigger.dev/python@d46aab2

@trigger.dev/react-hooks

npm i https://pkg.pr.new/@trigger.dev/react-hooks@d46aab2

@trigger.dev/redis-worker

npm i https://pkg.pr.new/@trigger.dev/redis-worker@d46aab2

@trigger.dev/rsc

npm i https://pkg.pr.new/@trigger.dev/rsc@d46aab2

@trigger.dev/schema-to-json

npm i https://pkg.pr.new/@trigger.dev/schema-to-json@d46aab2

@trigger.dev/sdk

npm i https://pkg.pr.new/@trigger.dev/sdk@d46aab2

commit: d46aab2

[coderabbitai]

Actionable comments posted: 1

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: 6d641654-57d5-483a-9738-2b47363d764c

📥 Commits

Reviewing files that changed from the base of the PR and between 8b126c6 and d46aab2.

⛔ Files ignored due to path filters (1)

• pnpm-lock.yaml is excluded by !**/pnpm-lock.yaml

📒 Files selected for processing (72)

• .server-changes/run-engine-store-routing-seam.md
• apps/webapp/app/presenters/v3/ApiBatchResultsPresenter.server.ts
• apps/webapp/test/presenters/ApiBatchResultsPresenter.split.test.ts
• apps/webapp/test/updateMetadataStoreRoutingHetero.test.ts
• internal-packages/run-engine/src/engine/controlPlaneResolver.ts
• internal-packages/run-engine/src/engine/errors.ts
• internal-packages/run-engine/src/engine/index.ts
• internal-packages/run-engine/src/engine/systems/batchSystem.test.ts
• internal-packages/run-engine/src/engine/systems/batchSystem.ts
• internal-packages/run-engine/src/engine/systems/checkpointSystem.ts
• internal-packages/run-engine/src/engine/systems/debounceSystem.test.ts
• internal-packages/run-engine/src/engine/systems/debounceSystem.ts
• internal-packages/run-engine/src/engine/systems/delayedRunSystem.test.ts
• internal-packages/run-engine/src/engine/systems/delayedRunSystem.ts
• internal-packages/run-engine/src/engine/systems/dequeueSystem.ts
• internal-packages/run-engine/src/engine/systems/enqueueSystem.test.ts
• internal-packages/run-engine/src/engine/systems/enqueueSystem.ts
• internal-packages/run-engine/src/engine/systems/executionSnapshotSystem.test.ts
• internal-packages/run-engine/src/engine/systems/executionSnapshotSystem.ts
• internal-packages/run-engine/src/engine/systems/pendingVersionSystem.test.ts
• internal-packages/run-engine/src/engine/systems/pendingVersionSystem.ts
• internal-packages/run-engine/src/engine/systems/runAttemptSystem.test.ts
• internal-packages/run-engine/src/engine/systems/runAttemptSystem.ts
• internal-packages/run-engine/src/engine/systems/systems.ts
• internal-packages/run-engine/src/engine/systems/ttlSystem.test.ts
• internal-packages/run-engine/src/engine/systems/ttlSystem.ts
• internal-packages/run-engine/src/engine/systems/waitpointSystem.test.ts
• internal-packages/run-engine/src/engine/systems/waitpointSystem.ts
• internal-packages/run-engine/src/engine/tests/blockEdgeResidency.test.ts
• internal-packages/run-engine/src/engine/tests/checkpointSystem.controlPlaneResolver.test.ts
• internal-packages/run-engine/src/engine/tests/checkpointSystemStore.test.ts
• internal-packages/run-engine/src/engine/tests/completeWaitpointCrossSeamGuard.test.ts
• internal-packages/run-engine/src/engine/tests/completeWaitpointReadResidency.test.ts
• internal-packages/run-engine/src/engine/tests/controlPlaneResolverInjectability.test.ts
• internal-packages/run-engine/src/engine/tests/datetimeWaitpointColocation.test.ts
• internal-packages/run-engine/src/engine/tests/debounce.test.ts
• internal-packages/run-engine/src/engine/tests/delayedRunSystem.controlPlaneResolver.test.ts
• internal-packages/run-engine/src/engine/tests/dequeueSystem.controlPlaneResolver.test.ts
• internal-packages/run-engine/src/engine/tests/dequeueSystem.recovery.controlPlaneResolver.test.ts
• internal-packages/run-engine/src/engine/tests/engineResidualInversions.controlPlaneResolver.test.ts
• internal-packages/run-engine/src/engine/tests/lifecycleRouter.test.ts
• internal-packages/run-engine/src/engine/tests/runAttemptSystem.controlPlaneResolver.test.ts
• internal-packages/run-engine/src/engine/tests/runStoreInjectability.test.ts
• internal-packages/run-engine/src/engine/tests/triggerCreateRouting.test.ts
• internal-packages/run-engine/src/engine/tests/ttl.test.ts
• internal-packages/run-engine/src/engine/tests/waitpointPublicRouter.test.ts
• internal-packages/run-engine/src/engine/tests/waitpointSystem.controlPlaneResolver.test.ts
• internal-packages/run-engine/src/engine/types.ts
• internal-packages/run-engine/src/index.ts
• internal-packages/run-store/package.json
• internal-packages/run-store/src/NoopRunStore.ts
• internal-packages/run-store/src/PostgresRunStore.batchProbeReadClient.test.ts
• internal-packages/run-store/src/PostgresRunStore.crossGenerationError.test.ts
• internal-packages/run-store/src/PostgresRunStore.dedicatedRepro.test.ts
• internal-packages/run-store/src/PostgresRunStore.dedicatedSelect.test.ts
• internal-packages/run-store/src/PostgresRunStore.dualSchema.test.ts
• internal-packages/run-store/src/PostgresRunStore.test.ts
• internal-packages/run-store/src/PostgresRunStore.ts
• internal-packages/run-store/src/PostgresRunStore.writeAtomicity.test.ts
• internal-packages/run-store/src/batchCompletionResidency.test.ts
• internal-packages/run-store/src/clientCompat.test.ts
• internal-packages/run-store/src/index.ts
• internal-packages/run-store/src/runOpsStore.flipWindowDuplicate.test.ts
• internal-packages/run-store/src/runOpsStore.forWaitpointCompletion.test.ts
• internal-packages/run-store/src/runOpsStore.idempotencyDedup.test.ts
• internal-packages/run-store/src/runOpsStore.mixedResidency.test.ts
• internal-packages/run-store/src/runOpsStore.readAfterWrite.test.ts
• internal-packages/run-store/src/runOpsStore.snapshots.test.ts
• internal-packages/run-store/src/runOpsStore.test.ts
• internal-packages/run-store/src/runOpsStore.ts
• internal-packages/run-store/src/runOpsStore.waitpoints.test.ts
• internal-packages/run-store/src/types.ts

💤 Files with no reviewable changes (1)

• internal-packages/run-store/src/NoopRunStore.ts

✅ Files skipped from review due to trivial changes (1)

• .server-changes/run-engine-store-routing-seam.md

🚧 Files skipped from review as they are similar to previous changes (51)

• internal-packages/run-engine/src/index.ts
• internal-packages/run-store/package.json
• internal-packages/run-engine/src/engine/tests/checkpointSystem.controlPlaneResolver.test.ts
• apps/webapp/app/presenters/v3/ApiBatchResultsPresenter.server.ts
• internal-packages/run-engine/src/engine/systems/pendingVersionSystem.ts
• internal-packages/run-engine/src/engine/systems/systems.ts
• internal-packages/run-engine/src/engine/errors.ts
• internal-packages/run-engine/src/engine/tests/controlPlaneResolverInjectability.test.ts
• internal-packages/run-engine/src/engine/tests/runStoreInjectability.test.ts
• internal-packages/run-store/src/PostgresRunStore.test.ts
• apps/webapp/test/presenters/ApiBatchResultsPresenter.split.test.ts
• internal-packages/run-store/src/PostgresRunStore.batchProbeReadClient.test.ts
• internal-packages/run-engine/src/engine/tests/waitpointSystem.controlPlaneResolver.test.ts
• internal-packages/run-store/src/PostgresRunStore.dedicatedSelect.test.ts
• internal-packages/run-engine/src/engine/systems/debounceSystem.ts
• internal-packages/run-engine/src/engine/tests/delayedRunSystem.controlPlaneResolver.test.ts
• internal-packages/run-engine/src/engine/tests/dequeueSystem.recovery.controlPlaneResolver.test.ts
• internal-packages/run-engine/src/engine/systems/delayedRunSystem.ts
• internal-packages/run-engine/src/engine/systems/enqueueSystem.test.ts
• internal-packages/run-engine/src/engine/types.ts
• internal-packages/run-engine/src/engine/tests/completeWaitpointReadResidency.test.ts
• internal-packages/run-engine/src/engine/tests/dequeueSystem.controlPlaneResolver.test.ts
• internal-packages/run-engine/src/engine/tests/completeWaitpointCrossSeamGuard.test.ts
• internal-packages/run-engine/src/engine/tests/waitpointPublicRouter.test.ts
• internal-packages/run-engine/src/engine/systems/checkpointSystem.ts
• internal-packages/run-engine/src/engine/systems/delayedRunSystem.test.ts
• internal-packages/run-engine/src/engine/systems/runAttemptSystem.test.ts
• internal-packages/run-engine/src/engine/systems/batchSystem.test.ts
• internal-packages/run-engine/src/engine/systems/enqueueSystem.ts
• internal-packages/run-engine/src/engine/systems/debounceSystem.test.ts
• internal-packages/run-store/src/PostgresRunStore.dualSchema.test.ts
• internal-packages/run-engine/src/engine/systems/executionSnapshotSystem.test.ts
• internal-packages/run-engine/src/engine/tests/debounce.test.ts
• internal-packages/run-store/src/PostgresRunStore.crossGenerationError.test.ts
• apps/webapp/test/updateMetadataStoreRoutingHetero.test.ts
• internal-packages/run-engine/src/engine/systems/batchSystem.ts
• internal-packages/run-engine/src/engine/tests/datetimeWaitpointColocation.test.ts
• internal-packages/run-engine/src/engine/tests/engineResidualInversions.controlPlaneResolver.test.ts
• internal-packages/run-engine/src/engine/systems/ttlSystem.test.ts
• internal-packages/run-engine/src/engine/systems/executionSnapshotSystem.ts
• internal-packages/run-engine/src/engine/tests/triggerCreateRouting.test.ts
• internal-packages/run-engine/src/engine/tests/runAttemptSystem.controlPlaneResolver.test.ts
• internal-packages/run-engine/src/engine/tests/lifecycleRouter.test.ts
• internal-packages/run-engine/src/engine/systems/waitpointSystem.test.ts
• internal-packages/run-engine/src/engine/tests/checkpointSystemStore.test.ts
• internal-packages/run-store/src/PostgresRunStore.dedicatedRepro.test.ts
• internal-packages/run-engine/src/engine/systems/pendingVersionSystem.test.ts
• internal-packages/run-engine/src/engine/tests/blockEdgeResidency.test.ts
• internal-packages/run-engine/src/engine/systems/dequeueSystem.ts
• internal-packages/run-engine/src/engine/index.ts
• internal-packages/run-engine/src/engine/systems/waitpointSystem.ts

📜 Review details

⏰ Context from checks skipped due to timeout. (28)

• GitHub Check: webapp / 🧪 Unit Tests: Webapp (3, 10)
• GitHub Check: internal / 🧪 Unit Tests: Internal (12, 12)
• GitHub Check: internal / 🧪 Unit Tests: Internal (3, 12)
• GitHub Check: internal / 🧪 Unit Tests: Internal (1, 12)
• GitHub Check: packages / 🧪 Unit Tests: Packages (3, 3)
• GitHub Check: internal / 🧪 Unit Tests: Internal (9, 12)
• GitHub Check: internal / 🧪 Unit Tests: Internal (11, 12)
• GitHub Check: internal / 🧪 Unit Tests: Internal (6, 12)
• GitHub Check: internal / 🧪 Unit Tests: Internal (5, 12)
• GitHub Check: webapp / 🧪 Unit Tests: Webapp (10, 10)
• GitHub Check: internal / 🧪 Unit Tests: Internal (7, 12)
• GitHub Check: webapp / 🧪 Unit Tests: Webapp (9, 10)
• GitHub Check: webapp / 🧪 Unit Tests: Webapp (6, 10)
• GitHub Check: webapp / 🧪 Unit Tests: Webapp (1, 10)
• GitHub Check: internal / 🧪 Unit Tests: Internal (2, 12)
• GitHub Check: internal / 🧪 Unit Tests: Internal (10, 12)
• GitHub Check: internal / 🧪 Unit Tests: Internal (4, 12)
• GitHub Check: internal / 🧪 Unit Tests: Internal (8, 12)
• GitHub Check: webapp / 🧪 Unit Tests: Webapp (7, 10)
• GitHub Check: webapp / 🧪 Unit Tests: Webapp (8, 10)
• GitHub Check: webapp / 🧪 Unit Tests: Webapp (5, 10)
• GitHub Check: webapp / 🧪 Unit Tests: Webapp (4, 10)
• GitHub Check: webapp / 🧪 Unit Tests: Webapp (2, 10)
• GitHub Check: e2e / 🧪 CLI v3 tests (blacksmith-4vcpu-windows-2025 - npm)
• GitHub Check: e2e / 🧪 CLI v3 tests (blacksmith-4vcpu-windows-2025 - pnpm)
• GitHub Check: packages / 🧪 Unit Tests: Packages (1, 3)
• GitHub Check: e2e-webapp / 🧪 E2E Tests: Webapp
• GitHub Check: typecheck / typecheck

⚠️ CI failures not shown inline (4)

GitHub Actions: 📝 CLAUDE.md Audit / 0_audit.txt: feat(run-ops): run-store routing seam + run-engine read seams

Conclusion: failure

View job details

alt@3.0.0-beta.46 -> `@trigger.dev/yalt`@3.0.0-beta.46
  * [new tag]             `@trigger.dev/yalt`@3.0.0-beta.47 -> `@trigger.dev/yalt`@3.0.0-beta.47
  * [new tag]             `@trigger.dev/yalt`@3.0.0-beta.48 -> `@trigger.dev/yalt`@3.0.0-beta.48
  * [new tag]             `@trigger.dev/yalt`@3.0.0-beta.49 -> `@trigger.dev/yalt`@3.0.0-beta.49
  * [new tag]             `@trigger.dev/yalt`@3.0.0-beta.5 -> `@trigger.dev/yalt`@3.0.0-beta.5
  * [new tag]             `@trigger.dev/yalt`@3.0.0-beta.50 -> `@trigger.dev/yalt`@3.0.0-beta.50
  * [new tag]             `@trigger.dev/yalt`@3.0.0-beta.51 -> `@trigger.dev/yalt`@3.0.0-beta.51
  * [new tag]             `@trigger.dev/yalt`@3.0.0-beta.52 -> `@trigger.dev/yalt`@3.0.0-beta.52
  * [new tag]             `@trigger.dev/yalt`@3.0.0-beta.53 -> `@trigger.dev/yalt`@3.0.0-beta.53
  * [new tag]             `@trigger.dev/yalt`@3.0.0-beta.55 -> `@trigger.dev/yalt`@3.0.0-beta.55
  * [new tag]             `@trigger.dev/yalt`@3.0.0-beta.6 -> `@trigger.dev/yalt`@3.0.0-beta.6
  * [new tag]             `@trigger.dev/yalt`@3.0.0-beta.7 -> `@trigger.dev/yalt`@3.0.0-beta.7
  * [new tag]             build-alert-hotfix.rc1      -> build-alert-hotfix.rc1
  * [new tag]             build-alert-hotfix.rc2      -> build-alert-hotfix.rc2
  * [new tag]             build-arm-builds-rc.1       -> build-arm-builds-rc.1
  * [new tag]             build-arm-builds-rc.2       -> build-arm-builds-rc.2
  * [new tag]             build-arm-builds-rc.3       -> build-arm-builds-rc.3
  * [new tag]             build-batchid-carryover-rc.0 -> build-batchid-carryover-rc.0
  * [new tag]             build-batching-rc.1         -> build-batching-rc.1
  * [new tag]             build-batching-rc.2         -> build-batching-rc.2
  * [new tag]             build-billing-0.0.1         -> build-billing-0.0.1
  * [new tag]             build-billing-0.0.2         -> build-billing-0.0.2
  * [new tag]             build-billing-0.0.3         -> build-billing-0.0.3
  * [new tag]             build-buildinfo-rc.0        -> b...

---

GitHub Actions: 📝 CLAUDE.md Audit / audit: feat(run-ops): run-store routing seam + run-engine read seams

Conclusion: failure

View job details

##[group]Run anthropics/claude-code-action@428971d2ecd6e3a7cb0ee0da2a3a8b33fdb3678d
 with:
   anthropic_***REDACTED***
   use_sticky_comment: true
   allowed_bots: devin-ai-integration[bot]
   claude_args: --max-turns 25
--model claude-opus-4-8
--allowedTools "Read,Glob,Grep,Bash(git diff:*)"
   prompt: You are reviewing a PR to check whether any CLAUDE.md files or .claude/rules/ files need updating.
## Your task
1. Run `git diff origin/main...HEAD --name-only` to see which files changed in this PR.
2. For each changed directory, check if there's a CLAUDE.md in that directory or a parent directory.
3. Determine if any CLAUDE.md or .claude/rules/ file should be updated based on the changes. Consider:
   - New files/directories that aren't covered by existing documentation
   - Changed architecture or patterns that contradict current CLAUDE.md guidance
   - New dependencies, services, or infrastructure that Claude should know about
   - Renamed or moved files that are referenced in CLAUDE.md
   - Changes to build commands, test patterns, or development workflows
## Response format
If NO updates are needed, respond with exactly:
✅ CLAUDE.md files look current for this PR.
If updates ARE needed, respond with a short list:
📝 **CLAUDE.md updates suggested:**
- `path/to/CLAUDE.md`: [what should be added/changed]
- `.claude/rules/file.md`: [what should be added/changed]
Keep suggestions specific and brief. Only flag things that would actually mislead Claude in future sessions.
Do NOT suggest updates for trivial changes (bug fixes, small refactors within existing patterns).
Do NOT suggest creating new CLAUDE.md files - only updates to existing ones.
   trigger_phrase: `@claude`
   label_trigger: claude
   branch_prefix: claude/
   use_bedrock: false
   use_vertex: false
   use_foundry: false
   classify_inline_comments: true
   use_commit_signing: false
   bot_id: 41898282
   bot_name: claude[bot]
   track_progress: false
   include_fix_links: true
   display_report: false...

---

GitHub Actions: 🔎 REVIEW.md Drift Audit / 0_audit.txt: feat(run-ops): run-store routing seam + run-engine read seams

Conclusion: failure

View job details

ild-legacy-run-engine.fix3
  * [new tag]             build-manual-checkpoints.rc1 -> build-manual-checkpoints.rc1
  * [new tag]             build-metadata-upgrade-logging.rc1 -> build-metadata-upgrade-logging.rc1
  * [new tag]             build-metadata-upgrade-logging.rc2 -> build-metadata-upgrade-logging.rc2
  * [new tag]             build-metadata-upgrade-logging.rc3 -> build-metadata-upgrade-logging.rc3
  * [new tag]             build-new-build-system.rc.1 -> build-new-build-system.rc.1
  * [new tag]             build-otel-upgrade-rc.0     -> build-otel-upgrade-rc.0
  * [new tag]             build-otel-upgrade-rc.1     -> build-otel-upgrade-rc.1
  * [new tag]             build-pre-pull-deployments-rc.1 -> build-pre-pull-deployments-rc.1
  * [new tag]             build-prod-rescue-rc.1      -> build-prod-rescue-rc.1
  * [new tag]             build-rate-limiter-fix-rc.1 -> build-rate-limiter-fix-rc.1
  * [new tag]             build-re2.rc0               -> build-re2.rc0
  * [new tag]             build-realtime-v2-stream-fix -> build-realtime-v2-stream-fix
  * [new tag]             build-realtime-v2-stream-fix-2 -> build-realtime-v2-stream-fix-2
  * [new tag]             build-realtime-v2-stream-fix-3 -> build-realtime-v2-stream-fix-3
  * [new tag]             build-realtime-v2-stream-fix-4 -> build-realtime-v2-stream-fix-4
  * [new tag]             build-realtime-v2-stream-fix-5 -> build-realtime-v2-stream-fix-5
  * [new tag]             build-realtimestreams-dedupe -> build-realtimestreams-dedupe
  * [new tag]             build-registry-maintenance-rc.1 -> build-registry-maintenance-rc.1
  * [new tag]             build-registry-maintenance-rc.2 -> build-registry-maintenance-rc.2
  * [new tag]             build-remote-ecr-rc.0       -> build-remote-ecr-rc.0
  * [new tag]             build-reschedule-hotfix.rc1 -> build-reschedule-hotfix.rc1
  * [new tag]             build-resume-fixes.rc1      -> build-resume-fixes.rc1
  * [new tag]             build-resume-fix...

---

GitHub Actions: 🔎 REVIEW.md Drift Audit / audit: feat(run-ops): run-store routing seam + run-engine read seams

Conclusion: failure

View job details

##[group]Run anthropics/claude-code-action@428971d2ecd6e3a7cb0ee0da2a3a8b33fdb3678d
 with:
   anthropic_***REDACTED***
   use_sticky_comment: true
   allowed_bots: devin-ai-integration[bot]
   claude_args: --max-turns 30
--allowedTools "Read,Glob,Grep,Bash(git diff:*)"
   prompt: You are auditing this PR for drift against `.claude/REVIEW.md`.
## Context
`.claude/REVIEW.md` is the repo's source of truth for what AI / agent code reviewers should treat as critical findings (rolling-deploy safety, hot-table indexes, recovery-path queries, testcontainers usage, Lua versioning, etc.). It is consumed by review agents to calibrate severity. If REVIEW.md goes stale, every future agent review degrades.
## Strategy — read this first
You have a hard turn budget. Spend it on signal, not coverage. The audit is allowed to miss things; it is NOT allowed to time out.
1. Read `.claude/REVIEW.md` once, in full.
2. Run `git diff origin/main...HEAD --name-only` to get the list of changed files. Do NOT read the diff content yet.
3. Scan the file-list for relevance to REVIEW.md scope. Relevance signals: changes to Prisma schema, Redis / queue / Lua code, hot tables, recovery / restart loops, new packages, deletions of paths REVIEW.md cites. Skim everything else.
4. Open at most **5 files** total — only the ones most likely to surface a real signal. If nothing in the file-list looks relevant to any REVIEW.md rule, do NOT read any files; go straight to the verdict.
5. Form a verdict and stop. Do not exhaust the turn budget exploring.
Large PRs (>50 files changed) are a strong signal to be MORE selective, not more thorough. Pick 3-5 files at most.
## What to look for
- **Stale references** — does any REVIEW.md rule cite a file, directory, function, table, Prisma model, or package name that has been removed or renamed in this PR (or is already gone from `main`)?
- **Contradictions** — does code in this PR clearly violate a current REVIEW.md rule? (Don't re-review the PR. Only flag if REVIE...

🧰 Additional context used

📓 Path-based instructions (10)

**/*.{ts,tsx}

📄 CodeRabbit inference engine (.github/copilot-instructions.md)

**/*.{ts,tsx}: Use types over interfaces for TypeScript
Avoid using enums; prefer string unions or const objects instead

Files:

• internal-packages/run-engine/src/engine/tests/ttl.test.ts
• internal-packages/run-engine/src/engine/systems/ttlSystem.ts
• internal-packages/run-engine/src/engine/controlPlaneResolver.ts
• internal-packages/run-engine/src/engine/systems/runAttemptSystem.ts

**/*.{ts,tsx,js,jsx}

📄 CodeRabbit inference engine (.github/copilot-instructions.md)

Use function declarations instead of default exports

Files:

• internal-packages/run-engine/src/engine/tests/ttl.test.ts
• internal-packages/run-engine/src/engine/systems/ttlSystem.ts
• internal-packages/run-engine/src/engine/controlPlaneResolver.ts
• internal-packages/run-engine/src/engine/systems/runAttemptSystem.ts

**/*.{test,spec}.{ts,tsx}

📄 CodeRabbit inference engine (.github/copilot-instructions.md)

Use vitest for all tests in the Trigger.dev repository

Files:

• internal-packages/run-engine/src/engine/tests/ttl.test.ts

**/*.ts

📄 CodeRabbit inference engine (.cursor/rules/otel-metrics.mdc)

**/*.ts: When creating or editing OTEL metrics (counters, histograms, gauges), ensure metric attributes have low cardinality by using only enums, booleans, bounded error codes, or bounded shard IDs
Do not use high-cardinality attributes in OTEL metrics such as UUIDs/IDs (envId, userId, runId, projectId, organizationId), unbounded integers (itemCount, batchSize, retryCount), timestamps (createdAt, startTime), or free-form strings (errorMessage, taskName, queueName)
When exporting OTEL metrics via OTLP to Prometheus, be aware that the exporter automatically adds unit suffixes to metric names (e.g., 'my_duration_ms' becomes 'my_duration_ms_milliseconds', 'my_counter' becomes 'my_counter_total'). Account for these transformations when writing Grafana dashboards or Prometheus queries

Files:

• internal-packages/run-engine/src/engine/tests/ttl.test.ts
• internal-packages/run-engine/src/engine/systems/ttlSystem.ts
• internal-packages/run-engine/src/engine/controlPlaneResolver.ts
• internal-packages/run-engine/src/engine/systems/runAttemptSystem.ts

internal-packages/run-engine/src/engine/tests/**/*.test.ts

📄 CodeRabbit inference engine (internal-packages/run-engine/CLAUDE.md)

Implement tests for RunEngine in src/engine/tests/ using testcontainers for Redis and PostgreSQL containerization

Files:

• internal-packages/run-engine/src/engine/tests/ttl.test.ts

**/*.{ts,tsx,js,jsx,mts,cts,mjs,cjs}

📄 CodeRabbit inference engine (CLAUDE.md)

**/*.{ts,tsx,js,jsx,mts,cts,mjs,cjs}: Use pnpm run typecheck for changes in apps (apps/*) and internal packages (internal-packages/*), and never use build to verify those changes.
Use Vitest for tests, and never mock anything; use testcontainers instead.
Prefer static imports over dynamic import(), and only use dynamic imports for unresolved circular dependencies, genuine code-splitting needs, or conditional runtime loading.

Files:

• internal-packages/run-engine/src/engine/tests/ttl.test.ts
• internal-packages/run-engine/src/engine/systems/ttlSystem.ts
• internal-packages/run-engine/src/engine/controlPlaneResolver.ts
• internal-packages/run-engine/src/engine/systems/runAttemptSystem.ts

**/*.{test,spec}.{ts,tsx,js,jsx,mts,cts,mjs,cjs}

📄 CodeRabbit inference engine (CLAUDE.md)

Place test files next to the source files they cover (for example, MyService.ts -> MyService.test.ts).

Files:

• internal-packages/run-engine/src/engine/tests/ttl.test.ts

**/*.{ts,tsx,js,jsx,mts,cts,mjs,cjs,md,mdx}

📄 CodeRabbit inference engine (CLAUDE.md)

Always import from @trigger.dev/sdk when writing Trigger.dev tasks; never use @trigger.dev/sdk/v3 or deprecated client.defineJob.

Files:

• internal-packages/run-engine/src/engine/tests/ttl.test.ts
• internal-packages/run-engine/src/engine/systems/ttlSystem.ts
• internal-packages/run-engine/src/engine/controlPlaneResolver.ts
• internal-packages/run-engine/src/engine/systems/runAttemptSystem.ts

**/*.{test,spec}.{js,jsx,ts,tsx}

📄 CodeRabbit inference engine (AGENTS.md)

**/*.{test,spec}.{js,jsx,ts,tsx}: Keep test files beside the files under test and write tests with descriptive describe and it blocks.
Avoid mocks or stubs in tests, and use helpers from @internal/testcontainers when Redis or Postgres are needed.

Files:

• internal-packages/run-engine/src/engine/tests/ttl.test.ts

internal-packages/run-engine/src/engine/systems/**/*.ts

📄 CodeRabbit inference engine (internal-packages/run-engine/CLAUDE.md)

Integrate OpenTelemetry tracer and meter instrumentation in RunEngine systems for observability

Files:

• internal-packages/run-engine/src/engine/systems/ttlSystem.ts
• internal-packages/run-engine/src/engine/systems/runAttemptSystem.ts

🧠 Learnings (11)

📚 Learning: 2026-03-22T13:26:12.060Z

Learnt from: ericallam
Repo: triggerdotdev/trigger.dev PR: 3244
File: apps/webapp/app/components/code/TextEditor.tsx:81-86
Timestamp: 2026-03-22T13:26:12.060Z
Learning: In the triggerdotdev/trigger.dev codebase, do not flag `navigator.clipboard.writeText(...)` calls for `missing-await`/`unhandled-promise` issues. These clipboard writes are intentionally invoked without `await` and without `catch` handlers across the project; keep that behavior consistent when reviewing TypeScript/TSX files (e.g., usages like in `apps/webapp/app/components/code/TextEditor.tsx`).

Applied to files:

• internal-packages/run-engine/src/engine/tests/ttl.test.ts
• internal-packages/run-engine/src/engine/systems/ttlSystem.ts
• internal-packages/run-engine/src/engine/controlPlaneResolver.ts
• internal-packages/run-engine/src/engine/systems/runAttemptSystem.ts

📚 Learning: 2026-03-22T19:24:14.403Z

Learnt from: matt-aitken
Repo: triggerdotdev/trigger.dev PR: 3187
File: apps/webapp/app/v3/services/alerts/deliverErrorGroupAlert.server.ts:200-204
Timestamp: 2026-03-22T19:24:14.403Z
Learning: In the triggerdotdev/trigger.dev codebase, webhook URLs are not expected to contain embedded credentials/secrets (e.g., fields like `ProjectAlertWebhookProperties` should only hold credential-free webhook endpoints). During code review, if you see logging or inclusion of raw webhook URLs in error messages, do not automatically treat it as a credential-leak/secrets-in-logs issue by default—first verify the URL does not contain embedded credentials (for example, no username/password in the URL, no obvious secret/token query params or fragments). If the URL is credential-free per this project’s conventions, allow the logging.

Applied to files:

• internal-packages/run-engine/src/engine/tests/ttl.test.ts
• internal-packages/run-engine/src/engine/systems/ttlSystem.ts
• internal-packages/run-engine/src/engine/controlPlaneResolver.ts
• internal-packages/run-engine/src/engine/systems/runAttemptSystem.ts

📚 Learning: 2026-05-18T08:21:27.694Z

Learnt from: d-cs
Repo: triggerdotdev/trigger.dev PR: 3632
File: apps/webapp/sentry.server.ts:4-21
Timestamp: 2026-05-18T08:21:27.694Z
Learning: When handling Prisma error P1001 ("Can't reach database server") in TypeScript, don’t assume a single error shape. Prisma can surface P1001 via two different error classes/fields: `PrismaClientKnownRequestError` exposes it as `err.code === "P1001"` (common during mid-query connection drops), while `PrismaClientInitializationError` exposes it as `err.errorCode === "P1001"` (common on client startup failure). Therefore, predicates should use `err.code === "P1001" || err.errorCode === "P1001"`. Do not flag `err.code === "P1001"` as “unreachable/never matches,” as it is expected in production.

Applied to files:

• internal-packages/run-engine/src/engine/tests/ttl.test.ts
• internal-packages/run-engine/src/engine/systems/ttlSystem.ts
• internal-packages/run-engine/src/engine/controlPlaneResolver.ts
• internal-packages/run-engine/src/engine/systems/runAttemptSystem.ts

📚 Learning: 2026-05-18T08:21:27.694Z

Learnt from: d-cs
Repo: triggerdotdev/trigger.dev PR: 3632
File: apps/webapp/sentry.server.ts:4-21
Timestamp: 2026-05-18T08:21:27.694Z
Learning: When handling Prisma errors for P1001 ("Can't reach database server"), do not assume it only appears under a single property name. Prisma may surface P1001 via either `PrismaClientKnownRequestError` (`err.code === "P1001"`, e.g., mid-query connection drops) or `PrismaClientInitializationError` (`err.errorCode === "P1001"`, e.g., client startup connection failure). To reliably detect the condition, check `err.code === "P1001" || err.errorCode === "P1001"`, and avoid review rules that would incorrectly flag `err.code === "P1001"` as unreachable/never-matching.

Applied to files:

• internal-packages/run-engine/src/engine/tests/ttl.test.ts
• internal-packages/run-engine/src/engine/systems/ttlSystem.ts
• internal-packages/run-engine/src/engine/controlPlaneResolver.ts
• internal-packages/run-engine/src/engine/systems/runAttemptSystem.ts

📚 Learning: 2026-06-13T19:53:13.759Z

Learnt from: ericallam
Repo: triggerdotdev/trigger.dev PR: 3937
File: packages/trigger-sdk/skills/realtime-and-frontend/SKILL.md:258-260
Timestamp: 2026-06-13T19:53:13.759Z
Learning: When reviewing code that uses `trigger.dev/react-hooks`’s `useRealtimeRun`, preserve the call signature where the first argument is the full realtime handle object (not `handle.id`). This is intentional to maintain type-safety and is consistent with the official docs; do not suggest changing the first argument from the handle object to `handle.id`.

Applied to files:

• internal-packages/run-engine/src/engine/tests/ttl.test.ts
• internal-packages/run-engine/src/engine/systems/ttlSystem.ts
• internal-packages/run-engine/src/engine/controlPlaneResolver.ts
• internal-packages/run-engine/src/engine/systems/runAttemptSystem.ts

📚 Learning: 2026-06-17T17:13:49.929Z

Learnt from: matt-aitken
Repo: triggerdotdev/trigger.dev PR: 3948
File: apps/webapp/app/routes/_app.orgs.$organizationSlug.projects.$projectParam.env.$envParam.bulk-actions.$bulkActionParam/route.tsx:48-62
Timestamp: 2026-06-17T17:13:49.929Z
Learning: In triggerdotdev/trigger.dev, within `dashboardLoader`/`dashboardAction` (or similar context resolver code) whenever you resolve an organization ID from an organization slug for RBAC/enterprise authorization scope, always read from the primary Prisma client (`prisma`), not `$replica`. Using `$replica` can hit replica-lag and cause the RBAC lookup/authorization to run without the correct org scope (bypassing intended role enforcement). Implement the slug→org lookup with `prisma.organization.findFirst(...)` (or equivalent primary-client query) and add an inline comment documenting why the primary client is required (replica lag could lead to unscoped RBAC checks).

Applied to files:

• internal-packages/run-engine/src/engine/tests/ttl.test.ts
• internal-packages/run-engine/src/engine/systems/ttlSystem.ts
• internal-packages/run-engine/src/engine/controlPlaneResolver.ts
• internal-packages/run-engine/src/engine/systems/runAttemptSystem.ts

📚 Learning: 2026-06-23T13:04:21.413Z

Learnt from: carderne
Repo: triggerdotdev/trigger.dev PR: 4023
File: apps/webapp/app/services/upsertBranch.server.ts:14-18
Timestamp: 2026-06-23T13:04:21.413Z
Learning: In TypeScript, it’s valid to `import { type X }` and then use `typeof X` in a type-only position, e.g. `type Alias = z.infer<typeof X>`. The `type` modifier suppresses the runtime import, but the type checker still has the full exported type so `z.infer<typeof X>` can resolve correctly. In code reviews, don’t flag this as a TypeScript compile error as long as `typeof X` is used in a type context (e.g., with `z.infer`, `type` aliases, generics), not as a runtime value.

Applied to files:

• internal-packages/run-engine/src/engine/tests/ttl.test.ts
• internal-packages/run-engine/src/engine/systems/ttlSystem.ts
• internal-packages/run-engine/src/engine/controlPlaneResolver.ts
• internal-packages/run-engine/src/engine/systems/runAttemptSystem.ts

📚 Learning: 2026-05-18T14:40:02.173Z

Learnt from: ericallam
Repo: triggerdotdev/trigger.dev PR: 3658
File: packages/core/src/v3/realtimeStreams/manager.test.ts:1-147
Timestamp: 2026-05-18T14:40:02.173Z
Learning: In the triggerdotdev/trigger.dev repo, the policy “Never mock anything — use testcontainers instead” should only be enforced for integration tests that interact with real external services (e.g., Redis, Postgres) via actual infrastructure. For unit tests that exercise pure in-memory logic (e.g., cache semantics) it is OK to stub collaborators such as `ApiClient` using Vitest (`vi.fn()`) to assert call counts or control behavior. Do not flag `vi.fn()`-based `ApiClient` stubs in unit tests as violations of the testcontainers policy.

Applied to files:

• internal-packages/run-engine/src/engine/tests/ttl.test.ts

📚 Learning: 2026-06-04T18:16:35.386Z

Learnt from: nicktrn
Repo: triggerdotdev/trigger.dev PR: 3836
File: apps/supervisor/src/backpressure/backpressureMonitor.ts:3-5
Timestamp: 2026-06-04T18:16:35.386Z
Learning: When reviewing TypeScript in this repo, apply the rule “prefer type aliases over interfaces” only to data/object shapes and union/intersection type modeling. If an interface is being used as a behavioral contract for collaborators to implement (e.g., method-shape interfaces that define required behavior, such as `BackpressureLogger` / `BackpressureSignalSource` in `apps/supervisor/src/backpressure/backpressureMonitor.ts`), keep it as an `interface` and do not flag it as a type-alias-vs-interface violation.

Applied to files:

• internal-packages/run-engine/src/engine/tests/ttl.test.ts
• internal-packages/run-engine/src/engine/systems/ttlSystem.ts
• internal-packages/run-engine/src/engine/controlPlaneResolver.ts
• internal-packages/run-engine/src/engine/systems/runAttemptSystem.ts

📚 Learning: 2026-06-09T17:58:04.699Z

Learnt from: 0ski
Repo: triggerdotdev/trigger.dev PR: 3879
File: apps/webapp/app/models/vercelIntegration.server.ts:619-630
Timestamp: 2026-06-09T17:58:04.699Z
Learning: In this codebase, outbound raw `fetch` calls should typically rely on Node/undici’s default request timeout (about ~300s) rather than adding a per-call `AbortController` + `setTimeout` wrapper inside individual functions (e.g. in files like `apps/webapp/app/models/vercelIntegration.server.ts`). During code review, do not flag the absence of a per-call timeout on a single `fetch` as an issue; if per-call timeouts are needed, they should be implemented via a codebase-wide convention (e.g., a shared fetch wrapper or documented pattern) rather than ad-hoc per-function changes.

Applied to files:

• internal-packages/run-engine/src/engine/tests/ttl.test.ts
• internal-packages/run-engine/src/engine/systems/ttlSystem.ts
• internal-packages/run-engine/src/engine/controlPlaneResolver.ts
• internal-packages/run-engine/src/engine/systems/runAttemptSystem.ts

📚 Learning: 2026-06-16T09:19:47.637Z

Learnt from: d-cs
Repo: triggerdotdev/trigger.dev PR: 3960
File: apps/webapp/test/prismaInfrastructureErrorCapture.test.ts:0-0
Timestamp: 2026-06-16T09:19:47.637Z
Learning: In this repo’s Vitest setup, `vitest.config.ts` uses `globals: true`, so identifiers like `vi`, `describe`, `it`, and `expect` are available as globals in Vitest test files. During code review, do not flag missing `vi`/`describe`/`it`/`expect` imports as a runtime error or correctness issue when they’re used in `*.test.ts/tsx` or `*.spec.ts/tsx` files. Explicit imports are still preferred for consistency, but they’re not required for runtime behavior.

Applied to files:

• internal-packages/run-engine/src/engine/tests/ttl.test.ts

🔇 Additional comments (6)

internal-packages/run-engine/src/engine/controlPlaneResolver.ts (2)

34-186: LGTM!

---

188-341: 🩺 Stability & Availability

Drop the env-scoping concern. BackgroundWorker.id is globally unique, so the id-only worker lookups don’t cross environments; the prior dequeue flow used the same pattern. The orderBy on the unique-id lookup is redundant.

			> Likely an incorrect or invalid review comment.

internal-packages/run-engine/src/engine/systems/ttlSystem.ts (1)

28-28: LGTM!

Also applies to: 84-84, 104-106, 119-121

internal-packages/run-engine/src/engine/tests/ttl.test.ts (1)

8-11: LGTM!

Also applies to: 1614-1733

internal-packages/run-engine/src/engine/systems/runAttemptSystem.ts (2)

208-226: LGTM!

Also applies to: 237-240, 272-272, 285-289, 317-317, 527-537, 549-552, 604-613, 835-835, 1077-1095, 1442-1454, 1638-1638, 1652-1707, 1222-1222

---

393-494: 🩺 Stability & Availability

Transaction wiring looks correct; verify runInTransaction semantics for the in-tx worker.ack.

The atomic bump + snapshot via runStore.runInTransaction wrapped in tryCatch is sound. Note the worker.ack(\expireRun:...`) (Line 478) runs inside the DB transaction callback; being a Redis side-effect it won't roll back if the transaction later fails. This mirrors prior behavior so it isn't a regression, but confirm the inner writes (store.startAttempt, createExecutionSnapshot(..., store)) are bound to the tx-scoped store/tx` and not the router, as the comment asserts.

Also, the error log at Line 486 still reads "prisma.$transaction error" though this now routes through runInTransaction.

[coderabbitai]

🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Inspect resolver contract: can resolveEnv/resolveAuthenticatedEnv legitimately return null for a live run's env?
fd controlPlaneResolver.ts internal-packages/run-engine/src/engine --exec cat -n

Repository: triggerdotdev/trigger.dev

Length of output: 13091

---

🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Inspect the surrounding control-flow in runAttemptSystem and all env resolver call sites.
file="internal-packages/run-engine/src/engine/systems/runAttemptSystem.ts"

echo "== call sites =="
rg -n "resolveEnv\\(|resolveAuthenticatedEnv\\(" "$file"

echo
echo "== line 470-560 =="
sed -n '470,560p' "$file"

echo
echo "== line 760-820 =="
sed -n '760,820p' "$file"

echo
echo "== line 1000-1065 =="
sed -n '1000,1065p' "$file"

echo
echo "== line 1410-1475 =="
sed -n '1410,1475p' "$file"

echo
echo "== line 1620-1685 =="
sed -n '1620,1685p' "$file"

Repository: triggerdotdev/trigger.dev

Length of output: 13069

---

🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Inspect runQueue acknowledgement and side-effect helpers referenced by the terminal paths.
root="internal-packages/run-engine/src/engine"

rg -n "acknowledgeMessage\\(|runSucceeded\\(|`#finalizeRun`\\(|completeAttemptSuccess\\(|waitpoint|event emission|emit" "$root" -g '*.ts'

echo
echo "== queue implementation matches =="
fd -a "runQueue" "$root" -t f

echo
echo "== likely queue files =="
rg -n "acknowledgeMessage\\(" "$root" -g '*.ts'

Repository: triggerdotdev/trigger.dev

Length of output: 50382

---

🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Look for any explicit guarantees/comments around run/env presence and transaction ordering.
file="internal-packages/run-engine/src/engine/systems/runAttemptSystem.ts"

rg -n "finished-but-unacknowledged|unacknowledged|Task run environment not found|assertEnvExists|resolveAuthenticatedEnv|completeAttemptSuccess|completeAttemptFailure|finalizeRun|acknowledgeMessage" "$file"

echo
echo "== repo-wide mentions of the error string =="
rg -n "Task run environment not found" .

Repository: triggerdotdev/trigger.dev

Length of output: 3295

---

🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Inspect the control-plane resolver tests for null-env scenarios and the terminal-path behavior.
testfile="internal-packages/run-engine/src/engine/tests/runAttemptSystem.controlPlaneResolver.test.ts"
systemfile="internal-packages/run-engine/src/engine/systems/runAttemptSystem.ts"

echo "== controlPlaneResolver test snippets =="
sed -n '330,390p' "$testfile"

echo
echo "== completeAttemptSuccess terminal path =="
sed -n '730,860p' "$systemfile"

echo
echo "== cancel/fail terminal path details =="
sed -n '1430,1735p' "$systemfile"

Repository: triggerdotdev/trigger.dev

Length of output: 16415

---

Resolve env before the terminal write. resolveEnv/resolveAuthenticatedEnv now runs after the run is already marked finished in these paths. If it returns null, the 404 skips queue acking, waitpoint completion, event emission, and #finalizeRun, which can leave a finished run unacknowledged. Move the lookup earlier where possible, or keep the cleanup running even when env lookup fails.