You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Local verification and CI are drifting — they run different commands with different flags, so "green locally" and "green in CI" are not the same verdict. Inventory:
No lint job in CI.make check (pre-commit: ruff-format, ruff, prettier) only runs as a local git hook. A contributor without hooks installed bypasses formatting/linting entirely and CI stays green.
pytest flag drift.ci.yml runs uv run pytest tests/ -v --cov=src --cov-report=term --cov-report=xml --junitxml=... while make test runs uv run pytest tests/ --cov=src --cov-report=term-missing.
Dependency-group mismatch. CI installs --group agents --group dev; make init adds --group eval.
AGENTS.md (#263) establishes make targets as the canonical verification interface ("scripts to rule them all"). CI should consume the same interface so the flags live in exactly one place.
Solution
Agreed design (grilled 2026-09-18):
Split make targets to preserve CI parallelism.make test becomes test: test-python test-ts (local behavior unchanged); CI jobs call make test-python, make test-ts, make synth respectively — one source of truth per command, parallel jobs preserved.
CI-only pytest output flags via PYTEST_ADDOPTS. CI sets PYTEST_ADDOPTS="-v --cov-report=xml --junitxml=test-results.xml" and calls the same make test-python — parametrization of one target, not a forked test-ci target.
Add a 4th CI job: lint, calling make check. Cache ~/.cache/pre-commit keyed on OS + Python version + hash of .pre-commit-config.yaml (the pattern from pre-commit's own docs; the pre-commit/action wrapper is in maintenance mode, and pre-commit.ci likely can't get org app approval on aws-samples). Intentional, accepted drift: machine-local wrappers (e.g. Code Defender) run only locally — CI runs exactly the repo's .pre-commit-config.yaml. Do not "fix" this. CI is check-only: fixer hooks (ruff-format, prettier) modify the runner's checkout but nothing commits them — the job fails and the fixes are discarded. Auto-committing fixes from CI is off the table (fork PRs: GITHUB_TOKEN can't push to contributor forks), so on failure the job just surfaces a "run make check locally" hint — pre-commit's own output already names the failing hooks, no extra diff step needed.
Dependency groups stay per-target — intentional, not drift. Each CI job installs only the groups its make target needs. The eval group is excluded from CI by policy (evals cost real Bedrock tokens, run only on explicit human request — see AGENTS.md) and mechanically (no AWS credentials are wired to the runners).
Acceptance
No raw pytest/vitest/synth invocations left in ci.yml — jobs call make check, make test-python, make test-ts, make synth
Local make check && make test && make synth and CI produce the same pass/fail verdict for the same tree
make test still runs the full suite locally with unchanged behavior
Why is this needed?
Local verification and CI are drifting — they run different commands with different flags, so "green locally" and "green in CI" are not the same verdict. Inventory:
make check(pre-commit: ruff-format, ruff, prettier) only runs as a local git hook. A contributor without hooks installed bypasses formatting/linting entirely and CI stays green.ci.ymlrunsuv run pytest tests/ -v --cov=src --cov-report=term --cov-report=xml --junitxml=...whilemake testrunsuv run pytest tests/ --cov=src --cov-report=term-missing.cdk-nag-testjob runsnpm run synthdirectly; amake synthtarget exists as of docs: wire up agent skill configuration (issue tracker, triage, domain docs) #263.--group agents --group dev;make initadds--group eval.AGENTS.md (#263) establishes make targets as the canonical verification interface ("scripts to rule them all"). CI should consume the same interface so the flags live in exactly one place.
Solution
Agreed design (grilled 2026-09-18):
make testbecomestest: test-python test-ts(local behavior unchanged); CI jobs callmake test-python,make test-ts,make synthrespectively — one source of truth per command, parallel jobs preserved.PYTEST_ADDOPTS. CI setsPYTEST_ADDOPTS="-v --cov-report=xml --junitxml=test-results.xml"and calls the samemake test-python— parametrization of one target, not a forkedtest-citarget.make check. Cache~/.cache/pre-commitkeyed on OS + Python version + hash of.pre-commit-config.yaml(the pattern from pre-commit's own docs; thepre-commit/actionwrapper is in maintenance mode, and pre-commit.ci likely can't get org app approval on aws-samples). Intentional, accepted drift: machine-local wrappers (e.g. Code Defender) run only locally — CI runs exactly the repo's.pre-commit-config.yaml. Do not "fix" this. CI is check-only: fixer hooks (ruff-format, prettier) modify the runner's checkout but nothing commits them — the job fails and the fixes are discarded. Auto-committing fixes from CI is off the table (fork PRs:GITHUB_TOKENcan't push to contributor forks), so on failure the job just surfaces a "runmake checklocally" hint — pre-commit's own output already names the failing hooks, no extra diff step needed.evalgroup is excluded from CI by policy (evals cost real Bedrock tokens, run only on explicit human request — see AGENTS.md) and mechanically (no AWS credentials are wired to the runners).Acceptance
pytest/vitest/synthinvocations left inci.yml— jobs callmake check,make test-python,make test-ts,make synthmake check && make test && make synthand CI produce the same pass/fail verdict for the same treemake teststill runs the full suite locally with unchanged behavior