Skip to content

Maintenance: CI workflows should call make targets instead of raw commands #265

Description

@rezabekf

Why is this needed?

Local verification and CI are drifting — they run different commands with different flags, so "green locally" and "green in CI" are not the same verdict. Inventory:

  1. No lint job in CI. make check (pre-commit: ruff-format, ruff, prettier) only runs as a local git hook. A contributor without hooks installed bypasses formatting/linting entirely and CI stays green.
  2. pytest flag drift. ci.yml runs uv run pytest tests/ -v --cov=src --cov-report=term --cov-report=xml --junitxml=... while make test runs uv run pytest tests/ --cov=src --cov-report=term-missing.
  3. cdk-nag via raw command. CI's cdk-nag-test job runs npm run synth directly; a make synth target exists as of docs: wire up agent skill configuration (issue tracker, triage, domain docs) #263.
  4. Dependency-group mismatch. CI installs --group agents --group dev; make init adds --group eval.

AGENTS.md (#263) establishes make targets as the canonical verification interface ("scripts to rule them all"). CI should consume the same interface so the flags live in exactly one place.

Solution

Agreed design (grilled 2026-09-18):

  1. Split make targets to preserve CI parallelism. make test becomes test: test-python test-ts (local behavior unchanged); CI jobs call make test-python, make test-ts, make synth respectively — one source of truth per command, parallel jobs preserved.
  2. CI-only pytest output flags via PYTEST_ADDOPTS. CI sets PYTEST_ADDOPTS="-v --cov-report=xml --junitxml=test-results.xml" and calls the same make test-python — parametrization of one target, not a forked test-ci target.
  3. Add a 4th CI job: lint, calling make check. Cache ~/.cache/pre-commit keyed on OS + Python version + hash of .pre-commit-config.yaml (the pattern from pre-commit's own docs; the pre-commit/action wrapper is in maintenance mode, and pre-commit.ci likely can't get org app approval on aws-samples). Intentional, accepted drift: machine-local wrappers (e.g. Code Defender) run only locally — CI runs exactly the repo's .pre-commit-config.yaml. Do not "fix" this. CI is check-only: fixer hooks (ruff-format, prettier) modify the runner's checkout but nothing commits them — the job fails and the fixes are discarded. Auto-committing fixes from CI is off the table (fork PRs: GITHUB_TOKEN can't push to contributor forks), so on failure the job just surfaces a "run make check locally" hint — pre-commit's own output already names the failing hooks, no extra diff step needed.
  4. Dependency groups stay per-target — intentional, not drift. Each CI job installs only the groups its make target needs. The eval group is excluded from CI by policy (evals cost real Bedrock tokens, run only on explicit human request — see AGENTS.md) and mechanically (no AWS credentials are wired to the runners).

Acceptance

  • No raw pytest/vitest/synth invocations left in ci.yml — jobs call make check, make test-python, make test-ts, make synth
  • Local make check && make test && make synth and CI produce the same pass/fail verdict for the same tree
  • make test still runs the full suite locally with unchanged behavior

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    ready-for-agentFully specified, ready for an AFK agent

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions