A scenario can emit only a thin subset of its checks while the reported denominator makes that subset look complete.
The concrete case from mcpkit: auth/authorization-server-migration reported 2 passed, 1 failed for months. The initial wire handshake failed, so roughly 76 other checks never emitted. PR #327 makes the setup failure explicit, but the result still cannot show how much of the scenario did not run.
The harder case has no setup failure: setup succeeds, only accept-path checkpoints emit, and the thin scenario result passes.
Add a per-scenario expected-vs-emitted coverage signal to the upstream results. It should be visible in both terminal summaries and machine-readable output. A count is enough to expose the gap; stable IDs would also identify the missing checks.
Today Scenario and ClientScenario do not declare their expected check IDs, and some checks are dynamic or version-gated. The implementation will need a stable expected set or another baseline that excludes checks which are not applicable to the selected spec version or extension.
Acceptance criteria:
- Every executed scenario reports expected and emitted check counts, or equivalent coverage data.
- Missing eligible checks remain visible when every emitted check passes.
- Machine-readable results expose the coverage signal for downstream consumers.
- Spec-version, extension, and documented applicability gates do not create false gaps.
- Tests cover both a setup failure and a passing thin-shadow result.
- SDKs do not need to maintain their own per-scenario count snapshots.
@panyam offered to implement this in #327.
Prior art: panyam/mcpkit#1115
A scenario can emit only a thin subset of its checks while the reported denominator makes that subset look complete.
The concrete case from mcpkit:
auth/authorization-server-migrationreported2 passed, 1 failedfor months. The initial wire handshake failed, so roughly 76 other checks never emitted. PR #327 makes the setup failure explicit, but the result still cannot show how much of the scenario did not run.The harder case has no setup failure: setup succeeds, only accept-path checkpoints emit, and the thin scenario result passes.
Add a per-scenario expected-vs-emitted coverage signal to the upstream results. It should be visible in both terminal summaries and machine-readable output. A count is enough to expose the gap; stable IDs would also identify the missing checks.
Today
ScenarioandClientScenariodo not declare their expected check IDs, and some checks are dynamic or version-gated. The implementation will need a stable expected set or another baseline that excludes checks which are not applicable to the selected spec version or extension.Acceptance criteria:
@panyam offered to implement this in #327.
Prior art: panyam/mcpkit#1115