Skip to content

docs(conformance): record Microsoft Agent Framework cross-validation report (33/47) - #66

Merged
MohammadHaroonAbuomar merged 1 commit into
mainfrom
mhabuomar/maf-conformance-claim
Aug 7, 2026
Merged

docs(conformance): record Microsoft Agent Framework cross-validation report (33/47)#66
MohammadHaroonAbuomar merged 1 commit into
mainfrom
mhabuomar/maf-conformance-claim

Conversation

@MohammadHaroonAbuomar

Copy link
Copy Markdown
Contributor

What

Adds a registry entry and reproducible claim artifacts (conformance/claims/maf/: per-part REPORT.md, harness.py, test_conformance.py) for the AGENT-HOOKS-0.1 middleware merged upstream as microsoft/agent-framework#7515.

  • Adapter: agent-framework-core 1.13.0, upstream main @ 4b1afd90520310547cb0e9cdc70f644d80161e82
  • SDK/CTK: agent-hooks-sdk[ctk] 0.1.0a4 (PyPI), Python 3.12, pytest plugin runner
  • Corpus: the 47 wheel-vendored vectors, byte-identical to this repo's conformance/vectors/ at the PR base

Declared surface

All eight interception points; model_calls, tool_calls, int64_json, bigint_json; all four §7.2 profiles and knobs (factory composition= passthrough, default sequential/first_deny, on_approval: stop); jcs-sha256 (+ null, vector-scoped); enforce and evaluate_only; buffered_output: true (streaming fully buffered and gated); hosted (service-side) tools disclosed as uninterceptable at the tool seam (surfaced in the post_model_call projection).

Result — recorded as partial cross-validation, not a §13.1 conformance claim

33 passed, 14 failed, 0 skipped. Every failure is root-caused in the report:

  • F1 (13 vectors, sole failing assertion run_outcome blocked-vs-completed): MAF deliberately halts the run on a host_error:* deny at the tool seam (documented + unit-tested upstream). §6.2's "unless the host's own semantics terminate the turn" permits this posture; the CTK's single-valued run_outcome cannot express it. CTK expressibility gap — issue material for this repo. Secondary MAF-side question: the halt keys on the host_error: prefix, so it also fires for composition-produced denies (transform_conflict, composition_disagreement).
  • F2 (AH-CTK-100): the vector pins the reference host's blocked-tool transcript (messages[1].role == "tool", content string blocked: <reason>); MAF keeps the protocol-required assistant function-call message and uses a structured error payload. §6.2 prescribes no transcript shape. CTK over-prescription — issue material for this repo.
  • F3 (observation, no vector failure): constructor-registered tools do not surface in agent_startup.tools_registered (Agent has no .tools attribute; they live in default_options). MAF-side fix material.

Production-path confirmation (per CLAIMS.md)

The harness drives a real Agent through create_agent_hooks_middleware (per-run session; create_agent_hooks_middleware_from_emitter only for the zero-interceptor and redact_for_approval vectors, with harness-owned session brackets in the documented host role). Only the chat client and tools are mocked; every emission, write-back, and block is the framework's. One disclosed host policy: an inner (inside-the-bundle) presentation middleware makes the caller-visible output the final assistant message before the output point guards it.

Reproduce:

pip install "agent-framework-core[agent-hooks] @ git+https://github.com/microsoft/agent-framework.git@4b1afd90520310547cb0e9cdc70f644d80161e82#subdirectory=python/packages/core" "agent-hooks-sdk[ctk]==0.1.0a4" pytest
cd conformance/claims/maf && pytest test_conformance.py --agent-hooks-harness=harness:AgentFrameworkHarness

…report (33/47)

Add a registry entry and claim artifacts for the agent-hooks middleware
merged upstream as microsoft/agent-framework#7515 (agent-framework-core
1.13.0, main @ 4b1afd9052), run against the 47-vector CTK from
agent-hooks-sdk 0.1.0a4 with a harness that drives the production
middleware bundle over a real Agent (only model/tool I/O mocked).

Result: 33 passed, 14 failed, 0 skipped. All 14 failures trace to two
documented host-semantics divergences, analysed in the report:

- F1 (13 vectors): MAF deliberately halts the run on a host_error:*
  deny at the tool seam; every vector fails solely on
  run_outcome blocked-vs-completed. Spec 6.2's "unless the host's own
  semantics terminate the turn" clause permits the posture, but the
  CTK's run_outcome grammar cannot express it.
- F2 (AH-CTK-100): the vector pins the reference host's blocked-tool
  transcript convention (tool message at index 1, "blocked: <reason>"
  string), which protocol-valid transcripts cannot match.
- F3 (observation, no vector failure): constructor-registered tools do
  not surface in agent_startup.tools_registered (MAF-side fix material).

The entry is explicitly recorded as partial cross-validation, not a
13.1 conformance claim; harness, runner entry point, and per-part
report live under conformance/claims/maf/ so the run is reproducible
from this repository alone.

Signed-off-by: MohammadHaroonAbuomar <40180927+MohammadHaroonAbuomar@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant