docs(conformance): record Microsoft Agent Framework cross-validation report (33/47) - #66
Merged
Merged
Conversation
…report (33/47) Add a registry entry and claim artifacts for the agent-hooks middleware merged upstream as microsoft/agent-framework#7515 (agent-framework-core 1.13.0, main @ 4b1afd9052), run against the 47-vector CTK from agent-hooks-sdk 0.1.0a4 with a harness that drives the production middleware bundle over a real Agent (only model/tool I/O mocked). Result: 33 passed, 14 failed, 0 skipped. All 14 failures trace to two documented host-semantics divergences, analysed in the report: - F1 (13 vectors): MAF deliberately halts the run on a host_error:* deny at the tool seam; every vector fails solely on run_outcome blocked-vs-completed. Spec 6.2's "unless the host's own semantics terminate the turn" clause permits the posture, but the CTK's run_outcome grammar cannot express it. - F2 (AH-CTK-100): the vector pins the reference host's blocked-tool transcript convention (tool message at index 1, "blocked: <reason>" string), which protocol-valid transcripts cannot match. - F3 (observation, no vector failure): constructor-registered tools do not surface in agent_startup.tools_registered (MAF-side fix material). The entry is explicitly recorded as partial cross-validation, not a 13.1 conformance claim; harness, runner entry point, and per-part report live under conformance/claims/maf/ so the run is reproducible from this repository alone. Signed-off-by: MohammadHaroonAbuomar <40180927+MohammadHaroonAbuomar@users.noreply.github.com>
MohammadHaroonAbuomar
requested a review
from Caitie McCaffrey (CaitieM20)
as a code owner
August 7, 2026 06:15
This was referenced Aug 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds a registry entry and reproducible claim artifacts (
conformance/claims/maf/: per-partREPORT.md,harness.py,test_conformance.py) for the AGENT-HOOKS-0.1 middleware merged upstream as microsoft/agent-framework#7515.agent-framework-core1.13.0, upstreammain@4b1afd90520310547cb0e9cdc70f644d80161e82agent-hooks-sdk[ctk]0.1.0a4 (PyPI), Python 3.12, pytest plugin runnerconformance/vectors/at the PR baseDeclared surface
All eight interception points;
model_calls,tool_calls,int64_json,bigint_json; all four §7.2 profiles and knobs (factorycomposition=passthrough, defaultsequential/first_deny,on_approval: stop);jcs-sha256(+null, vector-scoped);enforceandevaluate_only;buffered_output: true(streaming fully buffered and gated); hosted (service-side) tools disclosed as uninterceptable at the tool seam (surfaced in thepost_model_callprojection).Result — recorded as partial cross-validation, not a §13.1 conformance claim
33 passed, 14 failed, 0 skipped. Every failure is root-caused in the report:
run_outcomeblocked-vs-completed): MAF deliberately halts the run on ahost_error:*deny at the tool seam (documented + unit-tested upstream). §6.2's "unless the host's own semantics terminate the turn" permits this posture; the CTK's single-valuedrun_outcomecannot express it. CTK expressibility gap — issue material for this repo. Secondary MAF-side question: the halt keys on thehost_error:prefix, so it also fires for composition-produced denies (transform_conflict,composition_disagreement).messages[1].role == "tool", content stringblocked: <reason>); MAF keeps the protocol-required assistant function-call message and uses a structured error payload. §6.2 prescribes no transcript shape. CTK over-prescription — issue material for this repo.agent_startup.tools_registered(Agenthas no.toolsattribute; they live indefault_options). MAF-side fix material.Production-path confirmation (per CLAIMS.md)
The harness drives a real
Agentthroughcreate_agent_hooks_middleware(per-run session;create_agent_hooks_middleware_from_emitteronly for the zero-interceptor andredact_for_approvalvectors, with harness-owned session brackets in the documented host role). Only the chat client and tools are mocked; every emission, write-back, and block is the framework's. One disclosed host policy: an inner (inside-the-bundle) presentation middleware makes the caller-visible output the final assistant message before theoutputpoint guards it.Reproduce: