fix(sandbox): make tool decisions host-authoritative - #329
fix(sandbox): make tool decisions host-authoritative#329Jake Present (jakepresent) wants to merge 9 commits into
Conversation
Two hardening fixes on the trust boundary this PR introduces. RemoteActionMediator defaulted an untracked executor to real_executed=True. That value drives execution_status in the trusted ledger, which is the strongest claim in the evidence contract: whether a consequential action actually ran. An executor that never proved it executed should not be reported as having executed. Not reachable today (AgentHooksToolHost always supplies a tracking executor), but it is a fail-open default on the exact boundary this PR exists to close. The run_turn fail-closed gate re-derived 'did the host mediate this turn' by string-matching evidence_source inside a rendered evidence message. That value is known structurally at drain time, so matching it back out couples a security gate to a presentation detail: a later change to the evidence shape could disable the gate with every test still green. Record the count of drained host ledger rows instead. Also fixes an existing test that passed a bare lambda where production passes a tracking executor; it now mirrors the real call shape. Verified by sabotage: restoring the fail-open default fails the new execution-claim test, and both breaking the gate and dropping the counter fail the new gate test. 1346 passed, 23 skipped, 840 subtests outside the viewer suite. The 29 viewer failures are the pre-existing missing viewer/node_modules gap and reproduce identically on the unmodified PR head.
Chang Liu (changliu2)
left a comment
There was a problem hiding this comment.
The fail-closed gate on exact head a412143 accepts the presence of any host-ledger row, then drops the target tool events. An unmatched target action can therefore disappear when the host ledger contains only an unrelated action. Correlate every target action/tool-call ID against the trusted host-ledger IDs and fail closed on any unmatched action.
This PR is also stacked on #327, so please rebase and revalidate after the case-correlation blockers there are fixed. The exact head currently has only Tier 1 CI.
# Conflicts: # assert_ai/integrations/sandbox/stock/server.py # examples/sandbox_action_mediation/stock_agent/server.py
# Conflicts: # examples/sandbox_action_mediation/README.md # tests/test_sandbox_runtime.py
# Conflicts: # assert_ai/integrations/sandbox/session.py # tests/test_sandbox_runtime.py
|
I wanted to bring up with you, I think having something like RemoteActionMediator is better than nothing, but I think we should likely model this as a separate MCP tool gateway. RemoteActionMediator is moving the policy decision to the host, but the passed tools still execute inside the agents container and their results are target-reported, so we're depending on the agent to execute them, report the result, and not bypass our wrapper. We could have a separate docker container which acts as an mcp tool gateway. The caller just needs to configure this gateway as an mcp server for their agent. It will provide MCP tools/list and tools/call methods, so all tools route through this separate container. Everything is blocked other than the model proxy and this mcp tool gateway. It would reduce the integration code, since instead of relying on the integrator to wrap every tool call, it would only need to configure the MCP server. You could move tool implementations, credentials, policy enforcement, mocks, execution, and the ledger itself into this gateway. Besides this main point, I think policy should be context aware, so you can decide pass/mock/block using tool name, arguments, ASSERT case ID, and scenario state instead of mainly just tool name. You should also restrict egress by both host and port, so that allow-listing one hostname does not permit every service running on it. |
Problem
The stock sandbox trusted the evaluated target to choose pass/mock/block and return its own action evidence. Read-only policy files did not make that evidence authoritative: target code could omit or forge endpoint events, and the writable target ledger was not an independent record.
Approach
Add an explicit
target.host_action_mediation: truepath for stock containers:AgentHooksToolHostthroughRemoteActionMediatorover the existing narrow relay;The evidence contract remains bounded:
target_reportedandresult_authoritative: false; andReview hardening
Adversarial review found and this head fixes:
Host-mode reconciliation now requires every target-reported action ID, tool name, and argument object to agree with the host ledger. Host-only attempts remain visible when the target omits its compatibility event.
Verification
132 passed.1480 passed, 23 skipped, 840 subtests passed.A clean local stock-image rebuild remains blocked by the local Docker builder's external PyPI TLS failure. The local proof used a no-network derivative of the existing stock image with this exact branch source overlaid; #328's GitHub workflow passed the clean-image build path.
Stack and landing
This PR remains stacked on #327 because stable case identity is part of the host ledger contract. #326 is merged. Current landing order is #328, #327, then this PR after updating its base to
mainand rerunning the required Docker workflow on the final head.Deliberate follow-up
This PR does not move consequential real tool implementations and credentials behind a host-side adapter. That stronger boundary would make passed execution independently authoritative, but changes the customer onboarding and tool-packaging contract and should be reviewed separately.