Skip to content

fix(sandbox): make tool decisions host-authoritative - #329

Open
Jake Present (jakepresent) wants to merge 9 commits into
jake/fix-sandbox-case-correlationfrom
jake/fix-sandbox-host-action-ledger
Open

fix(sandbox): make tool decisions host-authoritative#329
Jake Present (jakepresent) wants to merge 9 commits into
jake/fix-sandbox-case-correlationfrom
jake/fix-sandbox-host-action-ledger

Conversation

@jakepresent

@jakepresent Jake Present (jakepresent) commented Aug 20, 2026

Copy link
Copy Markdown
Collaborator

Problem

The stock sandbox trusted the evaluated target to choose pass/mock/block and return its own action evidence. Read-only policy files did not make that evidence authoritative: target code could omit or forge endpoint events, and the writable target ledger was not an independent record.

Approach

Add an explicit target.host_action_mediation: true path for stock containers:

  • run policy evaluation, mock resolution, and the action ledger in an authenticated service on the ASSERT host;
  • route the target's AgentHooksToolHost through RemoteActionMediator over the existing narrow relay;
  • require registration of the ASSERT-owned case ID during startup;
  • append the attempted call and pass/mock/block decision before returning to the target;
  • keep the ledger outside every target-visible mount;
  • replace target-supplied tool events with host ledger rows;
  • preserve completed and incomplete action rows when a turn fails; and
  • fail startup or the turn rather than silently accepting an opted-in target that did not use the host mediator.

The evidence contract remains bounded:

  • attempts and pass/mock/block decisions are host-authoritative;
  • mock/block results are host-generated and authoritative;
  • passed tools execute against disposable state inside the target, so their results remain explicitly target_reported and result_authoritative: false; and
  • arbitrary in-process calls that bypass the mediator and are omitted by the target are not claimed as observed. Container isolation, deny-by-default egress, and host-only real credentials remain the containment backstop.

Review hardening

Adversarial review found and this head fixes:

  • target-forged or conflicting case IDs;
  • duplicate call IDs consuming scenario state;
  • completed and incomplete rows draining out of attempt order;
  • host evidence being lost when fail-closed validation raised;
  • an opted-in image starting without a host-mediator client;
  • an untracked pass executor running behind an incorrect execution claim;
  • a fail-closed gate coupled to rendered evidence wording;
  • an unrelated host row hiding an unmatched target action;
  • the same call ID being reused for different tool names or arguments;
  • target actions with missing IDs; and
  • target-controlled tool evidence being accepted without per-action host reconciliation.

Host-mode reconciliation now requires every target-reported action ID, tool name, and argument object to agree with the host ledger. Host-only attempts remain visible when the target omits its compatibility event.

Verification

  • Focused combined sandbox suites: 132 passed.
  • Full exact-head Python suite: 1480 passed, 23 skipped, 840 subtests passed.
  • ci(sandbox): run real Docker containment tests #328's latest clean-image workflow passed all four Docker tests, verified the receipt, and removed the build image.
  • The prior combined-stack proof passed before these final delegated case-evidence and receipt-hardening deltas; the required final remote-stack rerun remains a landing gate.
  • The Docker proof verified host decisions for pass/mock actions, target-reported pass results, host-generated mock results, host-only ledgers, egress containment, credential isolation, and cleanup.
  • No sandbox containers or networks remained.

A clean local stock-image rebuild remains blocked by the local Docker builder's external PyPI TLS failure. The local proof used a no-network derivative of the existing stock image with this exact branch source overlaid; #328's GitHub workflow passed the clean-image build path.

Stack and landing

This PR remains stacked on #327 because stable case identity is part of the host ledger contract. #326 is merged. Current landing order is #328, #327, then this PR after updating its base to main and rerunning the required Docker workflow on the final head.

Deliberate follow-up

This PR does not move consequential real tool implementations and credentials behind a host-side adapter. That stronger boundary would make passed execution independently authoritative, but changes the customer onboarding and tool-packaging contract and should be reviewed separately.

Two hardening fixes on the trust boundary this PR introduces.

RemoteActionMediator defaulted an untracked executor to real_executed=True.
That value drives execution_status in the trusted ledger, which is the
strongest claim in the evidence contract: whether a consequential action
actually ran. An executor that never proved it executed should not be
reported as having executed. Not reachable today (AgentHooksToolHost always
supplies a tracking executor), but it is a fail-open default on the exact
boundary this PR exists to close.

The run_turn fail-closed gate re-derived 'did the host mediate this turn'
by string-matching evidence_source inside a rendered evidence message. That
value is known structurally at drain time, so matching it back out couples
a security gate to a presentation detail: a later change to the evidence
shape could disable the gate with every test still green. Record the count
of drained host ledger rows instead.

Also fixes an existing test that passed a bare lambda where production
passes a tracking executor; it now mirrors the real call shape.

Verified by sabotage: restoring the fail-open default fails the new
execution-claim test, and both breaking the gate and dropping the counter
fail the new gate test.

1346 passed, 23 skipped, 840 subtests outside the viewer suite. The 29
viewer failures are the pre-existing missing viewer/node_modules gap and
reproduce identically on the unmodified PR head.

@changliu2 Chang Liu (changliu2) left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The fail-closed gate on exact head a412143 accepts the presence of any host-ledger row, then drops the target tool events. An unmatched target action can therefore disappear when the host ledger contains only an unrelated action. Correlate every target action/tool-call ID against the trusted host-ledger IDs and fail closed on any unmatched action.

This PR is also stacked on #327, so please rebase and revalidate after the case-correlation blockers there are fixed. The exact head currently has only Tier 1 CI.

# Conflicts:
#	assert_ai/integrations/sandbox/stock/server.py
#	examples/sandbox_action_mediation/stock_agent/server.py
# Conflicts:
#	examples/sandbox_action_mediation/README.md
#	tests/test_sandbox_runtime.py
# Conflicts:
#	assert_ai/integrations/sandbox/session.py
#	tests/test_sandbox_runtime.py
@liamcrumm

Copy link
Copy Markdown
Collaborator

I wanted to bring up with you, I think having something like RemoteActionMediator is better than nothing, but I think we should likely model this as a separate MCP tool gateway. RemoteActionMediator is moving the policy decision to the host, but the passed tools still execute inside the agents container and their results are target-reported, so we're depending on the agent to execute them, report the result, and not bypass our wrapper. We could have a separate docker container which acts as an mcp tool gateway. The caller just needs to configure this gateway as an mcp server for their agent. It will provide MCP tools/list and tools/call methods, so all tools route through this separate container. Everything is blocked other than the model proxy and this mcp tool gateway. It would reduce the integration code, since instead of relying on the integrator to wrap every tool call, it would only need to configure the MCP server. You could move tool implementations, credentials, policy enforcement, mocks, execution, and the ledger itself into this gateway.

Besides this main point, I think policy should be context aware, so you can decide pass/mock/block using tool name, arguments, ASSERT case ID, and scenario state instead of mainly just tool name. You should also restrict egress by both host and port, so that allow-listing one hostname does not permit every service running on it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants