Python: fix FIDES session isolation and runtime integration gaps

[Oscar Lobato Ruiz (lerelerele)]

• Closes #7455

This PR implements the A1 - C3 changes described in #7455.

The main architectural change is to make FIDES runtime state conversation-scoped. Context labels, audit records, pending approvals, counters, and variable stores are now associated with AgentSession.state instead of shared middleware or agent instances. This prevents state leaking between concurrent conversations that reuse the same agent or middleware.

What changed

• A1: Store context labels, audit logs, pending approvals, and turn/call counters in AgentSession.state; variable stores are isolated per session.
• A2: A blocked call now returns a correlated function_result instead of terminating the loop with empty output, allowing the model to explain the refusal or choose another action. Approval requests still pause for the user.
• A3: Audit records now carry a per-run turn and per-call call_index.
• B1: Add tool_labels configuration for externally-created tools, including harness and MCP tools.
• B2: Preserve approval-request additional_properties when reconstructing function calls.
• B3: Prevent standing approval rules and automatic approval callbacks from approving FIDES policy violations.
• B4: Add the public build_function_call_content() extension point while retaining the former private method as an alias.
• B5: Add deny_untrusted_tools, with deny rules taking precedence over allow rules.
• C1: Support enable_quarantine=False, allowing labels and policy enforcement without quarantine tooling.
• C2: Bind the quarantine client to the current async context instead of a process-global slot.
• C3: Apply a PUBLIC confidentiality cap to read-only MCP tools, with mark_read_tools_as_sinks as an opt-out.

The durable session.state["_fides"] schema is owned by _fides_session_state(). It creates the mapping and initializes the context label, audit log, pending approvals, and turn/call counters in one place. before_run(), the label
accessors, and the policy-enforcement accessors all route through it; their per-key initialization has been removed.

_current_middleware now uses a plain async-safe ContextVar, replacing the previous threading.local() storage. No-session fallbacks remain local, and direct middleware calls still return turn 1 on their first call.

5. Audit logs can be bounded with max_audit_log_entries, which defaults to 1000.

Additional API surface

This PR also includes two additions that were not explicitly listed in #7455:

• max_audit_log_entries, defaulting to 1000 and accepting None for an unlimited log.
• begin_turn(session), which provides the run boundary needed to populate meaningful audit turn numbers.

These additions are called out explicitly for maintainer review.

Scope

This PR is scoped to the Python core FIDES integration work described in #7455.

Tests and verification

Regression coverage was added to the existing test suites:

• python/packages/core/tests/test_security.py
• python/packages/core/tests/core/test_harness_tool_approval.py

The standalone regression test module was removed.

• Full core test suite excluding integration tests:
  pytest packages/core/tests -m "not integration" -q

[Oscar Lobato Ruiz (lerelerele)]

@microsoft-github-policy-service agree

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated no new comments.

Suppressed comments (2)

python/packages/core/agent_framework/security.py:2425

• PolicyEnforcementFunctionMiddleware.clear_audit_log() has the same session-resolution issue as get_audit_log(): when called without an explicit session, it uses get_current_session() which may be None after middleware execution. Fall back to the remembered active session so the public API behaves consistently.

    def clear_audit_log(self, session: Any = None) -> None:
        """Clear the audit log."""
        self._resolve_audit_log(session or get_current_session()).clear()

python/packages/core/agent_framework/security.py:2421

• PolicyEnforcementFunctionMiddleware.get_audit_log() only falls back to get_current_session(), which is cleared when the label-tracking middleware exits. This means calling get_audit_log() without explicitly passing session can incorrectly return the no-session fallback log even though a session was previously active in the current async context (contrary to the module’s new “remember last session” behavior). Consider also falling back to the weakly remembered _fides_active_session.

This issue also appears on line 2423 of the same file.

            List of violation records.
        """
        return self._resolve_audit_log(session or get_current_session()).copy()

[Evan Mattson (moonbox3)]

It's a bit confusing why the PR body links to an issue that says that this PR doesn't solve. Can you please link to the correct issue?

[Oscar Lobato Ruiz (lerelerele)]

You’re right. I updated the PR description to reference the correct issue, #7455, using Closes #7455. The unrelated #7466 reference has been removed.

I also addressed the other review suggestions:

“I don't think this net-new file is needed.”

Removed test_fides_7455_regressions.py and moved all 21 regression cases into the existing suites: 20 into test_security.py and the standing-approval case into core/test_harness_tool_approval.py.

“Please keep JSON safety behind the existing make_json_safe method…”

Removed the local _json_safe helper and extended make_json_safe() to handle Enum values and sets/frozensets. ContentLabel.to_dict() now delegates to it, leaving a single owner for JSON safety.

“Let’s look at having _current_middleware remain a plain ContextVar…”

_current_middleware is now a plain async-safe ContextVar, replacing the previous threading.local() storage. Production code uses set(), reset(), and get(), and the tests no longer rely on the .instance property.

“How about one helper owns initialization and access for session.state["_fides"]…”

_fides_session_state() is now the single owner of the durable session.state["_fides"] schema. It creates the mapping and initializes the context-label slot, audit log, pending approvals, and turn/call counters.

before_run(), the label accessors, and the policy-enforcement accessors all route through it, so their per-key initialization has been removed. No-session fallbacks remain local, and direct middleware calls still return turn 1 on
their first call.

One intentional consequence is that any FIDES session-state access materializes the complete schema, including empty audit, approval, and counter fields. This keeps the durable contract owned by one accessor.

While validating the changes, I also found and fixed a separate audit-log issue: after a run completed, get_audit_log() and clear_audit_log() could fall back to the middleware’s empty log because the active session had already been
cleared. Both accessors now resolve the active or last remembered session through _resolve_active_session(), with regression coverage in the existing TestPolicyEnforcementMiddleware suite.

Focused FIDES tests, the full core suite excluding integration tests, Ruff, and Pyright all pass.