Add Voice Agents to Foundry Python SDK

[xitzhang (xitzhang)]

Description

Adds Voice Agents to the Foundry Python SDK: a new agent kind for real-time, speech-to-speech conversational AI unified with the rest of the Agents API. Define a voice agent's model, audio, turn detection, greeting, and tools; manage it like any other agent; hold a live conversation with it over a WebSocket with barge-in and persisted conversation history/audio; and reach it through telephony (inbound bindings or outbound calls/campaigns).

This SDK is regenerated based on the following API spec changes:

Spec: foundrysdk_specs/specs/agents/voice_agents/spec.md
TypeSpec: Unify voice agents with the Agents API and add the v1 voice data plane · Azure/azure-rest-api-specs@9759481
Add Foundry Voice Agent batch 2 APIs by yulin-li · Pull Request #45852 · Azure/azure-rest-api-specs
Auto-generated (from TypeSpec)
The voice agent surface itself: VoiceAgentDefinition and its audio/turn-detection/greeting/tool/avatar config models, managed through the existing project_client.agents CRUD/versioning methods.
New agent_endpoint_conversations operation group to read back persisted transcripts and audio.
New agent_telephony operation group for outbound call jobs and campaigns, plus telephony bindings/call management on project_client.agents.
WebRTC transport, sub-agent consultation, and conversation-engine delegation on VoiceAgentDefinition.
Hand-written
A new client.realtime / async_client.realtime WebSocket client (not TypeSpec-generated, modeled after the OpenAI Python realtime client's ergonomics) for sending/receiving strongly-typed realtime events.
11 new samples under samples/agents/voice/ covering lifecycle, live audio/text conversation, function tools, versioning, and conversation read-back.
New unit (mocked) and live/recorded test coverage for the realtime client, voice agent CRUD/conversations/telephony.
CHANGELOG, README, and optional websockets/aiohttp dependency updates.

All SDK Contribution checklist:

• The pull request does not introduce [breaking changes]
• CHANGELOG is updated for new features, bug fixes or other significant changes.
• I have read the contribution guidelines.

General Guidelines and Best Practices

• Title of the pull request is clear and informative.
• There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, see this page.

Testing Guidelines

• Pull request includes test coverage for the included changes.

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).
10 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[Copilot]

🟡 Changes recommended

There are at least two concrete repo-impacting issues (a type-checking problem in a new sample and an incorrectly named tsp-location file) that should be corrected before approval.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR adds Voice Agents support to azure-ai-projects, introducing a new voice-agent kind integrated into the existing Agents API, plus realtime (WebSocket) client support, new operation groups for conversation readback and telephony, and accompanying samples/tests and test-resource provisioning updates.

Changes:

• Adds voice-agent API surface (CRUD/versioning, conversation read-back, telephony) and updates feature-header behaviors for preview opt-in.
• Introduces sync/async realtime streaming clients (client.realtime / async_client.realtime) with optional dependencies and extensive samples.
• Adds new recorded/unit/live tests plus test resource provisioning assets for live realtime scenarios.

File summaries

File	Description
sdk/ai/cspell.yaml	Adds voice-agent related terms to spellcheck allowlist.
sdk/ai/azure-ai-projects/tsp-location.yaml.saved	Tracks TypeSpec source location/commit for regeneration (but file naming needs attention).
sdk/ai/azure-ai-projects/tests/test_base.py	Adds a sanitized env var placeholder for voice model deployment in tests.
sdk/ai/azure-ai-projects/tests/samples/test_samples.py	Adds voice samples to sample test matrix with appropriate skips for WS/unrecordable flows.
sdk/ai/azure-ai-projects/tests/samples/test_samples_async.py	Adds async voice samples to sample test matrix with WS-related skips.
sdk/ai/azure-ai-projects/tests/foundry_features_header/test_foundry_features_header_on_ga_operations.py	Expands header override coverage to include different header casing variants.
sdk/ai/azure-ai-projects/tests/foundry_features_header/test_foundry_features_header_on_ga_operations_async.py	Async counterpart for header override casing tests.
sdk/ai/azure-ai-projects/tests/foundry_features_header/test_agent_telephony_protocol.py	Adds protocol-level request construction tests for telephony call-job/campaign routes.
sdk/ai/azure-ai-projects/tests/foundry_features_header/test_agent_telephony_protocol_async.py	Async protocol tests for telephony request construction.
sdk/ai/azure-ai-projects/tests/foundry_features_header/foundry_features_header_test_base.py	Updates expected Foundry-Features opt-ins and adds coverage for conversations/telephony groups.
sdk/ai/azure-ai-projects/tests/conftest.py	Sanitizer changes to reduce cassette drift (header matching) for stable playback.
sdk/ai/azure-ai-projects/tests/agents/test_voice_agent_telephony.py	Adds recorded tests for agent telephony bindings/calls/transfer targets and generated-audio reads (currently skipped pending service support).
sdk/ai/azure-ai-projects/tests/agents/test_voice_agent_telephony_campaign.py	Adds recorded tests for outbound telephony call jobs/campaigns (currently skipped pending API-version enablement).
sdk/ai/azure-ai-projects/tests/agents/test_voice_agent_telephony_campaign_async.py	Async counterpart for outbound telephony call-job/campaign tests (currently skipped).
sdk/ai/azure-ai-projects/tests/agents/test_voice_agent_realtime_live.py	Adds live-only realtime WebSocket tests for sync client.
sdk/ai/azure-ai-projects/tests/agents/test_voice_agent_realtime_live_async.py	Adds live-only realtime WebSocket tests for async client.
sdk/ai/azure-ai-projects/tests/agents/test_voice_agent_crud.py	Adds recorded CRUD/versioning tests for voice agents via project_client.agents.
sdk/ai/azure-ai-projects/tests/agents/test_voice_agent_crud_async.py	Async CRUD/versioning tests for voice agents.
sdk/ai/azure-ai-projects/tests/agents/test_voice_agent_conversations.py	Adds recorded tests for conversation transcript/audio read-back with live-only setup for recording.
sdk/ai/azure-ai-projects/tests/agents/test_voice_agent_conversations_async.py	Async counterpart for conversation read-back tests.
sdk/ai/azure-ai-projects/tests.yml	Adds a live-test pipeline configuration for voice-agent live tests in a supported region.
sdk/ai/azure-ai-projects/test-resources.bicep	Adds Bicep template to provision Foundry resources required for voice-agent live tests.
sdk/ai/azure-ai-projects/test-resources-post.ps1	Adds post-deploy script to create the gpt-realtime deployment with retry/wait logic.
sdk/ai/azure-ai-projects/samples/hosted_agents/sample_session_log_stream.py	Updates session log stream sample to explicitly request streaming.
sdk/ai/azure-ai-projects/samples/hosted_agents/sample_session_log_stream_async.py	Async version of explicit streaming request update.
sdk/ai/azure-ai-projects/samples/agents/voice/sample_voice_agent_with_tools.py	Adds voice-agent definition sample showing audio config and tool configuration.
sdk/ai/azure-ai-projects/samples/agents/voice/sample_voice_agent_versions.py	Adds sample showing voice-agent versioning flows.
sdk/ai/azure-ai-projects/samples/agents/voice/sample_voice_agent_read_conversation.py	Adds sample showing conversation transcript readback via agent_endpoint_conversations.
sdk/ai/azure-ai-projects/samples/agents/voice/sample_voice_agent_read_conversation_audio.py	Adds sample showing conversation audio readback + file streaming.
sdk/ai/azure-ai-projects/samples/agents/voice/sample_voice_agent_live_function_tool.py	Adds sample showing function tool round-trip during a live realtime session.
sdk/ai/azure-ai-projects/samples/agents/voice/sample_voice_agent_generate.py	Adds sample for guided authoring via agents.generate_agent(kind="voice").
sdk/ai/azure-ai-projects/samples/agents/voice/sample_voice_agent_basic.py	Adds basic sync voice-agent lifecycle sample.
sdk/ai/azure-ai-projects/samples/agents/voice/sample_voice_agent_basic_async.py	Adds basic async voice-agent lifecycle sample.
sdk/ai/azure-ai-projects/samples/agents/sample_workflow_multi_agent_with_mcp_approval.py	Adjusts typing ignores for workflow action item access.
sdk/ai/azure-ai-projects/samples/agents/sample_workflow_multi_agent_async.py	Async counterpart typing-ignore adjustment.
sdk/ai/azure-ai-projects/README.md	Adds voice agents (preview) to the operation group table and links to relevant docs.
sdk/ai/azure-ai-projects/pyproject.toml	Bumps package to beta classifier and adds realtime optional dependencies.
sdk/ai/azure-ai-projects/docs/public-methods.md	Updates public method inventory and includes new operation groups/methods.
sdk/ai/azure-ai-projects/dev_requirements.txt	Adds websockets to development requirements.
sdk/ai/azure-ai-projects/CHANGELOG.md	Adds a 2.7.0b1 entry describing voice agents, realtime, conversation readback, and telephony.
sdk/ai/azure-ai-projects/azure/ai/projects/operations/_patch.py	Exports new operation group patch types.
sdk/ai/azure-ai-projects/azure/ai/projects/operations/_patch_evaluators.py	Refactors typing aliases/imports and adds type ignores for overrides.
sdk/ai/azure-ai-projects/azure/ai/projects/operations/_patch_evaluation_rules.py	Adds override typing ignores and tightens typing for super calls.
sdk/ai/azure-ai-projects/azure/ai/projects/operations/_patch_datasets.py	Refactors typing alias import and adds typing ignores for overrides.
sdk/ai/azure-ai-projects/azure/ai/projects/operations/init.py	Exposes new operation group classes from operations namespace.
sdk/ai/azure-ai-projects/azure/ai/projects/models/_patch.py	Adds VoiceAgents preview opt-in to the agents header composition and documents client placement.
sdk/ai/azure-ai-projects/azure/ai/projects/aio/operations/_patch.py	Exports async patch types for new operation group.
sdk/ai/azure-ai-projects/azure/ai/projects/aio/operations/_patch_evaluators_async.py	Async typing alias refactor + override ignores.
sdk/ai/azure-ai-projects/azure/ai/projects/aio/operations/_patch_evaluation_rules_async.py	Async override typing ignores and arg-type ignore for super call.
sdk/ai/azure-ai-projects/azure/ai/projects/aio/operations/_patch_datasets_async.py	Async typing alias import + override ignores.
sdk/ai/azure-ai-projects/azure/ai/projects/aio/operations/init.py	Exposes new async operation group classes.
sdk/ai/azure-ai-projects/azure/ai/projects/aio/_patch.pyi	Exposes realtime types and realtime property in async client typing stubs.
sdk/ai/azure-ai-projects/azure/ai/projects/aio/_patch.py	Adds async realtime property + opt-in header proxying and accept-encoding workaround.
sdk/ai/azure-ai-projects/azure/ai/projects/aio/_client.py	Adds generated async operation group attributes for conversations and telephony.
sdk/ai/azure-ai-projects/azure/ai/projects/_version.py	Bumps package version to 2.7.0b1.
sdk/ai/azure-ai-projects/azure/ai/projects/_utils/utils.py	Adds ETag quoting and match-condition header preparation helpers.
sdk/ai/azure-ai-projects/azure/ai/projects/_unions.py	Adds new voice-agent related union types/literals.
sdk/ai/azure-ai-projects/azure/ai/projects/_patch.pyi	Exposes realtime types and realtime property in sync client typing stubs.
sdk/ai/azure-ai-projects/azure/ai/projects/_patch.py	Adds sync realtime property + opt-in header proxying and exports realtime types.
sdk/ai/azure-ai-projects/azure/ai/projects/_client.py	Adds generated operation group attributes for conversations and telephony.
sdk/ai/azure-ai-projects/assets.json	Updates azure-sdk-assets tag reference.
sdk/ai/azure-ai-projects/api.metadata.yml	Updates api.md hash and package version metadata for 2.7.0b1.
sdk/ai/azure-ai-projects/.env.template	Adds environment variables for voice tests and voice samples.
scripts/devops_tasks/test_run_samples.py	Excludes interactive/live-only voice samples from non-interactive sample runner.
eng/tools/azure-sdk-tools/azpysdk/samples.py	Excludes interactive/live-only voice samples from azpysdk sample execution.

Review details

• Files reviewed: 81/85 changed files
• Comments generated: 2
• Review effort level: Lite

---

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

🔵 Needs a closer look

The change set introduces substantial new public surface area (realtime WebSocket client + new operation groups) and modifies testing/packaging/infrastructure, requiring careful human validation of API and test strategy decisions.

Review details

• Files reviewed: 82/86 changed files
• Comments generated: 5
• Review effort level: Lite

[github-actions]

[Pilot] PR Pipeline Failure Analysis

What failed

Build 6808203 (python - pullrequest) failed on all 6 platforms (Ubuntu2404_313, Ubuntu2404_314, macos311, ubuntu2404_310, ubuntu2404_310_coverage, windows2022_312) x both build variants (sdist, whl) for package azure-ai-projects, totaling 4,704 failed test cases. All failures trace back to three related root causes introduced with this PR's new test coverage, none of which are infrastructure/transient issues.

1) Test-proxy recording/playback failure (~9,168 of the underlying error events, the overwhelming majority of failures)

Every test that uses @recorded_by_proxy/@recorded_by_proxy_async fails identically at start_record_or_playback:

test_id = 'sdk/ai/azure-ai-projects/tests/recordings/agents/telemetry/test_ai_agents_instrumentor.pyTestAiAgentsInstrumentortest_agent_creation_with_tracing_content_recording_enabled'

    def start_record_or_playback(test_id: str) -> "Tuple[str, Dict[str, str]]":
        ...
        else:
            result = http_client.request(
                method="POST",
                url=PLAYBACK_START_URL,
                body=encoded_payload,
            )
            if result.status != 200:
>               raise HttpResponseError(message=result.data)
E               azure.core.exceptions.HttpResponseError: Operation returned an invalid status 'None'

.venv/.../devtools_testutils/proxy_testcase.py:117: HttpResponseError

This affects essentially every recorded test added/touched by this PR (agents telemetry instrumentor tests, responses instrumentor tests, voice agent tests, session/file CRUD tests, samples tests, etc.) on every platform - the test-proxy cannot find or match a valid recording for the test IDs, most likely because the recordings for these newly added/renamed tests are missing or assets.json wasn't updated/pushed to match.

2) AttributeError: 'LegacyAPIResponse' object has no attribute 'close' (96 occurrences)

In sdk/ai/azure-ai-projects/tests/agents/telemetry/test_responses_instrumentor_raw_response.py, several TestRawResponseStreaming tests call raw.close() on the object returned by client.responses.with_raw_response.create(...), e.g.:

tests/agents/telemetry/test_responses_instrumentor_raw_response.py:208
    raw = await client.responses.with_raw_response.create(model="gpt-4o", input="hi", stream=True)
    await raw.close()  # pyright: ignore[reportAttributeAccessIssue]
E   AttributeError: 'LegacyAPIResponse' object has no attribute 'close'

The OpenAI SDK's LegacyAPIResponse object doesn't expose a close() method - this is a test bug (wrong API usage) in the newly written raw-response streaming tests, not an infrastructure issue. Affects test_async_with_raw_response_close_finalizes_unconsumed_stream, test_async_with_raw_response_close_finalizes_partially_consumed_stream, and their sync counterparts.

3) Missing telemetry spans for raw-response streaming (144 occurrences)

Same test file, related tests assert spans were recorded but get zero:

tests/agents/telemetry/test_responses_instrumentor_raw_response.py:168
    spans = self.exporter.get_spans()
>   assert len(spans) >= 1, f"Expected at least 1 span, got {len(spans)}"
E   AssertionError: Expected at least 1 span, got 0

This cascades from the same raw-response close/parse handling issue in #2 - the instrumentation isn't finalizing/emitting a span for the raw-response streaming path being exercised by these new tests.

Relevant pipeline output

azure.core.exceptions.HttpResponseError: Operation returned an invalid status 'None'
  at devtools_testutils/proxy_testcase.py:117 (start_record_or_playback -> PLAYBACK_START_URL)

AttributeError: 'LegacyAPIResponse' object has no attribute 'close'
  at tests/agents/telemetry/test_responses_instrumentor_raw_response.py:208

AssertionError: Expected at least 1 span, got 0
  at tests/agents/telemetry/test_responses_instrumentor_raw_response.py:168

Recommended next steps

• Re-record test-proxy recordings for the new/changed tests in azure-ai-projects (agents telemetry, voice agent, session/file, samples, foundry_features_header) and confirm sdk/ai/azure-ai-projects/assets.json is updated/pushed to point at those recordings. This workflow does not have Azure DevOps/test-proxy recording credentials, so re-recording must be done by the author.
• Fix sdk/ai/azure-ai-projects/tests/agents/telemetry/test_responses_instrumentor_raw_response.py - replace the invalid raw.close() calls (lines ~208, ~226 and sync equivalents) with the correct OpenAI LegacyAPIResponse API (e.g., closing the underlying .response/httpx stream) so raw-response close/finalize and the associated span-emission tests pass.
• See https://aka.ms/ci-fix

Copilot detected the failing pipeline and generated the analysis above. To have it attempt a
fix automatically, reply with @copilot please fix the failing pipeline on this PR.

Generated by Pipeline Analysis Next Steps · auto · 316.3 AIC · ⌖ 2.11 AIC · ⊞ 9.2K · ◷

[Howie Leung (howieleung)]

Found this PR difficult to review since it has many changes for unrelated.
I am trying to merge this PR with the latest emitted from feature/foundry-release
Then you can target to your PR to feature/azure-ai-projects/vnext, so you get a clean PR.
I also need to understand better about realtime