Skip to content

fix(aws): keep Nova Sonic replies alive with IGNORE_ON_ENTER - #6813

Open
uuzzrm wants to merge 8 commits into
livekit:mainfrom
uuzzrm:codex/fix-nova-ignore-on-enter
Open

fix(aws): keep Nova Sonic replies alive with IGNORE_ON_ENTER#6813
uuzzrm wants to merge 8 commits into
livekit:mainfrom
uuzzrm:codex/fix-nova-ignore-on-enter

Conversation

@uuzzrm

@uuzzrm uuzzrm commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

Closes #6812

What changed

Amazon Nova Sonic declares tools when a realtime session starts. ToolFlag.IGNORE_ON_ENTER temporarily changes the tool set around generate_reply, and the old path could queue overlapping session recycles. A later reply could then race the stream restart and stop producing audio.

This change:

  • coalesces pending tool-set recycle tasks
  • records the tool set actually sent in the prompt-start block
  • waits for an in-flight generation before rebuilding the session
  • bounds a stuck-generation wait and preserves the existing session recovery path
  • keeps waiting if a newer generation replaces the one being drained
  • makes the next reply wait for the pending recycle
  • re-checks tool changes that arrive while a recycle is in flight
  • skips a recycle when the tool set has already been restored

The regression coverage exercises consecutive replies, restored tool sets, active generations, stuck generations, generation replacement, prompt snapshot races, audio delivery, and reply gating without requiring AWS credentials.

Validation

Local validation on Windows, using the isolated worktree virtual environment:

  • pytest livekit-plugins/livekit-plugins-aws/tests -q (25 passed)
  • pytest tests/test_realtime_reply_chat_ctx.py tests/test_agent_session.py -q (82 passed)
  • ruff check livekit-plugins/livekit-plugins-aws/livekit/plugins/aws/experimental/realtime/realtime_model.py livekit-plugins/livekit-plugins-aws/tests/test_nova_sonic_tool_recycle.py (passed)
  • ruff format --check livekit-plugins/livekit-plugins-aws/livekit/plugins/aws/experimental/realtime/realtime_model.py livekit-plugins/livekit-plugins-aws/tests/test_nova_sonic_tool_recycle.py (passed)
  • git diff --check (passed)

Remote CI is running for the follow-up commit. The repository's current mypy checks and the existing AWS typing baseline are tracked by CI; no package manifests or changelog files are part of this change.

AI assistance

Codex assisted with investigating the reported behavior, drafting the regression coverage, and implementing the patch. I reviewed the resulting diff and ran the validation commands above.

Coalesce pending tool-set session recycles and wait for an active generation to finish before rebuilding the stream. This keeps repeated generate_reply calls from racing session initialization when IGNORE_ON_ENTER temporarily changes the tools.
@uuzzrm
uuzzrm requested a review from a team as a code owner August 12, 2026 01:03
devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration[bot]

This comment was marked as resolved.

@uuzzrm

uuzzrm commented Aug 12, 2026

Copy link
Copy Markdown
Contributor Author

Addressed the two new review findings in commit 642106d.

  • Re-check the session/tool state after a generation timeout before closing the active generation, so a tool restore during the timeout window does not truncate a live reply.
  • Keep shutdown cleanup running when an in-progress recycle has already cleared the session-active flag; this closes the streams and cancels the remaining tasks instead of returning early.

Added deterministic regression coverage for both cases.

Verification:

  • pytest -q livekit-plugins/livekit-plugins-aws/tests/test_nova_sonic_tool_recycle.py — 12 passed
  • Ruff check — passed
  • Ruff format check — passed
  • git diff --check — passed

AI assistance was used and the final diff and test results were reviewed.

Signed-off-by: Ruiming Zhao <uuzzrm@gmail.com>
devin-ai-integration[bot]

This comment was marked as resolved.

@uuzzrm

uuzzrm commented Aug 12, 2026

Copy link
Copy Markdown
Contributor Author

Addressed the latest Devin Review finding in a74f1843f.

The recycle task now propagates cancellation when shutdown interrupts the wait for its response or audio child task. Previously those CancelledError handlers could swallow cancellation from aclose() and allow the recycle to continue into initialize_streams(is_restart=True), opening a new Bedrock connection during shutdown.

Added a regression test that exercises cancellation during the child-task wait and asserts that stream initialization is not restarted.

Verification:

  • pytest -q livekit-plugins/livekit-plugins-aws/tests/test_nova_sonic_tool_recycle.py — 13 passed
  • Ruff check — passed
  • Ruff format check — passed
  • git diff --check — passed

AI assistance was used and the final diff and test results were reviewed.

devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration[bot]

This comment was marked as resolved.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ToolFlag.IGNORE_ON_ENTER Is Broken With Amazon Nova 2 Sonic

1 participant