feat: governed realtime media membrane

[sandl99]

Problem Statement

Related: NVIDIA/NemoClaw#6207.
VoiceClaw referencces

• Prioritized VoiceClaw platform-issues spreadsheet
• Detailed VoiceClaw platform-issues document

P0: OpenShell needs a governed realtime media path for audio streams.
OpenShell 0.0.72 can relay WebSocket binary bytes, but only the WebSocket upgrade and text messages have policy semantics today. WebRTC/TURN-style media is also outside the current TCP CONNECT proxy model.

VoiceClaw needs realtime audio without turning the sandbox boundary into an opaque raw socket escape. Media must remain deny-by-default, auditable, rate-limited, and bound to declared destinations and sessions.

Who will use this

• VoiceClaw realtime voice assistant.
• OpenClaw voice-call and Talk sessions.
• NemoClaw sandboxes running voice-enabled OpenClaw.
• NVIDIA speech integrations such as Parakeet ASR, Magpie TTS, and PersonaPlex once its serving wrapper is confirmed.
• Operators and security teams who need media egress governed by OpenShell policy instead of hidden behind host bridges.

Concrete protocol evidence

• Parakeet is served through NVIDIA ASR NIM / Riva Speech Recognition:
  • HTTP REST for offline transcription.
  • gRPC over HTTP/2 for RivaSpeechRecognition/Recognize and bidirectional StreamingRecognize.
  • Realtime WebSocket for /v1/realtime?intent=transcription, using JSON events with base64 audio.
• Magpie is served through NVIDIA TTS NIM / Riva Speech Synthesis:
  • HTTP REST for /v1/audio/synthesize and /v1/audio/synthesize_online.
  • gRPC over HTTP/2 for RivaSpeechSynthesis/Synthesize and server-streaming SynthesizeOnline.
  • Realtime WebSocket for /v1/realtime?intent=synthesize, using JSON events.
• PersonaPlex is publicly described as a full-duplex speech-to-speech model/checkpoint, not a documented Speech NIM API surface:
  • Treat its network protocol as unconfirmed until the serving wrapper is known.
  • If wrapped as Riva/NIM, expect gRPC or realtime WebSocket.
  • If exposed directly for browser voice, expect WebSocket or WebRTC.

Proposed Design

Add a governed media protocol surface to OpenShell.

This could be a new protocol: media, or staged support through:

1. WEBSOCKET_BINARY for realtime audio over WebSocket.
2. Realtime WebSocket JSON media policy.
3. protocol: grpc for streaming gRPC.
4. Optional TURN/WebRTC support later, only when there is a concrete sandbox use case.

The media policy should include:

• transport allowlist
• codec allowlist
• destination and session binding
• direction controls
• max frame/message size
• bitrate limits
• session duration limits
• concurrency limits
• OCSF media session logging

Alternatives Considered

• Keep using raw L4 passthrough. This moves bytes, but it cannot express media policy, session caps, gRPC method policy, or useful audit semantics.
• Keep VoiceClaw audio on the host forever. This avoids sandbox risk but forces every voice/media integration to rebuild a host-side shadow plane.
• Add raw UDP/RTP/WebRTC first. The current VoiceClaw POC does not require that inside the sandbox. Start with WebSocket binary and gRPC streaming, then add TURN/WebRTC only with a concrete requirement.

Agent Investigation

• Reviewed NemoClaw #6207, which frames this as the P0 OpenShell ask: governed realtime media without raw socket escape.
• Inspected local ../openclaw Voice Call code. Carrier and provider paths use WebSocket media streams; some providers use JSON events with base64 audio and some use binary audio frames.
• Checked OpenShell docs/code. protocol: websocket governs upgrade and client text frames; binary WebSocket frames are relayed but not policy-governed. protocol: grpc is not currently listed as an L7 protocol.
• Checked NVIDIA Speech NIM docs for Parakeet and Magpie protocols.

Sources:

• NVIDIA Speech NIM overview: https://docs.nvidia.com/nim/speech/latest/about/index.html
• ASR support matrix / Parakeet: https://docs.nvidia.com/nim/speech/latest/reference/support-matrix/asr.html
• ASR gRPC API: https://docs.nvidia.com/nim/speech/latest/reference/api-references/asr/protos.html
• ASR realtime WebSocket API: https://docs.nvidia.com/nim/speech/latest/reference/api-references/asr/realtime-asr.html
• TTS support matrix / Magpie: https://docs.nvidia.com/nim/speech/latest/reference/support-matrix/tts.html
• TTS HTTP API: https://docs.nvidia.com/nim/speech/latest/reference/api-references/tts/http-tts.html
• TTS gRPC API: https://docs.nvidia.com/nim/speech/latest/reference/api-references/tts/protos.html
• TTS realtime WebSocket API: https://docs.nvidia.com/nim/speech/latest/reference/api-references/tts/realtime-tts.html
• PersonaPlex paper: https://arxiv.org/abs/2602.06053

Acceptance Criteria

• Realtime audio can be allowed without granting arbitrary raw socket egress.
• WebSocket binary media is denied unless explicitly allowed.
• Realtime WebSocket JSON media can be governed by event type and size limits.
• gRPC streaming can be restricted by host, service, method, metadata, size, rate, duration, and concurrency.
• Media sessions have frame/message size, bitrate, duration, and concurrency caps.
• OCSF logs record allow, deny, and limit decisions without logging audio payloads, base64 media, or secrets.

Non-Goals

• Unrestricted raw socket escape.
• Generic UDP/RTP/WebRTC support in the first implementation.
• Audio transcription or semantic inspection inside OpenShell.
• OpenClaw provider implementation.
• NemoClaw plugin packaging work.

Checklist

• I've reviewed existing issues and the architecture docs
• This is a design proposal, not a "please build this" request

[johntmyers]

@sandl99 are you able to get media working by adding a target endpoint and using tls:skip to allow traffic to pass through un-inspected? I'd like to just know if the proxy will pass this traffic currently.

As far as adding new protocol: media|grpc types, we are unlikely to support that. This is a great candidate for using middleware: #1738 - so you could develop a small service that analyzes payloads and annotate allow/deny semantics that will trigger the OpenShell Proxy to take action.

What is the concrete MVP you would want to build here? Can we start with one of Parakeet, Magpie, etc and a small governance surface area? If the MVP requirements can be narrowed down we can work with you on using the upcoming middleware feature.