Repository navigation
Add the Session API crate - #1593
MaxHeimbrock wants to merge 41 commits into
Conversation
79fda31 to
97b4696
Compare
97b4696 to
ce21dc6
Compare
2d28c8d to
16e31a5
Compare
Changeset ✓This PR includes a changeset covering all affected packages:
|
This comment was marked as resolved.
This comment was marked as resolved.
48cb99a to
690e6bf
Compare
This comment was marked as resolved.
This comment was marked as resolved.
99c83dc to
5f9d744
Compare
There was a problem hiding this comment.
Note
Newer findings are available below. Devin Review posted a newer report on this PR, in addition to the findings presented here.
Devin Review found 2 potential issues.
1 flag not posted on this PR by your GitHub settings — view it in Devin Review. (Configure)
| livekit = { workspace = true } | ||
| livekit-token-source = { workspace = true } |
There was a problem hiding this comment.
🟡 Disabling defaults still enables native backends
Building livekit-session without default features still enables native networking. Both dependencies activate native by default, so the crate's native switch cannot disable either backend.
| livekit = { workspace = true } | |
| livekit-token-source = { workspace = true } | |
| livekit = { workspace = true, default-features = false } | |
| livekit-token-source = { workspace = true, default-features = false } |
Was this helpful? React with 👍 or 👎 to provide feedback.
| /// options, such as agent metadata. | ||
| pub fn with_agent( | ||
| token_source: impl TokenSourceConfigurable + Send + Sync + 'static, | ||
| agent_name: &str, | ||
| session_options: SessionOptions, | ||
| ) -> Self { |
There was a problem hiding this comment.
thought: It could be interesting to expose a builder type api here maybe? Session::new_configurable(configurable token source).with_agent("agent name").with_options(...) or something...
There was a problem hiding this comment.
I agree, I am of the opinion functions with more than two arguments should be avoided in Rust since there are no argument labels which creates ambiguity at the call site.
| /// A receiver of changes to the message feed. | ||
| /// | ||
| /// Unbounded, like [`Room::subscribe`]: the session must never block on a | ||
| /// slow subscriber, and the feed's rate is bounded by the conversation. | ||
| /// Drop the receiver to unsubscribe; it is removed on the next event. | ||
| pub fn subscribe(&self) -> mpsc::UnboundedReceiver<SessionEvent> { | ||
| self.inner.shared.subscribe() | ||
| } |
There was a problem hiding this comment.
thought: very interesting, this could play well into some stuff blaze has been talking about to make uniffi events more stream based...
| /// What the agent is doing. | ||
| pub agent: AgentState, | ||
| } |
There was a problem hiding this comment.
thought: We can talk about this in our 1:1, but some original thinking was done back in the day to try to make session able to support multiple agents in the future with minimal updates. It would be nice to try to carry over those decisions in here as well.
There was a problem hiding this comment.
Note
Newer findings are available below. Devin Review posted a newer report on this PR, in addition to the findings presented here.
Devin Review found 1 new potential issue.
1 flag not posted on this PR by your GitHub settings — view it in Devin Review. (Configure)
| } | ||
| // Dropped while the actor was publishing the microphone. Dropping its | ||
| // `stop` sender stops it. | ||
| drop(lifecycle.take_running(self.generation)); |
There was a problem hiding this comment.
🟡 Cancelled starts detach room tasks
Dropping start() during microphone publication drops the Running task handle without joining it. The actor closes in the background, but no owner observes its completion or failure.
Learn more
A running session keeps the actor's JoinHandle in Running, and Running::join normally drops the stop sender and awaits that handle. Here take_running removes the only handle, then drops it without awaiting. Tokio detaches the actor, so no owner can observe a failed shutdown or guarantee a join. The same unowned task path occurs when the lifecycle changes after spawning the actor.
Example: A caller times out session.start() while the microphone publication is pending. Its guard signals stop but discards the task handle; the actor closes the room asynchronously without a joiner.
Recommended fix: Arrange for both cancellation paths to transfer Running to a named shutdown owner that awaits Running::join, while preserving background teardown when start() is dropped. Add a lifecycle test that verifies the shutdown owner joins the task.
Was this helpful? React with 👍 or 👎 to provide feedback.
This comment was marked as outdated.
This comment was marked as outdated.
daa2e7d to
f79aa64
Compare
…imeouts This commit adds the unit tests of `AgentTracker`. The tests cover the precedence of the connection state, an agent that leaves, an agent missing during a reconnect, a recovery after `Left`, the reset after a disconnect, and the agent-connect timeout with and without an agent.
This commit adds `Dispatcher`. A `Dispatcher` sends each value to every subscriber. Each subscriber has its own unbounded channel. The dispatcher removes a subscriber on the next dispatch after its receiver is dropped. The `livekit` crate has the same type, but keeps it private.
This commit adds the unit test of `Dispatcher` and the `len` helper it uses. The test shows that a dropped receiver is removed on the next dispatch, and not on drop.
This commit adds `Shared` and the `lock` helper. `Shared` holds the session state, the message store and the event dispatcher. A `Session` and its actor both hold an `Arc<Shared>`. `Shared` holds no reference back to either, so it keeps neither alive. Each `start` begins a new generation. `publish` writes a state only when its generation is current. An actor of an earlier start that is still shutting down cannot overwrite the state of a newer start. `upsert` dispatches the message event under the store lock. Subscribers see updates in the order the store applied them. The `history` field exists only in tests. It records every published state, so tests can assert on states a `watch` receiver would skip. `lock` locks a mutex and recovers from poisoning.
This commit adds the public option, state and event types of the session. `SessionOptions` holds the agent-connect timeout, the room options, and the `platform_audio` flag. The flag defaults to `true`. The builder methods set each field. `SessionConnectionState` and `SessionState` describe the session at one point in time. `SessionEvent` reports a change to the message feed. All four types are `non_exhaustive`.
This commit adds the `Session` handle and the private types behind it. `SessionTokenSource` wraps a fixed or a configurable token source. The constructor that the caller uses selects the variant. `Session` is a cheap handle around `Inner`. `Inner` holds the token source, the options, the shared state and the `Lifecycle`. `Lifecycle` is `Idle`, `Starting` or `Running`. `Running` holds the room, the stop sender and the actor task. `join` drops the stop sender and waits for the actor. `Session::new` takes a fixed source. `Session::with_fetch_options` takes a configurable source and fetch options. `Session::with_agent` is a shorthand that sets the agent name.
This commit adds `SessionActor` and the `Connection` it runs. The actor runs one connected room for one `start`. It holds the room, the platform audio, the room events, the stop receiver, the agent deadline and the `AgentTracker`. It holds no `Session` handle, so dropping the last handle stops the actor. `run` selects between the stop channel, the room events and the agent deadline. A `Disconnected` room event or a closed stop channel ends the loop. On exit, the actor closes the stop receiver first. A `start` that follows then finds the actor gone. The actor closes the room when the session stopped it. It then publishes the default state. `on_room_event` tracks the connection state and refreshes the projection on participant and track changes. `on_agent_deadline` records the timeout in the tracker and refreshes the projection. Review the shutdown order and the ownership with care. See the "Memory lifecycle" section of AGENTS.md.
This commit adds the text-stream reading to `SessionActor`. `on_text_stream` takes the reader of a stream on the `lk.chat` or `lk.transcription` topic. It leaves streams on other topics to the app. A chat stream is read in full. A transcription stream is read chunk by chunk, followed by an end marker. The readers are polled in the actor loop, through a `SelectAll`. The `TranscriptAggregator` then has one writer. `on_reader_event` turns each chunk, end or chat into a `MessageUpdate` and upserts it into the shared store.
This commit adds `Session::start`, `Session::end`, `connect`, `publish_microphone` and `StartGuard`. `start` fails with `AlreadyStarted` while the session is starting or running. It begins a new generation, joins a stale actor, and then fetches credentials and connects the room. `end` can cancel this phase. Platform audio is created before the room connects. After the connect, `start` publishes the microphone. This step is not cancellable. A failed publish closes the room. `start` spawns the actor only when the lifecycle still holds its generation. Otherwise `end` was called during the connect. `start` then closes the room and returns `Cancelled`. `StartGuard` resets the lifecycle and the state when `start` fails or is dropped before the actor runs. `end` closes the room through the actor, or cancels a pending start. It does nothing when there is nothing to end. Review the cancellation paths with care. Each path must leave no room, task or audio device behind.
This commit adds the accessor methods of `Session`. `state` returns a `watch` receiver of the session state. `room` returns the room while it is connected or reconnecting. `agent_audio_track` and `agent_avatar_video_track` return the agent's media tracks, once subscribed. `send_text` sends chat to the agent on `lk.chat` and adds the text to the feed once sent. `messages`, `restore_messages` and `subscribe` read, replace and observe the feed.
This commit adds a unit test of `Session::with_agent`. A recording token source captures the fetch options. The test shows that the options carry the agent name.
This commit adds the end-to-end test module and its harness. The tests run against a `livekit-server --dev` and need the `__lk-e2e-test` feature. `TestEnv` builds tokens for a unique room. It joins a second room as the agent, as an avatar worker, or as an observer. The agent room sets the agent state attribute. The helper functions wait for a session state, poll a condition, and read the private lifecycle of a session. The root module declares the test module behind the feature.
This commit adds the first end-to-end tests of the agent projection. The tests show that a started session reports the agent once it joins, that `end` disconnects and is idempotent, that an agent that leaves is reported as `Left`, and that a full reconnect is not reported as `Left`.
This commit adds the end-to-end tests of the agent-connect timeout. The tests show that the timeout fires without an agent, that a late agent recovers the state, that the timeout is armed without an agent dispatch, and that the timeout fires for an agent stuck in `initializing`.
This commit adds the end-to-end test of the default platform audio. The test shares the platform-audio handle with the session. It shows that `start` publishes the microphone track and that `end` releases the audio devices. The test skips on a machine without a microphone.
This commit adds the end-to-end tests of the message feed. The first test shows that user input reaches the agent on `lk.chat`, that an agent transcript grows in place and is finalized by the trailer, that chat from the agent joins the feed, and that a failed send adds nothing. The second test shows that the session leaves streams on other topics to the app.
This commit adds the end-to-end tests of the agent media accessors. The helpers publish a silent audio track or a blank video track from the agent room. The tests show that only the microphone is the agent's audio, that only the avatar worker's camera is the avatar video, and that an avatar worker of kind agent is not taken for the agent.
This commit adds two end-to-end tests of the start path. The first test removes the user from the room through the server API. It shows that the session reports `Disconnected`, that a new `start` releases the old room, and that `end` after a disconnect is a no-op. The second test shows that a rejected token fails `start` with `Connect` and leaves the session idle.
This commit adds the end-to-end tests of a start that waits on the token source. `PendingSource` never answers. The tests show that `end` cancels the start with `Cancelled`, and that a dropped `start` future resets the session.
…ing behind This commit adds the end-to-end tests of a start that waits in `Room::connect`. `SilentServer` accepts one connection and never answers. The tests cancel the start with `end` or by a drop of the future. They show that the client hung up, that no actor was spawned, and that the task count returns to the baseline.
This commit adds the end-to-end lifecycle tests that AGENTS.md asks for. The tests show that dropping the last `Session` handle closes the room, that repeated start and end cycles leave no task or subscriber behind, and that failed starts spawn nothing. Each test checks observable destruction: a `Weak` that no longer upgrades, a strong count of one, or the task count of the runtime.
This commit adds the README of the crate. The README gives a usage example and explains the credentials, the lifecycle, the state table, the message feed, and the things the session owns.
This commit adds `examples/session` and registers it in the workspace. The example creates a session from a pre-generated token or from a development token server. It prints the session state and the completed messages. Lines typed on stdin go to the agent. By default, the example uses the platform audio. The `--text` flag turns it off. `Cargo.lock` adds the lock entry of the example.
This commit adds the knope changeset for the new crate. The summary is one line of prose, as the "Documenting changes" section of AGENTS.md requires.
9d07362 to
3697a81
Compare
New
livekit-sessioncrate withSession. Similar to the other session APIs in https://docs.livekit.io/frontends/build/sessions/.Look at the example or my PR in the Rust dev client for usage.
Disclaimer: I mostly used Claude to write this. I am still not a Rust expert. I tried to find wholes as much as possible, look at the developer facing APIs and test it myself through the Rust dev client.
Decisions I took:
The platform audio is activated for the user by default, opt out via the session options (
with_platform_audio = false).If you pass a cached token source, you will run into issues reusing the same token. This is a footgun we accept, since invalidating tokens after use is not easy to expose in the current design. Also fixed token sources are allowed, which cause the same problem.
We don't plan to expose this Rust session via FFI for now.
Using the actor pattern according to AGENT.md.
Public API
Every enum and the options/state structs are
#[non_exhaustive].Messageis not, so it can be built forrestore_messages.SessionCloneSession::new(token_source, options)TokenSourceFixed, e.g.literalSession::with_fetch_options(token_source, fetch_options, options)TokenSourceConfigurablewith fullTokenSourceFetchOptionsSession::with_agent(token_source, agent_name, options)with_fetch_optionsthat dispatchesagent_nameSession::start(&self) -> Result<(), SessionStartError>Session::end(&self)start; idempotentSession::state(&self) -> watch::Receiver<SessionState>borrow) and changes (changed,wait_for)Session::room(&self) -> Option<Arc<Room>>Session::agent_audio_track(&self) -> Option<RemoteAudioTrack>Session::agent_avatar_video_track(&self) -> Option<RemoteVideoTrack>Session::send_text(&self, text: &str) -> Result<Message, MessageSendError>lk.chatand adds it to the feed once sentSession::messages(&self) -> Vec<Message>seqSession::restore_messages(&self, messages: &[Message])Session::subscribe(&self) -> mpsc::UnboundedReceiver<SessionEvent>SessionOptions { agent_connect_timeout, room_options, platform_audio }DefaultRoomOptions, and whetherstartuses the platform audio devices (trueby default: the microphone is published and the agent's audio played); builderswith_agent_connect_timeout,with_room_options,with_platform_audioSessionState { connection, agent }Copy,Defaultstate()SessionConnectionStateDisconnected,Connecting,Connected,Reconnecting;is_connected()AgentStateCopyDisconnected,Connecting,Initializing,Idle,Listening,Thinking,Speaking,Failed(AgentFailure),Unknown;is_connected(),is_pending(),is_finished()AgentFailureTimeout,LeftSessionEventMessageUpserted(Message),MessagesRestoredMessage { id, seq, timestamp, origin, participant_identity, text, is_final }id+participant_identityMessageOriginAgentTranscript,UserTranscript,UserInput,ChatSessionStartErrorAlreadyStarted,Cancelled,Token(TokenSourceError),Connect(RoomError),PlatformAudio(AudioError)(no audio devices),Microphone(RoomError)(the microphone failed to publish; the room is closed again)MessageSendErrorNotConnected,Stream(StreamError)Commits
Group 1. Crate scaffolding
Group 2. Message feed
5. feat(session): add the Message model and the MessageStore feed
6. test(session): cover MessageStore ordering, upsert and restore
7. IMPORTANT(session): aggregate lk.transcription streams into transcript messages
8. test(session): cover transcript aggregation
Group 3. Agent projection
9. feat(session): add AgentState and AgentFailure
10. feat(session): find the agent, its avatar worker and their tracks
11. IMPORTANT(session): project AgentState with timeout and leave detection
12. test(session): cover agent discovery and state parsing
13. test(session): cover agent projection across reconnects, leaves and timeouts
Group 4. Session runtime
14. feat(session): add the Dispatcher fan-out
15. test(session): cover Dispatcher pruning
16. feat(session): add the state shared between a session and its actor
17. feat(session): add SessionOptions, SessionState and SessionEvent
18. feat(session): add the Session handle and its constructors
19. IMPORTANT(session): add the session actor and its agent-state projection
20. IMPORTANT(session): read chat and transcript streams in the actor
21. IMPORTANT(session): start and end a session
22. feat(session): expose state, room, agent tracks and the message feed
23. test(session): assert with_agent fetches with the agent name
Group 5. End-to-end tests
24. test(session): add the e2e harness with a stand-in agent
25. test(session): e2e agent state follows joins, leaves and reconnects
26. test(session): e2e agent connect timeout
27. test(session): e2e start publishes the platform microphone
28. test(session): e2e the feed merges chat and transcripts
29. test(session): e2e agent audio and avatar video tracks
30. test(session): e2e restart after a disconnect and a rejected token
31. test(session): e2e end and drop cancel a pending start
32. test(session): e2e a start cancelled during the handshake leaves nothing behind
33. test(session): e2e memory lifecycle
Group 6. Docs, example and changeset
34. docs(session): add the crate README
35. feat(example): add the session voice and text chat example
36. chore(session): add the changeset