feat: report tool calls on $ai_generation

[soukiassianb]

Problem

PostHog's AI observability Tools tab is empty for every OpenCode user of this plugin, even when the agent is calling tools constantly.

The plugin captures tool calls as $ai_span events with $ai_span_name set to the tool name. That's a documented use of spans and it makes the trace timeline render correctly. But PostHog extracts tool usage exclusively from $ai_generation — the ingestion pipeline's first branch is:

// nodejs/src/ingestion/pipelines/ai/tools/index.ts
if (event.event !== '$ai_generation') {
    return event
}

So $ai_tools_called and $ai_tool_call_count are never set, and the Tools tab, Tool trends, and Tool co-occurrence have nothing to aggregate. The data is captured — it's just in a shape those views can't see.

Measured on a project with heavy tool use: 0 of 169 $ai_generation events carried tool data, while $ai_span held the complete tool breakdown.

Changes

buildAiGeneration now sets $ai_tools_called to the names of the tools that step called, in call order:

• TraceState gains stepToolCalls: string[], reset in handleStepStart alongside the existing per-step state.
• handleToolPart records the tool name after emitting its span.
• buildAiGeneration emits a copy of the array, or null when the step called no tools.

Spans are untouched, so the trace tree keeps its current shape.

Why set $ai_tools_called directly rather than adding tool_calls to $ai_output_choices: ingestion respects a user-provided value and normalizes arrays, JSON-stringified arrays, and comma strings, deriving $ai_tool_call_count when absent. The plugin already knows the exact names, so reporting them is more robust than emitting a provider-shaped payload and relying on format detection.

Privacy: names only, no arguments. $ai_span_name already carries tool names unredacted in privacy mode, so this exposes nothing new. Covered by a test.

Compatibility: additive. A generation that called no tools sends null, which ingestion treats exactly as the property being absent today.

How did you test this code?

Automated tests, run locally:

• pnpm test — 47 passed (42 pre-existing + 5 new).
• Negative control: reverted src/events.ts to its original content and re-ran. Exactly the 5 new tests failed, with expected undefined to deeply equal [ 'read', 'edit' ] and similar; all 42 pre-existing tests still passed. The tests fail for the intended reason rather than passing vacuously.
• pnpm typecheck — clean.
• pnpm lint — clean (one pre-existing no-await-in-loop warning on src/index.test.ts:23, untouched here).

The new integration test drives the plugin through a real two-step OpenCode event sequence and asserts step 1 reports ['read', 'edit'] while step 2 reports null, and that the tool spans are parented to the same generation the names were attributed to — so both views of a step agree. The unit tests cover call order, the empty case, privacy mode, and that the array is copied so a later step can't mutate an already-captured event.

That the names then reach the Tools tab follows from the ingestion path quoted above; I verified the plugin side, not a live round-trip.

One assumption is load-bearing and worth naming: tool parts arrive between step-start and step-finish. It isn't new — buildAiSpan already parents every tool span to trace.currentGenerationSpanId, which holds only under the same ordering, and the existing handleStepStart comments state it outright. The integration test now encodes it, so a future change to that ordering fails loudly instead of silently misattributing.

Checklist

• Tests for new code (if applicable)
• Accounted for backwards compatibility of any changes (no breaking changes!)

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

Found while investigating why a production OpenCode agent's tool usage wasn't reaching PostHog's Tools tab: confirmed events were flowing, found no tool fields on $ai_generation, traced that to buildAiGeneration, then read PostHog's ingestion extractor to establish what it looks for. Adding tool_calls to $ai_output_choices would also work — extract-tool-calls.ts handles that shape — but it leans on format detection, so setting $ai_tools_called directly won.

Written by Claude Opus 5 in Claude Code, directed by @soukiassianb.

[marandaneto]

you can also check how https://github.com/PostHog/posthog-js/tree/main/packages/ai is done

[marandaneto]

Commits must have verified signatures.

make sure you verify your commits, the previous one as well
moving to draft until all comments are resolved

[marandaneto]

thanks for the PR @soukiassianb
i also made sure that the current version is under npm https://github.com/PostHog/posthog-opencode/releases/tag/0.1.0
this package wasnt published yet, now it is

[marandaneto]

Addressed the signing requirement as well: the original contributor commit was rewritten with a verified signature, and the merge plus review-fix commits are also verified. All commits currently on the PR branch now show verified signatures.