🤖 feat: emit compaction_completed telemetry

[ThomasK33]

Backend now emits a compaction_completed telemetry event (PostHog) after successful history compaction, including privacy-preserving (base-2 rounded) token bucket metrics.

Also adds typed payload + oRPC schema validation and unit coverage for the backend emission.

---

📋 Implementation Plan

🤖 Compaction completed telemetry (PostHog)

Goals

• Capture a dedicated telemetry event whenever a history compaction completes successfully.
• Include token deltas so we can quantify effectiveness:
  • starting token count (pre-compaction)
  • new token count (post-compaction summary)
• Send data to PostHog so we can later:
  • build dashboards (compaction size distribution, reduction ratios)
  • segment by model / compaction type
  • trigger in-app UX (e.g., show a feedback survey after compaction)

Non-goals (for this change)

• No new UI (survey modal, toast, etc.).
• No raw user content / summary text ever leaves the machine.
• No changes to compaction behavior itself.

---

Recommended approach (backend-only capture in CompactionHandler.handleCompletion())

Why: Compaction already happens in the main process, and you explicitly want the backend to be the sole emitter (no renderer dependency).

Net new LoC (product code): ~120–180

1) Define a new telemetry event payload (transparency + typing)

Add a new payload type + union member:

• src/common/telemetry/payload.ts
  • export interface CompactionCompletedPayload { ... }
  • extend TelemetryEventPayload with:
    • { event: "compaction_completed"; properties: CompactionCompletedPayload }

Suggested properties (privacy-preserving, base-2 rounded):

• model: string
• duration_b2: number (seconds)
• input_tokens_b2: number (proxy for pre-compaction history size)
• output_tokens_b2: number (proxy for post-compaction history size)
• compaction_source: "manual" | "idle"

Token source of truth: prefer provider-reported usage from the compaction stream metadata (same tokenizer/model as the compaction):

• inputTokens = event.metadata.contextUsage?.inputTokens ?? event.metadata.usage?.inputTokens ?? 0
• outputTokens = event.metadata.contextUsage?.outputTokens ?? event.metadata.usage?.outputTokens ?? 0

2) Keep Zod telemetry schema in sync (even if the renderer never emits this event)

• src/common/orpc/schemas/telemetry.ts
  • add CompactionCompletedPropertiesSchema
  • add { event: z.literal("compaction_completed"), properties: ... } to TelemetryEventSchema

3) Emit from the backend after successful compaction

• src/node/services/compactionHandler.ts

  • Extend CompactionHandlerOptions with optional telemetryService?: TelemetryService.
  • After performCompaction(...) succeeds (and after the dedupe check), call:
    • telemetryService?.capture({ event: "compaction_completed", properties: { ... } })
  • Apply privacy rounding using roundToBase2 (src/common/telemetry/utils.ts).
  • compaction_source should be derived from the already-existing isIdleCompaction check.
  • Ensure the event is emitted once per compaction-request (reuse processedCompactionRequestIds).

• Wire the TelemetryService through:

  • WorkspaceService → new AgentSession({ telemetryService: ... })
  • AgentSession → new CompactionHandler({ telemetryService: ... })
  • Keep the option optional to avoid rewriting unit tests that don’t care about telemetry.

Alternative approach (renderer-emitted) — NOT recommended per requirements

The existing telemetry pipeline is renderer → oRPC → backend → PostHog. While that’s usually fine (and still avoids classic adblocker problems because it’s not PostHog JS), you explicitly want only backend emission, so this plan does not rely on trackEvent() from the renderer.

---

Tests + validation

• Unit tests (recommended): extend src/node/services/compactionHandler.test.ts

  • Inject a fake telemetryService with a spy capture().
  • Assert compaction_completed is captured exactly once on successful compaction.
  • Assert it is not captured for non-compaction stream-end events.

• Repo checks:

  • make fmt-check && make lint && make typecheck && make test

• Manual smoke test:

  • Run packaged, or set MUX_ENABLE_TELEMETRY_IN_DEV=1.
  • Trigger /compact.
  • Confirm compaction_completed appears in PostHog with rounded properties.

---

PostHog: create dashboard + insights (via MCP tools)

Net new LoC (product code): 0 (PostHog objects only)

1) Identify the correct PostHog org/project

Use the MCP tools to confirm we’re operating in the PostHog project that receives mux telemetry:

• posthog_organizations-get
• posthog_projects-get

2) Create a dashboard

• posthog_dashboard-create with:
  • name: Mux • Compaction
  • description: Compaction completion volume and effectiveness
  • pinned: true (optional)
  • tags: ["mux", "compaction"]

3) Create insights (query-run → create → attach to dashboard)

For each insight:

1. posthog_query-run (validate the query returns successfully)
2. posthog_insight-create-from-query
3. posthog_insight-update to attach it to the new dashboard (set dashboard: <id>)

Suggested insights:

• Trend: daily count of compaction_completed (last 30 days)
• Breakdown: compaction_completed by compaction_source (manual vs idle)
• Table (HogQL): effectiveness by model and compaction_source

Example HogQL for the effectiveness table

SELECT
  properties['model']               AS model,
  properties['compaction_source']   AS compaction_source,
  count()                           AS compactions,
  avg(toFloat(properties['input_tokens_b2']))  AS avg_input_tokens_b2,
  avg(toFloat(properties['output_tokens_b2'])) AS avg_output_tokens_b2,
  avg(toFloat(properties['output_tokens_b2']) / nullIf(toFloat(properties['input_tokens_b2']), 0)) AS avg_ratio
FROM events
WHERE event = 'compaction_completed'
  AND timestamp >= now() - INTERVAL 30 DAY
GROUP BY model, compaction_source
ORDER BY compactions DESC

4) Verify dashboard wiring

• posthog_dashboard-get (ensure the dashboard exists)
• posthog_insight-query for each insight (ensure the query executes; it may legitimately be empty until events arrive)

---

Future follow-ups (not in scope)

• Survey after compaction: gate a post-compaction prompt with a PostHog feature flag (evaluated via the existing backend ExperimentsService), then record survey answers to PostHog as a separate event.
• Consider migrating other telemetry events to backend-only for consistency.

---

Generated with mux • Model: openai:gpt-5.2 • Thinking: high