🤖 feat: emit compaction_completed telemetry (#1201)

Backend now emits a `compaction_completed` telemetry event (PostHog)
after successful history compaction, including privacy-preserving
(base-2 rounded) token bucket metrics.

Also adds typed payload + oRPC schema validation and unit coverage for
the backend emission.

---

<details>
<summary>📋 Implementation Plan</summary>

# 🤖 Compaction completed telemetry (PostHog)

## Goals

- Capture a dedicated telemetry event whenever a history **compaction
completes successfully**.
- Include **token deltas** so we can quantify effectiveness:
  - **starting token count** (pre-compaction)
  - **new token count** (post-compaction summary)
- Send data to **PostHog** so we can later:
  - build dashboards (compaction size distribution, reduction ratios)
  - segment by model / compaction type
  - trigger in-app UX (e.g., show a feedback survey after compaction)

## Non-goals (for this change)

- No new UI (survey modal, toast, etc.).
- No raw user content / summary text ever leaves the machine.
- No changes to compaction behavior itself.

---

## Recommended approach (backend-only capture in
`CompactionHandler.handleCompletion()`)

**Why:** Compaction already happens in the main process, and you
explicitly want the backend to be the sole emitter (no renderer
dependency).

**Net new LoC (product code):** ~120–180

### 1) Define a new telemetry event payload (transparency + typing)

Add a new payload type + union member:

- `src/common/telemetry/payload.ts`
  - `export interface CompactionCompletedPayload { ... }`
  - extend `TelemetryEventPayload` with:
- `{ event: "compaction_completed"; properties:
CompactionCompletedPayload }`

Suggested properties (privacy-preserving, base-2 rounded):

- `model: string`
- `duration_b2: number` (seconds)
- `input_tokens_b2: number` (proxy for pre-compaction history size)
- `output_tokens_b2: number` (proxy for post-compaction history size)
- `compaction_source: "manual" | "idle"`

**Token source of truth:** prefer provider-reported usage from the
compaction stream metadata (same tokenizer/model as the compaction):

- `inputTokens` = `event.metadata.contextUsage?.inputTokens ??
event.metadata.usage?.inputTokens ?? 0`
- `outputTokens` = `event.metadata.contextUsage?.outputTokens ??
event.metadata.usage?.outputTokens ?? 0`

### 2) Keep Zod telemetry schema in sync (even if the renderer never
emits this event)

- `src/common/orpc/schemas/telemetry.ts`
  - add `CompactionCompletedPropertiesSchema`
- add `{ event: z.literal("compaction_completed"), properties: ... }` to
`TelemetryEventSchema`

### 3) Emit from the backend after successful compaction

- `src/node/services/compactionHandler.ts`
- Extend `CompactionHandlerOptions` with optional `telemetryService?:
TelemetryService`.
- After `performCompaction(...)` succeeds (and after the dedupe check),
call:
- `telemetryService?.capture({ event: "compaction_completed",
properties: { ... } })`
- Apply privacy rounding using `roundToBase2`
(`src/common/telemetry/utils.ts`).
- `compaction_source` should be derived from the already-existing
`isIdleCompaction` check.
- Ensure the event is emitted **once** per compaction-request (reuse
`processedCompactionRequestIds`).

- Wire the TelemetryService through:
  - `WorkspaceService` → `new AgentSession({ telemetryService: ... })`
  - `AgentSession` → `new CompactionHandler({ telemetryService: ... })`
- Keep the option optional to avoid rewriting unit tests that don’t care
about telemetry.

<details>
<summary>Alternative approach (renderer-emitted) — NOT recommended per
requirements</summary>

The existing telemetry pipeline is renderer → oRPC → backend → PostHog.
While that’s usually fine (and still avoids classic adblocker problems
because it’s not PostHog JS), you explicitly want *only* backend
emission, so this plan does not rely on `trackEvent()` from the
renderer.

</details>

---

## Tests + validation

- Unit tests (recommended): extend
`src/node/services/compactionHandler.test.ts`
  - Inject a fake `telemetryService` with a spy `capture()`.
- Assert `compaction_completed` is captured exactly once on successful
compaction.
  - Assert it is **not** captured for non-compaction stream-end events.

- Repo checks:
  - `make fmt-check && make lint && make typecheck && make test`

- Manual smoke test:
  - Run packaged, or set `MUX_ENABLE_TELEMETRY_IN_DEV=1`.
  - Trigger `/compact`.
- Confirm `compaction_completed` appears in PostHog with rounded
properties.

---

## PostHog: create dashboard + insights (via MCP tools)

**Net new LoC (product code):** 0 (PostHog objects only)

### 1) Identify the correct PostHog org/project

Use the MCP tools to confirm we’re operating in the PostHog project that
receives mux telemetry:

- `posthog_organizations-get`
- `posthog_projects-get`

### 2) Create a dashboard

- `posthog_dashboard-create` with:
  - `name`: `Mux • Compaction`
  - `description`: `Compaction completion volume and effectiveness`
  - `pinned`: `true` (optional)
  - `tags`: `["mux", "compaction"]`

### 3) Create insights (query-run → create → attach to dashboard)

For each insight:

1) `posthog_query-run` (validate the query returns successfully)
2) `posthog_insight-create-from-query`
3) `posthog_insight-update` to attach it to the new dashboard (set
`dashboard: <id>`)

Suggested insights:

- **Trend:** daily count of `compaction_completed` (last 30 days)
- **Breakdown:** `compaction_completed` by `compaction_source` (manual
vs idle)
- **Table (HogQL):** effectiveness by `model` and `compaction_source`

<details>
<summary>Example HogQL for the effectiveness table</summary>

```sql
SELECT
  properties['model']               AS model,
  properties['compaction_source']   AS compaction_source,
  count()                           AS compactions,
  avg(toFloat(properties['input_tokens_b2']))  AS avg_input_tokens_b2,
  avg(toFloat(properties['output_tokens_b2'])) AS avg_output_tokens_b2,
  avg(toFloat(properties['output_tokens_b2']) / nullIf(toFloat(properties['input_tokens_b2']), 0)) AS avg_ratio
FROM events
WHERE event = 'compaction_completed'
  AND timestamp >= now() - INTERVAL 30 DAY
GROUP BY model, compaction_source
ORDER BY compactions DESC
```

</details>

### 4) Verify dashboard wiring

- `posthog_dashboard-get` (ensure the dashboard exists)
- `posthog_insight-query` for each insight (ensure the query executes;
it may legitimately be empty until events arrive)

---

## Future follow-ups (not in scope)

- **Survey after compaction:** gate a post-compaction prompt with a
PostHog feature flag (evaluated via the existing backend
`ExperimentsService`), then record survey answers to PostHog as a
separate event.
- Consider migrating other telemetry events to backend-only for
consistency.


</details>

---
_Generated with `mux` • Model: `openai:gpt-5.2` • Thinking: `high`_

Signed-off-by: Thomas Kosiewski <[email protected]>

Author: ThomasK33

src/common/orpc/schemas/telemetry.ts

@@ -78,6 +78,14 @@ const StreamCompletedPropertiesSchema = z.object({
   output_tokens_b2: z.number(),
 });
 
+const CompactionCompletedPropertiesSchema = z.object({
+  model: z.string(),
+  duration_b2: z.number(),
+  input_tokens_b2: z.number(),
+  output_tokens_b2: z.number(),
+  compaction_source: z.enum(["manual", "idle"]),
+});
+
 const ProviderConfiguredPropertiesSchema = z.object({
   provider: z.string(),
   keyType: z.string(),
@@ -125,6 +133,10 @@ export const TelemetryEventSchema = z.discriminatedUnion("event", [
     event: z.literal("stream_completed"),
     properties: StreamCompletedPropertiesSchema,
   }),
+  z.object({
+    event: z.literal("compaction_completed"),
+    properties: CompactionCompletedPropertiesSchema,
+  }),
   z.object({
     event: z.literal("provider_configured"),
     properties: ProviderConfiguredPropertiesSchema,

src/common/telemetry/payload.ts

@@ -122,6 +122,22 @@ export interface StreamCompletedPayload {
   output_tokens_b2: number;
 }
 
+/**
+ * Compaction completion event - tracks when history compaction finishes
+ */
+export interface CompactionCompletedPayload {
+  /** Model used for compaction */
+  model: string;
+  /** Duration in seconds, rounded to nearest power of 2 */
+  duration_b2: number;
+  /** Input tokens (pre-compaction history size), rounded to nearest power of 2 */
+  input_tokens_b2: number;
+  /** Output tokens (post-compaction summary size), rounded to nearest power of 2 */
+  output_tokens_b2: number;
+  /** Whether this compaction was user-triggered vs idle */
+  compaction_source: "manual" | "idle";
+}
+
 /**
  * Provider configuration event - tracks when users set up providers
  * Note: Only tracks that a key was set, never the actual value
@@ -211,6 +227,7 @@ export type TelemetryEventPayload =
   | { event: "workspace_switched"; properties: WorkspaceSwitchedPayload }
   | { event: "message_sent"; properties: MessageSentPayload }
   | { event: "stream_completed"; properties: StreamCompletedPayload }
+  | { event: "compaction_completed"; properties: CompactionCompletedPayload }
   | { event: "provider_configured"; properties: ProviderConfiguredPayload }
   | { event: "command_used"; properties: CommandUsedPayload }
   | { event: "voice_transcription"; properties: VoiceTranscriptionPayload }

src/node/services/agentSession.ts

@@ -30,6 +30,7 @@ import { createRuntime } from "@/node/runtime/runtimeFactory";
 import { MessageQueue } from "./messageQueue";
 import type { StreamEndEvent } from "@/common/types/stream";
 import { CompactionHandler } from "./compactionHandler";
+import type { TelemetryService } from "./telemetryService";
 import type { BackgroundProcessManager } from "./backgroundProcessManager";
 import { computeDiff } from "@/node/utils/diff";
 import { AttachmentService } from "./attachmentService";
@@ -89,6 +90,7 @@ interface AgentSessionOptions {
   partialService: PartialService;
   aiService: AIService;
   initStateManager: InitStateManager;
+  telemetryService?: TelemetryService;
   backgroundProcessManager: BackgroundProcessManager;
   /** Called when compaction completes (e.g., to clear idle compaction pending state) */
   onCompactionComplete?: () => void;
@@ -147,6 +149,7 @@ export class AgentSession {
       partialService,
       aiService,
       initStateManager,
+      telemetryService,
       backgroundProcessManager,
       onCompactionComplete,
       onPostCompactionStateChange,
@@ -169,6 +172,7 @@ export class AgentSession {
     this.compactionHandler = new CompactionHandler({
       workspaceId: this.workspaceId,
       historyService: this.historyService,
+      telemetryService,
       partialService: this.partialService,
       emitter: this.emitter,
       onCompactionComplete,

src/node/services/compactionHandler.test.ts

@@ -5,6 +5,8 @@ import type { PartialService } from "./partialService";
 import type { EventEmitter } from "events";
 import { createMuxMessage, type MuxMessage } from "@/common/types/message";
 import type { StreamEndEvent } from "@/common/types/stream";
+import type { TelemetryService } from "./telemetryService";
+import type { TelemetryEventPayload } from "@/common/telemetry/payload";
 import { Ok, Err, type Result } from "@/common/types/result";
 
 interface EmittedEvent {
@@ -117,6 +119,8 @@ describe("CompactionHandler", () => {
   let mockHistoryService: ReturnType<typeof createMockHistoryService>;
   let mockPartialService: ReturnType<typeof createMockPartialService>;
   let mockEmitter: EventEmitter;
+  let telemetryCapture: ReturnType<typeof mock>;
+  let telemetryService: TelemetryService;
   let emittedEvents: EmittedEvent[];
   const workspaceId = "test-workspace";
 
@@ -125,12 +129,18 @@ describe("CompactionHandler", () => {
     mockEmitter = emitter;
     emittedEvents = events;
 
+    telemetryCapture = mock((_payload: TelemetryEventPayload) => {
+      void _payload;
+    });
+    telemetryService = { capture: telemetryCapture } as unknown as TelemetryService;
+
     mockHistoryService = createMockHistoryService();
     mockPartialService = createMockPartialService();
 
     handler = new CompactionHandler({
       workspaceId,
       historyService: mockHistoryService as unknown as HistoryService,
+      telemetryService,
       partialService: mockPartialService as unknown as PartialService,
       emitter: mockEmitter,
     });
@@ -160,6 +170,37 @@ describe("CompactionHandler", () => {
       expect(result).toBe(false);
     });
 
+    it("should capture compaction_completed telemetry on successful compaction", async () => {
+      const compactionReq = createCompactionRequest();
+      setupSuccessfulCompaction(mockHistoryService, [compactionReq]);
+
+      const event = createStreamEndEvent("Summary", {
+        duration: 1500,
+        // Prefer contextUsage (context size) over total usage.
+        contextUsage: { inputTokens: 1000, outputTokens: 333, totalTokens: undefined },
+      });
+
+      await handler.handleCompletion(event);
+
+      expect(telemetryCapture.mock.calls).toHaveLength(1);
+      const payload = telemetryCapture.mock.calls[0][0] as TelemetryEventPayload;
+      expect(payload.event).toBe("compaction_completed");
+      if (payload.event !== "compaction_completed") {
+        throw new Error("Expected compaction_completed payload");
+      }
+
+      expect(payload.properties).toEqual({
+        model: "claude-3-5-sonnet-20241022",
+        // 1.5s -> 2
+        duration_b2: 2,
+        // 1000 -> 1024
+        input_tokens_b2: 1024,
+        // 333 -> 512
+        output_tokens_b2: 512,
+        compaction_source: "manual",
+      });
+    });
+
     it("should return true when successful", async () => {
       const compactionReq = createCompactionRequest();
       mockHistoryService.mockGetHistory(Ok([compactionReq]));

src/node/services/compactionHandler.ts

@@ -9,6 +9,8 @@ import { Ok, Err } from "@/common/types/result";
 import type { LanguageModelV2Usage } from "@ai-sdk/provider";
 
 import { createMuxMessage, type MuxMessage } from "@/common/types/message";
+import type { TelemetryService } from "@/node/services/telemetryService";
+import { roundToBase2 } from "@/common/telemetry/utils";
 import { log } from "@/node/services/log";
 import {
   extractEditedFileDiffs,
@@ -20,6 +22,7 @@ interface CompactionHandlerOptions {
   workspaceId: string;
   historyService: HistoryService;
   partialService: PartialService;
+  telemetryService?: TelemetryService;
   emitter: EventEmitter;
   /** Called when compaction completes successfully (e.g., to clear idle compaction pending state) */
   onCompactionComplete?: () => void;
@@ -37,6 +40,7 @@ export class CompactionHandler {
   private readonly workspaceId: string;
   private readonly historyService: HistoryService;
   private readonly partialService: PartialService;
+  private readonly telemetryService?: TelemetryService;
   private readonly emitter: EventEmitter;
   private readonly processedCompactionRequestIds: Set<string> = new Set<string>();
   private readonly onCompactionComplete?: () => void;
@@ -50,6 +54,7 @@ export class CompactionHandler {
     this.workspaceId = options.workspaceId;
     this.historyService = options.historyService;
     this.partialService = options.partialService;
+    this.telemetryService = options.telemetryService;
     this.emitter = options.emitter;
     this.onCompactionComplete = options.onCompactionComplete;
   }
@@ -130,6 +135,24 @@ export class CompactionHandler {
       return false;
     }
 
+    const durationSecs =
+      typeof event.metadata.duration === "number" ? event.metadata.duration / 1000 : 0;
+    const inputTokens =
+      event.metadata.contextUsage?.inputTokens ?? event.metadata.usage?.inputTokens ?? 0;
+    const outputTokens =
+      event.metadata.contextUsage?.outputTokens ?? event.metadata.usage?.outputTokens ?? 0;
+
+    this.telemetryService?.capture({
+      event: "compaction_completed",
+      properties: {
+        model: event.metadata.model,
+        duration_b2: roundToBase2(durationSecs),
+        input_tokens_b2: roundToBase2(inputTokens ?? 0),
+        output_tokens_b2: roundToBase2(outputTokens ?? 0),
+        compaction_source: isIdleCompaction ? "idle" : "manual",
+      },
+    });
+
     // Notify that compaction completed (clears idle compaction pending state)
     this.onCompactionComplete?.();
 

src/node/services/serviceContainer.ts

@@ -116,6 +116,7 @@ export class ServiceContainer {
     this.menuEventService = new MenuEventService();
     this.voiceService = new VoiceService(config);
     this.telemetryService = new TelemetryService(config.rootDir);
+    this.workspaceService.setTelemetryService(this.telemetryService);
     this.experimentsService = new ExperimentsService({
       telemetryService: this.telemetryService,
       muxHome: config.rootDir,

src/node/services/workspaceService.ts

@@ -13,6 +13,7 @@ import type { PartialService } from "@/node/services/partialService";
 import type { AIService } from "@/node/services/aiService";
 import type { InitStateManager } from "@/node/services/initStateManager";
 import type { ExtensionMetadataService } from "@/node/services/ExtensionMetadataService";
+import type { TelemetryService } from "@/node/services/telemetryService";
 import type { ExperimentsService } from "@/node/services/experimentsService";
 import { EXPERIMENT_IDS, EXPERIMENTS } from "@/common/constants/experiments";
 import type { MCPServerManager } from "@/node/services/mcpServerManager";
@@ -120,6 +121,7 @@ export class WorkspaceService extends EventEmitter {
     this.setupMetadataListeners();
   }
 
+  private telemetryService?: TelemetryService;
   private experimentsService?: ExperimentsService;
   private mcpServerManager?: MCPServerManager;
   // Optional terminal service for cleanup on workspace removal
@@ -133,6 +135,10 @@ export class WorkspaceService extends EventEmitter {
     this.mcpServerManager = manager;
   }
 
+  setTelemetryService(telemetryService: TelemetryService): void {
+    this.telemetryService = telemetryService;
+  }
+
   setExperimentsService(experimentsService: ExperimentsService): void {
     this.experimentsService = experimentsService;
   }
@@ -301,6 +307,7 @@ export class WorkspaceService extends EventEmitter {
       historyService: this.historyService,
       partialService: this.partialService,
       aiService: this.aiService,
+      telemetryService: this.telemetryService,
       initStateManager: this.initStateManager,
       backgroundProcessManager: this.backgroundProcessManager,
       onCompactionComplete: () => {