feat(capture,store): join input events into T1 cards so slots record what the user did

AGENTS.md

@@ -20,7 +20,7 @@ Before editing, read every `AGENTS.md` along the path from root to leaf. Every `
 - [scripts/](scripts/AGENTS.md) — dev loop, signing/notarization/Sparkle release, publish; the root `Makefile` is the entry point
 - [site/](site/AGENTS.md) — afterray.com: React+Vite, Cloudflare Pages, R2-backed appcast/download functions
 - [docs/](docs/AGENTS.md) — specs and plans (some plans are historical; code wins)
-- [context/](context/) — navigation articles: [capture-pipeline](context/capture-pipeline.md), [wire-protocol](context/wire-protocol.md), [compute-governance](context/compute-governance.md), [agent-tools](context/agent-tools.md), [ocr-text-selection](context/ocr-text-selection.md); [CONTEXT-GAPS.md](context/CONTEXT-GAPS.md) — gaps backlog
+- [context/](context/) — navigation articles: [capture-pipeline](context/capture-pipeline.md), [wire-protocol](context/wire-protocol.md), [compute-governance](context/compute-governance.md), [agent-tools](context/agent-tools.md), [acts-join](context/acts-join.md), [ocr-text-selection](context/ocr-text-selection.md); [CONTEXT-GAPS.md](context/CONTEXT-GAPS.md) — gaps backlog
 - `skills/afterray/` — the shipped Agent Skill for the read-only CLI surface; keep in sync with `afterray-cli`
 
 ## Working agreements

apps/AfterRayCaptureShim/AGENTS.md

@@ -1,30 +1,35 @@
 # AGENTS.md — apps/AfterRayCaptureShim
 
-The ScreenCaptureKit boundary for the Rust daemon. It exists because the Rust workspace denies `unsafe_code` and ScreenCaptureKit delegates need unsafe FFI (see `README.md` here). A **standalone SwiftPM package** — its own `Package.swift` and `.build/`, deliberately not a target of the root package. The whole shim is one file: `Sources/AfterRayCaptureShim/main.swift` (~1350 lines).
+The ScreenCaptureKit boundary for the Rust daemon. It exists because the Rust workspace denies `unsafe_code` and ScreenCaptureKit delegates need unsafe FFI (see `README.md` here). A **standalone SwiftPM package** — its own `Package.swift` and `.build/`, deliberately not a target of the root package. The whole shim is one file: `Sources/AfterRayCaptureShim/main.swift` (~2160 lines).
 
 ## Key anchors
 
-- `main.swift:25` `Options` (`parse` at :31) — CLI flags (`--output-dir`, `--jpeg-quality`, audio, …)
-- `main.swift:99` `Event` — JSON-line event protocol emitted on stdout (`ready`, `artifact`, `warning`, `failed`, `stopped`)
-- `main.swift:1213` `InputCommand` — stdin commands; main loop at :1289-1320 handles `capture_screen` (requires `request_id`), `set_excluded_bundles` (carries `bundle_ids`), and `stop`
-- `main.swift:894` `ExcludedAudioGate` — drops audio while an excluded app is frontmost (see Invariants)
-- `main.swift:1332` `log()` — logging goes to **stderr only**
+- `main.swift:26` `Options` (`parse` at :32) — CLI flags (`--output-dir`, `--jpeg-quality`, audio, …)
+- `main.swift:104` `Event` — JSON-line event protocol emitted on stdout (`ready`, `artifact`, `warning`, `failed`, `input_events`, `stopped`)
+- `main.swift:1461` `InputEventMonitor` — listen-only tap + coalescing worker (see Invariants)
+- `main.swift:1978` `InputCommand` — stdin commands; main loop at :2071-2098 handles `capture_screen` (requires `request_id`), `set_excluded_bundles` (carries `bundle_ids`), and `stop`
+- `main.swift:971` `ExcludedAudioGate` — drops audio while an excluded app is frontmost (see Invariants)
+- `main.swift:2112` `log()` — logging goes to **stderr only**
 
 ## Invariants
 
 - stdout is reserved for JSON-line events — never print anything else there; the daemon parses it.
 - Screenshots are pull-based: Rust decides timing (`capture_screen`), the shim adds no hidden frame scheduler.
-- Output dir is hardened to `0700`, artifact files to `0600` (`main.swift:13,20`).
-- The shim excludes AfterRay's own windows from capture (`main.swift:1258-1265`).
+- Output dir is hardened to `0700`, artifact files to `0600` (`main.swift:12,19`).
+- The shim excludes AfterRay's own windows from capture (`main.swift:2034-2036`).
 - Each screenshot uses the display with the largest intersection with the AX focused window (`main window` is the AX fallback); no usable window frame falls back to `CGMainDisplayID`. The foreground PID, window id, and frame are rechecked around the screenshot. Keep the continuous audio stream separate from this per-tick display filter.
-- **A screen artifact is never emitted without its accessibility artifact** (`main.swift:1157`). The daemon's only exclusion check lives in the accessibility branch, so an unpaired screenshot can never be evaluated and would be kept whatever the user excluded. Every path that cannot produce a snapshot returns before the screenshot — keep it that way.
-- **Audio exclusions are enforced here, screen exclusions in the daemon.** A moment can be deleted once the snapshot names the app; a finished five-minute `m4a` cannot be sliced. `ExcludedAudioGate` (`main.swift:901`) therefore answers "which stretch of the recent past had no excluded app in front", not "is one in front now": samples are **held** (`AudioSegmentWriter.hold`) until a check vouches for the moment they arrived, and dropped otherwise. Writing first and cutting on the next check would leave every sample since the previous check inside a file the daemon imports and transcribes. The frontmost app is polled (100 ms — latency, not exposure) because the main thread blocks in `readLine` and never services a run loop, so `NSWorkspace` notifications would not arrive; the helper also holds all audio until the daemon's list arrives, since an app in front before that cannot be judged.
+- **A screen artifact is never emitted without its accessibility artifact** (`main.swift:1323`). The daemon's only exclusion check lives in the accessibility branch, so an unpaired screenshot can never be evaluated and would be kept whatever the user excluded. Every path that cannot produce a snapshot returns before the screenshot — keep it that way.
+- **Audio exclusions are enforced here, screen exclusions in the daemon.** A moment can be deleted once the snapshot names the app; a finished five-minute `m4a` cannot be sliced. `ExcludedAudioGate` (`main.swift:971`) therefore answers "which stretch of the recent past had no excluded app in front", not "is one in front now": samples are **held** (`AudioSegmentWriter.hold`) until a check vouches for the moment they arrived, and dropped otherwise. Writing first and cutting on the next check would leave every sample since the previous check inside a file the daemon imports and transcribes. The frontmost app is polled (100 ms — latency, not exposure) because the main thread blocks in `readLine` and never services a run loop, so `NSWorkspace` notifications would not arrive; the helper also holds all audio until the daemon's list arrives, since an app in front before that cannot be judged.
+- Input events: a listen-only `CGEventTap` on its own thread emits coalesced `input_events` batches — typing-burst counts (key codes classify command keys and never leave the callback), command keys (⌘-combos, Return/Tab/Esc), click/scroll targets resolved to element identity (coordinates dropped after resolution); a typing burst whose focus is not a text-entry role is attributed to the last click instead (`TypingTarget` — Electron and Zed report `AXWebArea`/`AXWindow`, and a landing point that coarse drags the run's scope to the whole window). Excluded apps and AfterRay itself are never recorded; fails closed before the daemon's list arrives, fails open (warning) when the tap cannot be created. See docs/input-events-and-t1-acts-plan.md.
+- R3 edge snapshots (`captureEdgeSnapshot`, :1801): a frontmost-bundle change or a click arms a candidate, paced by the pure `EdgeSnapshotPacing` (settle 500ms re-armed by any input, ≥5s apart, ≤6/min; a refused candidate is dropped, not queued). Spend goes through `fire(nowMs:walk:)` only, so a walk the guards decline cannot burn the minute's allowance. Walks the trigger's AXWindow with the same bounded encoder, emits `accessibility_edge`, and **never a screenshot** — an event-driven frame would outlive the 48h events behind it. Excluded apps, AfterRay, and all known browsers are skipped (the private-browsing gate needs an async probe a 1s tick cannot afford). Why: [acts-join](../../context/acts-join.md).
+- AX walk costs are bounded: the `AXMenuBar` subtree is stubbed (menus were 80–90% of walked nodes in native apps; every consumer treats them as chrome; deliberately not `truncated`), and the walk is time-boxed — process-global 100ms `AXUIElementSetMessagingTimeout` at startup + 500ms whole-walk deadline → `truncated`, same as the 20k node cap. A fresh Electron app's first snapshot may time out once while it builds its AX tree; the next heartbeat recovers.
 - Missing microphone input **or missing microphone TCC authorization** must never disable system-audio capture: gate `captureMicrophone`, its writer, and its stream output with `AudioCapturePlan`, while leaving `capturesAudio` enabled. Adding the microphone output without authorization makes `SCStream.startCapture` fail wholesale.
 - Requires **macOS 15** (`Package.swift:6`) while the rest of the app targets macOS 14 — intentional, not a bug.
 
 ## Build / test
 
 - `make capture-shim` → `swift build --package-path apps/AfterRayCaptureShim --product AfterRayCaptureShim` (Makefile:14-15); binary at `.build/release/AfterRayCaptureShim` under this directory
+- `swift test --package-path apps/AfterRayCaptureShim` — the package's own XCTest suite (`Tests/AfterRayCaptureShimTests`), covering the pure policy target `Sources/AfterRayCapturePolicy` (browser privacy, display selection, R3 pacing). `make test` runs it; plain `swift test` at the root does not. Logic that must be tested belongs in that target — the executable needs live TCC permissions.
 - Smoke test: run the binary with `--output-dir /tmp/…`, then send `{"command":"capture_screen","request_id":"smoke-1"}` on stdin (see this directory's `README.md`)
 
 ## Watch out

apps/AfterRayCaptureShim/Sources/AfterRayCapturePolicy/EdgeSnapshotPacing.swift

@@ -0,0 +1,96 @@
+/// When an R3 edge snapshot may be walked (docs/input-events-and-t1-acts-plan.md
+/// phase 4).
+///
+/// R1 heartbeats capture on a fixed cadence, so they miss content a person only
+/// looked at between two ticks — stepping into a conversation for eight seconds
+/// and leaving. R3 fills exactly that hole, and the whole decision of *when* is
+/// this state machine: a candidate (frontmost app changed, or a click), a settle
+/// window that any further input re-arms, and a token bucket.
+///
+/// Kept pure and separate from the tap so the timing rules are unit-testable:
+/// the failure modes here are "walked the tree while the user was still typing"
+/// and "walked it thirty times a minute", and neither is observable in a test
+/// that needs a live `CGEventTap`.
+package struct EdgeSnapshotPacing: Equatable {
+    /// Silence required after the last input before the tree may be walked.
+    /// An AX walk is synchronous IPC into the app the user is working in, so
+    /// walking mid-interaction is felt as lag in that app.
+    package static let settleMs: Int64 = 500
+    /// Floor between two walks.
+    package static let minSpacingMs: Int64 = 5_000
+    /// Ceiling per rolling minute.
+    package static let maxPerWindow = 6
+    /// Width of the rolling window `maxPerWindow` applies to.
+    package static let windowMs: Int64 = 60_000
+
+    /// The armed candidate's most recent re-arm instant, if one is armed.
+    private var candidateAtMs: Int64?
+    /// Fire instants inside the rolling window, oldest first.
+    private var fires: [Int64] = []
+
+    package init() {}
+
+    /// Arms a candidate: the frontmost bundle changed, or a click landed.
+    ///
+    /// A later candidate replaces an earlier one rather than queueing: the
+    /// snapshot's value is the state of the screen *now*, and one walk describes
+    /// the newest trigger as well as it describes any older one.
+    package mutating func arm(atMs: Int64) {
+        candidateAtMs = atMs
+    }
+
+    /// Records any input observation. While a candidate is armed this restarts
+    /// the settle window — the walk waits for the interaction to finish, however
+    /// long that takes. Input with nothing armed is not itself a trigger.
+    package mutating func observeInput(atMs: Int64) {
+        guard candidateAtMs != nil else { return }
+        candidateAtMs = atMs
+    }
+
+    /// Whether a walk may run now, consuming the candidate when it answers.
+    ///
+    /// A candidate refused by the bucket is **dropped**, not held: it would
+    /// otherwise fire seconds later against a screen that has moved on, and the
+    /// snapshot would be attributed to a trigger it no longer describes. The
+    /// next candidate is at most one interaction away.
+    ///
+    /// Answering yes does not spend the budget — `recordFire` does. The caller
+    /// still has cheap reasons to walk away (the frontmost app is a browser or
+    /// excluded, the window cannot be resolved), and a walk that never happened
+    /// must not starve the ones that would: clicking around one excluded app
+    /// would otherwise burn the whole minute's allowance.
+    package mutating func shouldFire(nowMs: Int64) -> Bool {
+        guard let candidate = candidateAtMs else { return false }
+        guard nowMs - candidate >= Self.settleMs else { return false }
+        candidateAtMs = nil
+        fires.removeAll { nowMs - $0 >= Self.windowMs }
+        if let last = fires.last, nowMs - last < Self.minSpacingMs {
+            return false
+        }
+        return fires.count < Self.maxPerWindow
+    }
+
+    /// Runs `walk` if the budget allows, and spends a walk only when one
+    /// actually happened.
+    ///
+    /// The permission check and the accounting are exposed only through this,
+    /// so no caller can consume the minute's allowance with a walk it then
+    /// declined to do — which is how clicking around an excluded app or a
+    /// browser used to starve every other app for a minute.
+    package mutating func fire(nowMs: Int64, walk: () -> Bool) -> Bool {
+        guard shouldFire(nowMs: nowMs) else { return false }
+        guard walk() else { return false }
+        recordFire(atMs: nowMs)
+        return true
+    }
+
+    /// Spends one of the window's walks. Called only once a tree has actually
+    /// been walked.
+    package mutating func recordFire(atMs: Int64) {
+        fires.removeAll { atMs - $0 >= Self.windowMs }
+        fires.append(atMs)
+    }
+
+    /// Whether a candidate is waiting — for logging and tests only.
+    package var isArmed: Bool { candidateAtMs != nil }
+}

apps/AfterRayCaptureShim/Sources/AfterRayCapturePolicy/TypingTarget.swift

@@ -0,0 +1,56 @@
+/// Which landing point a keystroke belongs to (decision 4 of
+/// docs/input-events-and-t1-acts-plan.md).
+///
+/// System focus is the obvious answer and is right whenever the app gives a
+/// real answer. The apps this pipeline exists for do not: measured on the
+/// 2026-08-17 vault, Feishu reports `AXWebArea` for its entire web view and
+/// Zed reports `AXWindow`. Attributing a typing burst to a landing point that
+/// coarse drags the run's engaged scope up to the whole window through the
+/// LCA, which re-creates the sidebar-noise bug this branch exists to remove —
+/// and does it precisely in the case the user is typing, the strongest
+/// evidence of engagement there is.
+///
+/// A click, by contrast, resolves to a real element (measured depth 21–39 in
+/// Feishu), so when focus declines to be specific the last click is the better
+/// evidence of where the caret is.
+///
+/// This is a role decision, never an application decision: `AXWebArea` and
+/// `AXWindow` are generic accessibility roles, and no bundle identifier
+/// reaches this file. Kept in the pure target because the executable needs
+/// live Accessibility permission to run at all.
+package enum TypingTarget {
+    /// A click older than this no longer describes where the caret is: the
+    /// user may have moved on with ⌘-Tab, a shortcut, or arrow keys.
+    package static let lastClickMaxAgeMs: Int64 = 120_000
+
+    /// Roles a person can type into. The list is the definition of "the app
+    /// answered specifically"; anything outside it is the app declining to say.
+    package static let typeableRoles: Set<String> = [
+        "AXTextArea",
+        "AXTextField",
+        "AXSecureTextField",
+        "AXComboBox",
+        "AXSearchField",
+    ]
+
+    package enum Choice: Equatable {
+        /// Focus named something typeable; use it.
+        case focus
+        /// Focus was coarse or absent and a recent click resolved precisely.
+        case lastClick
+    }
+
+    /// `lastClickAgeMs` is `nil` when no click has been resolved yet.
+    package static func choose(focusedRole: String?, lastClickAgeMs: Int64?) -> Choice {
+        if let focusedRole, typeableRoles.contains(focusedRole) {
+            return .focus
+        }
+        guard let lastClickAgeMs, lastClickAgeMs <= lastClickMaxAgeMs else {
+            // Nothing better to offer: report the coarse focus honestly rather
+            // than inventing a scope. The join fails open on a scope it cannot
+            // resolve, which is the correct outcome.
+            return .focus
+        }
+        return .lastClick
+    }
+}