feat(capture,store): join input events into T1 cards so slots record what the user did - #41
Open
zxch3n wants to merge 21 commits into
Open
feat(capture,store): join input events into T1 cards so slots record what the user did#41zxch3n wants to merge 21 commits into
zxch3n wants to merge 21 commits into
Conversation
The AX walk was spending most of its nodes on the menu bar — measured 205/257 in Ghostty, 245/276 in Zed, ~170/1115 in Feishu — encrypted and stored every 10s tick with no consumer: digests collect text roles only and the store's text extraction asserts menus are chrome. Stub the AXMenuBar subtree instead of descending. The walk also had a node cap but no time bound, while every attribute read is synchronous IPC into the target app (system default timeout 6s per call): one wedged app could stall ticks unboundedly and lag the app the user is working in. Bound both sides: process-global 100ms messaging timeout at startup plus a 500ms whole-walk deadline that sets `truncated`, same as the node cap. Verified: build green; startup + JSON event protocol smoke-tested (TCC-denied context, so real-walk behavior needs a signed dev run). Groundwork for the input-events/T1-acts program (phase 0). Model: claude-fable-5 Harness: lody Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
New docs/input-events-and-t1-acts-plan.md capturing the 2026-08-17 investigation and decisions: two-fact-streams principle (screen state vs input events, joined by time and tree position; T1 never infers), the 3-model x 2-scenario evidence matrix, R1/R2/R3 capture cadence, CAP-005 amendments (Return/Tab/Esc as command keys, burst granularity, element-identity persistence for pointer events), T1 reorganization around acts, and the explicit not-doing list with experimental reasons. Amends slot-summaries-and-ax-pipeline.md §7 tables in place and marks the superseded items; indexes the plan in docs/AGENTS.md. Model: claude-fable-5 Harness: lody Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
New InputEventMonitor in the capture shim: a listen-only CGEventTap on
its own thread whose callback only classifies and enqueues; all AX work
happens on a worker queue bounded by the process-global 100ms messaging
timeout. Emits coalesced `input_events` batches on stdout:
- typing bursts {start, end, count, ended_with} — key codes are read
solely to classify command keys and never leave the callback
- command keys: ⌘-combos plus Return/Tab/Esc (2026-08-17 decision:
they carry no character content; "submit/execute" semantics)
- clicks and scroll bursts resolved at event time to element identity
(role/label/frame/ancestor chain); coordinates die with the
resolving stack frame — never serialized
- producer-side coalescing (2s burst gap, 1s scroll gap, 40 records
per flush window, overflow counted in `dropped`)
- exclusion: AfterRay itself and daemon-excluded apps are never
recorded; fails closed before the exclusion list arrives (same
posture as the audio hold), fails open with a warning event when
the tap cannot be created
- liveness: taps die silently on code-signature changes, so the
worker cross-checks CGEventSource idle time and emits
`input_tap_stalled` + re-enables — absence of events must never
read as "the user did nothing"
Rust side: CaptureEvent::InputEvents + record structs in
afterray-platform-macos (parse test included); the daemon logs and
drops batches for now — persistence (events table, 48h retention,
seal-time acts materialization) is phase 2 of
docs/input-events-and-t1-acts-plan.md.
Verified: shim builds; afterray-platform-macos 17/17 tests green;
afterrayd 127 tests green with only the pre-existing GOP
compression-ratio failure (fails identically on the base commit);
clippy adds zero new findings (5 pre-existing pedantic errors in
untouched files under nightly clippy, identical before/after).
Runtime behavior needs a signed dev run (TCC) — not yet exercised.
Model: claude-fable-5
Harness: lody
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ise fixes Concrete specs so implementation can proceed without the design conversation: input_events DDL + Vault API + retention/cascade invariants (phase 2), the acts join/hysteresis/materialization contract with its fail-open invariant (phase 3), and the exact targets for the independent T1 noise fixes. Materialization moves from phase 2 to phase 3 — the acts shape is defined there. Model: claude-fable-5 Harness: lody Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…bnails Two independent T1 noise fixes measured against the production vault (2026-08-17 15:20 slot), both confined to slot.rs. Place noise: `target_key` accepted url/document/window_title unfiltered, so Feishu's `native-resource://sdk/avatar?...` became the slot's `theme_key`, and its bundled `file:///Applications/Lark.app/.../en-US.html` became the run title and a `facts.top_documents` entry — `shorten_place` reduces that path to a plausible-looking `en-US.html`, which passed the label filters. Every place candidate now runs through one gate, `is_place_noise` (`is_chrome_noise` + `is_opaque_id` + the new `is_app_bundle_resource`), applied to the raw value *and* its shortened form, at all four call sites: `target_key`, `place_label`, and the `top_documents`/`top_urls` aggregation. A rejected candidate falls through to the next; with every candidate rejected the key degrades to the app-only form. Anchor frame: the day summary anchored on `moment_ids.first()`. On real data a slot's opening frame is the worst thumbnail available — a window that just gained focus dumps its whole sidebar into that one frame (1399 characters of navigation). `SlotCard` now carries `anchor_moment_id`, computed by the pure `anchor_frame_id` as the middle frame of the longest run (ties to the earliest run); the first moment only stands in for a card built without one. It lives on `SlotCard`, not `RunRow`, because a `RunRow` field pushes `TimelineEntry` past clippy's `large_enum_variant` threshold — and the piece's own row list gives the exact middle rather than an estimate. Verified: `cargo test -p afterray-store` green, 154 passed / 0 failed / 1 ignored, up from 145 at HEAD (three new tests here; the rest arrived with concurrent work in the same tree). New tests replay both real-world failures verbatim, the all-noise degradation, and a 3-run slot whose longest run sits in the middle. `cargo clippy -p afterray-store --all-targets -- -D warnings` still errors only on lints that a pristine HEAD copy reproduces identically; slot.rs keeps exactly its two pre-existing findings and gains none. `cargo check --workspace` clean. Updated `assemble_keeps_t1_only_slots_and_overlays_t2_titles`: its anchor assertion asserted only `is_some()` under the name "opening frame", so it now names and pins the new rule (middle frame of the longest run, `b` rather than `a`). Not verified: no run against a real vault or the app UI — the thumbnails this changes were judged from the recorded slot, not re-rendered. Model: claude-opus-5 Harness: lody
Schema 22: new input_events table (at_ms/end_ms/kind/count/ended_with/ command/bundle_identifier/target_json) plus slot_summaries.acts_json, added now so phase 3 needs no second migration. kind and target_json are stored uninterpreted — the T1 join owns their meaning, and a newer shim's kinds must round-trip. Never holds typed characters. Vault API: insert_input_events (one transaction per batch, writer connection), input_events_between (reader pool, overlap semantics, ordered), prune_input_events (INPUT_EVENT_RETENTION_MS = 48h). delete_history now cascades input_events — one privacy invariant, three layers: forgetting a window takes the cards and the acts with the frames. Not exposed via SharedReadOnlyVault. Daemon: the InputEvents arm goes from log-only to run_store batch insert (target serialized verbatim — InputTargetRef gains Serialize so the store needs no second schema); prune runs beside enforce_retention. Verified: afterray-store 154/154; afterrayd 127 green plus only the pre-existing GOP compression-ratio failure; platform-macos 17/17; concurrency test 10/10 consecutive green via make test-repeat; zero clippy findings touching new code (the 114 workspace warnings are pre-existing nightly-pedantic noise, none mention new symbols). Implementation by an Opus subagent against the phase-2 contract in docs/input-events-and-t1-acts-plan.md; verification re-run and docs completed by the orchestrating session. Model: claude-opus-5 Harness: lody Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The shim has always written a `frame` on every accessibility node (1106 of 1115 on a measured frame) and nothing ever read it. The input-event join needs it: to say which region of a window a click landed in, the tree has to carry geometry. `memory.rs` gains a flattened, parent-linked arena — every question the join asks is an upward walk, which the nested shape makes a search — and `accessibility_text_lines` now delegates to the same traversal. That is the load-bearing part: the `AX_TEXT_MIN_CHARS` text-source decision counts the whole line vector and the join partitions that same vector, so the two must never be able to disagree about what a line is. `acts.rs` holds the geometry, pure: deepest node containing a landing point, LCA of several, expansion to the smallest ancestor covering ENGAGED_MIN_WINDOW_AREA_RATIO of its window — the single knob in the whole join. It fails open everywhere it cannot answer honestly: nothing landed, no window, an unmeasurable window frame all yield no scope, because an invented scope reads downstream as "the user was here". Model: claude-opus-5 Harness: lody
…itting The acts block is what the user *did* over a stretch: keys, submits, clicked labels, scrolls, and whether the input stream could be observed at all. Fixed shape, every field always serialised — a reader has to be able to tell "zero keys" from "keys unknown", and a missing field cannot. Two decisions worth the space: `ended_with` is not a submit. The shim emits both a burst carrying the key that closed it and a separate `command` row for that same key; counting both would double every Return in the slot, so only the command row counts. Pinned by a test. Run splitting is hysteretic. A new scope becomes a boundary only once sustained — two events or fifteen seconds — because triage (glancing at four conversations, answering one) otherwise shatters into four runs of one click each, which is how a real slot came out as "multi-group scan" with the actual 1:1 unmentioned. An un-promoted excursion folds back into the run it interrupted, where its clicked labels survive as the record of the glancing. An unresolved scope never forces a boundary: not knowing where an event landed is not evidence that it landed somewhere new. `unavailable` is not idle. A signal gap runs until input is observed again and no further, and no engaged assertion may be made inside it. Model: claude-opus-5 Harness: lody
Captured from the pipeline as it stands now, so the fail-open invariant is anchored on behaviour that predates the change rather than on whatever the change produces. A slot with no input events has nothing to partition by and must come out byte-identical. Only the clock-derived fields are normalised — day and HH:MM are the card's one locale-dependent surface, and the fixture normalises both sides the same way. Everything the partition could plausibly break (line selection, ordering, line_frames, total_chars, more_chars, budgets) is pinned exactly. Model: claude-opus-5 Harness: lody
A T1 card now says what was done, not only what was on screen. Each run carries an `acts` block joined from the input stream, the text it introduced is split into the engaged region and the merely visible, and the card names the regions that received no input all slot. This is the fix for a measured failure: on a real Feishu slot, 67% of the prompt budget went to a conversation list the user never touched, and the card came out as "multi-group scan" with the actual 1:1 unmentioned. The engaged region now takes the whole existing budget through infoscore, so IDF de-chromes within the bucket that matters instead of ranking a sidebar against a conversation; peripheral text folds to 200 characters and a line count, because "40 lines, not shown" is what a model needs from it. Three things are load-bearing and each is pinned: The text-source gate still counts every line. `AX_TEXT_MIN_CHARS` decides AX-vs-OCR on the unfiltered vector, so a frame whose engaged pane is small keeps its exact accessibility text instead of being silently demoted to whole-screen OCR — which is the frame the join works best on. Fail-open is structural, not conventional. Partitioning is gated on the event stream itself, not on callers leaving `ax_join` unset, and the pinned fixture from 19ae9f8 still passes byte-for-byte. `unavailable` suppresses every engaged claim — including the partition itself. Splitting text into operated and visible is an assertion about agency, so a frame from a stretch the tap could not observe makes none, and no region is called untouched there. A run with an `ok` signal and zero acts is a fact, not a gap: "22 minutes here, no keys" can only be stated by a stream that was running, and the model is told the difference. Model: claude-opus-5 Harness: lody
…xpire Two holes closed at the ends of the acts pipeline. The daemon was logging `input_tap_stalled` / `input_tap_unavailable` and throwing them away. A dead tap has to be recorded *in* the event stream, because T1 reads an absence of events as "the user did nothing here" — the one inference this pipeline exists to prevent. The marker rides the same table as a `signal_gap` row; the vault stores `kind` uninterpreted, so this needed no schema change and the gap arrives in its place in time. Events are deleted after 48 hours and T1 is computed lazily, so acts that are not frozen simply vanish from history: two days on, a card would keep the half that says what was on screen and silently lose the half that says what the user did. The sweeper now freezes sealed slots into `slot_summaries.acts_json`, and `slot_card` reads the frozen copy once the events are gone. The freeze runs before the T2 gate and independently of it. It is a short read and one small write with no model in it, and the deadline it races is physical — the events expire whether or not the machine was ever on AC power with a charged battery. Gating it behind T2's conditions would lose acts on exactly the laptops that stay unplugged. What the frozen copy deliberately does not restore is the engaged/peripheral partition: it was computed by hit-testing rects that no longer exist, so the text goes back to whole rather than being partitioned against a guess. Pinned, along with the roundtrip through expiry, the idempotence the five-minute sweeper needs, and the delete_history cascade. Model: claude-opus-5 Harness: lody
The store's AGENTS.md was already 1.5k over its ~4000-char budget before this phase, so the join's detail goes where the repo's own rule sends it — `context/acts-join.md` — and the index lines stay pointers. The article carries the four invariants that may not be weakened (fail-open byte-for-byte, the text-source gate, `unavailable` suppressing every engaged claim, T1 purity), the measurements behind the 0.10 knob and the hysteresis, and the reason `ended_with` is not counted as a submit. The plan doc records what phase 3 did *not* do, which matters more than what it did: timeline rows are still cut by `target_key` rather than re-cut by engaged scope, only the v2 system prompt changed, and the facts.apps acts summary and revisits/theme_key rework are untouched. Also the two places the implementation came out stricter than the contract, both because a test forced it. Filed the remaining AGENTS.md overflow in CONTEXT-GAPS with a concrete next extraction rather than leaving it as a silent violation. Model: claude-opus-5 Harness: lody
…hots Trigger/debounce/token-bucket parameters, the two v1 simplifications (window-scoped walk instead of engaged-subtree; no shim-side power degrade yet, bucket bounds the cost), the accessibility_edge artifact kind with its no-moment/no-screenshot/no-OCR boundary, schema 23 edge_snapshots with 48h event-aligned retention and the fourth layer of the delete_history cascade, and how edge trees join T1 as extra partition frames only. Model: claude-fable-5 Harness: lody Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The 10s heartbeat misses content a person only looked at between two ticks — stepping into a conversation for eight seconds and leaving. R3 fills exactly that hole: a candidate (frontmost bundle change, or a click), a 500ms settle window that any further input re-arms, and a token bucket of >=5s spacing and <=6 walks per rolling minute. The pacing rules are a pure state machine in `AfterRayCapturePolicy` (`EdgeSnapshotPacing`) because both failure modes — walking the tree while the user is still typing, and walking it thirty times a minute — are invisible to any test that needs a live `CGEventTap`. Never a screenshot: an event-driven frame would outlive the events that triggered it and keep exposing interaction instants after the record of the interaction was erased. Two v1 simplifications, per the plan: the walk root is the whole AXWindow the trigger landed in (focused window fallback) rather than the engaged subtree, and known browsers are skipped entirely, because the heartbeat's private-browsing gate needs an async automation probe a 1s worker tick cannot afford. Model: claude-opus-5 Harness: lody
SCHEMA_VERSION 22 -> 23. An R3 edge snapshot is an accessibility tree with no moment: no screenshot, no thumbnail, no OCR. Its own table rather than a column on `moments`, because hanging it off a frame would drag it through every retention and export path that treats a moment as a picture of the screen. Retention is the events' own 48h, at the events' own call site: a tree triggered by an event and outliving it would keep saying "the user was here at 03:14" after the record of the input was erased. Pruning deletes the encrypted files, not just the rows. `delete_history` gains a fourth layer — frames, cards, acts, and now R3 trees. Nothing else could have reached them: they belong to no moment. Artifacts have no purpose column, so the purpose rides the content type (`purpose=edge-ax`). It is a constant, never a string copied off the capture event: the encryption AAD binds the content type, so an artifact stored under one spelling and read back under another is undecryptable. Model: claude-opus-5 Harness: lody
`ArtifactKind::AccessibilityEdge` on the Rust side of the shim protocol, and an import branch beside the accessibility one. It stores the tree as an encrypted `edge-ax` artifact plus one `edge_snapshots` row: no moment, no thumbnail, no OCR job — an edge snapshot is not a frame of the screen. The exclusion check is the accessibility branch's, one notch stricter: `edge_snapshot_identity` refuses a snapshot that does not name its app, because the exclusion list is keyed by bundle identifier and an unnamed app cannot be judged. The heartbeat branch has a screenshot already on disk and must decide what to delete; this one loses nothing by dropping — the next trigger is one interaction away. That decision is a pure function so the fail-closed posture is testable without an AppState. Edge retention runs at the events' own call site, from the same clock. Model: claude-opus-5 Harness: lody
An edge tree is text and only text: it adds the lines no heartbeat frame ever carried to the run it fell inside, partitioned into engaged and peripheral by its own join exactly as a frame's lines are. It contributes no moment_id, no anchor, no OCR evidence and no `facts` count — those all answer "which frames does this card stand on", and an edge tree is not one. Pinned by a test that diffs every one of them against the same card built without edges. Two deliberate limits. The join does not write resolved scopes back onto the events from an edge tree: run splitting segments on those scopes, and R3's job is to widen the text a run shows, not to re-cut the runs. And a tree landing in a capture gap belongs to no run and is dropped, rather than being attached to the nearest one — that would claim a window was on screen during a stretch nothing was captured in. Edge trees are gated on the event stream like the partition itself, so the fail-open pin extends to them: a slot with edge trees and no events produces the byte-for-byte pre-acts card. It cannot hold them by construction — only input triggers a walk — and now a card cannot depend on that staying true. Model: claude-opus-5 Harness: lody
…-offs Phase 4 marked done with six recorded deviations: known browsers are skipped entirely (the heartbeat path's private-browsing verdict needs an async probe a 1s tick cannot afford — skip rather than half-judge), import rejects snapshots without a bundle identifier (the exclusion list is keyed by it, and a dropped edge snapshot costs one interaction), `edge-ax` rides the content type because artifacts have no purpose column (AAD binds content type, so it must be a constant), edge frames neither re-cut runs nor feed `not_engaged` nor pass the AX-vs-OCR threshold, and trees landing in a capture gap are discarded rather than attached to the nearest run. Covers apps/AfterRayCaptureShim/AGENTS.md (R3 invariant + the new XCTest suite), the platform/store/daemon AGENTS.md anchors, the capture-pipeline stage notes, and the acts-join article's edge-frame paragraph. Model: claude-opus-5 Harness: lody
Deploying afterray with
|
| Latest commit: |
62bf5c4
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://64916c53.afterray.pages.dev |
| Branch Preview URL: | https://ax-walk-menu-skip-timebox.afterray.pages.dev |
main's #42 made summary slot length a user setting and took schema 22 for `summary_slot_geometry`, which collided with this branch's own 22. Git merged the two `migrate_schema_22` bodies into one file without a conflict marker — a duplicate symbol that only the compiler catches, and the more dangerous half of this merge. main's 22 is already released, so it keeps the number: `input_events` becomes migration 23 and `edge_snapshots` 24, with SCHEMA_VERSION 24. The two migration tests simulated a pre-migration vault by stamping the version below their own step; both still passed after the renumber because the migrations are idempotent, but their names and stamps then described the wrong step. Renamed and re-stamped — in this repo a test name is the specification. Both sides' tests kept where they collided at the top of afterrayd's test module (edge-snapshot identity, and the pin that the settings picker only offers lengths the vault accepts). The store AGENTS.md anchors were rewritten to carry both stories — geometry history and `slot_bounds_in` from main, the two fact streams and `acts.rs` from here — with the long entries compressed toward context/acts-join.md, and capture-pipeline.md's schema line refreshed (it had been stale at 18 on both sides). Verified on the merged tree: afterray-store 217/217 (both sides' new tests present, including the byte-for-byte zero-event fail-open pin), afterrayd 129 pass with only the pre-existing GOP compression-ratio failure, afterray-protocol 41, Swift root 340 and shim 24, workspace `cargo check --all-targets` clean, and no clippy warning naming any merged symbol. Model: claude-opus-5[1m] Harness: lody
…time Three defects from review, and one reported defect that is not one. **Typing was attributed to whatever focus said.** System focus is only as precise as the app chooses to be, and the apps this pipeline exists for are the imprecise ones: measured, Feishu reports `AXWebArea` for its whole web view and Zed reports `AXWindow`. A landing point that coarse drags the run's engaged scope up to the window through the LCA, which re-creates the sidebar-noise bug this branch removes — and does it exactly when the user is typing, the strongest evidence of engagement there is. Bursts and command keys now fall back to the last click, which resolved to a real element (measured depth 21–39). The rule is a role decision, never an application one, and lives in the pure policy target as `TypingTarget` with five tests. **The 48h expiry rode the screen-import path**, so it only ran while recording. Stop capture and the raw click targets and R3 trees stayed on disk indefinitely, contradicting the contract. Expiry is a promise about time: it now runs inside `enforce_retention` — above its size-sweep early return, which fires whenever the vault is under its limit — so a mere `Vault::open` expires them, and on the sweeper's ungated tick beside `freeze_slot_acts`, which already exists because the same deadline is physical. A test opens a vault with nothing recorded and asserts the expired row is gone. **A failed event batch vanished silently.** A stretch with no rows reads as "the user did nothing", the one thing this pipeline may never say by accident, so a failed insert now writes a `signal_gap` marker spanning the batch. Extracted `record_signal_gap` so the dead-tap path and this one cannot drift. Also: the shim only stopped its monitor on the explicit `stop` command, so a daemon crash closing the pipe lost the events buffered since the last tick; `stop()` is idempotent and now runs on both exits. **Not changed: the reported pointer-coordinate flip.** The claim was that `CGEvent.location` is bottom-left origin and must be flipped for `AXUIElementCopyElementAtPosition`. Measured instead of assumed: a null-source `CGEvent` reports the cursor at y=562.6 while `NSEvent.mouseLocation` — documented bottom-left — reports y=554.4 on a 1117pt screen, summing to exactly the screen height. The two spaces are mirrors, so `CGEvent.location` is already top-left, like AX. Flipping it would have introduced the very bug the review was trying to prevent. Verified: afterray-store 218/218, shim XCTest 29/29, afterrayd 128 pass. Two afterrayd failures are pre-existing and reproduce with these changes stashed: the GOP compression-ratio assertion, and a live-Ollama stream test that asks a local model to echo a token (this session swapped the loaded model while running experiments). Model: claude-opus-5[1m] Harness: lody
The remaining four review findings. **A declined walk spent the minute's allowance.** `shouldFire` recorded the fire, then `captureEdgeSnapshot` could still bail on its own guards — a browser, an excluded app, a window that would not resolve. Clicking around one such app burned all six walks and starved every other app for the minute. Spending now happens only through `fire(nowMs:walk:)`, which consumes the budget after the walk reports it happened; the permission check and the accounting are no longer separately callable, because an API that asks callers to pair them is one that eventually drifts. **Unavailable spans were closed at both ends.** A gap ends at the next observed input — the very event proving the tap was alive again — so that instant was being reported as unobservable. Spans are half-open now. A gap that never recovered still covers its horizon inclusively, which is why that fallback is one past the end rather than the end. **Materialisation missed slots that a span only reached into.** The work list was keyed on `at_ms` alone, so a burst from 09:59:58 to 10:00:20 enqueued only the first slot; with nothing else in the second one it was never frozen, and once the events expired it failed open forever — silently dropping typing the user did. Every slot a span touches is enqueued. The test was checked against the unfixed function first: it fails there, so it tests what it claims to. Also trimmed the duplicated rationale on `lastClick` now that `TypingTarget` owns that rule, and recorded the half-open span and the touched-slot rule in context/acts-join.md (invariant six). Verified: afterray-store 219/219, shim XCTest 30/30 (the three pacing tests now exercise the same `fire` path production does, plus a new one pinning that twenty declined walks leave the budget untouched), Swift root 340/340. afterrayd unchanged at 128 pass with the two pre-existing failures (GOP compression ratio; a live-Ollama stream test). Model: claude-opus-5[1m] Harness: lody
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
T1 cards misread IM apps: on the 2026-08-17 15:20 slot of the real vault, 67% of the prompt budget went to a Feishu sidebar the user never touched, and the resulting T2 card read "Feishu multi-group scan" while the actual 1:1 conversation went unmentioned. Between 15:20 and 16:20 the user was in essentially one conversation; five of six cards described scanning many groups.
The root cause was not the scoring — it was that T1 had only one fact stream and was inferring what the user did from what was on screen. Every fix attempted against that single stream failed: geometry heuristics (break per app), placeholder parsing (overfit to Feishu), churn (measures "something happened", not "the user did it" — in the group case it pointed at the sidebar while the engaged pane changed by zero characters), and handing the raw tree to the model (works on Haiku, fails on a 35B local model, inverts on a 4B — and T2 runs the user's configured model).
So this adds the second fact stream and joins them.
The evidence that picked the design
Three models × two real scenarios (Feishu 1:1 and a 3-person group), asked to identify the conversation the user was actually in:
The last two rows differ by three lines of text. The asserted fact — "input landed in region 1; no input landed elsewhere" — is the only thing that flips results across model tiers, and its only honest source is input events. Local cost measured at ~5s/slot for the 4.8 KB shape on qwen3.6:35b-mlx, so the ceiling here is correctness, not latency.
Design and every number:
docs/input-events-and-t1-acts-plan.md. Code map:context/acts-join.md.What landed (18 commits, phases 0–4)
AXMenuBarstubbed, 100ms per call, 500ms whole-walk budget.CGEventTapin the shim on its own thread. Typing bursts as counts (key codes classify command keys and never leave the callback), command keys incl. Return/Tab/Esc, clicks/scrolls resolved at event time to element identity with coordinates dropped in the resolving stack frame. Fails closed for excluded apps before the daemon's list arrives; fails open with a warning if the tap cannot be created; cross-checksCGEventSourceidle time because taps die silently on code-signature changes — absence of events must never read as "the user did nothing".input_eventstable (schema 22), 48h retention,delete_historycascade.kind/target_jsonstored uninterpreted so a newer shim round-trips.frame; Rust never read it), landing points hit-tested per frame, engaged scope = LCA expanded to ≥10% window area (the system's one knob). Runs carryacts; text splits into engaged vs peripheral; cards gainnot_engagedandno_input_ratio.theme_keywas picking up Feishu's avatarnative-resource://…URL anden-US.html(a path insideLark.app, whichshorten_placelaundered into a plausible document name) was becoming the run title and atop_documentsentry. One gate now, applied to raw and shortened values at all four call sites. The slot anchor moved off the first frame — measured, that frame carries the one-time sidebar dump.Verified
afterray-store210/210 (154 → 210; no pre-existing test rewritten),afterrayd128 pass,afterray-platform-macos18, protocol 39, codec 11, cli 2, infer 1. Swift: shim package 24, root 336.origin/mainin a clean worktree:afterrayd::tests::packer_encodes_closed_gop_and_serves_poster(GOP compression-ratio assertion) and 4afterray-models::remote::streamtests (sandbox blocks localhost HTTP).line_frames,total_chars,more_chars, budgets all exact.make test-repeat N=10).Not verified — please read before merging
make devrun — look forinput tap startedandcapture input events batch=Nin daemon stderr.mainbuild afterwards may refuse to open it. UseAFTERRAY_DATA_DIRfor an isolated trial.target_key(hysteresis splitting is implemented and tested, acts attach by overlap; re-cutting would movemoment_idanchors, gaps and revisits),facts.apps[]acts summary /revisits/theme_keyregion derivation and theidle_ratiorename are deferred, and R3 skips all known browsers because the private-browsing verdict needs an async probe a 1s tick cannot afford.Privacy
CAP-005 is enforced at the source, and the amendments are recorded in
docs/slot-summaries-and-ax-pipeline.md§7.1/§7.2b rather than applied silently: Return/Tab/Esc are reclassified as command keys (no character content; "submit/execute" semantics), typing exists only as burst counts, pointer coordinates are never serialized — only the element identity they resolve to.AXSecureTextFieldvalues stay nulled. Forgetting a window now takes four layers with it: frames, cards, acts, and edge snapshots.Two implementation choices were made stricter than the contract because a test caught the gap: partitioning is gated on the event stream itself rather than on caller convention, and an
unavailablesignal suppresses the engaged/peripheral text split too — splitting text into "operated" and "merely visible" is itself an agency claim.Confidence
High on the deterministic layers (pure functions, pinned fail-open, 210 store tests). Medium on the end-to-end system, entirely because no runtime path has run. The measurement that motivated all of this is real and reproducible from the vault; the measurement that would confirm the fix is phase 5 and still owed.
🤖 Generated with Claude Code