feat(capture,store): event-driven capture, AX tree diffs, and a Markdown summary card - #46
Merged
Conversation
Records the reverse-engineering evidence (144 segments, 207 output docs) and the decisions it produced: event-driven capture with a 10s heartbeat fallback, per-kind tree attachment, numbered-text AX encoding with keyframe+diff (median 913B vs our 200KB full trees), screenshot throttling with citability marking, text_input carrying the composed field value (the keystream is pinyin fragments for a CJK user — 1796 events, zero Chinese), drag with both endpoints, element-level citations anchored to a single frame, and the T2 card v3 contract (frontmatter+Markdown; JSON demonstrably breaks three model tiers). CAP-005's keystroke-content prohibition is retired under the local trust model (all processing local, encrypted vault, export only with user approval) — decided 2026-08-18. The secure-field guard stays. R3 edge snapshots are absorbed into the keyframe policy. Model: claude-fable-5 Harness: lody
WS1 of docs/event-capture-v2-plan.md §4, pure and in AfterRayCapturePolicy only — nothing here is wired into main.swift yet. Three pieces. CaptureTreeNode is the shim's AccessibilityNode as a value type, so the encoding is testable without live Accessibility permissions. TreeText renders it to numbered indented lines with a humanized role vocabulary, inline URL/Document/Description/Value, whitespace collapsed (a newline in a title would otherwise forge a tree line) and a 300-char clip that announces itself. Container chains carrying no text below collapse to one `(collapsed)` line that keeps the best label; a named AXGroup is chrome and does not by itself hold a subtree open, which is what makes Electron's container soup collapse at all. TreeDiff aligns two rendered trees by (role, label) in sibling order, falls back to role alone so a rename reads as one `~` instead of a removal plus an addition, keeps every ancestor of a change as `~` context, and coalesces removals into a leading `Removed element IDs: 97-100, 103-137`. KeyframePolicy answers skip / keyframe / diff: a window switch always re-bases (the previous tree describes another window, so it is not a valid diff base), chains cap at 30, an unchanged fingerprint emits nothing. Numbers count emitted lines only, so a collapsed-away node takes no number and every `#el<N>` citation resolves inside its own frame. Verified: `swift test --package-path apps/AfterRayCaptureShim` green, 87 tests (30 before, 57 new), five consecutive runs identical; `make capture-shim` exits 0. Not verified: behavior against a real AX tree — no walk feeds this yet. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Model: claude-opus-5 Harness: lody
WS1 built the encoding; nothing emitted it. Every accessibility artifact
the shim writes — heartbeat and edge walk alike — now carries a
`tree_text` envelope beside the untouched `root` and `digest`: the whole
numbered tree, a diff against this window's previous one, or `unchanged`
when the fingerprint says the screen did not move.
Chains are keyed per (pid, window title, walk root). The walk root is
part of the identity because the heartbeat walks the application element
while an attached walk starts at a window; diffing one against the other
aligns AXApplication with AXWindow and produces a "delete everything,
add everything" diff larger than the keyframe it replaced.
Because the chains are per window, a diff's base is not in general the
previous artifact in time, so the envelope names it: `chain` + `seq`,
beyond the plan's {mode, text}. A diff whose base cannot be named is not
decodable.
Staging is split from committing: the foreground can move between the
walk and the write, and the screenshot path deletes an accessibility
artifact it could not pair. Only a caller that actually emitted the
artifact advances the chain — otherwise the next diff would be taken
against a tree the consumer never received.
Verified: swift test --package-path apps/AfterRayCaptureShim, 100 tests
green (87 pre-existing + 13 new chain tests); make capture-shim builds.
Unverified: the chains against real windows — that needs a signed dev run.
Model: claude-opus-5
Harness: lody
…vault
OCR reads the whole screen but a moment is about one window. On a measured
WeChat frame 58 of 123 regions (47%) were the menu bar, a weather widget, or
background windows clipped at the window edge; WeChat's own accessibility
tree has zero text nodes, so for apps like it that noise *was* the text
evidence. docs/event-capture-v2-plan.md §7.
ocr_crop.rs maps each Vision box onto screen points with the shim's `Ready`
display size (now kept in `AppState::capture_display`, the daemon's only
source of a screenshot's dimensions), flips Y, and drops regions whose centre
falls outside the frame of the first window node in the moment's paired AX
snapshot. Survivors lose two more classes of junk: regions with no letter, no
digit and no common CJK/kana/Hangul character, and regions under 8 characters
that also touch the window boundary — clipped neighbours like "Conversatio".
Short text away from the edge ("Issues", "19") stays: length alone says
nothing, it is length plus position that identifies a fragment.
Every geometric uncertainty fails open and keeps all regions byte-identical:
no snapshot (AX is attached after the screenshot lands), no window node, an
unmeasurable frame, no display size, or a window frame that misses the
assumed display bounds. `text` and `layout_json` are rebuilt from the kept
regions together, and only when something was actually dropped.
Verified: 19 new unit tests cover the crop, the Y flip, both fragment rules
and every fail-open branch. `cargo test -p afterrayd` — 147 passed, 2 failed,
both of which also fail on this base (`packer_encodes_closed_gop_and_serves_poster`,
and the live-Ollama `live_ollama_answers_from_a_stored_tool_result`, which
failed 3 of 4 runs on the unmodified base). `cargo test -p afterray-store` —
219 passed. Clippy names no new symbol. Not verified: no daemon was run, so
the crop has never seen a real frame.
Model: claude-opus-5
Harness: lody
Event vocabulary v2 in the shim's tap (docs/event-capture-v2-plan.md §2, §3). CAP-005 is retired, so a typing run now carries the characters it typed and the composed value of the field they went into — the value being the primary channel, since a CJK user's keystream is pinyin fragments and 1,796 measured text_input events held no Chinese at all while 451 target values did. The guard that remains is absolute and fails closed: SecureInputGuard answers from the subrole, the ancestors' subroles, and a label that looks like a secret (Electron and web apps render password boxes as plain text fields), and a focus that cannot be resolved counts as secret too. A guarded run keeps its count and loses its content. New records: `drag`, with both ends resolved — the source at mouse-down, the destination at mouse-up — because a drag is a causal edge and one end of it says nothing; and `window_changed`, promoting the frontmost poll from something that merely armed a walk into the event a chronicle reads as "the user moved to X". Kinds `burst` and `command` keep their names though the plan calls them text_input and submit: the store's act join matches on those strings, and this workstream is additive by contract. Everything new is an optional field an old consumer ignores. Tree attachment follows §3 through one `requestTreeWalk`: click, drag, window_changed and submit ask for a walk; typing and scrolling never do; shortcuts ask every sixth time (~17%). Asking is all it is — the R3 invariants hold, pacing still refuses, walks still spend only through fire(nowMs:walk:), and no attached walk ever takes a screenshot. Dragged events join the tap mask but never reach the worker queue: the callback answers them with one distance comparison, and only the Bool travels — the coordinates die on the tap thread as they always have. Verified: swift test --package-path apps/AfterRayCaptureShim, 122 tests green (87 pre-existing + 35 new); make capture-shim builds. Unverified, and needing a signed dev run: live typed-character extraction, IME behaviour, how well both drag ends resolve in real apps. Model: claude-opus-5 Harness: lody
The shim now sends a typing run's characters and the field's composed value, a drag's two ends, and explicit window changes. All of it lands as optional fields with serde defaults: the daemon can be newer than the helper it spawns during an update, so a pre-v2 batch has to keep parsing, and a test pins that. `kind` stays a plain string — the vault keeps it uninterpreted and a newer shim's vocabulary must round-trip through an older daemon — so `drag` and `window_changed` need no enum here. `subrole`, `value` and `secure` join InputTargetRef (and `subrole` InputAncestorRef) because the daemon serializes targets verbatim into `target_json`; nothing between the shim and the vault re-derives them, so a round-trip test guards the shape. The secure guard is not re-checked here and cannot be: by the time a record reaches a parser the password is already absent, and a parser that tried to judge a field it never saw would be guessing. The test says so where a reader will look for it. Verified: cargo test -p afterray-platform-macos, 22 passed; cargo check --workspace clean; cargo clippy -p afterray-platform-macos --all-targets reports the same 5 pre-existing pedantic warnings as before this change, none naming a new symbol. Model: claude-opus-5 Harness: lody
New article context/event-capture-v2.md: the tree_text envelope and its per-window chains, the v2 input vocabulary with the plan's names mapped onto the kinds the code actually emits, the secure guard, and the attachment tiers. The shim's AGENTS.md was already 6.3k against a ~4000 budget before this change, so the depth went to the article and the AGENTS.md came out slightly smaller than it started while covering more — the remaining overflow is logged in CONTEXT-GAPS with what should move next. The plan doc gains a status column (WS1 e871ca3, WS2 11c401e + cf5e01d, WS3 partial — its screenshot-throttle half is daemon work this workstream was not allowed to touch) and a 实现偏差 section recording all ten places the code diverged from the design and why, in the doc that would otherwise quietly become wrong. Verified: swift test 122 green, cargo test -p afterray-platform-macos 22 green, make capture-shim builds. Model: claude-opus-5 Harness: lody
The WS6 agent's isolation worktree was cut from the main line, so its work landed on a side branch based on this one; only the plan doc's WS table conflicted (both sides recorded their own completion). Kept both: WS1/2/3 status from HEAD, WS6 marked done at 1a7ed62. Model: claude-fable-5 Harness: lody
Schema 25, additive: `input_events` gains `text` (the typed run the shim coalesced) and `extra_json` (the record's remaining fields — `application_name`, `window_title`, `source`, `destination` — as one object holding only the keys that are present). The target keeps flowing into `target_json` through the platform crate's own `Serialize`, which now carries `value` / `secure` / `subrole` with no mapping code here. Two columns rather than five because the vault stores the input vocabulary without modelling it: the fields readers filter on have columns, the rest travel together, and the next field the shim invents costs a mapping line in the daemon instead of a migration. A schema-24 row reads back with both NULL, which is exactly what the shim that wrote it sent. CAP-005's content ban lapsed with the local trust model (docs/event-capture-v2-plan.md §信任模型变更); the one guard left is the shim's secure guard, at the source, and nothing here re-checks it because by this point the content is already absent. Verified: `cargo test -p afterray-store --lib` 221 pass (219 pre-existing + a v24→v25 migration test that keeps its rows and a v2-batch round trip); `cargo test -p afterrayd` 148 pass (147 + the record-to-row mapping test) with only the two known failures (`packer_encodes_closed_gop_and_serves_poster`, the live-Ollama stream test). Model: claude-opus-5 Harness: lody
…their era The 48h channel existed because the event stream was the sharpest thing the vault held and had to be the shortest-lived. The trust model changed (docs/event-capture-v2-plan.md §信任模型变更), so the rule that came with it goes too: observations and `edge_snapshots` are captured content and now expire inside `enforce_retention`'s oldest-first sweep, measured against the retention horizon — the oldest frame the vault still holds. What the user did in a stretch survives exactly as long as what was on screen during it. Two decisions the plan left open, both written down in context/event-capture-v2.md §3b: - The horizon is the oldest surviving moment, mirroring how orphaned audio is swept: content whose surrounding frames are gone has nothing to attach to. It is not a second clock — under the size limit nothing expires, like frames. - No frames left means no horizon and no sweep. "Everything is older than nothing" would take live events off a vault that had simply never captured a frame. The cost: edge-tree artifacts on a frameless vault are not reclaimable by retention; `delete_history` still reaches them. `prune_input_events` becomes `prune_input_events_before(horizon)`; `prune_edge_snapshots` becomes `prune_edge_snapshots_before(horizon)`; `INPUT_EVENT_RETENTION_MS` becomes `SIGNAL_MARKER_RETENTION_MS`, feeding `prune_signal_gaps` alone. That is all the daemon's ungated sweeper tick and `Vault::open` still run on a clock — a marker's whole meaning is a deadline. R3 trees left the events' 48h because that rule assumed the events were the shortest-lived thing here; following them now would make the trees the longest. Verified: `cargo test -p afterray-store --lib` 224 pass (219 pre-existing + 5), including a sweep test that pins events and trees to the surviving frames' horizon, the frameless fail-safe, marker-only clock expiry, and `delete_history` now asserting no content column outlives its window — checked in SQL as well as through the reader. `cargo test -p afterrayd` 148 pass with only the two known failures. Clippy names none of the new symbols. Model: claude-opus-5 Harness: lody
The heartbeat used to be the only thing that could take a frame, so a frame landed wherever the timer's phase happened to fall — up to ten seconds away from the click that made the screen worth looking at. An `input_events` batch can now pull the next capture forward (`event_capture_is_due`), throttled to `max(EVENT_CAPTURE_MIN_INTERVAL_MS 10s, the configured interval)` since the last request. The cadence is unchanged; only its phase follows the user (docs/event-capture-v2-plan.md §1). The scheduler stops being a `tokio::time::interval` and sleeps until `last_capture_ms + interval` instead. That is what makes the heartbeat a fallback rather than a second clock: any capture, from either path, re-phases it, and the atomic every tick already writes is the whole handshake between the two tasks — no channel, nothing to keep in step. Both paths go through `fire_capture_tick`, the only door to `capture_screen`: `capture_paused`, `capture_busy` (now a compare-exchange, since the two callers genuinely race where the heartbeat's own spacing used to be the only guard) and `recording_active`. A held tick moves `last_capture_ms` nowhere, so the caller decides when to ask again — that is why the scheduler waits one interval after a hold instead of spinning. `event_capture_is_due` is a pure function of (batch size, last-capture age, configured interval) and is tested as one: the throttle boundary, a longer configured interval winning over the 10s floor, a shorter one losing to it, and "nothing captured yet" being past any throttle without depending on what the epoch makes the subtraction. Verified: `cargo test -p afterrayd` 149 pass (147 pre-existing + the throttle decision + the record mapping) with only the two known failures. Clippy names none of the new symbols. **Not verified:** the wiring itself — a live shim is needed to see a batch actually produce a frame, and this session may not touch a running daemon. Reasoned, not measured: `capture_screen` only writes a command and never awaits an event, so calling it from the event consumer cannot deadlock against the stream it is reading. Model: claude-opus-5 Harness: lody
Plan WS table: WS4 ✅ (schema 25 + retention unification), WS3 now says the screenshot throttle and the heartbeat demotion landed daemon-side while `screenshot_id` / citable marking are **deferred to WS5** — their only consumer is the T2 prompt, so shipping the field before the thing that reads it would ship something unverifiable. New "WS4 实现偏差" section for the decisions the plan left open: the horizon is the oldest surviving frame rather than a second clock, a frameless vault is not swept, `INPUT_EVENT_RETENTION_MS` was renamed rather than deleted because a marker's whole meaning is a deadline, no separate keyframe/diff artifact type, and the event throttle is `max(10s, configured interval)`. The `capture_paused` × throttle interaction moves from "open PoC" to "same gate in code, unverified on a live shim". `context/event-capture-v2.md` gains §3b (what the vault stores and for how long) and §3c (when the screenshot lands); its "the daemon drops these fields" note is now false and says so. `acts-join` invariant 6 changes from "never holds typed characters" — retired with CAP-005 — to "acts carry no content", which is what the join still guarantees; the retention wording throughout drops the 48h claim. Two budget entries in CONTEXT-GAPS, and one real gap logged: "how long does the vault keep things" had no answer anywhere, because there is no time-based retention at all — `enforce_retention` is size-driven, and under the limit nothing expires. That is now a sentence in the store's AGENTS.md. Verified: `cargo test -p afterray-store --lib` 224, `-p afterray-platform-macos` 22, `-p afterrayd` 149 (two known failures only); `make test-repeat N=10` on the three IO-touching new tests = 10/10 consecutive green (30 test runs). Clippy names none of the new symbols. Model: claude-opus-5 Harness: lody
Persisted summary schema 3: `title` / `description` / `details`, where `details` is one Markdown document with `afterray://moment/<id>` citations. The carrier is measured, not stylistic — a long body inside a JSON string broke three of four model tiers, frontmatter + Markdown came back valid 4/4 (docs/event-capture-v2-plan.md §5) — and threads/entities/decisions/ category/confidence go with it: threads forced three bullets whatever the evidence held, and a self-reported confidence was wrong at 0.95. `parse_t2_card_v3` is deliberately tolerant of the three ways the measured models missed the shape (no FINAL line, a code fence, no frontmatter at all); a recovered header is flagged `low_trust` rather than thrown away. `ground_t2_details` strips every citation this slot cannot show and keeps the label — one bad id is not worth the card around it. Three card shapes now live in `slot_summaries`, told apart by `schema_version` and never by null columns. The gates that meant "at least v2" now say so (`V2_SLOT_SUMMARY_SCHEMA_VERSION`); bumping the current constant in `find_slot_mentions` alone would have un-indexed every stored card. A v3 body stays searchable through `details_sections`, and a mention answers with the matching section, clipped, not the whole document. Verified: `cargo test -p afterray-store --lib` 230 pass / 0 fail (224 before, +6 new: v3 parse, tolerance, grounding, sections, a v3/v2 round-trip, and the schema-25→26 upgrade). One old-contract assertion updated — `slot_summary_export_is_structured…` writes a v2 card, so it now expects the v2 version constant rather than "the newest shape". Clippy warning count unchanged from HEAD (21, all pre-existing). Model: claude-opus-5 Harness: lody
The v3 prompt contract, its budget, and the tool catalog land together
because the renderer's signature couples them.
Prompt: `T2_SYSTEM_PROMPT_V3` asks for frontmatter + a Markdown document,
gives the five-part shape as advice rather than schema ("depth follows the
evidence, and nothing else"), and states the citation rules once —
``, standalone renders the frame and inline
stays a link, `#el<N>` anchors belong to the frame that numbered them, and
ids never need fetching because they are already in the input.
Budget: `PROMPT_LINES_BUDGET_CHARS` is gone. `render_t2_prompt` takes the
budget and the daemon derives it from `resolve_context_budget()` — the same
probe chat uses. That asymmetry was a bug: T2 planned against a 16k default
while chat measured the real window, so on a 256k model the summariser wrote
from a twelfth of its context and nothing said so. Conservative 2.5 B/token,
floored at the old constant (a small window buys fewer rounds, not a worse
card) and capped at 4× it until the corpus eval lifts it.
Catalog: `get_run_text` and `get_prev_cards` are removed from the host and
the prompt — measured, the small models never called either and the large
one burned rounds on them. `get_transcript` is named only when the slot
recorded audio. Prev cards are injected with their descriptions instead.
Acts content: the join now carries what the user *wrote*. `ActContent` sits
beside `Acts`, never inside it, so `slot_summaries.acts_json` keeps its
counts-and-labels shape and needs no version; content lives exactly as long
as the events do and is never materialised. The field's value beats the
keystream (a CJK keystream is pinyin fragments), a sentence composed and
then sent appears once, and a field the shim called secure contributes
nothing even if a value rode along with the row.
The fail-open pin was NOT regenerated: everything v3 adds is omitted rather
than emitted empty, so a zero-event slot still produces the pre-acts card
and prompt byte-for-byte. Its comment now says so.
Verified: `cargo test -p afterray-store --lib` 236 pass / 0 fail (+6 over
M1: wrote-blocks, unframed counts, the audio-gated catalog, budget movement,
the secure guard, the content caps). `cargo test -p afterrayd` 151 pass, 1
fail — `packer_encodes_closed_gop_and_serves_poster`, the known GOP failure
on this machine, unrelated. No new clippy warnings in the changed files.
Model: claude-opus-5
Harness: lody
Day panel: a card with `details` renders its Markdown body on expand, split by `StreamingMarkdown` — the same parser the chat uses, so the two surfaces cannot disagree on what a heading or a citation is. The panel is one AppKit text view, so inline syntax resolves to what it says: `[label](afterray://moment/id)` renders as `label`, a standalone citation becomes its own line, and an `#el<N>` fragment stays text (it is outside the media regex by design). Loading the frame and highlighting the element are the chat's today and this panel's later. v1 bullets and v2 threads render exactly as before — three shapes decode at once and `schema_version` says which one a row is. The agent's day view reads a v3 body as its own sections too, clipped to 200 characters each: the derived bullet list is only the headings, and a day of headings answers "what did I do today" with a table of contents. PROTOCOL_VERSION 14 → 15 (both ends). The new field is additive, but a client that cannot read it draws a v3 card as a title with nothing under it, and a silent empty panel is what the strict handshake exists to turn loud. Docs: WS5 marked done in the plan with its ten deviations (why the version constant split in two, why acts content is a sibling of `Acts`, why citability landed as `unframed_lines`, why the fail-open pin was not regenerated); `context/acts-join.md` invariant 6 restated; `agent-tools.md` gains the T2 slot surface and the numbers behind cutting it to two tools; `event-capture-v2.md` no longer says nothing reads the new fields; store, daemon and AfterRayRecall anchors updated. Verified: `swift test` 343 pass / 0 fail (340 before, +3: v3 decode and sections, citation rendering, v1+v2 still rendering). `swift test --package-path apps/AfterRayCaptureShim` 122 pass — untouched. `cargo test -p afterray-store --lib` 236 pass. `cargo test -p afterrayd` 151 pass, 1 fail (`packer_encodes_closed_gop_and_serves_poster`, the known GOP failure) plus the live-Ollama tests, which fail only under a parallel workspace run and pass on their own. Clippy warning counts back at HEAD's. Model: claude-opus-5 Harness: lody
Brings main's chat rework, compute governor, OCR text selection, the microphone consent fixes, and its slot-length budget scaling into the v2 branch. Nine files conflicted; two were real design reconciliations: - slot.rs: main scaled the prompt budget with slot length (capped at a pessimistic 24k because "the window is routinely 4096"); WS5 derives the budget from the model's probed window. These solve different variables, so they compose: `prompt_budget_for(duration, window)` — duration decides what a slot deserves, the probed window is the absolute ceiling, the per-run cap keeps its share. Main's 24k constant is superseded by the probe. The floor stays the daemon's (`t2_prompt_budget_chars`): flooring again in slot.rs made a deliberately narrow budget silently wide, which a WS5 test caught. - PROTOCOL_VERSION: main reached 14, v3's `details` stays 15; merged the version history docs on both sides of the wire. One semantic conflict git could not flag: WS5's day-panel v3 rendering was written against the old StreamingMarkdown block vocabulary, which main's MarkdownUI rework collapsed to markdown/momentImage/code. `markdownSections` now splits headings on the raw lines instead. The remaining six were two-sided doc/test additions, resolved by keeping both (compute-governor anchors + v3 anchors; the mic-TCC invariant; both new daemon tests, third time in this shape). Verified on the merged tree: afterray-store 242/242, afterrayd 181/181, platform-macos 25/25, shim package 126/126 and builds, root Swift 467/467. Zero known failures left anywhere — main's d4a573d fixed the GOP assertion and the flaky live-Ollama test passed. Model: claude-fable-5 Harness: lody
…itoring PoC Everything measured live against the production vault on the first night the v2 build ran: the 21->26 migration, the OCR crop stopping widget junk at the exact restart instant, the per-window diff chains at their predicted ~1KB against 260-380KB full trees, submit values carrying the user's sentences verbatim with the IME behaving exactly as designed, and 226 untouched chat lines correctly labelled not_engaged — the bug this whole program set out to fix, fixed on real data. The three-day-old §7.3 PoC closes: a listen-only tap leaves no trace in System Settings > Input Monitoring even while demonstrably capturing keystrokes. Recorded with its meaning inverted post-CAP-005: the OS will not tell the user AfterRay observes input, so disclosure is the app's own obligation now, and the old advice to drop the Permission Center item deserves re-evaluation. Model: claude-fable-5 Harness: lody
…SR is alive Two read-only queries the T2 summariser needs before sealing a card it can never revise: `has_untranscribed_audio_between` (per window) and `asr_health` (global, one snapshot). Both read through a new `AUDIO_UNTRANSCRIBED_PREDICATE`, a documented strict superset of `AUDIO_CLAIMABLE_PREDICATE`: it drops the retry-backoff clause and adds `running`, because "may the sweeper claim this now" and "is a transcript still coming" are different questions. It keeps the state list rather than the `NOT EXISTS` alone — `complete_audio_transcription` marks silence `done` with no evidence row, so the `NOT EXISTS` half alone would call every quiet segment untranscribed forever. There is no retry cap in this codebase: a failed segment retries forever, with the daemon's delay saturating at `1 << min(attempts, 6)` minutes. That saturation point is therefore what `exhausted_segments` counts against, now a shared `AUDIO_BACKOFF_SATURATION_ATTEMPTS` const rather than a fresh cap. Verified: `cargo test -p afterray-store --lib` 246 passed (242 pre-existing + 4 new), 0 failed; clippy clean for the new symbols. Reader pool only, no writer lock taken. Model: claude-opus-5 Harness: lody
…alive The T2 sweeper had three gates — the settle window, the `ocr_in_flight()` yield and the governor — and none of them asked whether this slot's audio had been transcribed. Measured on a live vault: with ten ASR jobs backlogged (~50 minutes of audio) one card was written before its transcript existed and says so in `not_captured` forever, while a later card swept after its transcript landed carries the whole meeting. Which one you get was luck. `asr_wait_verdict` is the decision, pure and testable: wait only while something is actually pending, ASR is demonstrably alive, and the cap has not elapsed. Every other outcome is a named `AsrProceed` branch, because each is a different way a card would otherwise never arrive — never succeeded (cold start, model absent, broken worker), failing more recently than succeeding, every pending segment run out to backoff saturation, cap elapsed. Liveness compares the last success against the last *failure*, never the clock alone: a machine that slept eight hours has a stale success and a healthy worker. The clock is only a one-week backstop for a worker that stopped succeeding without recording a failure. Wiring: the sweeper takes `slots_ready_for_t2`, which skips a waiting slot without touching its state — it stays `Degraded`, the only state `due_slot_windows` ever picks back up, so waiting is "skip this round". One `asr_health` per sweep (global) short-circuits the per-slot query entirely when nothing anywhere owes a transcript. `slot_backfill` keeps the ungated `slots_awaiting_t2`: it is an explicit request to fill history, where audio either transcribed long ago or never will. Held-back slots also keep a "run now" override alive rather than reporting a drained backlog. Two fixes carried along: the sweep now goes through `run_store` (it was calling `Vault` synchronously from a tokio worker), and `fail_claimed_audio` takes its backoff saturation point from the store const rather than a hand-copied 6. Verified: `cargo test -p afterrayd` 191 passed (181 pre-existing + 10 new), 0 failed; `cargo test -p afterray-store --lib` 246 passed. Clippy names none of the new symbols. Two of the new tests drive a real vault end to end: the sweeper holds the slot, backfill does not, the state stays `Degraded`, and the slot is swept once the transcript lands or the cap expires. Model: claude-opus-5 Harness: lody
The T2 anchor in the daemon's AGENTS.md gains the fourth gate and its rule;
the store's gains the two queries, the predicate they share and why its state
list is load-bearing. §2.3 of the slot-summaries doc had the requirement
("rather two minutes late than a summary that missed the meeting") with only
its OCR half implemented — it now says what "worth waiting for" means, names
the 30-minute cap, and carries the measurement behind it.
The known gap is written down in both the daemon's "Watch out" and §2.3: a
card already written is not re-run when its transcript lands late. Only
`slot backfill` can redo it, and that overwrites a card the user may have
read.
Verified: text only, no code touched.
Model: claude-opus-5
Harness: lody
zxch3n
added
risk:middle
LodyAI
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #41. Rebuilds capture around two independent fact streams and rewrites what a summary card is. Every parameter here was measured — against a competing product's on-disk data, against this vault, or against four model tiers — and the measurements are in
docs/event-capture-v2-plan.mdbeside the decisions they produced.Why
#41 gave T1 input events so a card could say what the user did. Three things were still wrong, all measured on this vault:
Conversatio,ter,• Gi). For apps whose AX tree is empty (WeChat: 196 nodes, 0 text nodes) OCR is the only text source, so that noise went straight into cards.threadsmade every model write three bullets regardless of evidence, and a long Markdown body inside a JSON string field broke three of four model tiers outright.What changed
Capture. AX trees now also ship as numbered indented text with a per-window keyframe/diff chain (
+/~/Removed element IDs: 97-100, 103-137). Measured live: ~1 KB diffs against 260–380 KB full trees, 250–350×. Event-driven capture with the 10s heartbeat as fallback; screenshots throttled to ≥10s and phase-aligned to interaction.root/digeststill ship byte-identical — nothing downstream had to change yet.Input vocabulary v2. Typing bursts carry the typed text and the field's value at the event instant; submits read the focused value; drags carry both endpoints. The field value is the primary content channel because a CJK user's keystream is pinyin fragments — measured on the reference product: 1,796 text-input events, zero Chinese. Secure fields contribute nothing, and an unresolvable focus counts as secure.
OCR window cropping with fragment filtering, every geometric uncertainty failing open. Verified live: weather-widget text last entered the vault at 21:02:13, the new daemon started 21:05:30, zero hits since.
Card v3 — YAML front matter + Markdown body, replacing
threads/entities/decisions/category/confidence. Depth follows the evidence, with no target section count. Prompt budget derives from the model's probed context window composed with slot length, replacing a fixed 12k constant. Two tools instead of four, one audio-gated.T2 waits for ASR while ASR is demonstrably alive (last success newer than last failure), capped at 30 minutes. This fixes a race caught on live data: the 18:10 card was written before its transcript existed and says so permanently, while the 22:10 card got the whole meeting by luck.
Trust model change — please read
CAP-005's keystroke-content prohibition is retired (decided 2026-08-18, recorded in
docs/event-capture-v2-plan.mdand amended intoslot-summaries-and-ax-pipeline.md§7.1). Rationale: all processing is local, the vault is encrypted, export needs explicit approval. Typed characters and field values are now stored. TheAXSecureTextFieldguard is the one thing kept, and it was implemented stricter than specified — it also catches password-looking labels in Electron/web fields, and fails closed when focus cannot be resolved.A settled PoC belongs with this: a listen-only tap leaves no trace in System Settings → Privacy → Input Monitoring, even while demonstrably capturing keystrokes. Under the old policy that was a feature to advertise; now it means the OS will not tell the user AfterRay observes input, so disclosure is entirely the app's own obligation. The old advice to drop the Input Monitoring item from the Permission Center should be re-evaluated.
This is why the PR carries
security— not because it is unsafe, but because a reviewer should see the change rather than find it inside twenty commits.Verified
afterray-store246/246,afterrayd191/191,afterray-platform-macos25/25, shim package 126/126, root Swift 467/467. Zero known failures anywhere. New IO tests 10/10 consecutive viamake test-repeat.not_engagedinstead of impersonating activity, which is the bug this whole program started from.Not verified
ASR_ALIVE_STALENESS_MS(7 days) is a judgement call; only the 30-minute cap has a measurement behind it.slot backfillcan redo it, and that overwrites a card the user may have read. Recorded incrates/afterrayd/AGENTS.md.#el<N>element citations round-trip only in unit tests; element highlight in the UI is future work.Confidence
High on the deterministic layers and on capture, which ran a full day on real data. Medium on the summarisation contract: the format is validated across four model tiers and 20 real cards, but its quality against the old contract has not been blind-scored yet.
🤖 Generated with Claude Code