Integer-proxy chunks: backend-agnostic join and reduce over (key_hash, value_id)

[frankmcsherry]

A boundary where only integers cross: a storage backend presents each record as ((key_hash, value_id), time, diff) — integer proxies for data it keeps in its own layout — the operators own all the lattice/time logic over those integers, and the backend supplies value semantics via callbacks. Any columnar (or otherwise opaque-to-DD) value store can then reuse join and reduce without materializing values as Rust types.

The two integers carry different contracts. key_hash is a content hash of the key, stable across the whole system with no registry: the same key hashes identically in every operator, including across the output→input boundary (a reduce output re-ingested downstream). value_id is an intra-key identifier, consistent within one operator computation — equal ids mean equal values there, which is all that consolidation and presence need. Output ids are minted by hashing the produced value, which is what lets a reduce output become a real arrangement whose values present with the same ids downstream. Hash collisions are an accepted risk (birthday bound; the module doc quantifies it, and the upgrade path is a wider id, never a registry).

Contents:

• trace/chunk/int_proxy: ProxyChunk, a cursor-less Chunk of proxy columns (slots into the Chunk navigation capability #778 split; batches, fueled merging, and grading come free, entirely in integer space), plus from_unsorted — integer sort+consolidate with representative provenance — as the helper a backend uses to build presentations.
• operators/int_proxy: ProxyJoinTactic / ProxyReduceTactic for the join_with_tactic and reduce_with_tactic seams (made pub here — these tactics are the first out-of-crate-shaped consumers), and the backend traits: present-as-proxies (read), value callback with hash-minted output ids (write), materialize (egress). The reduce tactic keeps its cross-retire pending keyed by the stable key_hash, and restricts presentations to changed keys so incremental cost tracks the delta, not the accumulation. The module doc carries the design notes, including why value_id is deliberately not order-preserving (min/max belong to the value callback).
• operators/int_proxy/reference: an in-memory backend over VecChunk arrangements with fnv content hashes, so the framework is exercised without any columnar engine.
• Tests: an equi-join and count/distinct/min reduces checked against the row-based operators over multi-round inputs with retractions, and a scripted Product-time retire sequence exercising synthetic time corrections both within an interval and pended across retires.

🤖 Generated with Claude Code