Skip to content

join: reshape JoinTactic to an iterator of output containers#790

Open
frankmcsherry wants to merge 5 commits into
master-nextfrom
join-tactic-iter
Open

join: reshape JoinTactic to an iterator of output containers#790
frankmcsherry wants to merge 5 commits into
master-nextfrom
join-tactic-iter

Conversation

@frankmcsherry

@frankmcsherry frankmcsherry commented Jul 8, 2026

Copy link
Copy Markdown
Member

Replaces the two-method JoinTactic (defer + fuel-metered work) with a single

fn prep(&mut self, input0: Vec<B0>, input1: Vec<B1>, fresh: Fresh, meet: B0::Time)
    -> Box<dyn Iterator<Item = CB::Container>>;

The tactic now maps two batch lists to an iterator of output containers and holds neither capabilities nor a fuel budget — the same data-to-data division of labor as ReduceTactic. The iterator is the suspension mechanism: a container the driver is free to stop reading.

What moves where

  • Driver owns the two per-direction queues, pairs each iterator with its shipping capability, pulls under a split fuel budget, ships each yielded container, and drops a unit when its iterator goes dry.
  • "Work remains" becomes driver-observable (an iterator has yet to yield None) instead of a fuel-sign protocol the tactic must self-report. The footgun documented on the old work method is gone.
  • Fairness (the two-queue split by arrival direction) moves to the driver.

Sink change

Join closures now receive &mut CB and call push_into directly, rather than a timely JoinSession. A Session is bound to a live operator output and is not locally constructible, so it can't back a self-contained iterator. The arbitrary-container capability #478 wanted comes from the ContainerBuilder itself, not the session, so handing the closure the bare builder suffices. JoinSession is removed.

The EffortBuilder wrapper is also removed: its record counter existed only to meter fuel from a closure's output volume, and the driver now meters fuel from shipped containers' record_count(), leaving the type nothing to do but rename push_into to give.

Cursor tactic

CursorTactic shares its logic closure across units via Rc<RefCell<_>> — each unit is a 'static iterator and cannot borrow the tactic — preserving the single-mutable-state semantics of one closure threaded through every match. Deferred becomes DeferredIter: Iterator; next() plays the same merge-join loop forward (still #[inline(never)]), suspending at container boundaries rather than under a fuel budget.

Tests

Whole workspace builds warning-free. All differential-dataflow tests pass, including SCC and BFS — iterative, recursive-scope dataflows, so capabilities are released correctly when iterators drain (no deadlock).

Open for review

  • DeferredIter::next uses mem::take on extract (needs Container: Default), stealing the builder's recycled allocation; a stash-swap would avoid the churn.
  • logic's public sink type changed (JoinSession&mut CB); Mz exposure unverified.

🤖 Generated with Claude Code

frankmcsherry and others added 3 commits July 8, 2026 13:04
Replace the two-method JoinTactic (defer + fuel-metered work) with a single
`prep(input0, input1, fresh, meet) -> Box<dyn Iterator<Item = CB::Container>>`.

The tactic now maps two batch lists to an iterator of output containers and
holds neither capabilities nor a fuel budget, mirroring ReduceTactic's
data-to-data division of labor. The driver owns the two per-direction queues,
pairs each iterator with its shipping capability, pulls under a split fuel
budget, ships each yielded container, and drops a unit when its iterator goes
dry. "Work remains" becomes driver-observable (an iterator has yet to yield
None) rather than a fuel-sign protocol the tactic must self-report.

Join closures now push into a bare `&mut EffortBuilder<CB>` rather than a
timely `JoinSession`. The session is bound to a live operator output and is
not locally constructible, so it could not back a self-contained iterator;
the arbitrary-container capability #478 wanted comes from the ContainerBuilder,
not the session, so a bare builder suffices. `JoinSession` is removed.

The cursor tactic shares its logic closure across units via Rc<RefCell<_>>
(each unit is a 'static iterator and cannot borrow the tactic), preserving the
single-mutable-state semantics of one closure threaded through every match.
Deferred becomes DeferredIter: next() plays the same merge-join loop forward,
suspending at container boundaries instead of under a fuel budget.

Open for review:
- EffortBuilder's record counter is now vestigial; the type reduces to a bare
  give-providing sink, or could be removed for push_into on CB directly.
- DeferredIter::next uses mem::take on extract (needs Container: Default),
  stealing the builder's recycled allocation; a stash-swap would avoid it.
- logic's public sink type changed (JoinSession -> EffortBuilder<CB>); Mz
  exposure unverified.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Its only remaining job was renaming `push_into` to `give`; the record counter
that justified the wrapper went dead when the driver began metering fuel from
shipped containers' `record_count()`. Join closures now receive `&mut CB` and
call `push_into` directly. `EffortBuilder` is deleted.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The reachability example pushed into the join sink via `session.give(..)`, an
inherent method on the removed `EffortBuilder`. The sink is now `&mut CB`, so it
calls `push_into` directly and imports `timely::container::PushInto` in the
`reachability` module (the old inherent `give` needed no trait in scope).

Only surfaces under `--all-targets`, which is why `cargo build --workspace`
missed it but CI did not.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@frankmcsherry

Copy link
Copy Markdown
Member Author

DeferredIter::next uses mem::take on extract (needs Container: Default), stealing the builder's recycled allocation; a stash-swap would avoid the churn.

This is fixable, but seemingly pre-Polonius requires a bit of a dance to return the &mut borrow. A grim loop { .. } and a done variable to bail out of it.

frankmcsherry and others added 2 commits July 8, 2026 14:43
The trait used `CB` only as `CB::Container` (the iterator's Item); it never
constructs or pushes into a builder. Parameterizing by the container `C`
instead keeps the builder where it is actually used — a private concern of the
cursor tactic, which now carries it as `CursorTactic<B0, B1, L, CB>` and
implements `JoinTactic<B0, B1, CB::Container>`.

Two things fall out. The `Container: Default` bound (needed only for the cursor
tactic's `mem::take`) leaves the trait for that impl. And the driver, which
only ships finished containers via `give_container`, no longer needs the
caller's builder: it pins the operator output to `NoopBuilder<C>` (a builder
for any `C`, no bounds) and requires only `C: Container`. This is the groundwork
for tactics that produce containers without a push-builder (proxy `cross`,
columnar bulk), which would otherwise carry a builder they never use.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant