join: reshape JoinTactic to an iterator of output containers#790
Open
frankmcsherry wants to merge 5 commits into
Open
join: reshape JoinTactic to an iterator of output containers#790frankmcsherry wants to merge 5 commits into
frankmcsherry wants to merge 5 commits into
Conversation
Replace the two-method JoinTactic (defer + fuel-metered work) with a single `prep(input0, input1, fresh, meet) -> Box<dyn Iterator<Item = CB::Container>>`. The tactic now maps two batch lists to an iterator of output containers and holds neither capabilities nor a fuel budget, mirroring ReduceTactic's data-to-data division of labor. The driver owns the two per-direction queues, pairs each iterator with its shipping capability, pulls under a split fuel budget, ships each yielded container, and drops a unit when its iterator goes dry. "Work remains" becomes driver-observable (an iterator has yet to yield None) rather than a fuel-sign protocol the tactic must self-report. Join closures now push into a bare `&mut EffortBuilder<CB>` rather than a timely `JoinSession`. The session is bound to a live operator output and is not locally constructible, so it could not back a self-contained iterator; the arbitrary-container capability #478 wanted comes from the ContainerBuilder, not the session, so a bare builder suffices. `JoinSession` is removed. The cursor tactic shares its logic closure across units via Rc<RefCell<_>> (each unit is a 'static iterator and cannot borrow the tactic), preserving the single-mutable-state semantics of one closure threaded through every match. Deferred becomes DeferredIter: next() plays the same merge-join loop forward, suspending at container boundaries instead of under a fuel budget. Open for review: - EffortBuilder's record counter is now vestigial; the type reduces to a bare give-providing sink, or could be removed for push_into on CB directly. - DeferredIter::next uses mem::take on extract (needs Container: Default), stealing the builder's recycled allocation; a stash-swap would avoid it. - logic's public sink type changed (JoinSession -> EffortBuilder<CB>); Mz exposure unverified. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Its only remaining job was renaming `push_into` to `give`; the record counter that justified the wrapper went dead when the driver began metering fuel from shipped containers' `record_count()`. Join closures now receive `&mut CB` and call `push_into` directly. `EffortBuilder` is deleted. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The reachability example pushed into the join sink via `session.give(..)`, an inherent method on the removed `EffortBuilder`. The sink is now `&mut CB`, so it calls `push_into` directly and imports `timely::container::PushInto` in the `reachability` module (the old inherent `give` needed no trait in scope). Only surfaces under `--all-targets`, which is why `cargo build --workspace` missed it but CI did not. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Member
Author
This is fixable, but seemingly pre-Polonius requires a bit of a dance to return the |
The trait used `CB` only as `CB::Container` (the iterator's Item); it never constructs or pushes into a builder. Parameterizing by the container `C` instead keeps the builder where it is actually used — a private concern of the cursor tactic, which now carries it as `CursorTactic<B0, B1, L, CB>` and implements `JoinTactic<B0, B1, CB::Container>`. Two things fall out. The `Container: Default` bound (needed only for the cursor tactic's `mem::take`) leaves the trait for that impl. And the driver, which only ships finished containers via `give_container`, no longer needs the caller's builder: it pins the operator output to `NoopBuilder<C>` (a builder for any `C`, no bounds) and requires only `C: Container`. This is the groundwork for tactics that produce containers without a push-builder (proxy `cross`, columnar bulk), which would otherwise carry a builder they never use. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replaces the two-method
JoinTactic(defer+ fuel-meteredwork) with a singleThe tactic now maps two batch lists to an iterator of output containers and holds neither capabilities nor a fuel budget — the same data-to-data division of labor as
ReduceTactic. The iterator is the suspension mechanism: a container the driver is free to stop reading.What moves where
None) instead of a fuel-sign protocol the tactic must self-report. The footgun documented on the oldworkmethod is gone.Sink change
Join closures now receive
&mut CBand callpush_intodirectly, rather than a timelyJoinSession. ASessionis bound to a live operator output and is not locally constructible, so it can't back a self-contained iterator. The arbitrary-container capability #478 wanted comes from theContainerBuilderitself, not the session, so handing the closure the bare builder suffices.JoinSessionis removed.The
EffortBuilderwrapper is also removed: its record counter existed only to meter fuel from a closure's output volume, and the driver now meters fuel from shipped containers'record_count(), leaving the type nothing to do but renamepush_intotogive.Cursor tactic
CursorTacticshares its logic closure across units viaRc<RefCell<_>>— each unit is a'staticiterator and cannot borrow the tactic — preserving the single-mutable-state semantics of one closure threaded through every match.DeferredbecomesDeferredIter: Iterator;next()plays the same merge-join loop forward (still#[inline(never)]), suspending at container boundaries rather than under a fuel budget.Tests
Whole workspace builds warning-free. All
differential-dataflowtests pass, including SCC and BFS — iterative, recursive-scope dataflows, so capabilities are released correctly when iterators drain (no deadlock).Open for review
DeferredIter::nextusesmem::takeonextract(needsContainer: Default), stealing the builder's recycled allocation; a stash-swap would avoid the churn.logic's public sink type changed (JoinSession→&mut CB); Mz exposure unverified.🤖 Generated with Claude Code