Skip to content

refactor phase 3c: decompose the pre-aggregation pipeline into named phases - #300

Merged
hussainsultan merged 4 commits into
mainfrom
refactor/phase3c-preagg-decompose
Aug 19, 2026
Merged

refactor phase 3c: decompose the pre-aggregation pipeline into named phases#300
hussainsultan merged 4 commits into
mainfrom
refactor/phase3c-preagg-decompose

Conversation

@hussainsultan

Copy link
Copy Markdown
Collaborator

Phase 3c, stacked on #299 — the riskiest single refactor in the plan, executed with the safety net built for it. The 1,089-line to_untagged_with_preagg body is now a ~25-line orchestrator over ten named phase functions, with explicit data flow instead of 15 loose accumulators threading through one scope.

Shape

scope       = _build_scope(...)                # prologue + phase 1: metadata merge,
                                               #   joined table, filter application
raw_tables,
filter_owners = _resolve_filter_ownership(scope)   # 1b
filter_legs  = _split_cross_table_legs(...)        # 1c
plan         = _build_plan(scope)                  # 2
partitioned  = _partition_by_source(scope, plan)   # 3
acc          = _preaggregate_sources(...)          # 4: the 556-line per-source loop
result       = _combine_preaggregates(...)         # 5 (or _aggregate_joined_fallback)
result       = _attach_deferred_count_distincts(scope, acc, result)  # 5b
result       = _apply_calc_phase(scope, plan, acc, result)           # 6
return _project_requested(plan, result)                              # 7

Two explicit dataclasses replace the loose state: _PreaggScope (read-only after phase 1 — op, join tree info, merged fields, the filtered joined table, filter bookkeeping) and _PreaggAccumulators (what phase 4 produces — pre-agg results, decomposed means, re-agg ops, count identities, deferred count-distincts, totals sources).

Method

Every phase body moved verbatim: each function unpacks the scope fields into the original local names at the top, so the diff inside each phase is pure indentation. No logic was rewritten. Extraction went one commit per phase group (scope → 1b–3 → 4 → 5–7), each gated on the SQL-snapshot corpus, the full soundness directory, and the pre-agg stress + BI-trap suites; the final commit ran the full suite.

The follow-up this enables (not in this PR): phase 4's interior — per-table filter pushdown, measure classification, grain computation — can now be decomposed against a 556-line function with a typed contract, and the ~40 remaining broad exception handlers in the pre-agg path can be triaged per phase.

Verification

  • Full suite: 1703 passed, exit 0
  • SQL snapshots byte-identical across all 4 commits
  • Soundness (152), pre-agg stress, BI-traps green at every step
  • SCC ratchet green (graph remains a DAG); ruff + format clean

🤖 Generated with Claude Code

@hussainsultan
hussainsultan force-pushed the refactor/phase3b-ops-split branch from 42c8229 to ee75f26 Compare August 18, 2026 23:15
@hussainsultan
hussainsultan force-pushed the refactor/phase3c-preagg-decompose branch from fd55b6c to 5d0d4e8 Compare August 18, 2026 23:15
@hussainsultan
hussainsultan force-pushed the refactor/phase3b-ops-split branch from ee75f26 to 39183ce Compare August 18, 2026 23:26
@hussainsultan
hussainsultan force-pushed the refactor/phase3c-preagg-decompose branch from 5d0d4e8 to c4fc4cd Compare August 18, 2026 23:27
@hussainsultan
hussainsultan force-pushed the refactor/phase3b-ops-split branch from 39183ce to abff27b Compare August 18, 2026 23:37
@hussainsultan
hussainsultan force-pushed the refactor/phase3c-preagg-decompose branch from c4fc4cd to 6cb2296 Compare August 18, 2026 23:37
@hussainsultan
hussainsultan force-pushed the refactor/phase3b-ops-split branch from abff27b to d3c17e3 Compare August 18, 2026 23:43
@hussainsultan
hussainsultan force-pushed the refactor/phase3c-preagg-decompose branch from 6cb2296 to 98e8955 Compare August 18, 2026 23:43
hussainsultan and others added 4 commits August 18, 2026 19:49
…aggScope

The metadata merge, joined-table construction, wrapper-dimension
ownership inference, and joined-table filter application move verbatim
into _build_scope, returning an explicit read-only _PreaggScope. The
_PreaggAccumulators dataclass is introduced for phase 4's outputs.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
_resolve_filter_ownership, _split_cross_table_legs, _build_plan,
_partition_by_source — verbatim bodies over the explicit scope.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The 556-line per-source pre-aggregation loop moves verbatim into its
own function; its six accumulators travel back as an explicit
_PreaggAccumulators instead of loose locals.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…rchestrator

_aggregate_joined_fallback, _combine_preaggregates,
_attach_deferred_count_distincts, _apply_calc_phase, and
_project_requested complete the decomposition: to_untagged_with_preagg
is a ~25-line orchestrator over ten named phase functions, each with an
explicit signature over _PreaggScope/_PreaggAccumulators instead of 15
loose accumulators threading through 1,089 lines. Bodies verbatim.

Full suite green (1703 passed, exit 0); SQL snapshots byte-identical.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@hussainsultan
hussainsultan force-pushed the refactor/phase3c-preagg-decompose branch from 98e8955 to aa32af3 Compare August 18, 2026 23:50
@hussainsultan
hussainsultan changed the base branch from refactor/phase3b-ops-split to main August 18, 2026 23:50
@hussainsultan hussainsultan reopened this Aug 18, 2026
@hussainsultan
hussainsultan marked this pull request as ready for review August 19, 2026 00:04
@hussainsultan
hussainsultan merged commit 42e5aa3 into main Aug 19, 2026
9 checks passed
hussainsultan added a commit that referenced this pull request Aug 21, 2026
Main's phase 3c (#300) already decomposed the pre-aggregation pipeline
into named phase functions in ops/_compile_preagg.py, superseding this
branch's preagg.py staging. Resolve by keeping main's implementation and
dropping the branch's preagg.py, its stage tests, and ADR 0002.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant