Skip to content

refactor phase 3b: split ops.py — package conversion + compile strategies extracted - #299

Merged
hussainsultan merged 7 commits into
mainfrom
refactor/phase3b-ops-split
Aug 18, 2026
Merged

refactor phase 3b: split ops.py — package conversion + compile strategies extracted#299
hussainsultan merged 7 commits into
mainfrom
refactor/phase3b-ops-split

Conversation

@hussainsultan

Copy link
Copy Markdown
Collaborator

Phase 3b of the SCC-driven refactor, stacked on #298. The 7,720-line ops.py monolith becomes an ops/ package: five helper zones and the three compile strategies move into focused modules, all as verbatim moves — every commit is green on the full suite and the SQL snapshots are byte-identical throughout. _core.py shrinks 7,720 → 5,369 lines.

Shape

Stage 1 — package conversion. ops.pyops/_core.py verbatim; ops/__init__.py is the stable import surface (explicit re-exports of every name any module or test pulls from boring_semantic_layer.ops, plus a __getattr__ fallback to _core), so the split can never break an importer.

Stage 2 — helper zones out of _core, one commit each:

  • _compat.py — xorq/plain-ibis backend interop (conversion, connection identity, backend rebinding)
  • _tracking.py — projection-pushdown column tracking (ColumnTracker & friends)
  • _reductions.py — the phase-2-deferred ReductionKind consolidation: reduction sniffers, re-aggregation op selection, count-identity fills — one module answers "how does this reduction re-aggregate from a finer pre-aggregate"
  • _values.pyDimension/Measure/CalcMeasure, _CallableWrapper, the dimension proxies, column-error formatting
  • _normalize.py — join/group-by input normalization and collision-free join-name allocation

Stage 3 — compile strategies out of SemanticAggregateOp:

  • _compile_preagg.py (1,288 ln) — fan-out-safe source-grain pre-aggregation, the dimension-bridge re-join helpers, calc-spec application
  • _compile_deferred.py (235 ln) — aggregate-first + deferred dimension LEFT JOINs
  • _compile_nest.py (86 ln) — nest= aggregate entries

Strategy modules import _core downward; the op methods dispatch through a call-time module accessor, so the import graph stays acyclic (the #298 ratchet stayed green through every commit — lazy in-package imports would have tripped it).

What stays in _core and why

The op classes are mutually referential (SemanticJoinOp.index() constructs SemanticIndexOp; the join analysis helpers isinstance-check the op types and are called from to_untagged dispatchers), so the index cluster and the join-tree analysis belt can't leave without inverting those references — that's the next seam. The # --- N. phase decomposition of the preagg pipeline is deliberately not in this PR: everything here is a verbatim move, and the decomposition is behavior-adjacent surgery that deserves its own diff (phase 3c) — now against a 1,288-line module instead of a 7,700-line one, with the snapshot corpus as its gate.

Verification

  • Full suite after every stage; final: 1703 passed, exit 0
  • SQL snapshots byte-identical across all 9 commits
  • SCC ratchet green throughout (graph remains a DAG); ruff + format clean
  • One extraction bug caught mid-flight by the soundness suites (a method captured into the wrong module still being called method-style) and fixed before commit

🤖 Generated with Claude Code

hussainsultan and others added 7 commits August 18, 2026 19:43
Mechanical stage 1 of the split: ops.py moves to ops/_core.py verbatim
(relative imports adjusted one level), and ops/__init__.py becomes the
stable import surface — explicit re-exports of every name any module
or test pulls from boring_semantic_layer.ops, with a __getattr__
fallback to _core so the zone-by-zone split can never break an
importer. Full suite green; SQL snapshots byte-identical.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
_patch_xorq_sortkey_compat, _ensure_xorq_table, _connection_identity,
_rebind_to_backend, _rebind_to_canonical_backend move verbatim out of
_core.py into their own leaf module.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ColumnTracker, ColumnExtractionResult, JoinColumnExtractionResult,
_make_tracking_proxy, _extract_columns_from_callable,
_extract_join_key_columns move verbatim out of _core.py; a stale
banner comment from the deleted requirements cluster goes with them.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The phase-2-deferred ReductionKind consolidation: _reductions_for_expr
plus the mean/count/count-distinct sniffers, re-aggregation op
selection, and count-identity fills move verbatim into one leaf module
— the single answer to 'how does this reduction re-aggregate from a
finer pre-aggregate'.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The value types plus _CallableWrapper/_ensure_wrapped, _is_deferred,
_reject_bool_resolution, the dimension proxies, and the column-error
formatter move verbatim into their own module.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…allocation)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
SemanticAggregateOp's strategy methods move out of _core.py as
functions taking the op:

- _compile_preagg.py (1,288 ln): fan-out-safe source-grain
  pre-aggregation + the dimension-bridge re-join helpers and calc-spec
  application
- _compile_deferred.py (235 ln): aggregate-first + deferred dimension
  LEFT JOINs
- _compile_nest.py (86 ln): nest= aggregate entries

Strategy modules import _core downward; the op methods dispatch through
a call-time module accessor, keeping the import graph acyclic (ratchet
green). Bodies moved verbatim (self -> op); the # --- N. phase
decomposition of the preagg pipeline is the follow-up, now against a
1,288-line module instead of a 7,700-line one.

Full suite green (1703 passed, exit 0); SQL snapshots byte-identical.
_core.py: 7,720 -> 5,369 lines.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@hussainsultan
hussainsultan force-pushed the refactor/phase3b-ops-split branch from abff27b to d3c17e3 Compare August 18, 2026 23:43
@hussainsultan
hussainsultan changed the base branch from refactor/phase3-core-knot to main August 18, 2026 23:44
@hussainsultan hussainsultan reopened this Aug 18, 2026
@hussainsultan
hussainsultan marked this pull request as ready for review August 18, 2026 23:48
@hussainsultan
hussainsultan merged commit b611509 into main Aug 18, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant