Skip to content

refactor phase 4: serialization consolidation — utils.py dissolved, codec on public constructors - #301

Merged
hussainsultan merged 3 commits into
mainfrom
refactor/phase4-serialization
Aug 19, 2026
Merged

refactor phase 4: serialization consolidation — utils.py dissolved, codec on public constructors#301
hussainsultan merged 3 commits into
mainfrom
refactor/phase4-serialization

Conversation

@hussainsultan

Copy link
Copy Markdown
Collaborator

Phase 4 of the SCC-driven refactor, stacked on #300. The serialization stack becomes one self-contained package, the hand-built ibis internals are gone, and the codec provably works on plain ibis. Full suite green (1749 passed, exit 0 — up 46 tests from the new codec coverage).

utils.py no longer exists

The seven-cluster grab-bag dissolved into its real homes, bodies verbatim:

  • safe_eval.py — the AST-allowlist sandbox for YAML/agent expression strings (its own trust boundary, now visibly separate from serialization's)
  • io.py — YAML/URL config loading
  • serialization/_trust.py — the callable allowlist for ("fn", module, qualname) payload arms, enforced on write and read; trust_callable_module/UntrustedCallableError export from the serialization package and the refusal message names the new path
  • serialization/codec.py — the resolver-tree expression codec (scalar codec, serialize/deserialize_resolver, structured expr + join-predicate API). The layering inversion the audit flagged is fixed: the serialization stack's lowest layer finally lives inside the package.

Codec rebuilt on public constructors — and the plain-ibis Item bug is fixed

deserialize_resolver used object.__new__ + object.__setattr__ + a hand-computed __precomputed_hash__ to bypass validation — the source of an entire bug class (rebuilt resolvers raising AttributeError when hashed). Verified against both flavors: the public constructors are positionally identical and accept already-built child resolvers, so all seven arms now use them and _finalize_frozen_slotted is deleted. BinaryOperator/UnaryOperator gain upstream's own operator validation for free.

This also fixes the latent flavor bug: plain ibis 11 renamed Item's key slot nameindexer, so _["x"]-style expressions crashed on the HAS_XORQ=False branch — invisible because every serialization suite importorskips xorq. The serializer reads whichever slot exists; the deserializer passes the key positionally.

New coverage where the audit found none

  • test_codec_no_xorq.py — runs the round-trip in a subprocess with xorq import-blocked (including the Item case), so the plain-ibis branch is exercised even on xorq machines
  • test_codec_units.py — scalar-codec symmetry (dates, Decimal, timedelta…), freeze/thaw symmetry, and a 16-shape resolver round-trip battery with hashability checks (the precomputed-hash bug class now has direct unit coverage)

One format constant; less indirection

  • SCHEMA_VERSION + SUPPORTED_PAYLOAD_MAJORS live together in context.py — writer stamps and reader gates from one place; a format bump is a one-file change
  • helpers.py dissolved into the codec; extract_simple_column_name rewritten with real isinstance checks instead of type(x).__name__ == "Attr" string comparisons

Deliberately kept: trust roots stay a process-global registry — trust_callable_module is the documented, tested API and additive process-wide trust is the intended semantic; context-scoping it would thread a parameter through every codec call without a driving use case.

Verification

  • Full suite: 1749 passed, exit 0; SQL snapshots byte-identical
  • SCC ratchet green — serialization's internal graph stays acyclic through the reshuffle
  • ruff + format clean

🤖 Generated with Claude Code

@hussainsultan
hussainsultan force-pushed the refactor/phase3c-preagg-decompose branch from fd55b6c to 5d0d4e8 Compare August 18, 2026 23:15
@hussainsultan
hussainsultan force-pushed the refactor/phase4-serialization branch from 495bbba to 8eeb595 Compare August 18, 2026 23:15
@hussainsultan
hussainsultan force-pushed the refactor/phase3c-preagg-decompose branch from 5d0d4e8 to c4fc4cd Compare August 18, 2026 23:26
@hussainsultan
hussainsultan force-pushed the refactor/phase4-serialization branch from 8eeb595 to 9be1b17 Compare August 18, 2026 23:27
@hussainsultan
hussainsultan force-pushed the refactor/phase3c-preagg-decompose branch from c4fc4cd to 6cb2296 Compare August 18, 2026 23:37
@hussainsultan
hussainsultan force-pushed the refactor/phase4-serialization branch from 9be1b17 to 048161e Compare August 18, 2026 23:37
@hussainsultan
hussainsultan force-pushed the refactor/phase3c-preagg-decompose branch from 6cb2296 to 98e8955 Compare August 18, 2026 23:43
@hussainsultan
hussainsultan force-pushed the refactor/phase4-serialization branch from 048161e to 29bcdc4 Compare August 18, 2026 23:44
@hussainsultan
hussainsultan force-pushed the refactor/phase3c-preagg-decompose branch from 98e8955 to aa32af3 Compare August 18, 2026 23:50
@hussainsultan
hussainsultan force-pushed the refactor/phase4-serialization branch from 29bcdc4 to e790de3 Compare August 18, 2026 23:50
hussainsultan and others added 3 commits August 18, 2026 20:05
The grab-bag is gone; each cluster now lives with its concern:

- safe_eval.py: the AST-allowlist sandbox for YAML/agent expression
  strings (its own trust boundary, unrelated to serialization's)
- io.py: YAML/URL config loading
- serialization/_trust.py: the callable allowlist for ('fn', module,
  qualname) payload arms, enforced on write and read; the public
  trust_callable_module/UntrustedCallableError now export from the
  serialization package (the error hint names the new path)
- serialization/codec.py: the resolver-tree expression codec (scalar
  encode/decode, serialize/deserialize_resolver, the structured expr
  and join-predicate API) — the serialization stack's lowest layer
  finally lives inside the package

All importers updated; bodies verbatim. Full suite 1703 passed, exit 0.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…lain ibis

deserialize_resolver used object.__new__ + object.__setattr__ +
a hand-computed __precomputed_hash__ to bypass validation — an entire
bug class (rebuilt resolvers raising AttributeError when hashed) that
existed only because of the bypass. The public constructors accept
already-built child resolvers on both ibis flavors (verified against
xorq's vendored ibis and plain ibis 11), so all seven arms now use
them and _finalize_frozen_slotted is deleted.

This also fixes the latent flavor bug: plain ibis 11 renamed Item's
key slot from 'name' to 'indexer', so serializing or rebuilding
_["x"]-style expressions crashed on the HAS_XORQ=False branch —
invisible because every serialization suite importorskips xorq. The
serializer reads whichever slot exists; the deserializer passes the
key positionally.

New coverage: test_codec_no_xorq.py runs the round-trip in a
subprocess with xorq import-blocked (including the Item case);
test_codec_units.py pins scalar-codec symmetry, freeze/thaw, and a
16-shape resolver round-trip battery with hashability checks.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- SCHEMA_VERSION + SUPPORTED_PAYLOAD_MAJORS live together in context.py:
  the writer stamps and the reader gates from one place; bumping the
  format is now a one-file change
- helpers.py dissolves: deserialize_structured joins the codec beside
  structured_to_expr, and extract_simple_column_name is rewritten with
  real isinstance checks against the flavor classes instead of
  type-name string comparisons (the third, stringly-typed resolver
  traversal is gone)

Trust roots deliberately stay a process-global registry:
trust_callable_module is the documented, tested API and additive
process-wide trust is the intended semantic; threading a context
through every codec call would be churn without a driving use case.

Full suite 1749 passed, exit 0.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@hussainsultan
hussainsultan force-pushed the refactor/phase4-serialization branch from e790de3 to a0865e5 Compare August 19, 2026 00:05
@hussainsultan
hussainsultan changed the base branch from refactor/phase3c-preagg-decompose to main August 19, 2026 00:05
@hussainsultan hussainsultan reopened this Aug 19, 2026
@hussainsultan
hussainsultan marked this pull request as ready for review August 19, 2026 00:33
@hussainsultan
hussainsultan merged commit cbb4679 into main Aug 19, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant