Milestone 12 (part): benchmarks vs cgo pg_query_go + allocation profiling

[kyleconroy]

Lands the benchmarking + memory-profiling portion of milestone 12: a benchmark suite comparing the pure-Go parser against the cgo pg_query_go, and fixes for the two allocation hotspots the profiles surfaced.

Benchmarks

• benchmark_test.go (root) mirrors pg_query_go v6.2.2's benchmark file name-for-name — same queries, same globals trick — plus Scan/ParseToJSON variants and *Stress benchmarks over the corpus's largest input, the 1.1 MB multi-VALUES INSERT (fingerprint suite, case 073).
• oracle/benchmark_test.go is the cgo twin: identical benchmark names and inputs against the pinned pg_query_go v6.2.2, so the two runs diff directly with benchstat (after normalizing the pkg: line).

Measured on 4 vCPU / go1.24.7, after the fixes below:

Area	pure Go vs cgo
Raw parse, single-threaded	3–10% faster; 29% faster on the 1.1 MB stress query
Scan	parity
Parse parallel (small queries)	6–26% slower (GC pays for every AST node; C-side arena allocations are invisible to Go accounting)
Normalize	1.6–3.6× slower
Fingerprint	4.3–6.3× slower
ParseToJSON	~5× slower

The normalize/fingerprint/JSON gaps share one root cause: those walks are protobuf-reflection-driven. PLAN.md already carried "generated per-node emitters remain an option for milestone 12 if profiling wants them" — the numbers say that's where the remaining time is, if those entry points ever matter to a consumer. (cgo's ~1 alloc/op in benchmark output is only the Go-side result copy, so allocs/op comparisons are only meaningful within the pure-Go column.)

Allocation fixes from profiling

• Pre-size the parser's token buffer to len(input)/3 — append regrowth of parser.toks was 72% of parse alloc_space on the stress query. Stress parse: 92 MB → 39 MB and 185 ms → 146 ms per op; small queries lose a few allocs each.
• Memoize the fingerprint walk's per-message-type field order instead of re-sorting field descriptors at every node visit: fingerprint runs 28–31% faster with ~25% fewer allocations.

What remains is the AST itself (~2 allocs per node: struct + ast.Node oneof wrapper) plus the protobuf marshal — 20–138 allocs on the upstream benchmark queries — which is the cost of the pg_query_go-compatible protobuf AST, not overhead to engineer away.

Also

• Removed the accidentally committed 13 MB oracle/oracle binary and added a .gitignore for it.
• PLAN.md gains "As-built notes (milestone 12, benchmarks + profiling)"; still open from the milestone: difftest mutation fuzzing in CI, go test -race over the corpus run, and the wasilibs comparison.

go test ./... is green — all 308,561 corpus cases pass byte-identically.

🤖 Generated with Claude Code

https://claude.ai/code/session_017iCgDQzsNy8EHzKGhY7P8S

---

Generated by Claude Code