Advance the pin to libpg_query 18.0.0 (PostgreSQL 18.4) - #10
Merged
Conversation
No pg_query_go release ships libpg_query 18 yet, so the oracle module now builds against a locally generated tree: pg_query_go's Go wrapper (main, pinned commit) with its vendored C sources replaced by libpg_query 18.0.0, reproducing pg_query_go's own 'make update_source'. The tree is populated by oracle/update-pg-query-go.sh (gitignored) and wired in with a replace directive. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017CxEHH1UHZXsnhGpYQ9Fip
…eSQL 18.4) srcdata/*.json and pg_query.proto verbatim from the 18.0.0 tag; internal/reference re-derived from the postgresql-18.4 tarball with the tag's patches 01/03/04/09 applied (patch 04 now also touches gram.y — the comment-token declarations moved there); kwlist.h, pl_*_kwlist.h, plerrcodes.h copied from the tag's extracted includes. kwlist.h grows to 494 keywords. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017CxEHH1UHZXsnhGpYQ9Fip
cmd/generate over the 18.0.0 inputs: ast/pg_query.pb.go, aliases.go, keyword tables (494 keywords; RECHECK is gone), plpgsql tables. Ports for the PG 18 node changes, from the pinned gram.y / postgres_deparse.c / nodeFuncs.c: - returningList becomes ReturningClause on the four DML statements, with the new RETURNING WITH (OLD/NEW AS alias) options (gram.y returning_clause/returning_with_clause/returning_option); parser, deparser (deparseReturningClause), rawwalk and normalize walkers updated, ReturningOption walked as a primitive node. - AT_CheckNotNull no longer exists (removed no-op deparse arm). - opclass_item no longer accepts RECHECK. - IntoClause.viewQuery is typed Query in the PG 18 proto; the walkers treat a non-NULL value as unknown (C default arm), as before it was always NULL in raw trees. - ParseResult.Version is 180004; ported test literals updated to match (verified against the PG 18 oracle). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017CxEHH1UHZXsnhGpYQ9Fip
cmd/regenerate over the 18.0.0 corpora (postgres_regress and friends at PostgreSQL 18.4): 2,130 files / 330,817 cases across the eleven suites. Cases whose input+expectation are unchanged keep their passing status; the 17→18 diff (187,735 cases, dominated by the version field in every parse golden) returns to the todo list to be driven down. cmd/regenerate learns to strip // line comments from pg_query_go/testdata/fingerprint.json — the 18.0.0 file carries a comment, which encoding/json rejects. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017CxEHH1UHZXsnhGpYQ9Fip
…ates - stmtmulti now sets each RawStmt's stmt_location to its own first token's position (makeRawStmt($3, @3)); the PL/pgSQL raw-parse modes follow (makeRawStmt($2, @2)). - VariableSetStmt gains location and jumble_args per the 18.4 grammar: generic SET/RESET, the SQL-standard special syntaxes, ALTER SYSTEM SET/RESET, and TRANSACTION SNAPSHOT all ported with their exact @n or -1 values. - Fingerprint walk follows 18.0.0's generator: list_start/list_end and A_Expr rexpr list bounds are no longer jumbled, jumble_args is ignored, and RangeVar gets the hand-written body that mirrors PostgreSQL's post-analysis query jumble (alias contributes and suppresses relname in DML contexts, schemaname dropped there). - Version constant in the internal parser is 180004 as well. Todo count: 51,951 → 5,664. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017CxEHH1UHZXsnhGpYQ9Fip
…poral keys, ATAlterConstraint - ConstraintAttributeSpec/Elem gain ENFORCED / NOT ENFORCED (CAS bits 0x40/0x80) with the conflicting-properties check; processCASbits takes the is_enforced output (defaulting true, NOT ENFORCED also marks the constraint not valid) and its callers pass the 18.4 argument sets. - Column constraints: NOT NULL takes opt_no_inherit and records is_enforced/initially_valid; CHECK and REFERENCES record is_enforced; GENERATED ... AS (expr) takes opt_virtual_or_stored (VIRTUAL default) into generated_kind. - Table constraints: new NOT NULL ColId ConstraintAttributeSpec production (NOT joins the constraint dispatch sets); UNIQUE/PRIMARY KEY accept WITHOUT OVERLAPS; FOREIGN KEY accepts PERIOD columns on both sides (fk_with_period/pk_with_period), disambiguated one token after PERIOD exactly as the LALR tables do. - Domain constraints follow the new processCASbits shapes (NOT NULL no longer supports NO INHERIT; CHECK records is_enforced). - ALTER TABLE ALTER CONSTRAINT builds the new ATAlterConstraint node (alter_enforceability/deferrability/inheritability, the INHERIT variant, and the NOT VALID error); deparser gains deparseATAlterConstraint. - FunctionParameter records location = @1 (func_arg, table_func_column). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017CxEHH1UHZXsnhGpYQ9Fip
…string JSON omission - makeAArrayExpr records list_start/list_end; scalar IN/NOT IN records rexpr_list_start/rexpr_list_end (in_expr is inlined into a_expr at 18). - SelectLimit carries offsetLoc/countLoc/optionLoc and insertSelectOptions reports its errors at them. - ConstraintAttributeSpec's location is its first element's (PG 18's YYLLOC_DEFAULT scans for the first valid RHS location), so processCASbits errors carry cursors now. - vacuum_relation uses relation_expr: VACUUM/ANALYZE ONLY tbl (and trailing *) parse, with inh set accordingly. - opt_target_list treats RETURNING as a follow token (INSERT ... SELECT RETURNING with an empty target list). - unrecognized JSON encoding reports parser_errposition(@4). - The JSON emitter omits empty strings entirely, as 18.0.0's WRITE_STRING_FIELD does (alwaysEmitString special cases removed). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017CxEHH1UHZXsnhGpYQ9Fip
The C fingerprint sees the original tree, where COMMENT/SECURITY LABEL ... IS '' is a non-NULL empty string and IS NULL is NULL — a distinction proto3 drops. Since Fingerprint re-parses its input, the parser now records present-but-empty strings (ParseTracked) and the walk emits the field name for those nodes (TreeWithEmpties). parsePartitionStrategy reports its cursor at the strategy token (its PG 18 signature gained the location). Parse, scan, normalize, normalize_utility, fingerprint, and both split suites are green again; deparse, summary, and plpgsql remain. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017CxEHH1UHZXsnhGpYQ9Fip
- deparseConstraint takes a context: the domain VALUE key elides only in ALTER DOMAIN, NOT NULL/NULL table constraints print bare column names, UNIQUE/PRIMARY KEY print WITHOUT OVERLAPS, FK sides print PERIOD columns (deparseColumnListWithPeriod), GENERATED prints STORED/VIRTUAL by generated_kind, ENFORCED/NOT ENFORCED constraint attributes and the FK/CHECK NOT ENFORCED trailer print. - VariableSetStmt deparse keys the special TIME ZONE / XML OPTION / TIME ZONE DEFAULT forms off jumble_args (isSetTimeZoneInterval is gone upstream). - ALTER DEFAULT PRIVILEGES ... ON LARGE OBJECTS parses and deparses. - JOIN_RIGHT_SEMI joins the planner-only jointype arm; A_Indirection parenthesizes an A_ArrayExpr argument; DETACH PARTITION FINALIZE spacing fix; COPY (freeze) with no argument stays in WITH form. - The summary truncation deparse receives the parser's present-but-empty string set so COMMENT/SECURITY LABEL ... IS '' round-trips (the C summary deparses the original tree); rawwalk keeps ReturningClause out of its switch — 18.0.0 did not extend pg_query_raw_tree_walker_supports, so summary walks must not descend into RETURNING. Only the plpgsql suite (298 todos) remains. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017CxEHH1UHZXsnhGpYQ9Fip
libpg_query 18.0.0 rebuilt its PL/pgSQL support around the real compile
path: plpgsql_compile_callback runs against a forged pg_proc tuple, with
a mini pg_type catalog served by mocked syscache lookups. Ported:
- internal/reference vendors pg_query_pg_type.c + pg_type_d.h;
cmd/generate -plpgsql emits the builtinTypes table (193 entries).
- Real parse_datatype: typeStringToTypeName via the core parser's new
RAW_PARSE_TYPE_NAME mode (parse.ParseTypeName), the mocked
LookupTypeName pipeline (search path [pg_catalog, public]; unknown
public-namespace names resolve to RECORD), and build_datatype with
canonical catalog typenames (int -> int4, bool, ...). %TYPE/%ROWTYPE
stubs keep the reference text; array decoration uses the real
plpgsql_build_datatype_arrayof (whose syscache miss on a stub's
InvalidOid reproduces the mock's 'Not implemented' error).
- compileCreateFunctionStmt follows pg_query_create_function +
do_compile: interpret_function_parameter_list (typed, moded, VARIADIC
must-be-array via the always-empty mocked typelem), compute_return_type,
OUT-parameter row building, polymorphic resolution in validator mode,
pseudo-type rejections, trigger/event-trigger datums (new/old records
plus the tg_* promise variables, serialized as bare {}), and the real
add_dummy_return condition.
- make_return_stmt is the real one: SETOF/VOID/OUT-parameter errors, and
RETURN of a simple var/rec/row datum captures retvarno (not expr).
- Records honor CONSTANT; format_type_be renders SQL-standard spellings
with element[] only for base-type arrays.
- Parameter-list and return-type errors escape before the compile error
context is installed, so they carry no 'near line' context.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017CxEHH1UHZXsnhGpYQ9Fip
- CLAUDE.md/PLAN.md/README.md and the per-file attribution comments now name libpg_query 18.0.0 (PostgreSQL 18.4); PLAN.md § pin records the 2026-08-17 advance and the local-oracle arrangement. - The regenerate workflow clones the 18.0.0 tag, installs protoc, and builds the oracle's pg_query_go dependency via oracle/update-pg-query-go.sh before regenerating. - internal/reference/README.md documents the two new generator inputs (pg_query_pg_type.c, pg_type_d.h). - The ported pg_query_go compat literals updated to the 18 shapes (rexpr list bounds on scalar IN, canonical PL/pgSQL type names, RETURN-variable statements), verified against the PG 18 oracle. go build/vet clean; go test ./... green; corpus todo lists empty. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017CxEHH1UHZXsnhGpYQ9Fip
Both jobs of the previous run died in action download (503/429 from codeload.github.com) before any step ran. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017CxEHH1UHZXsnhGpYQ9Fip
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Advances the pin from libpg_query 17-6.2.2 (PostgreSQL 17.7) to libpg_query 18.0.0 (PostgreSQL 18.4), following PLAN.md § Regeneration.
ParseResult.Versionis now 180004. Every corpus suite is green: 0 todos across 2,130 files / 330,817 cases, andgo test ./...(including-raceover the corpus) passes.The oracle
pg_query_go has no PG 18 release yet (latest tag is v6.2.2 on libpg_query 17-6.2.2), so the cgo oracle now builds pg_query_go's wrapper against the 18.0.0 sources locally:
oracle/update-pg-query-go.shreproduces pg_query_go's ownmake update_sourceagainst the pinned tag into a gitignored tree thatoracle/go.modpicks up via areplacedirective. The weekly regenerate workflow installsprotocand runs the script before verifying golden reproducibility. When pg_query_go cuts a PG 18 release, the script and replace can be dropped for a normal module requirement.Re-vendored inputs
srcdata/*.json+pg_query.protoverbatim from the 18.0.0 tag;internal/referencere-derived from the postgresql-18.4 tarball with the tag's patches (patch 04 now also touchesgram.y).pg_query_pg_type.c+pg_type_d.h(the mini pg_type catalog libpg_query 18's PL/pgSQL mocks serve);cmd/generate -plpgsqlemits the 193-entry builtin type table.cmd/regeneratelearned to strip//line comments fromfingerprint.json(the 18.0.0 file carries one).Grammar and behavior changes ported
returningList→ the newReturningClausenode on all four DML statements (parser, deparser, walkers, fingerprint). Upstream did not extendpg_query_raw_tree_walker_supportsto it, so the summary walks visit but never descend into RETURNING.stmt_locationnow points at each statement's first token, and PG 18's rewrittenYYLLOC_DEFAULTscans for the first valid RHS location — givingConstraintAttributeSpec,insertSelectOptions,processCASbits, andparsePartitionStrategyreal error cursors.ENFORCED/NOT ENFORCED(CAS bits +processCASbitssignature), table-levelNOT NULL colname, temporal keys (WITHOUT OVERLAPS, FKPERIODcolumns),GENERATED ... VIRTUAL(the new default), and theATAlterConstraintnode forALTER TABLE ALTER CONSTRAINT(incl. theINHERITvariant).list_start/list_end, rexpr list bounds,jumble_args). The proto3-unrepresentableCOMMENT ... IS ''vsIS NULLdistinction is preserved by tracking present-but-empty strings at parse time and threading them into the fingerprint and summary-truncation walks.interpret_function_parameter_list/compute_return_type/do_compile, and a builtin pg_type catalog. Types resolve to canonical names (int→int4), unknown public-namespace types become RECORD, trigger functions getnew/oldrecords plus the tentg_*promise variables (serialized as bare{}),RETURN simple_varcaptures a varno, and the real SETOF/VOID/OUT-parameter RETURN errors apply. The core parser gainedRAW_PARSE_TYPE_NAMEmode fortypeStringToTypeName.VACUUM/ANALYZE ONLY(vacuum_relation → relation_expr),ALTER DEFAULT PRIVILEGES ... ON LARGE OBJECTS,VariableSetStmtlocation/jumble_argsfields (and the deparser keying its TIME ZONE / XML OPTION forms off them), array-literallist_start/list_end, scalar IN's rexpr list bounds (in_exprwas inlined intoa_expr), the JSON emitter omitting all empty strings (matching 18'sWRITE_STRING_FIELD),FunctionParameter.location,JOIN_RIGHT_SEMI, andCOPY (freeze)staying inWITH (...)form.Corpus
Regenerated from the 18.0.0 oracle: 2,130 files / 330,817 cases (~113 MB). The 17→18 diff put 187,735 cases back on the todo list; all were driven to zero through the
next-test/-check-parseloop. The ported pg_query_go compat literals (parse_test.go) were updated to the 18 shapes, verified against the oracle.Still open from milestone 12 (unchanged by this PR): difftest mutation fuzzing as a scheduled job and the wasilibs comparison.
🤖 Generated with Claude Code
https://claude.ai/code/session_017CxEHH1UHZXsnhGpYQ9Fip
Generated by Claude Code