ast: tagged, snake_case JSON output for parse and analyze - #4592
Merged
Conversation
Node is an interface, so the JSON that parse and analyze --ast print carried
no record of which node a given object was. Nodes with no fields of their own
all encoded as "{}": a star in a RETURNING clause was indistinguishable from
an untranslated clause, and an empty List was indistinguishable from both.
Every node struct now declares a zero-sized marker as its first field:
Tag NodeTag[T] `json:"tag"`
with T the node's own type. encoding/json calls MarshalJSON on the field, and
the generic type parameter carries the node's identity to it at compile time,
so plain json.Marshal of any node emits its type name with no custom encoder
walking the tree. The zero value is valid, so the engines' struct literals
are unchanged, and the field is zero-sized, so nodes cost the same memory.
The field's UnmarshalJSON accepts only the containing node's name, making a
document decoded into the wrong node type an error instead of a silently
misfielded tree. A test checks that every node struct declares the field,
that its type parameter names the containing struct (the one mistake the
compiler cannot catch, since a copy-pasted NodeTag[OtherNode] compiles), and
that no node declares a colliding second field.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01T9LCt1mwdY3mE14x3Zz5Vw
Most of what parse printed was absent fields. On a four-query file, 25% of
the keys were null and another 9% were empty containers, so reading an AST
meant scanning past clauses the statement never had.
Mark every pointer, slice, interface and map field of the node structs with
json:",omitempty". Scalars keep no such tag on purpose: StmtLocation is 0 for
the first statement in a file and LIMIT 0 parses to an Ival of 0, so omitting
zero-valued scalars would lose what the parser found rather than what it did
not. Field names are unchanged.
Beyond nil fields this also drops empty lists, which is a small normalization:
the dolphin converter builds Items as an empty slice where the postgresql one
leaves it nil, so the same SQL printed different JSON per engine. Both now
print as a bare tagged List.
This depends on the type tags. Omitting a nil Items turns an empty List into
"{}", which without a tag would be indistinguishable from A_Star and TODO.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01T9LCt1mwdY3mE14x3Zz5Vw
Give every node field an explicit json tag with the snake_case form of its name: StmtLocation prints as stmt_location, FromClause as from_clause. The keys now read as JSON rather than as Go leaking through, and they match the names libpg_query uses for the same fields in its own JSON output, so anyone coming from pg_query's tree finds the fields where they expect them. The tags also stop the output from being coupled to the Go field names: a field rename is now a compile-time-visible decision about the JSON, not a silent output change. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T9LCt1mwdY3mE14x3Zz5Vw
The source-level test checked that every node struct declares the Tag marker with the right type parameter. Drop it: the end-to-end goldens pin the output that matters, and the declarations are one convention, not behavior. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T9LCt1mwdY3mE14x3Zz5Vw
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reworks the JSON that
sqlc parseandsqlc analyze --astprint, in three commits. Replaces #4591, which got the same tagging from a hand-written reflective encoder; this version gets it from stockencoding/jsonwith no custom marshaller in the tree walk.Tag every node with its type
ast.Nodeis an interface, so the JSON encoding carried no record of which node a given object was. Nodes with no fields of their own —A_Star,Null,TODO— all encoded as{}, and an emptyListwas indistinguishable from them. OnINSERT ... RETURNING *, the star and four untranslated clauses printed as the same empty object.Every node struct now declares a zero-sized marker as its first field:
encoding/jsoncallsMarshalJSONon the field, and the generic type parameter carries the node's identity to it at compile time — no encoder walks the tree, plainjson.Marshalof any node is tagged. The zero value is valid, so the engines' struct literals are untouched, and the field is zero-sized, so nodes cost the same memory.The marker's
UnmarshalJSONaccepts only the containing node's name, so a document decoded into the wrong node type is an error (node tagged "RangeVar" decoded into SortBy) instead of a silently misfielded tree.tagas the key avoids every collision:encoding/jsonmatches field names case-insensitively on decode, sokindwould captureA_Expr.Kind,typethe eight nodes with aTypefield, andnodewould silently captureSortBy.Node. Because the tag is a real field rather than a key injected beside the fields, nothing can shadow it.Omit absent fields
25% of the keys in typical output were
nulland another 9% were empty containers. Pointer, slice, interface and map fields now carry,omitempty; scalars deliberately don't, because zero is a value the parser can find:stmt_locationis 0 for the first statement in a file, andLIMIT 0parses to anivalof 0.This also drops empty (not just nil) lists, a small normalization: the dolphin converter builds
Itemsas an empty slice where the postgresql one leaves it nil, so the same SQL printed different JSON per engine.snake_case field names
Every node field gets an explicit json tag with the snake_case form of its name:
stmt_location,from_clause,relname. The keys read as JSON rather than Go leaking through, match the names libpg_query uses for the same fields in its own JSON output, and matchanalyze's existingcolumns/paramskeys. Field renames become a compile-time-visible decision about the JSON rather than a silent output change.Sample, for
INSERT ... RETURNING *:Notes
Pos()), correctly excluding the 76 non-structNodeimplementers (enum types withPosmethods) and the fmt/comment structs. 702 reference-typed fields carryomitempty; 528 scalar fields carry a name-only tag; no two fields in any struct collapse to the same snake key.docs/howto/parse.mdalready notes the JSON shape is beta; it gains sections on node tags, absent fields, and the naming convention.TODOtags are documented as "parsed but not represented in the AST", not "absent from the query".Testing
Covered end to end:
go test --tags=examples -timeout 20m ./...passes with PostgreSQL and MySQL running, plusgo vetandgofmt. Eight goldens regenerated (sevenparse_basic,analyze_ast/postgresql); the other 26parse/analyzecases regenerate byte-identical, confirming nothing outside AST output moved.