Give ntConstant and ntVariable end positions - #9
Closed
partouf wants to merge 3 commits into
Closed
Conversation
A constant declaration produced a node with no end position at all, so a
consumer working in line ranges could not tell how far the declaration
reached. That silently truncates any constant whose value spans lines:
const
Banner = 'first part ' +
'second part';
reported only the first line, and a tool slicing that range dropped the
continuation.
ntConstant was built with FStack.Push, so it was a plain TSyntaxNode with
nowhere to record an end. Two changes are needed, because the node the
caller finally sees is not the node that was parsed:
ConstantDeclaration now pushes a compound node and records its end, the
same way TypeDeclaration already does. This gives the intermediate
ConstList an accurate extent.
ConstSection rebuilds each constant from that ConstList, so it also has
to push a compound node and inherit the end from it. The start still
comes from the name, which is what a caller looking for the constant
expects; only the end is new.
TCompoundSyntaxNode.AssignEndPositionFrom is the counterpart to the
existing AssignPositionFrom, for exactly this rebuild case.
Single-line constants keep ending on their own line - the new test asserts
both directions, since an end that ran on to the next declaration would be
no more useful than one that stopped short.
Verified against the existing suite: 42 passing, with the one pre-existing
Serialization.BinaryRoundTrip failure unchanged (line_seq holds a pointer
value that does not survive a round trip, unrelated to this change).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
partouf
force-pushed
the
fix/const-var-endline
branch
from
August 19, 2026 11:39
1a08cb7 to
378d904
Compare
Same gap as the previous commit, same shape of fix. A variable declaration
produced a node positioned at its name with no end at all, so a declaration
whose type or initialiser spans lines reported only its first line:
var
Grid: array[0..1] of
Integer;
VarDeclaration pushed ntVariables with FStack.Push, so the VarList that
RearrangeVarSection rebuilds each variable from had no extent to pass on.
Both now push compound nodes, and the rebuilt ntVariable inherits the end
via AssignEndPositionFrom - the second caller for the helper added in the
previous commit, which is the pattern it exists for.
The compound ntVariables nodes are the throwaway VarSect children that
VarSection frees, so the output footprint matches the constant change
exactly: VARIABLE gains begin/end, and the VARIABLES section node that
reaches the tree is untouched.
Names sharing one line (`A, B: Integer;`) each get the declaration's extent,
which is the whole declaration they share.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An array bound may be any constant expression, but OrdinalType only ever
accepted a constant or a type name:
const
mlab = 4;
mlog = 12;
type
TRanges = record
iu: array[mlab + 1..mlog] of Integer;
end;
failed with 'SquareClose' expected found '+'. OrdinalType decided on a single
token of lookahead: an identifier followed by anything but '(' or '..' was a
type name, so it read the bound as the type `mlab`, returned, and left
ArrayBounds looking for the ']' it found a '+' at.
The upper bound already worked, which is what makes the gap easy to miss.
`array[mlab..mlog + 1]` sends the first bound to ConstantExpression on the
'..' lookahead, and OrdinalType's own trailing '..' branch parses the rest as
an expression. Only the first bound of a subrange went down the type-name
path.
The lookahead now also routes the operators SimpleExpression and Term accept
to ConstantExpression. Every one of those tokens is a parse error in this
position today, so no input that parses now takes a different path: an
identifier in an OrdinalType is followed by ']', ',', '..', 'of' or ';', never
by an operator. Set types and variant record tag types go through the same
procedure and gain the same forms.
A single token is enough here and a full ahead-parse would be worse. Coming
from the identifier, SimpleType's `AheadParse.NextToken; AheadParse.Simple-
Expression` idiom would meet the ']' of the common `array[TIndex] of Byte` and
hand it to Factor as a set constructor.
Parenthesized bounds are deliberately left alone. `array[(mlab + 1)..mlog]`
goes to EnumeratedType, and dcc32 reads it the same way - it reports
"Identifier redeclared: 'mlab'" - so the parser already agrees with the
compiler.
Test/Snippets/arrayboundexpression.pas covers the first bound, both bounds,
two dimensions, `*`, `-` and `shl`, and a set of a computed subrange; dcc32
compiles it clean. Suite: 45 tests, 44 passing, with the pre-existing
Serialization.BinaryRoundTrip failure unchanged (line_seq holds a pointer
value that does not survive a round trip).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Constant and variable declarations produce nodes with no end position at all, so a consumer working in line ranges cannot tell how far a declaration reaches. That silently truncates any declaration that spans lines:
Both report only their first line, so a tool slicing that range drops the continuation.
ntTypeDeclalready records an end; this brings constants and variables in line with it.I hit this writing a refactoring tool that moves declarations between units by slicing source text — a multi-line constant came out truncated in the destination and orphaned in the source, leaving both files unparseable, with nothing in the AST to indicate why.
Two commits, one per node type, each independently buildable and testable.
Why each fix is two changes
The node a caller finally sees is not the node that was parsed, so fixing the declaration handler alone has no effect (my first attempt, and the test still failed):
ConstantDeclaration/VarDeclarationnow push compound nodes and record their end, the wayTypeDeclarationalready does. This gives the intermediateConstList/VarListan accurate extent.ConstSection/RearrangeVarSectionrebuild each declaration from that intermediate, so they also push compound nodes and inherit the end from it. The start still comes from the name — which is what a caller looking for the declaration expects — and only the end is new.TCompoundSyntaxNode.AssignEndPositionFromis added as the counterpart to the existingAssignPositionFrom, for exactly this rebuild case; both fixes use it. Theis TCompoundSyntaxNodeguards keep things safe if a declaration ever arrives by another path.Names sharing one line (
A, B: Integer;) each get the extent of the declaration they share.Output footprint
Only the per-declaration node gains position attributes. The section nodes that reach the tree are untouched, because the compound
ntConstants/ntVariablesnodes created above are the throwaway intermediates thatConstSectionandVarSectionfree:Verification
Test/UnitTestson Delphi 13 Win32, at each commit:The one failure is unchanged and pre-existing —
Serialization.BinaryRoundTrip, whereline_seqholds a pointer value that does not survive a round trip. Unrelated to these changes.FPC via the repo's own
FPC testsworkflow, on the branch tip: 43 tests, 43 passed, 0 failed.The two new tests each assert both directions:
The second assertion matters as much as the first: an end position that overran into the next declaration would be no more useful than one that stopped short.
Checked against a real file rather than only synthetic strings:
I also rebuilt the tool that prompted this against the patched parser and ran its full golden-file suite at each commit — every command's output and every lint rule byte-identical. So this does not disturb an existing consumer that reads constants or variables today.
Deliberately not included
ntMethod.EndLineis a different problem and I have left it alone. It is populated, but with the lexer position after the construct, so for a bodyless declaration it points at the next declaration's first token:A routine with a body can be measured from its
ntStatementschild, which is exact. Unlike constants and variables — which had no value at all, so there was nothing to break — changing whatntMethod.EndLinemeans would be a behavioural break for any consumer already relying on it. Happy to look at it separately if you think it is worth changing.