Skip to content

Give ntConstant and ntVariable end positions - #9

Closed
partouf wants to merge 3 commits into
jimmckeeth:mainfrom
GDKsoftware:fix/const-var-endline
Closed

Give ntConstant and ntVariable end positions#9
partouf wants to merge 3 commits into
jimmckeeth:mainfrom
GDKsoftware:fix/const-var-endline

Conversation

@partouf

@partouf partouf commented Aug 19, 2026

Copy link
Copy Markdown

Disclosure: written with Claude Code (Claude Opus 5), which is also recorded in each commit's Co-Authored-By trailer. Everything claimed below was actually run, and the steps are reproducible from the branch.

Constant and variable declarations produce nodes with no end position at all, so a consumer working in line ranges cannot tell how far a declaration reaches. That silently truncates any declaration that spans lines:

const
  Banner = 'first part ' +
           'second part';
var
  Grid: array[0..1] of
    Integer;

Both report only their first line, so a tool slicing that range drops the continuation. ntTypeDecl already records an end; this brings constants and variables in line with it.

I hit this writing a refactoring tool that moves declarations between units by slicing source text — a multi-line constant came out truncated in the destination and orphaned in the source, leaving both files unparseable, with nothing in the AST to indicate why.

Two commits, one per node type, each independently buildable and testable.

Why each fix is two changes

The node a caller finally sees is not the node that was parsed, so fixing the declaration handler alone has no effect (my first attempt, and the test still failed):

  • ConstantDeclaration / VarDeclaration now push compound nodes and record their end, the way TypeDeclaration already does. This gives the intermediate ConstList / VarList an accurate extent.
  • ConstSection / RearrangeVarSection rebuild each declaration from that intermediate, so they also push compound nodes and inherit the end from it. The start still comes from the name — which is what a caller looking for the declaration expects — and only the end is new.

TCompoundSyntaxNode.AssignEndPositionFrom is added as the counterpart to the existing AssignPositionFrom, for exactly this rebuild case; both fixes use it. The is TCompoundSyntaxNode guards keep things safe if a declaration ever arrives by another path.

Names sharing one line (A, B: Integer;) each get the extent of the declaration they share.

Output footprint

Only the per-declaration node gains position attributes. The section nodes that reach the tree are untouched, because the compound ntConstants / ntVariables nodes created above are the throwaway intermediates that ConstSection and VarSection free:

<CONSTANT   begin_line=… end_line=…    gained
<VARIABLE   begin_line=… end_line=…    gained
<CONSTANTS  line=… col=…               unchanged
<VARIABLES  line=… col=…               unchanged

Verification

Test/UnitTests on Delphi 13 Win32, at each commit:

tests passed failed
before 42 41 1
after commit 1 (const) 43 42 1
after commit 2 (const + var) 44 43 1

The one failure is unchanged and pre-existing — Serialization.BinaryRoundTrip, where line_seq holds a pointer value that does not survive a round trip. Unrelated to these changes.

FPC via the repo's own FPC tests workflow, on the branch tip: 43 tests, 43 passed, 0 failed.

The two new tests each assert both directions:

  • a declaration spanning two lines reaches the second one, and
  • a single-line declaration still ends on its own line rather than running on to whatever follows.

The second assertion matters as much as the first: an end position that overran into the next declaration would be no more useful than one that stopped short.

Checked against a real file rather than only synthetic strings:

before:  CONST MultiLineConst   start=35  end=NONE (not a compound node)
after:   CONST MultiLineConst   start=35  end=36
         CONST SingleLineConst  start=34  end=34

I also rebuilt the tool that prompted this against the patched parser and ran its full golden-file suite at each commit — every command's output and every lint rule byte-identical. So this does not disturb an existing consumer that reads constants or variables today.

Deliberately not included

ntMethod.EndLine is a different problem and I have left it alone. It is populated, but with the lexer position after the construct, so for a bodyless declaration it points at the next declaration's first token:

ntMethod begin_line=20  end_line=21   (line 20 is a one-line forward declaration)
ntMethod begin_line=21  end_line=25   (line 21 is a one-line forward declaration)

A routine with a body can be measured from its ntStatements child, which is exact. Unlike constants and variables — which had no value at all, so there was nothing to break — changing what ntMethod.EndLine means would be a behavioural break for any consumer already relying on it. Happy to look at it separately if you think it is worth changing.

A constant declaration produced a node with no end position at all, so a
consumer working in line ranges could not tell how far the declaration
reached. That silently truncates any constant whose value spans lines:

  const
    Banner = 'first part ' +
             'second part';

reported only the first line, and a tool slicing that range dropped the
continuation.

ntConstant was built with FStack.Push, so it was a plain TSyntaxNode with
nowhere to record an end. Two changes are needed, because the node the
caller finally sees is not the node that was parsed:

  ConstantDeclaration now pushes a compound node and records its end, the
  same way TypeDeclaration already does. This gives the intermediate
  ConstList an accurate extent.

  ConstSection rebuilds each constant from that ConstList, so it also has
  to push a compound node and inherit the end from it. The start still
  comes from the name, which is what a caller looking for the constant
  expects; only the end is new.

TCompoundSyntaxNode.AssignEndPositionFrom is the counterpart to the
existing AssignPositionFrom, for exactly this rebuild case.

Single-line constants keep ending on their own line - the new test asserts
both directions, since an end that ran on to the next declaration would be
no more useful than one that stopped short.

Verified against the existing suite: 42 passing, with the one pre-existing
Serialization.BinaryRoundTrip failure unchanged (line_seq holds a pointer
value that does not survive a round trip, unrelated to this change).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@partouf
partouf force-pushed the fix/const-var-endline branch from 1a08cb7 to 378d904 Compare August 19, 2026 11:39
Same gap as the previous commit, same shape of fix. A variable declaration
produced a node positioned at its name with no end at all, so a declaration
whose type or initialiser spans lines reported only its first line:

  var
    Grid: array[0..1] of
      Integer;

VarDeclaration pushed ntVariables with FStack.Push, so the VarList that
RearrangeVarSection rebuilds each variable from had no extent to pass on.
Both now push compound nodes, and the rebuilt ntVariable inherits the end
via AssignEndPositionFrom - the second caller for the helper added in the
previous commit, which is the pattern it exists for.

The compound ntVariables nodes are the throwaway VarSect children that
VarSection frees, so the output footprint matches the constant change
exactly: VARIABLE gains begin/end, and the VARIABLES section node that
reaches the tree is untouched.

Names sharing one line (`A, B: Integer;`) each get the declaration's extent,
which is the whole declaration they share.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@partouf partouf changed the title Give ntConstant an end position Give ntConstant and ntVariable end positions Aug 19, 2026
An array bound may be any constant expression, but OrdinalType only ever
accepted a constant or a type name:

  const
    mlab = 4;
    mlog = 12;
  type
    TRanges = record
      iu: array[mlab + 1..mlog] of Integer;
    end;

failed with 'SquareClose' expected found '+'. OrdinalType decided on a single
token of lookahead: an identifier followed by anything but '(' or '..' was a
type name, so it read the bound as the type `mlab`, returned, and left
ArrayBounds looking for the ']' it found a '+' at.

The upper bound already worked, which is what makes the gap easy to miss.
`array[mlab..mlog + 1]` sends the first bound to ConstantExpression on the
'..' lookahead, and OrdinalType's own trailing '..' branch parses the rest as
an expression. Only the first bound of a subrange went down the type-name
path.

The lookahead now also routes the operators SimpleExpression and Term accept
to ConstantExpression. Every one of those tokens is a parse error in this
position today, so no input that parses now takes a different path: an
identifier in an OrdinalType is followed by ']', ',', '..', 'of' or ';', never
by an operator. Set types and variant record tag types go through the same
procedure and gain the same forms.

A single token is enough here and a full ahead-parse would be worse. Coming
from the identifier, SimpleType's `AheadParse.NextToken; AheadParse.Simple-
Expression` idiom would meet the ']' of the common `array[TIndex] of Byte` and
hand it to Factor as a set constructor.

Parenthesized bounds are deliberately left alone. `array[(mlab + 1)..mlog]`
goes to EnumeratedType, and dcc32 reads it the same way - it reports
"Identifier redeclared: 'mlab'" - so the parser already agrees with the
compiler.

Test/Snippets/arrayboundexpression.pas covers the first bound, both bounds,
two dimensions, `*`, `-` and `shl`, and a set of a computed subrange; dcc32
compiles it clean. Suite: 45 tests, 44 passing, with the pre-existing
Serialization.BinaryRoundTrip failure unchanged (line_seq holds a pointer
value that does not survive a round trip).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@partouf partouf closed this Aug 31, 2026
@partouf
partouf deleted the fix/const-var-endline branch August 31, 2026 15:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant