fix(#1954): split on real newlines in _strip_single_line_comments, not str.splitlines() - #2385
Merged
Merged
Conversation
…t str.splitlines() _strip_single_line_comments (the comment stripper for the whole "line_exclusive" family -- ~20 languages) split its input with str.splitlines(), which also breaks on vertical tab, form feed, the file/group/record separators, NEL and U+2028/U+2029. Any of those inside a physical line -- most often inside a comment -- became a literal "\n" when the code/comment streams were rejoined with "\n".join(...), so every downstream newline-counting line-number calculation (detector.py's _slice_by_labels line tracker, etc.) drifted by the cumulative count of such characters -- a monotonic drift that never resets. Confirmed on language-crucible's assembly/cosmopolitan/ape.S (real Cosmopolitan libc, which uses form feed as a page-break idiom in its comments): every function's recorded start_line was too high by exactly the number of form feeds before it (kernel: 1752 vs the real 1743, +9). Fix: normalise CR/CRLF the way splitlines() did, then split on "\n" only. Form feed / vertical tab / the other splitlines()-special characters now stay inline as ordinary content, matching how ctags and tree-sitter read the file. Closes #1954. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
crucible_check.py --update: all changes confined to assembly/cosmopolitan/ape.S (the only corpus file with form feeds) and its assembly-ecosystem aggregates. Every ape.S function's start_line/ end_line drops by its preceding form-feed count, now matching ctags exactly (e.g. kernel 1752 -> 1743, dsknfo 348 -> 346, realmodeloader 1290 -> 1283). Structural Tab Indentations 1029 -> 1021 and small downstream impact/mass recalcs follow from the corrected line structure. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
squid-protocol
added
core-engine
Contributor
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1954.
Problem
_strip_single_line_comments— the comment stripper for the entire"line_exclusive"lexical family (~20 languages: assembly, python, ruby,shell, perl, makefile, yaml, …) — split its input with
for line in text.splitlines():and rejoined with"\n".join(...).str.splitlines()breaks on far more than\n/\r\n: vertical tab,form feed,
\x1c–\x1e, NEL (\x85), U+2028, U+2029. Any of thoseinside what a human reads as one physical line — most often inside a
comment — was silently converted to a literal
\non rejoin. Everydownstream line-number computation that counts
\nin the resultingstream (
detector.py's_slice_by_labelsline tracker, etc.) thendrifted by the cumulative count of such characters — a monotonic drift
that never resets.
Confirmed evidence
language-crucible/data/assembly/cosmopolitan/ape.S(real Cosmopolitanlibc, which uses form feed as a deliberate page-break idiom in its
comments — 9 of them) reproduced this exactly. Every function's recorded
start_linewas too high by precisely the number of form feeds beforeit:
Fix
Normalise
\r\n/\r→\n(the one useful thingsplitlines()wasdoing), then
split("\n"). Form feed / vertical tab / the othersplitlines()-special characters now stay inline as ordinary content,matching how ctags and tree-sitter read the file.
Verification
test_prism_issue_1954_form_feed_in_comment_preserves_line_count(python/shell/ruby with form feeds in comments + a CRLF round-trip
check). Confirmed fails without the fix, passes with it.
tests/core_engine/pass; ruff / mypy / dead-key / ast-accuracyclean.
crucible_check.py --update: all drift confined toape.Sand itsassembly-ecosystem aggregates — every function line number moves toward
the correct ctags value, no other corpus file touched. Both golden
masters re-blessed.
tri_comparison_chart.py --all --ci: all OK, no precision regressions.🤖 Generated with Claude Code