Fix shell Mode-D quote/heredoc shielding; validate all shell tri-comparison shapes by squid-protocol · Pull Request #2406 · squid-protocol/gitgalaxy

squid-protocol · 2026-08-29T00:38:48Z

What

tri-comparison-ledger-sweep for shell — investigate every GitGalaxy-vs-tree-sitter-vs-ctags
disagreement shape for shell, fix GitGalaxy where it's the one that's wrong, and land a real
verdict on every shape.

Engine fixes (`gitgalaxy/core/detector.py`, `_apply_literal_shield`)

Two bugs were desyncing _slice_by_keywords' Mode-D depth stack so badly that one mishandled
token swallowed the rest of the file into a single <name>_[Truncated] satellite:

-- and // were in shell's comment-marker set. In shell -- is the end-of-options
delimiter (alert --stop …, git checkout -- file) and // has no comment meaning — both
appear constantly in ordinary commands. --flag "… got eaten as a "comment", orphaning the
opening quote of a real (often multi-line) string and corrupting every subsequent quote/keyword
count. One alert --stop "…" in haiku/HardwareChecker.sh cost 10 top-level functions.
Fixed: shell/bash comment marker is # only (mirrors the existing matlab special-case).
Quoted heredoc delimiters were destroyed before the heredoc state-machine saw them.
sed 's/^X//' << 'SHAR_EOF' → the quote pass rewrote 'SHAR_EOF' to '' first, so step 2's
heredoc detector saw << '' and never registered the heredoc — the whole body (C source, a
shar archive) stayed live. Fixed: match a real heredoc opener before the quote alternatives
and return it verbatim (preserve_newlines).

Effect: the shell/function/existence/agree[ctags,tree_sitter]_vs[gitgalaxy] recall-gap shape
drops 16 → 4; the agree[gitgalaxy]_vs[ctags,tree_sitter] over-detection shape 7 → 3.

Both golden masters re-blessed (shell func counts up + global spatial-map ripple);
crucible_check.py PASS both modes; shell/ruby/perl/core extraction + strict suites green;
ruff / mypy / dead-key baselines clean.

Ledger — all 5 shell shapes now `status: validated` with real verdicts

The prior sweep's scratch script had written its findings into a non-schema notes field and left
verdict null on 4 of 5 shapes, and set a malformed credit_tools: ["gitgalaxy"] on an
agree[none] shape (no defined effect — printed a warning every chart run). All corrected:

shape	verdict summary
`agree[gitgalaxy]_vs[ctags,tree_sitter]` (3)	all 3 are `_[Truncated]` desync artifacts, zero real GG-only funcs — residual triggers filed as #2405
`agree[ctags,tree_sitter]_vs[gitgalaxy]` (4)	`zgrep` is the deliberate `scope_depth==0` limit; `t`/`check`/`__complete_job_spec` are #2405
`agree[ctags,gitgalaxy]_vs[tree_sitter]` (11)	GG+ctags right; tree-sitter-bash cascades to zero functions (see below)
`agree[ctags]_vs[gitgalaxy,tree_sitter]` (44)	ctags mis-tags scalar (not "array") assignments as functions; plus a genuine ctags-only recall win on `sub`-style helpers
`args/agree[none]_vs[gitgalaxy,tree_sitter]` (10)	shell args = positional-`$N` proxy (Claim 1); tree-sitter's flat `1` is the artifact; malformed credit cleared

Not GitGalaxy bugs — documented

docs/why_gitgalaxy_beats_ast_here.md — Claim 3 gains a fourth confirmed instance: a ;
inside a ${var:+word} / ${var:-word} parameter expansion makes tree-sitter-bash set
has_error and emit zero function_definition nodes for the whole file. ${codes:+$codes;}
in moby/check-config.sh costs it 11 ordinary top-level functions GitGalaxy and ctags both find.
tests/tools/ctags_reader.py — shell KIND-MAP note: ctags' Sh parser tags bare scalar
assignments (GREP_OPTS=, FILTERED_ENV=) as f-kind functions.

Follow-up

Shell Mode-D scope desync: bare keyword in an argument (and SerenityOS brace syntax) swallows the rest of the file #2405 — residual shell Mode-D desync: a bare shell keyword used as a plain word in a command
argument (echo … limit **for** $routine) is counted as a scope opener; SerenityOS Shell brace
syntax; plus a suggested blast-radius containment (re-scan an EOF-open span at depth 0).

Housekeeping

.gitignore: backstop /language-crucible*/ so a stray in-repo corpus clone can't poison the
golden masters (see .claude/rules/golden-master-guidelines.md).

🤖 Generated with Claude Code

…arison shapes Shell tri-comparison-ledger-sweep. Two engine bugs in `_apply_literal_shield` were desyncing `_slice_by_keywords`' depth stack so hard that one bad token swallowed the rest of the file into a single `<name>_[Truncated]` satellite: 1. `--` and `//` were in shell's comment-marker set. In shell `--` is the end-of-options delimiter (`alert --stop ...`) and `//` has no comment meaning; both are everywhere in ordinary commands. ` --flag "..."` got eaten as a "comment", orphaning the opening quote of a multi-line string and corrupting every subsequent quote/keyword count. One `alert --stop "..."` in haiku/HardwareChecker.sh cost 10 top-level functions. 2. A quoted heredoc delimiter (`sed 's/^X//' << 'SHAR_EOF'`) was rewritten to `<< ''` by the quote pass before the heredoc state-machine could register it, leaving the whole heredoc body (C source, a shar archive) live to corrupt scope. Fixed by matching a real heredoc opener before the quote alternatives and returning it verbatim. Shell func recall against tree-sitter+ctags: the `agree[ctags,tree_sitter]_vs[gitgalaxy]` shape drops 16 -> 4, and `agree[gitgalaxy]_vs[ctags,tree_sitter]` (over-detection) 7 -> 3. Both golden masters re-blessed (shell func counts up + global spatial-map ripple); crucible_check PASS both modes. Ledger: all 5 shell shapes now `status: validated` with real `verdict` text (the prior sweep's scratch script had written the investigation into a non-schema `notes` field and left `verdict` null). Cleared a malformed `credit_tools: ["gitgalaxy"]` on the `args/agree[none]` shape. Corrected the `agree[ctags]` verdict ("array" -> "scalar" assignments). Residual shell misses filed as #2405 (bare keyword in a command argument; SerenityOS brace syntax). Not GitGalaxy bugs, documented instead: - why_gitgalaxy_beats_ast_here.md Claim 3 gains a fourth instance: a `;` inside `${var:+word;}` cascades tree-sitter-bash into a zero-function parse. - ctags_reader.py shell KIND-MAP note: ctags' Sh parser tags scalar assignments (`GREP_OPTS=`) as functions. .gitignore: backstop `/language-crucible*/` so a stray in-repo corpus clone can't poison the golden masters. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

github-actions · 2026-08-29T00:41:00Z

🐦‍⬛ Muninn Security Scan

✅ No security issues found.

🐦‍⬛ Powered by Muninn · Skald Lab

…2422) #2415 (aperture infra_path_pattern left-boundary fix) newly included ~20 real source files across the corpus, but tree-sitter-accuracy-audit.yml does not trigger on aperture.py, so it merged without regenerating the affected baselines. `tree_sitter_accuracy_audit.py --all --ci` has been red on main for every detector.py/prism.py/language_standards.py PR since. Every number here is `measure()`'s own output, not hand-edited. No engine code changes. - shell: 31.6% -> 99.1% func recall, 68.3% -> 97.0% precision. This is #2401 / #2406's shell Mode-D fixes finally reflected in the baseline (found_functions 140 -> 449, extra 65 -> 14); the +6 files / +10 real functions from #2415 are a small part of it. - powershell: func recall/precision unchanged (100%/100%); +3 files, args_exact_match 234 -> 252 of 257. - c 99.0 -> 99.7% · lua 78.2 -> 78.7% · tcl 98.6 -> 100.0% · ruby args_exact_match 115 -> 121 -- all `found_functions` improvements from the +1 file each, already passing --ci ("improved on ..."), locked in here so the summary table stays honest. - html: N/A -> 0.0% func precision. `cpython_jinja/layout.html` (a Jinja template, new via #2415) has a `<style>` block; html's func_start intentionally anchors on `<style>`/`<script>` element boundaries, which tree-sitter-html models as inert raw_text, so it shows as extra_functions 0 -> 1. Baseline written from measure() output with the regression gate overridden -- same maintainer-override shape as commit 9011de4 (lua class precision 0.0%). Whether html func-precision should be forced N/A like css/html class-precision already is: filed as #2421. Summary table in language_standards.py's docstring regenerated to match. Claude-Session: https://claude.ai/code/session_019Zm1uVBpVEZJ9SB7bWFR8S Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>

squid-protocol merged commit 1b23569 into main Aug 29, 2026
28 checks passed

squid-protocol deleted the fix-shell-mode-d-shielding-2405 branch August 29, 2026 00:38

squid-protocol mentioned this pull request Aug 29, 2026

Shell Mode-D scope desync: bare keyword in an argument (and SerenityOS brace syntax) swallows the rest of the file #2405

Closed

squid-protocol mentioned this pull request Aug 29, 2026

chore: regenerate tree-sitter-accuracy baselines orphaned by #2415 #2422

Merged

Navigation Menu

Fix shell Mode-D quote/heredoc shielding; validate all shell tri-comparison shapes - #2406

Fix shell Mode-D quote/heredoc shielding; validate all shell tri-comparison shapes#2406
squid-protocol merged 1 commit into
mainfrom
fix-shell-mode-d-shielding-2405

squid-protocol commented Aug 29, 2026

Uh oh!

Uh oh!

github-actions Bot commented Aug 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

squid-protocol commented Aug 29, 2026

What

Engine fixes (gitgalaxy/core/detector.py, _apply_literal_shield)

Ledger — all 5 shell shapes now status: validated with real verdicts

Not GitGalaxy bugs — documented

Follow-up

Housekeeping

Uh oh!

Uh oh!

github-actions Bot commented Aug 29, 2026

🐦‍⬛ Muninn Security Scan

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Engine fixes (`gitgalaxy/core/detector.py`, `_apply_literal_shield`)

Ledger — all 5 shell shapes now `status: validated` with real verdicts