fix(query): emit the ranking it already computes — ANSWER block, camelCase sub-words, test demotion, honest truncation - #3417
Conversation
`_score_query` ranked every node against the question and only `_pick_seeds` read the result. The renderer ordered by graph topology and printed no score, so every line carried identical apparent weight and a reader could not tell the node that answered the question from the neighbourhood it was found in. Measured on a 51k-node TypeScript graph, an eight-query session fell back to grep every time. An ANSWER block leads the output: the best five nodes with a relative score, the file and line, one line saying which query terms matched at which tier, and the source line itself — the reader's next move was always to open the file. It is exempt from the token budget and its nodes are pinned, so the answer survives any `--budget`. Every NODE line also carries `rel=`. Two scoring defects made the ranking not merely invisible but inverted. Identifiers never split on camelCase. `_search_tokens` left `handleTeamConfigWrite` as one token, so a question saying "team config write" could only reach it through the substring tier — worth 1/1000th of an exact match. Measured: that node scored 15 while a bare `worker` variable matching one of the six query terms scored 126. A sub-word tier now sits between prefix and substring and counts toward term coverage, and a trailing-plural fold applied to both sides lets "writes" meet `Write`. Test, fixture, generated, vendored, config and archived paths are weighted down rather than filtered out, so a `*.test.tsx` can still answer "which test covers X" but cannot outrank the component it exercises. With equal labels and equal tiers the winner used to be decided by node id alone. Truncation now says what was actually lost. The old banner always read "The answer may be among the N cut nodes"; when every cut node scored zero against the query that was false, and it cost the reader a `--budget` retry that could only return more unmatched context. A query whose wording shares nothing with the corpus says so. A lexical graph cannot find a node with no word in common with the question, but it can decline to present five near-misses as an answer; naming the terms that matched nothing tells the caller which way the vocabulary gap runs. Two nags, both measured on the same graph. The pre-Graphify-Labs#1504 rebuild nudge fired on every query for any older graph; it now fires only when a legacy id is genuinely claimed by more than one file, which is the harm the new scheme prevents — on the measured graph 591 file nodes carried legacy ids and none collided. The community-relabel message demanded `graphify label` for work the same function had already done deterministically one line above; it now reports the rename and says the LLM pass is optional. The renderer's sort key is `(hop, -relevance, -degree, id)` — the key PR Graphify-Labs#3284 proposes, adopted deliberately so the two changes compose rather than compete. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CTFmYpnadn56gaEY9wfk1r
There was a problem hiding this comment.
Graphify reviewed this change.
Worth a look — the grounded gate found no coupling regressions or blocking issues, but 5 advisory finding(s) below merit a look before merge.
Formal verification. 2 change(s) tested, no difference found (not proven).
Graphify review — findings
Replaces the "graph predates #1504" rebuild nudge with legacy_id_collisions, so both the CLI and _load_graph warn only when a legacy node ID is actually claimed by more than one file, and report the count of ambiguous IDs instead of firing on every query of any older graph. Rewords the community-relabel message to state that hub-derived renames already happened and that the graphify label LLM pass is optional, rather than demanding it as a required repair. Adds identifier sub-word matching to the query scorer via _subword_tokens (camelCase/PascalCase/SCREAMING_SNAKE splitting) and a plural-folding _fold_plural, scored at a new _SUBWORD_MATCH_BONUS tier between prefix and substring that counts toward term coverage, plus relevance shaping so a test file no longer outranks the implementation it exercises.
Worth a look
- _relevance_weight computed but never applied to score —
graphify/serve.py:712· Escalate · high- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
- Query answer snippets can read arbitrary local files via graph-controlled source_file —
graphify/serve.py:890· Escalate · high- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
- Low-confidence miss detection uses shown subset, not full ranking, for coverage —
graphify/serve.py· Escalate · medium- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
- Path traversal via unvalidated source_file in snippet reader —
graphify/serve.py· Escalate · medium- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
- Relevance weight is not applied to per-term singleton scores —
graphify/serve.py:800· Escalate · medium- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
Analysis details — impact, health, verification
Impact & health
Graphify review
Impact — 1684 functions depend on the 396 functions this change touches.
Health — this change adds coupling hotspots:
- new:
_rebuild_code()— 115 callers, 51 callees - new:
build_from_json()— 199 callers, 19 callees - new:
build_merge()— 62 callers, 13 callees - new:
to_obsidian()— 38 callers, 14 callees - new:
_query_graph_text()— 37 callers, 13 callees - new:
extract_files_direct()— 17 callers, 20 callees - new:
_call_claude_cli()— 33 callers, 9 callees - new:
to_wiki()— 41 callers, 7 callees - …and 48 more — each is listed as a finding
Verification — 1684 functions in the blast radius were not formally verified this run (proofs are advisory here).
Gate & verification
graphify gate
PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.
Advisory (not blocking):
- verification_scope: 1504 function(s) in the blast radius were not formally verified this run
Formal verification
Could not verify: Could not verify dispatch\_command.
The verifier did not have enough to check dispatch\_command, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: not verifiable: all 23 sampled inputs raised on both versions — the function never executed, so 'no divergence' would be vacuous (mostly SystemExit — names the real obstacle, not a sampling gap)
Could not verify: Could not verify \_load\_graph.
The verifier did not have enough to check \_load\_graph, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: not verifiable: all 23 sampled inputs raised on both versions — the function never executed, so 'no divergence' would be vacuous (mostly SystemExit — names the real obstacle, not a sampling gap)
No difference found (not proven): No behavior difference found in \_query\_graph\_text (not a proof).
The verifier ran both versions of \_query\_graph\_text on many inputs and saw identical behavior every time. Strong evidence the change is safe, but evidence, not a proof.
Guarantee: Empirical: differential testing (both versions run on many generated inputs). A divergence on an untested input remains possible, so this is 'no counterexample found', not 'proven equivalent'.
Note: An input the sampler did not try could still differ.
Could not verify: Could not verify \_score\_query.
The verifier did not have enough to check \_score\_query, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: not verifiable: all 45 sampled inputs raised on both versions — the function never executed, so 'no divergence' would be vacuous (mostly TypeError — names the real obstacle, not a sampling gap)
No difference found (not proven): No behavior difference found in \_subgraph\_to\_text (not a proof).
The verifier ran both versions of \_subgraph\_to\_text on many inputs and saw identical behavior every time. Strong evidence the change is safe, but evidence, not a proof.
Guarantee: Empirical: differential testing (both versions run on many generated inputs). A divergence on an untested input remains possible, so this is 'no counterexample found', not 'proven equivalent'.
Note: An input the sampler did not try could still differ.
Could not verify: Could not verify \_rebuild\_code.
The verifier did not have enough to check \_rebuild\_code, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: parameter `watch_path` is annotated `Path` — outside the synthesizable primitive/collection set
· 2 grounded finding(s) anchored inline below; 54 more finding(s) on lines outside this diff (see the check run).
| return False | ||
|
|
||
|
|
||
| def legacy_id_collisions(nodes: list, root: str | Path | None = None) -> int: |
There was a problem hiding this comment.
legacy_id_collisions()
fans out to 7 callees (efferent coupling); 8 callers depend on it (afferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| return "" | ||
|
|
||
|
|
||
| def _answer_block( |
There was a problem hiding this comment.
_answer_block()
high coupling complexity (Ca·Ce = 12).
Grounded coupling-delta finding (deterministic), not an LLM guess.
Two follow-ups from re-reading the original report against what landed. **Relevance is now the primary sort key**, `(-relevance, hop, -degree, id)`, with hop demoted to the tie-break that keeps a seed's neighbourhood together among nodes the query cannot separate. The first commit made hop primary to match the key PR Graphify-Labs#3284 proposes; that put a node the reader is looking for below a nearer node that matched nothing, which is the complaint this whole change exists to answer. #BUG2's guarantee is carried by `pinned`/`seeds`, not by hop order, so nothing depends on hop being first. **A raised budget no longer buys unmatched context.** The renderer emitted every traversed node, so `--budget` bought filler: measured at `--budget 20000` on a 51k-node graph, one query rendered 122 matched nodes and **266** with `rel=0`. Nodes matching none of the query's terms are now capped at 15 after the matched set — the same query renders 128 matched and 15 context — and the trim is announced rather than silent. Edges render only between surviving nodes, so no dangling half-edge appears. The cap applies only when there is a ranking to trim by, leaving `path`/`explain` and every non-query caller untouched. **`rel=0` now means exactly one thing: this node matched none of your terms.** A weak-but-real match rounded down to zero, which made it indistinguishable from pure traversal context and contradicted the trimming note. Real matches floor at 1. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CTFmYpnadn56gaEY9wfk1r
There was a problem hiding this comment.
Graphify reviewed this change.
Worth a look — the grounded gate found no coupling regressions or blocking issues, but 5 advisory finding(s) below merit a look before merge.
Formal verification. 2 change(s) tested, no difference found (not proven).
Graphify review — findings
Reworks the "pre-#1504 node ID" nudge in both the CLI and serve to fire only when legacy_id_collisions finds an ID actually claimed by more than one file, replacing the old warning that triggered on any older graph on every query, and reports the collision count with wording about answering for the wrong file. Rewrites the post-reclustering message so it states that communities were renamed from their hubs and treats graphify label as optional rather than a required repair. Adds identifier-aware search matching via _subword_tokens (camelCase/PascalCase/SCREAMING_SNAKE splitting) scored at a new _SUBWORD_MATCH_BONUS tier between prefix and substring, plus _fold_plural so prose queries and singular identifiers agree on number, and shapes ranking so a support/test file no longer outranks the implementation it exercises.
Worth a look
- Query answer snippets can read files outside the graph root —
graphify/serve.py· Escalate · high- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
- Relevance weight computed but never applied to final score —
graphify/serve.py:713· Escalate · high- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
- Legacy collision count uses raw source_file spellings —
graphify/build.py:803· Escalate · medium- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
- Relevance weight is computed but not applied in query scoring —
graphify/serve.py:800· Escalate · medium- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
- Per-term singleton multiplies score by IDF weight twice —
graphify/serve.py:803· Escalate · medium- agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review
Analysis details — impact, health, verification
Impact & health
Graphify review
Impact — 1694 functions depend on the 406 functions this change touches.
Health — this change adds coupling hotspots:
- new:
_rebuild_code()— 115 callers, 51 callees - new:
build_from_json()— 199 callers, 19 callees - new:
build_merge()— 62 callers, 13 callees - new:
_query_graph_text()— 42 callers, 13 callees - new:
to_obsidian()— 38 callers, 14 callees - new:
extract_files_direct()— 17 callers, 20 callees - new:
_call_claude_cli()— 33 callers, 9 callees - new:
to_wiki()— 41 callers, 7 callees - …and 48 more — each is listed as a finding
Verification — 1694 functions in the blast radius were not formally verified this run (proofs are advisory here).
Gate & verification
graphify gate
PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.
Advisory (not blocking):
- verification_scope: 1514 function(s) in the blast radius were not formally verified this run
Formal verification
Could not verify: Could not verify dispatch\_command.
The verifier did not have enough to check dispatch\_command, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: not verifiable: all 23 sampled inputs raised on both versions — the function never executed, so 'no divergence' would be vacuous (mostly SystemExit — names the real obstacle, not a sampling gap)
Could not verify: Could not verify \_load\_graph.
The verifier did not have enough to check \_load\_graph, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: not verifiable: all 23 sampled inputs raised on both versions — the function never executed, so 'no divergence' would be vacuous (mostly SystemExit — names the real obstacle, not a sampling gap)
No difference found (not proven): No behavior difference found in \_query\_graph\_text (not a proof).
The verifier ran both versions of \_query\_graph\_text on many inputs and saw identical behavior every time. Strong evidence the change is safe, but evidence, not a proof.
Guarantee: Empirical: differential testing (both versions run on many generated inputs). A divergence on an untested input remains possible, so this is 'no counterexample found', not 'proven equivalent'.
Note: An input the sampler did not try could still differ.
Could not verify: Could not verify \_score\_query.
The verifier did not have enough to check \_score\_query, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: not verifiable: all 45 sampled inputs raised on both versions — the function never executed, so 'no divergence' would be vacuous (mostly TypeError — names the real obstacle, not a sampling gap)
No difference found (not proven): No behavior difference found in \_subgraph\_to\_text (not a proof).
The verifier ran both versions of \_subgraph\_to\_text on many inputs and saw identical behavior every time. Strong evidence the change is safe, but evidence, not a proof.
Guarantee: Empirical: differential testing (both versions run on many generated inputs). A divergence on an untested input remains possible, so this is 'no counterexample found', not 'proven equivalent'.
Note: An input the sampler did not try could still differ.
Could not verify: Could not verify \_rebuild\_code.
The verifier did not have enough to check \_rebuild\_code, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: parameter `watch_path` is annotated `Path` — outside the synthesizable primitive/collection set
· 2 grounded finding(s) anchored inline below; 54 more finding(s) on lines outside this diff (see the check run).
| return False | ||
|
|
||
|
|
||
| def legacy_id_collisions(nodes: list, root: str | Path | None = None) -> int: |
There was a problem hiding this comment.
legacy_id_collisions()
fans out to 7 callees (efferent coupling); 8 callers depend on it (afferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| return "" | ||
|
|
||
|
|
||
| def _answer_block( |
There was a problem hiding this comment.
_answer_block()
high coupling complexity (Ca·Ce = 12).
Grounded coupling-delta finding (deterministic), not an LLM guess.
Summary
queryscores every node against the question, uses the ranking to pick seeds, and then discards it. The renderer orders by graph topology (seeds, hop distance, degree) and prints no score, so every line carries identical apparent weight — the ranking is invisible, not absent, and the noise floor is indistinguishable from the signal.This adds an ANSWER block, puts a relevance figure on every node line, fixes two scoring defects that make the ranking actively inverted on natural-language queries, and stops two warnings that fire when nothing is wrong.
Measured throughout on a real 51,084-node / 69,383-edge TypeScript monorepo graph.
Problem
returns 49 flat
NODElines. The right answer is at position 2; position 1 is a test file, positions 3-6 are aproject.json, an unrelated types file, and an unrelated component. Nothing says which line is the answer.Two defects make it worse than unranked:
1. Identifiers never split on camelCase.
_search_tokenssplits on punctuation and_, sohandleTeamConfigWritestays ONE token. A question saying "team config write" can only reach it through the substring tier — worth 1/1000th of an exact match. For"which worker route validates team config writes":worker(a local variable in a service worker)handleTeamConfigWrite()(the answer)An 8× inversion. The coverage penalty for matching 1-of-6 is 36×, but the tier gap is 1000×, so a lone exact hit on a common word always buries a node that contains most of the question.
2. No notion of file kind.
LanguagesCard.test.tsxholds a bareLANGUAGESconstant that exact-matches a query term, while the component it exercises isSectionLanguagesCard()and only prefix-matches. With equal labels and equal tiers the winner is decided by node id alone.Fix
ANSWER block. The top five nodes with a relative score,
file:line, one line naming which query terms matched at which tier, and the source line itself — the reader's next move was always to open the file. Exempt from the token budget and pinned in the node list, so the answer survives any--budget.Sub-word tier.
_subword_tokenssplits camelCase, PascalCase and SCREAMING_SNAKE (handleTeamConfigWrite→handle, team, config, write;HTTPServer→http, server). A new tier between prefix and substring, counting toward term coverage. A trailing-plural fold applied to both sides lets "writes" meetWrite— a fold, not a stemmer, sostatus→statuis harmless because the label folds identically.File-kind weighting. Multiplicative, not a filter, so a test can still answer "which test covers X" but cannot outrank the source it exercises. Test classification reuses
paths._is_test_pathrather than adding a third definition.Honest truncation. The old banner always read "The answer may be among the N cut nodes". With the list ordered by relevance the cut falls on the tail, so when every cut node scored zero that sentence was false, and it cost the reader a
--budgetretry that could only return more unmatched context. It now reports how many cut nodes actually matched, and stays quiet when none did.Low-confidence signal. A lexical graph cannot find a node with no word in common with the question — but it can decline to present five near-misses as an answer. Naming the terms that matched nothing tells the caller which way the vocabulary gap runs.
Two nags. The pre-#1504 rebuild nudge fired on every query for any older graph. It now fires only when a legacy id is genuinely claimed by more than one file, which is the harm the new scheme prevents — on the measured graph 591 file nodes carried legacy ids and zero of them collided, so it was recommending a full re-extraction that would change nothing. The community-relabel message demanded
graphify labelfor work the same function had already done deterministically one line above; it now reports the rename and says the LLM pass is optional.Results
Nine queries against the 51k-node graph — the first three are natural-language, the last six are identifier lookups included as regression guards. Position of the correct file:
.test.tsx)langLadderhandleTeamConfigWriteSectionLanguagesCardgetTenantPrimaryLanguageuseTeamEditLangsmatchWriteRouteA raised budget no longer buys filler. The renderer emitted every traversed node, so
--budgetbought noise rather than answer — the original report's "more noise, same problem". Nodes matching none of the query's terms are capped at 15 after the matched set, and the trim is announced:--budgeton the same queryrel>0)rel=0)rel=0now means exactly one thing — this node matched none of your terms. A weak-but-real match used to round down to zero, which made it indistinguishable from traversal context and contradicted the trimming note.The one query that still fails is worth stating plainly. All four nodes of that file match zero query terms — "chrome" appears nowhere in it, and the bridging prose lives in a doc comment (
rationaleis populated on 107 of 51,084 nodes here). No lexical ranking can bridge that; it needs semantic retrieval. What changed is that the output now saysLOW CONFIDENCE, names the terms nothing matched, and tells the caller not to trust the top line.Tests
tests/test_query_ranking_is_visible.py, 62 tests: file-kind weighting including thelatest/contest/distributionnon-matches, sub-word splitting, plural folding, the measuredworkervshandleTeamConfigWriteinversion, identifier lookup still winning outright, the ANSWER block's presence/ordering/score/reason, the reason never claiming a tier the score did not credit, low confidence firing and staying quiet, both truncation branches, the answer surviving the budget, snippet reads including missing/binary/megaline cases, and the legacy-id collision gate.I watched the old behaviour hold before changing it: six of seven baseline assertions confirmed on pristine
v8and broken after. The seventh was a weak test of mine — it passed on baseline too — so I replaced it with one that reproduces the real inversion._subgraph_to_textwith no relevance map is byte-identical to today, sopath/explainand every non-query caller are unaffected.Full suite: 5,153 passed. The 24 failures (
test_ollama_retry_cap.py,test_skillgen.py,test_terraform.py) reproduce identically on pristinev8in this environment — I diffed the two failure sets and they are equal — and are unrelated to this change.Relation to existing work
#3284 proposes threading the same ranking into
_subgraph_to_text, with the key(hop, -score, -degree, id). I first adopted that key exactly so the two would not compete, then changed my mind and reverted to relevance-primary —(-relevance, hop, -degree, id)— in the second commit, and have corrected the note I left on that PR.The reason: with hop primary, a node the reader is looking for still renders below a nearer node that matched nothing, which is the complaint this change exists to answer. #BUG2's guarantee is carried by
pinned/seedsrather than by hop order, so nothing depends on hop being first; it survives as the tie-break among nodes the query cannot separate.That is a genuine disagreement about one sort key and it is the maintainer's call — #3284's diagnosis is right either way, and if you prefer hop-primary I will take that key back and keep the rest. The camelCase measurement above is offered as evidence for that PR regardless.
#2384 ("local variables win seeds") is the same territory from the seed side. The file-kind weighting here is a partial answer to its proposal 1 — a weight on the path, not yet a prior on node kind, which would still help.
#3245 adds
descriptionto the NODE line; it will conflict textually with therel=suffix on the same f-string, but the two compose — happy to rebase whichever lands second.Deliberately not attempted: community-diversity caps (#2384 proposal 2), confidence/weight-aware ranking (#2384 proposal 3), or node-kind priors.
🤖 Generated with Claude Code
https://claude.ai/code/session_01CTFmYpnadn56gaEY9wfk1r