deduplicate_entities's cross-file exact-match pass (in build.py, gated to concept-type nodes sharing a normalized label) merges two nodes for the same real-world entity into one survivor. The survivor is chosen by dedup.py's _pick_winner():
def _pick_winner(nodes: list[dict]) -> dict:
"""Pick the canonical survivor: prefer no chunk suffix, then shorter ID."""
def _score(n): return (1 if _CHUNK_SUFFIX.search(n["id"]) else 0, len(n["id"]))
return min(nodes, key=_score)This scores purely on ID shape/length — it has no regard for which node actually carries content (attributes, confidence, _merged_from history, etc.). In practice, a dedicated page for an entity tends to sit in a nested category folder, giving it a longer node ID (e.g. topics/networking/widget-x.md), while a passing mention of that same entity on an unrelated, flat summary page mints a short, shallow node (e.g. sources/notes.md) purely because it has one fewer path segment — regardless of how the two filenames themselves compare in length. Since the shorter ID always wins, the established, enriched node is discarded and replaced by the shallow one — every time this pattern occurs.
Repro:
- Create
topics/networking/widget-x.md, a dedicated page for an entity "Widget X" that produces a rich node (multiple attributes, high confidence, etc.).
- Extract it —
Widget X's node now has real content.
- Create
sources/notes.md, an unrelated, top-level page that only mentions "Widget X" in passing (one line, no real detail).
- Extract it. The exact-match cross-file dedup pass fires (same normalized label, both
concept type), and — because sources/notes.md sits one directory level shallower, making its node ID shorter regardless of the actual filename — the shallow node from step 3 wins. Diff graph.json before/after: Widget X's attributes (and any _merged_from history) from step 1 are gone, replaced by the shallow stub.
Note this is pure content loss, not structural corruption — edges pointing at the old survivor are correctly rewired first (confirmed via graphify diagnose / the health check: zero dangling edges).
Suggested fix: score candidates by content richness first (e.g. number of populated fields, presence of _merged_from/attributes/confidence) and fall back to ID length only as a tiebreaker among equally-rich candidates, rather than using ID length as the primary signal.
deduplicate_entities's cross-file exact-match pass (inbuild.py, gated toconcept-type nodes sharing a normalized label) merges two nodes for the same real-world entity into one survivor. The survivor is chosen bydedup.py's_pick_winner():This scores purely on ID shape/length — it has no regard for which node actually carries content (
attributes,confidence,_merged_fromhistory, etc.). In practice, a dedicated page for an entity tends to sit in a nested category folder, giving it a longer node ID (e.g.topics/networking/widget-x.md), while a passing mention of that same entity on an unrelated, flat summary page mints a short, shallow node (e.g.sources/notes.md) purely because it has one fewer path segment — regardless of how the two filenames themselves compare in length. Since the shorter ID always wins, the established, enriched node is discarded and replaced by the shallow one — every time this pattern occurs.Repro:
topics/networking/widget-x.md, a dedicated page for an entity "Widget X" that produces a rich node (multiple attributes, high confidence, etc.).Widget X's node now has real content.sources/notes.md, an unrelated, top-level page that only mentions "Widget X" in passing (one line, no real detail).concepttype), and — becausesources/notes.mdsits one directory level shallower, making its node ID shorter regardless of the actual filename — the shallow node from step 3 wins. Diffgraph.jsonbefore/after:Widget X'sattributes(and any_merged_fromhistory) from step 1 are gone, replaced by the shallow stub.Note this is pure content loss, not structural corruption — edges pointing at the old survivor are correctly rewired first (confirmed via
graphify diagnose/ the health check: zero dangling edges).Suggested fix: score candidates by content richness first (e.g. number of populated fields, presence of
_merged_from/attributes/confidence) and fall back to ID length only as a tiebreaker among equally-rich candidates, rather than using ID length as the primary signal.