fix(dedup): pick the richer duplicate as survivor and keep the losers' fields (#3372)

[abhay-codes07]

Closes #3372.

The problem

_pick_winner chose the dedup survivor by ID shape alone — no chunk suffix, then shorter ID. A passing one-line mention of an entity on a shallow page mints a shorter node id than the entity's dedicated page in a nested category folder, so the shallow stub won every time the pattern occurred, and the losers are dropped wholesale in the component collapse — the established node's description, attributes, confidence and _merged_from history vanished from the graph. Reproduced on 0.9.56 with the issue's two-node shape: the sources/notes.md stub survives and every rich field is gone.

The change

Two complementary pieces, both in dedup.py:

• Richness decides the survivor. _content_richness counts populated fields beyond the identity/placement baseline (id, label, source_file, …), weighting attributes by entry count and _merged_from by history length — a node that already absorbed merges is the established record. _pick_winner scores (chunk-suffix, −richness, id length): the chunk-suffix rule still dominates (a _c3 fragment never wins, however rich), and ID length still breaks ties between equally-rich candidates, preserving the old deterministic ordering exactly where the old signal was actually doing its job. All 139 existing dedup tests pass unchanged.
• Losers' fields fold into the survivor. Even with the right winner, dropping the losers loses whatever only they carried. At component collapse the survivor is back-filled with each loser's fields via the same never-override _merge_missing_attributes the same-source collision path already uses (Default dedup=True discards a same-ID node's attributes instead of merging them — LLM semantic summaries are silently lost #2091), in deterministic unique_nodes order. Content loss becomes a union.

Tests

tests/test_dedup_survivor_richness.py — 6 tests: the issue's exact shape keeps the enriched node (id, attributes, merge history intact); a field only the loser carried is folded into the survivor without overriding anything; equally-bare candidates keep the shorter-id tiebreak; a chunk-suffixed id never beats a clean one regardless of richness; the richness measure ignores placement fields; and edges rewire to the rich survivor. With the fix reverted, 4 of 6 fail (the tiebreak and chunk-suffix controls rightly keep passing). The dedup suites are unchanged (139 passed); the full suite matches a fresh same-version v8 (0.9.56) baseline.

🤖 Generated with Claude Code

https://claude.ai/code/session_01JJbLfztSxm2tH5cJwBbe9q

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[graphify-labs]

Graphify reviewed this change.

Worth a look — the grounded gate found no coupling regressions or blocking issues, but 1 advisory finding(s) below merit a look before merge.

Formal verification. No changes could be formally verified in this run.

---

Graphify review — findings

Reworks dedup survivor selection in _pick_winner to rank candidates by content richness before falling back to ID length, so the established, field-rich node wins instead of whichever duplicate has the shortest ID; the chunk-suffix check still takes precedence, and ID length only breaks ties between equally-rich nodes. Adds _content_richness, which scores populated non-identity/placement fields and weights attributes and _merged_from by entry count. Back-fills the chosen survivor with any fields only its losers carried via _merge_missing_attributes (never-override, deterministic by unique_nodes order), substituting the enriched node into the returned list so previously-discarded loser data is preserved.

Worth a look

• _pick_winner survivor and _collision_rank survivor can disagree, causing edges to rewire to a dropped node — graphify/dedup.py:864 · Escalate · medium
  • agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review

Analysis details — impact, health, verification

Impact & health

Graphify review

Impact — 418 functions depend on the 71 functions this change touches.

Health — this change adds coupling hotspots:

• new: deduplicate_entities() — 75 callers, 22 callees
• new: build_merge() — 62 callers, 13 callees
• new: dispatch_command() — 2 callers, 124 callees
• new: build() — 40 callers, 4 callees
• new: _prune() — 9 callers, 3 callees
• new: _llm_tiebreak() — 1 callers, 10 callees
• new: test_poisoned_manifest_is_healed() — 0 callers, 6 callees

Verification — 418 functions in the blast radius were not formally verified this run (proofs are advisory here).

Gate & verification

graphify gate

PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.

Advisory (not blocking):

• verification_scope: 197 function(s) in the blast radius were not formally verified this run

Test selection

Test selection

15 of 260 test file(s) selected (6%) via static blast radius.

• tests/test_build.py — impact
• tests/test_build_merge_hyperedges_and_prune.py — impact
• tests/test_build_merge_shrink_guard.py — impact
• tests/test_carried_hyperedge_remap.py — impact
• tests/test_corrupt_graph_json.py — impact
• tests/test_cross_extension_reexport_self_cycle.py — impact
• tests/test_dedup.py — impact
• tests/test_dedup_remaps_hyperedges.py — impact
• tests/test_dedup_survivor_richness.py — impact, changed-test
• tests/test_global_graph.py — impact
• tests/test_go_qualified_resolution.py — impact
• tests/test_no_dedup_flag.py — impact
• tests/test_non_string_node_ids.py — impact
• tests/test_prune_sweeps_orphans.py — impact
• tests/test_unverified_semantic_shrink.py — impact

Selection is safe under the controlled-regression assumption; always-run tests + a periodic full run are the backstops. Advisory — it never changes the check verdict.

Formal verification

Could not verify: Could not verify deduplicate\_entities.

The verifier did not have enough to check deduplicate\_entities, so it is saying so rather than guessing. No false assurance is the whole point.

Guarantee: No guarantee either way, this is an honest abstention, not a pass.

Note: Reason: not verifiable: all 9 sampled inputs raised on both versions — the function never executed, so 'no divergence' would be vacuous (mostly TypeError — names the real obstacle, not a sampling gap)

Could not verify: Could not verify \_pick\_winner.

The verifier did not have enough to check \_pick\_winner, so it is saying so rather than guessing. No false assurance is the whole point.

Guarantee: No guarantee either way, this is an honest abstention, not a pass.

Note: Reason: not verifiable: all 9 sampled inputs raised on both versions — the function never executed, so 'no divergence' would be vacuous (mostly TypeError — names the real obstacle, not a sampling gap)

· 7 more finding(s) on lines outside this diff (see the check run).

[abhay-codes07]

On the advisory: the _collision_rank/_pick_winner relationship is unchanged by this PR — _collision_rank resolves same-ID collisions when building seen_ids, and _pick_winner only ever runs over members of unique_nodes (the post-collision survivors), so every remap target is a node that survives into deduped_nodes; edges cannot rewire to a dropped id through either path. This PR only reorders _pick_winner's scoring within the same candidate set (richness before id length) and back-fills the survivor, so any disagreement between the two rankings would predate it and still resolve to a live node.

[safishamsi]

Shipped in v0.9.57 — landed on v8 via cherry-pick with your authorship preserved. Thanks @abhay-codes07 — when duplicates merge, the richer node is kept as survivor and the losers' non-empty fields are folded in. Release: https://github.com/Graphify-Labs/graphify/releases/tag/v0.9.57