fix(store): index BM25 on a search_text column carrying the abstract and summary layers

[gorkem2020]

Summary

Fixes #913. The BM25 index sat on the text column, which only holds the L0 abstract, so a keyword that survived only in the overview or the content layer was invisible to indexed full-text search. The lexical scan did see those layers, but only when the indexed query returned nothing at all. The vector column already embeds the abstract plus the content and is untouched.

Changes

• src/search-text.ts (new): buildSearchText(text, metadata) joins the abstract with the l0_abstract, l1_overview and l2_content metadata layers, folding identical layers once; missing or broken metadata degrades to the abstract.
• src/store.ts: new search_text column and the FTS index now targets it. Every write path (store/bulkStore, upsert, importEntry, update, bulkUpdateExact, the recall-metadata merge, the legacy-scope repair, storeSuperseding) passes rows through one toRow boundary that recomputes the column from the row's own text and metadata, so no rewrite path can let it drift. Tables from before the column existed get it through the existing legacy-column migration: addColumns seeds it with the abstract, then the same locked step backfills it from the metadata layers. createFtsIndex moves the index off the abstract column and drops the leftover index so a bare FTS query stays unambiguous; bm25Search names the column explicitly; rebuildFtsIndex recreates on the current column. A store whose migration did not run keeps the previous text index (the column is resolved from the schema and the index list, never assumed).
• MemoryStore.backfillSearchText({ dryRun }): report or rewrite rows whose column is missing or stale. It checks out the latest table version first, so a CLI process sees the gateway's writes.
• cli.ts: memory-pro backfill-search-text (report only; --apply rewrites), the repair path next to reindex-fts.
• Tests: test/fts-search-text-column.test.mjs covers the builder, a content-only token found through the real index with the lexical fallback forbidden, the rewrite paths keeping the column current, a pre-upgrade table being migrated, backfilled and re-indexed on open, the backfill reporting and repairing a stale row, and reindex-fts ending with a single index on search_text. test/fts-index-fold.test.mjs and the CLI attachment test follow the column move. Registered in the npm test chain and the CI manifest.

Notes

• LanceDB 0.26 evaluates addColumns value expressions against existing columns only, so the schema step can seed the abstract but cannot parse the metadata JSON; the app-level backfill in the same migration step does that.
• On 0.26 a mergeInsert(...).whenMatchedUpdateAll() source row that lacks the column keeps the target value, while add without it writes NULL; the write boundary sets the column on every path, so the behavior does not depend on either.
• The result shape is unchanged: entry.text keeps returning the abstract; the new column is index plumbing only.

Verification

• npm run build, npm test (full chain), the new test file, fts-index-fold, store-lexical-metadata-search, cli-subcommand-attachment, redis-lock, migrate-legacy-schema, node scripts/verify-ci-test-manifest.mjs.

[gorkem2020]

Ready for review. Rebased onto the current master (93899f88, post #946); only the two test-registration files conflicted (ordered union), the source is unchanged from the draft.

Fleet evidence: this change has run on our gateway since 2026-09-12 (the store migration ran once on first start and backfilled search_text for every existing row, index search_text_idx), with hybrid and FTS retrieval healthy through today. npm run build and the full npm test chain pass on the rebased head.

[rwmjhb]

PR #998 Review: fix(store): index BM25 on a search_text column carrying the abstract and summary layers

Verdict: APPROVE | 6 rounds completed | Value: 55% | Size: XL | Author: gorkem2020

Value Assessment

Problem: Indexed BM25 search currently indexes only the abstract (text), so keywords present only in metadata summary/content layers can be missed despite being stored.

Dimension	Assessment
Value Score	55%
Value Verdict	review
Issue Linked	true
Project Aligned	true
Duplicate	false
AI Slop Score	1/6
User Impact	medium
Urgency	medium

Scope Drift: 1 flag(s)

• PR #998 claims to fix issue #913, but #913 describes smart extraction producing no candidates and skipping regex fallback; this diff changes only storage/FTS retrieval behavior, not extraction or fallback gating.

AI Slop Signals:

• The PR claims “Fixes #913,” but issue #913 concerns smart extraction failures while the changed implementation is limited to src/store.ts indexing and retrieval.

Open Questions:

• Should PR #998 be relinked to a retrieval-specific issue rather than claiming to fix #913’s extraction/fallback failure?
• Does the project want backfill-search-text as a permanent supported CLI command, or should migration plus reindex-fts be the only operational surface?

Summary

Indexed BM25 search currently indexes only the abstract (text), so keywords present only in metadata summary/content layers can be missed despite being stored.

Evaluation Signals

Signal	Value
Blockers	0
Warnings	2
PR Size	XL
Verdict Floor	approve
Risk Level	high
Value Model	codex
Primary Model	codex
Adversarial Model	claude

Nice to Have

• F1: The PR does not fix linked issue #913's extraction failure
• F2: Rolling old-version writers create permanently unindexed rows
• F3: A failed migration backfill is logged and then treated as a usable migration
• F4: Startup can delete unrelated FTS indexes
• F5: Automated check reported a new any type
• F6: Automated check reported excessive console logging
• MR1: Moving the FTS index at startup is not atomic: the legacy text index is dropped before the search_text index is created, so a failed create leaves no FTS index
• MR2: The migration backfill loads the whole table into memory and rewrites stale rows while holding the write lock during init
• MR3: The claim that every write goes through toRow is false: backfillLegacySecondTimestamps still re-adds raw rows

Recommended Action

Ready to merge.

---

Reviewed at 2026-09-25T04:56:42Z | 6 rounds | Value: codex | Primary: codex | Adversarial: claude