CJK: unspaced multi-word queries miss silently (no token split, relaxed fallback not triggered)

[yrwd999]

Summary

CJK (Chinese) multi-word queries written without spaces return zero results, and the relaxed OR fallback does not trigger for them. Single CJK words and space-separated CJK words work fine.

This is a sharp, reproducible edge: the same information is found or missed depending only on whether a space is typed.

Environment

• basic-memory: 4.0.0b1 (Docker, kb-basic-memory:sealed)
• SQLite 3.53.1, FTS5 tokenize='unicode61 tokenchars 0x2F'
• semantic_search_enabled: true (default hybrid retrieval)
• Corpus: 201 entities, mixed Chinese/English

Reproduction

Real corpus note contains the text 09-07 雾凇词库部署时实测 (a two-word Chinese compound "rime dictionary" written adjacent, no space).

Query	Result
雾凇 (single word)	✅ hit, correct note, score 1.0
雾凇 拼音 (spaced)	✅ hit, correct note
雾凇拼音 (unspaced compound)	❌ No results found
卸载 (single word)	✅ hit
transmission 卸载 (spaced)	✅ hit

Same failure reproduces via raw SQL:

sqlite> select count(*) from search_index where search_index match '雾凇';    -- 0
sqlite> select count(*) from search_index where search_index match '雾凇*';   -- 4
sqlite> select count(*) from search_index where search_index match '雾凇拼音*'; -- 0

Mechanism (source-level, 4.0.0b1)

Two layers interact:

1. Auto prefix wildcard saves single words. sqlite_search_repository.py:375 (term = f"{term}*"), so a user query 雾凇 actually runs as 雾凇* and matches by character prefix inside 雾凇词库. That is why bare single-word CJK appears to work — and why a raw match '雾凇' (exact token) returns 0, since CJK is never split into word tokens in content_stems.

2. Unspaced compounds get no fallback. search_service.py:_is_relaxed_fts_fallback_eligible gates the relaxed OR retry on relaxed_query_words(). Measured on this build:

relaxed_query_words('雾凇拼音')  -> None          # 1 token, no fallback
relaxation_word_tokens('雾凇拼音') -> ['雾凇拼音']
relaxed_query_words('雾凇 拼音')  -> ['雾凇','拼音']  # 2 tokens, fallback fires

relaxation_word_tokens splits only on whitespace/punctuation, so an unspaced CJK compound is one token; the three-token guard plus single-token handling yields None; no relaxed retry happens; strict FTS returns 0. Result: silent miss.

Note the asymmetry: the prefix wildcard only rescues the case where the query is a prefix of a stored run. It cannot rescue the reverse (query 雾凇拼音 vs stored 雾凇词库), and it cannot see a stored word that begins mid-query.

Why this matters operationally

• The failure is silent: the user gets "No results found" or a screen of low-relevance noise, and reasonably concludes the knowledge base has no such entry. There is no signal that the query form was the problem.
• Semantic/vector retrieval masks it much of the time, which makes it hard to notice and hard to count — which is likely why it has not surfaced before.

Not a bug (works as intended)

• Unicode61 not splitting CJK is expected; I am not asking for a CJK tokenizer dependency.
• Trigrams were deliberately disabled upstream (issue Performance: Optimize sync/indexing for cloud deployments with ephemeral databases #351, noted in _generate_variants), understood.

Possible directions (up to maintainers)

Any of these would remove the cliff; listed without preference:

1. When a strict FTS query returns 0 and the query is a single unspaced CJK run of length ≥ 4, retry with character n-gram / substring expansion (or a LIKE-backed fallback on content_stems).
2. Relax the fallback guard for CJK: allow a single CJK run to be split into overlapping bigrams for the relaxed OR pass.
3. At minimum, when strict FTS returns empty for a CJK query, surface a hint in the response ("try spacing Chinese terms") — turns a silent miss into a recoverable one.

Workaround we adopted

Documented in our client-side rules: always space Chinese multi-word queries (雾凇 词库 部署, not 雾凇词库部署). Zero cost, verified effective. Happy to keep this if upstream decides the current behavior is intended.

Thanks for basic-memory — it has been running well for us; this is the one rough edge we hit in a week of heavy use.