Skip to content

feat(knowledge): add full-catalog selection, applicability checks and bounded reads - #534

Open
amy-why-3459 wants to merge 2 commits into
mainfrom
codex/issue-529-catalog-selection
Open

amy-why-3459 wants to merge 2 commits into
mainfrom
codex/issue-529-catalog-selection

Conversation

@amy-why-3459

@amy-why-3459 amy-why-3459 commented Nov 20, 2026 •

Copy link
Copy Markdown
Collaborator

Governing Issue and delivery slice

Refs #529

The requester authorized implementation, a PR, and a ten-query knowledge retrieval comparison (not ten hardware optimization rounds), with selection performed in the current conversation.

This adds an opt-in complete-catalog retrieval path over the shared Skill/Record/Case corpus. The current Agent selects snapshot-bound qualified references; Knowledge checks conditions, expands explicit references within budgets, and delivers original resources with pinned revisions. Find and the seven primary task routes remain unchanged. Over-budget catalogs fall back explicitly to Find.

  • catalog-select and the Apps Python selector seam share snapshot generation, identity validation and read assembly.
  • Required procedure/resources/notices stay atomic; optional expansion has independent depth/item/byte budgets and records omissions, cycles and unresolved/versioned links.
  • Conditions distinguish match, analogue, unknown and conflict. Post-read assessments require an exact quote from delivered bytes and never upgrade source validation.
  • The retrieval observation records coverage, selections, original resource hashes, expansion paths, omissions, accounting and caller-reported model information. Delivery does not imply model reading or adoption.

Changed artifacts and evidence

Usage and contracts · ten-query report · qualified results · source-quote assessments

The comparison pins baseline 1b826b5f9666a8b734000e1eb2990e582cdef876 and implementation c457c85; this PR's following documentation commit only records results. Knowledge bytes are unchanged: 80 supporting Skills, 48 Records and 7 Cases. Queries reuse the repository's seven task fixtures plus three labelled Qwen/quality-boundary questions.

Find returned 50 entries and delivered 39 owners. The Agent selected 25 entries and the final implementation delivered 19 owners; six selected owners could not fit their required resources. Each query's IDs and omissions are retained. The confirmed full-image quality failure question deliberately selects no extra evidence. Full transport size is 77,086–77,113 o200k_base tokens per query under the declared 128,000 budget and 4,096 reserve.

These are observational outputs, not accuracy or speedup claims. The same current conversation was used; the Agent inspected all 135 ID/summary entries, then relevant source excerpts. Actual model usage and decision latency are unavailable. Fixed Agent decisions were retained through a correction to deterministic reference reads; the final replay reused the same baseline and selections. No outcome-driven model reruns were used. Task benefit remains INCONCLUSIVE.

改动前后对比(10 个固定查询)

本次对照是旧版 Find 与显式启用的新模式。合入本 PR 后,未启用新模式的调用方仍沿用原有 Find。知识内容、题面与任务入口保持一致。

项目 改动前:Find 改动后:启用 catalog-select
支持证据选择 词法排名后返回 top-5 将全部 135 条证据的目录简述提供给当前 Agent 选择,无词法预筛,允许空选择
条件与来源 调用方读取材料并判断适用范围 结构化条件比较,区分匹配、类比、未知与冲突;原文确认绑定实际递送的 ref/revision/path/quote
读取与引用 沿命中入口执行现有 reader 读取 校验目录快照,按显式引用有界扩展;保留完整必需依赖及预算遗漏原因
全部 10 题返回/选中 50 条 25 条
实际成功递送 39 个材料条目 19 个材料条目;6 条选中材料因必需资源超预算而未递送

上述数量按题累加,同一材料跨题重复计算;“成功递送”表示 reader 实际返回原文字节,不表示模型已完整阅读或采用。返回数量不同不能作为精确率、召回率或效果提升的依据。

逐题全部命中与选择

技能采用稳定短 ID;Record/Case 保留原 ID。未递送表示新模式选中了材料,但受读取预算限制没有返回完整必需资源。最后一列只计实际成功递送。

# 查询 改动前:全部 Find top-5 改动后:全部选择 实际递送(旧 / 新)
1 Ascend Omni Day-0 ascend-model-adaptation
vllm_omni.ascend.diffusion.qwen_image_21_eager_t2i
sglang_omni.common.audio.serving_and_streaming_quality
vllm_omni.common.audio.streaming_serving_measurement
ascend-video-diffusion-optimization
ascend-model-adaptation
vllm_omni.common.omni.stage_ownership_and_search
vllm_omni.common.diffusion.production_integration_gates
5 / 3
2 Diffusion 单卡性能 ascend-video-diffusion-optimization
vllm_omni.common.diffusion.measurement_and_profiling
inference-benchmark-design
video-diffusion-optimization
host-overhead-optimization
vllm_omni.common.diffusion.measurement_and_profiling
performance-ceiling-analysis (未递送)
host-overhead-profiling
4 / 2
3 ROCm graph hang case.qwen-b300-rejected-directions
ascend-graph-mode
pytorch.cuda.compiler.cudagraph_shapes_and_lifetimes
flashinfer.cuda.llm.paged_decode_dispatch
ascend-aclgraph-capture-replay
pytorch
pytorch.cuda.compiler.cudagraph_shapes_and_lifetimes (未递送)
5 / 1
4 TTS 音质回归 inference-benchmark-design
qwen3_tts_engine.ascend.audio.fixed_customvoice_prompt
qwen3_tts_engine.ascend.audio.codec_constants_and_state
qwen3_tts_engine.ascend.audio.graph_batching_state_ownership
torch-npu
inference-benchmark-design (未递送)
sglang_omni.common.audio.serving_and_streaming_quality
vllm_omni.common.audio.streaming_serving_measurement
4 / 2
5 引擎/插件版本对齐 vllm-ascend
vllm-metal
vllm_omni.ascend.diffusion.minimax_h3_reference_kv_reuse
ascend-model-adaptation
audit-afd-p2p-exchanges
repository-evolution-analysis
vllm-ascend
torch-npu (未递送)
1 / 2
6 DeepInfer 衍生引擎 deepinfer.common.llm.zero_overhead_deferred_scheduling
deepinfer.common.llm.bounded_windowed_replay
vllm-metal
host-overhead-optimization
deepinfer.common.llm.speculative_verification_budget
deepinfer.common.llm.zero_overhead_deferred_scheduling
vllm.common.llm.runtime_and_kv_ownership
vllm.common.llm.extension_correctness_gates
repository-evolution-analysis
4 / 4
7 Windows XPU 独立引擎 inference-benchmark-design
project-maya
weight-loading-optimization
strata.common.llm.engine_validation_contract
cann-attribution-overlap
inference-benchmark-design (未递送)
kernel-writing-guide
claim-contract
3 / 2
8 Qwen 四卡 B300 融合 case.qwen-b300-rejected-directions
vllm_omni.ascend.diffusion.qwen_image_21_eager_t2i
case.qwen-ascend-rejected-directions
vllm_omni.cuda.diffusion.minimax_h3_qk_norm_rope
inference-benchmark-design
case.qwen-b300-rejected-directions
inference-benchmark-design (未递送)
4 / 1
9 Qwen 910B1 RoPE 缓存 vllm_omni.ascend.diffusion.qwen_image_21_eager_t2i
case.qwen-ascend-rejected-directions
case.qwen-b300-rejected-directions
ascend-graph-mode
ascend-model-adaptation
case.qwen-ascend-rejected-directions
vllm_omni.ascend.diffusion.qwen_image_21_eager_t2i
5 / 2
10 整图质量门槛已失败 case.qwen-b300-rejected-directions
inference-benchmark-design
case.drama3-rejected-directions
vllm_omni.ascend.diffusion.qwen_image_21_eager_t2i
ascend-operator-tuning
空选择 4 / 0

逐题解释、退化与结论边界

  1. 更集中于适配、阶段契约和上线边界;版本/真实权重未知。
  2. 性能上限技能必需材料 213,952 字节,超过 131,072 字节预算,未递送;不应把选中视为已读。
  3. PyTorch 材料占用大部分预算,选中的 CUDA graph Record 未递送;泛化方法不能证明 ROCm 根因。
  4. benchmark-design 必需材料 320,040 字节而未递送;实际读到的是两套音频测量方法,目标引擎未知。
  5. 读到历史分析与 vLLM-Ascend;torch-npu 未递送。具体兼容组合仍缺版本。
  6. 补充了 vLLM KV/state 所有权与扩展正确性;DeepInfer 离线调度不能直接迁移为服务调度。
  7. 仅有通用方法,缺 Windows/XPU 原生证据;没有把其他平台材料升级为目标支持。
  8. 命中同 cohort 案例:初始融合算子通过、整图失败并回退;benchmark-design 未递送。
  9. 命中同 cohort 案例和 Record:缓存 cos/sin 图像一致但更慢;仍为 author-reported,不是本地复验。
  10. 空选择:题面已经确认质量门槛失败,不能以算子通过或额外案例覆盖该失败。
  • 相同知识版本:80 Skill、48 Record、7 Case。每题每臂一次检索决策;修正引用闭包后,确定性读取重放复用了原选择和原基线,没有重新挑选更有利的模型结果。
  • 两组共用当前会话,未盲测、未隔离会话历史;实际模型 token 用量和决策耗时未知。完整输入的 o200k_base 预算计数不能替代实际会话用量。
  • 最终结果没有成功递送新的跨 owner 条目;部分 Record 通过 owner-local 技能资源扩展获得原文。未量化独立标注的证据命中率、错误迁移率或首次有效证据时间。
  • 结论保持 INCONCLUSIVE(任务收益);这些结果不证明检索优越性、优化收益或推理服务性能提升。

完整材料:对比报告 · 逐题机器结果 · 原文确认 · 固定题面。

Checks and dependencies

  • ruff check src tests tools: passed.
  • git diff --check: passed.
  • Catalog/Find/reader/CLI/Apps plus smoke/claim tests: 168 passed, 2 skipped.
  • Independent read-only worktree code review: passed after fixing authority collisions, store fallback, reference closure and exact CLI token accounting. This is not a formal GitHub approval or an efficacy verdict.
  • Full suite attempted, then interrupted with 1,143 passed, 12 failed, 12 skipped. Git 2.33 does not meet tests requiring Git >=2.45; Node 12 fails two browser-script cases; benchmark comparison fake-task evaluation also fails. Representative failures in all three groups reproduced on unchanged baseline. The full suite is not claimed green; remaining tests were not run.
  • Depends on the already merged Unify inference knowledge in skill_library with seven task routes and direct search #516. No knowledge corpus edits, device experiments, new model-service dependency, default-entry promotion or issue closure.

Remaining work and Issue closure

Formal independent PR review and CI remain separate. The report explicitly retains large-resource omissions, missing target-native evidence, absent independent scoring and conversation-isolation limits. Goal acceptance/closure remains with the requester and maintainers; this PR uses a non-closing issue reference.

Knowledge authorship declaration: not applicable to the configured protected paths; this change adds retrieval implementation and documentation without editing catalog/read implementations or knowledge data. Agent contributions are identified in signed-off commits.

Add full evidence cards, explicit Agent selection, condition checks, source-bound assessments and reference expansion with atomic required resources.

Agent-assisted-by: Codex (implementation)
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
Retain fixed inputs, Agent selections, qualified read results and source-quote assessments. Report observed omissions and unverified task benefit.

Agent-assisted-by: Codex (implementation and evaluation)
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>

@Dong1017 Dong1017 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR review verdict — DO NOT PASS

Reviewed base 1b826b5f9666a8b734000e1eb2990e582cdef876 and head 3f6d5d32ff42e5fbab1644988d28f02fd0d499c0 (first review on opened; pr/prior-reviews.md carries no formal review on this head, so there is no prior verdict to weigh either way). I read governing Issue #529, including its five 完成条件.

Only docs are visible in the seeded patch, and the patch is truncated. pr/meta.json has diffComplete: false, and pr/diff.patch stops mid-array inside docs/reviews/catalog-selection-529/queries.json. Visible: docs/catalog-selection.md (new), docs/operations.md (+1 table row), and docs/reviews/catalog-selection-529/{README.md, assessments.json, comparison.json, protocol.json, queries.json (partial)}. I cannot PASS on a truncated diff, and the missing portion is exactly the part that carries #529's的实现 acceptance.

Blockers

  1. The feat(knowledge) surface is not in what I could see. The title and docs/catalog-selection.md name a CLI (inferrsi catalog-select), inferrsi.apps.knowledge.select_catalog_evidence, confirm_catalog_evidence, inferrsi.knowledge.catalog_selection.CatalogSelection, and tools/compare_catalog_selection.py. No src/ module, no CLI wiring, no test file appears in the visible patch. That leaves #529 完成条件 2 (one real end-to-end chain, "交付不能仅停留在导出目录或提示词") and 条件 5 (端到端测试) unverifiable here. Either republish the complete diff or point me at the exact paths and revisions so the next pass can read them.

  2. The recorded run is scoped to a different revision than this head. docs/reviews/catalog-selection-529/README.md pins the code candidate as c457c85, while the reviewed head is 3f6d5d32. The README says it explicitly: after fixing the reference closure only deterministic reads were replayed, reusing the same baseline and the pre-fix selections ("修正引用闭包后只重放确定性读取", "候选决定在代码修正前保留"). So assessments.json and comparison.json characterize c457c85, not this head. Nothing in the visible patch maps c457c85 onto 3f6d5d3. State that relationship (ancestor? equivalent for these artifacts?) or re-run. Good news on the dependency: the baseline 1b826b5 is this PR's own base and the README records #516 as merged there, which clears the unmerged-dependency blocker from the #529 triage.

  3. Delivered packages and context-limit omissions are indistinguishable in the trace. In comparison.json, entry id: "version-alignment" lists package:inferrsi.ascend.skills.vllm_ascend in candidate_delivered (expansion path, status: "delivered", 86,668 bytes) and also in candidate_omissions with reason: "context_limit", required_bytes: 21756, available_bytes: 9828. Entry id: "day-0" does the same for package:vllm_omni.common.omni.stage_ownership_and_search (delivered 3,044 bytes; also omitted at 18,293/5,223). Those omission records carry no resource/path and no scope discriminator, so a reader cannot tell which declared resource was dropped. That collides with docs/catalog-selection.md, "Reads, expansion and trace": "Required skill procedures/resources, source attribution and license closure are atomic: insufficient space produces an omission, never a partial procedure marked complete", and with #529 条件 4 (终止原因、缺失材料与预算不足均可观察). Add the specific resource/path (or an explicit scope: package|resource) to omission records, and confirm the 86,668-byte delivery did not include a partial required procedure.

  4. Byte accounting does not say whether the typed reader metadata is inside the budget. docs/catalog-selection.md states "read_bytes is the complete bundle size, not a promise about budgeted context size", and the CLI "counts that exact transport, including its accounting envelope". #529's dependency note records that #516's read --material payloads grew ~45% because the metadata.inferrsi block is delivered inside content. If context_bytes/expansion_bytes accounting excludes that block, required_bytes/available_bytes undercount and the over-budget omissions above would be wrong. Please state one way or the other in the doc; this is a question, not a proven defect.

Non-blocking

  • In all 9 assessments.json entries, claim and reason are byte-identical. The doc describes them as distinct fields ("the specific claim, verdict, reason"). Confirm the interface populates them from separate inputs rather than one variable.
  • README.md reports 168 passed / 2 skipped and a full-repo attempt stopped at 1,143 passed / 12 failed / 12 skipped with all three representative failures reproduced on the unmodified baseline. That reporting style is right, but I ran nothing — no test file is visible, and CI state is not seeded.

Credited from what I could read: cards are capped per field with explicit truncation flags while the candidate list is never cut (that answers the uncapped-limitations/card-size concern from the #529 triage); over-budget input returns catalog_over_budget with ordinary Find results and no selector call; unqualified, outside-snapshot and duplicate refs fail rather than resolve silently; match/analogue/unknown/conflict is kept separate from source verification level; and benefit is stated as INCONCLUSIVE rather than claimed. The review artifacts also disclose their own degradations per query honestly, including the two queries where the selected Record was never delivered.

Not verified here: the implementation itself, its tests, CI status, and the ten-query run. This review does not approve the change or alter the scope of #529. I will re-read the full head after the author pushes or once the complete diff and the c457c85 → 3f6d5d3 relationship are in the seed.


Posted by Dong-bot, an automated agent acting on behalf of @Dong1017 (always-on-bot, model claude-sonnet-5-5).

@Dong1017 Dong1017 added area:knowledge Knowledge line: packs, skill library, knowledge admission kind:task Bounded delivery task with a delegated slice labels Nov 20, 2026

@zengchuang-hw zengchuang-hw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NO PASS

初次独立检视 1b826b5f → 3f6d5d32,按 #529 的完整目录和可恢复引用契约检查。批准前需修两项:

  1. catalog_selection.py:222–249:可选 Markdown URL 直接交给 urlsplit(),其解析异常会中止整个 snapshot,预算超限时也不能正常回退到 Find。应只把坏引用记为不可解析,继续生成其余候选;加测试证明同集合的有效候选和回退均仍可用。
  2. catalog_selection.py:70–83:词法检查把 1.4.x 识别为精确版本,和实际 1.4.2 比较得到 conflict(方法用途下 analogue),与文档承诺的非精确版本归 unknown 不符。请区分明确版本与通配/范围,覆盖对照用例。

已静态核对实现、文档和测试定义;未执行 CLI、测试或查询 CI。#529 的设计结论不变。

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent third-party tip COMMENT (cursor[bot]; not Approve / not Request changes).

Head 3f6d5d32ff42e5fbab1644988d28f02fd0d499c0 still carries tip DO NOT PASS from Dong1017 (always-on) and NO PASS from zengchuang-hw. I am not clearing those.

Blockers I independently treat as still open before any cursor formal vote:

  1. Dong: complete feat(knowledge) surface must be reviewable (seed/diff completeness; #529 end-to-end acceptance), recorded-run revision vs tip head relationship, omission records that cannot distinguish delivered vs context-limit drops, and explicit byte-accounting scope for typed reader metadata.
  2. Zeng: catalog_selection.py URL parse failures must not abort the whole snapshot (isolate bad refs; keep Find fallback + tests), and lexical version checks must not treat 1.4.x-style ranges as exact versions.

Ready checks are green on this tip, but that does not override the review blockers. Waiting on author push; will re-verify after tip moves.

Technical verify only — collaborator permission none; not write+ / dual / merge-ready.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:knowledge Knowledge line: packs, skill library, knowledge admission kind:task Bounded delivery task with a delegated slice

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants