fix(den-api): rank create routes findable in /mcp/agent capability search - #3428
Draft
benjaminshafii wants to merge 1 commit into
Draft
fix(den-api): rank create routes findable in /mcp/agent capability search#3428benjaminshafii wants to merge 1 commit into
benjaminshafii wants to merge 1 commit into
Conversation
…arch
Agents searching 'create marketplace' never surfaced postMarketplaces:
REST names spell creation 'post', plural resources ('marketplaces') demote
exact name hits to prefix hits, and equal-score ties broke alphabetically
so post* routes lost to delete*/get* siblings and were cut by small limits.
- derive name-level verb tokens from the closed world of HTTP methods
(POST create/add/new/register, GET list|get/read, PUT/PATCH update/edit/set,
DELETE delete/remove/revoke) plus static POST item-action segments
- make exact token equality plural-insensitive (prefix logic unchanged)
- short-circuit exact operation-name queries (+100)
- break score ties by fewer path params, then shorter name, alpha last
- add test/mcp-search-golden.test.ts as the ranking behavioral contract
Contributor
|
The latest updates on your projects. Learn more about Vercel for GitHub.
1 Skipped Deployment
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
search_capabilitieson/mcp/agentcould not surface REST create routes. Discovered live: an org admin asking for "create marketplace" got zero results pointing atpostMarketplacesacross four query variants, concluded the capability didn't exist, and was told (wrongly) it was dashboard-only.execute_capability("postMarketplaces")worked the whole time — the route was in the catalog; ranking buried it.Root causes (reproduced with the production scorer)
create marketplace(4)mcp:…slack_create_conversationalphabetically, cut by one slotnew marketplace register(6)delete… < get… < post…, cutmarketplace management admin(8)postMarketplaces(6)post*route and loses the alphabetpost. The +5 name hit can never fire for creation intent.marketplacevs name tokenmarketplacesearns prefix +3 instead of exact +5, deflating the whole resource family.localeCompare, sopost*structurally loses todelete*/get*/mcp:*/skill:*.The hard-coded "fast path" capability tables in
builtin-skills.ts("Skip broad search…") are the existing workaround for this recall problem.Fix (index-time structure, no dictionaries)
…/{id}/archive→archive), added as name-level tokens at scoring time. New routes get correct tokens automatically; the mapping cannot grow.marketplace==marketplaces) for name/summary/extra matches; prefix logic untouched.test/mcp-search-golden.test.tsis the new behavioral contract for ranking;src/mcp/README.mddocuments it.Deliberately not query-side synonym dictionaries (unbounded curation, silent reordering regressions) and not embeddings (self-hosted/air-gapped deployments, per-org live catalogs, non-deterministic rankings).
After (real catalog, 170 operations, via
buildMcpCatalog(openapi.json))Tests run
From
ee/apps/den-api(bun 1.3.10, deps viapnpm install --filter "@openwork-ee/den-api..."+ workspace buildsden-db,enterprise-mcp-client,connect-link,email,install-config):bun test test/mcp-search-golden.test.tsbun test test/mcp-url-guard.test.tsbun test test/external-capabilities-search-divergence.test.tsbun test test/mcp-catalog-parameter-refs.test.ts test/mcp-agent-timeouts.test.ts test/mcp-agent-config-policy.test.tsorigin/devwith the same deps built (assertion expectsmarketplaceIdinrequiredfor the GitHub import body; the live schema has it optional)pnpm exec tsc --noEmitgit stashA/B on the failing suitetest/google-workspace-capabilities.test.tswas not run to green: it needs a live MySQL (DATABASE_URL), unrelated to this diff.Validation status (honest)
Scoring-function change verified by unit contract + a live-catalog diagnostic (table above) rather than a full fraimz; this is a draft — before marking ready I'll attach the fraimz driving
search_capabilities("create marketplace")against a running Den and assertingpostMarketplacesis returned (pnpm fraimz --flow <id> --pr). Reviewer repro without a stack:Discovered while wiring a real org's MCP connection through the agent surface; the failing queries in the table are verbatim from that session.