Skip to content

fix(den-api): rank create routes findable in /mcp/agent capability search - #3428

Draft
benjaminshafii wants to merge 1 commit into
devfrom
fix/mcp-agent-search-ranking
Draft

fix(den-api): rank create routes findable in /mcp/agent capability search#3428
benjaminshafii wants to merge 1 commit into
devfrom
fix/mcp-agent-search-ranking

Conversation

@benjaminshafii

Copy link
Copy Markdown
Member

What

search_capabilities on /mcp/agent could not surface REST create routes. Discovered live: an org admin asking for "create marketplace" got zero results pointing at postMarketplaces across four query variants, concluded the capability didn't exist, and was told (wrongly) it was dashboard-only. execute_capability("postMarketplaces") worked the whole time — the route was in the catalog; ranking buried it.

Root causes (reproduced with the production scorer)

Live query (limit) Score Result
create marketplace (4) 7 rank 5 — lost a tie to mcp:…slack_create_conversation alphabetically, cut by one slot
new marketplace register (6) 5 rank 8 — six-way tie, alphabet orders delete… < get… < post…, cut
marketplace management admin (8) 5 rank 9 — same tie shape, cut
postMarketplaces (6) 3 buried — query tokenizer doesn't split camelCase, so it prefix-ties with every post* route and loses the alphabet
  1. Verb vocabulary gap — humans say create/new/add/register; operation names say post. The +5 name hit can never fire for creation intent.
  2. Plural mismatch — query marketplace vs name token marketplaces earns prefix +3 instead of exact +5, deflating the whole resource family.
  3. Adversarial tiebreak — equal scores broke by localeCompare, so post* structurally loses to delete*/get*/mcp:*/skill:*.
  4. No exact-name fast path — searching the literal capability name scored 3.

The hard-coded "fast path" capability tables in builtin-skills.ts ("Skip broad search…") are the existing workaround for this recall problem.

Fix (index-time structure, no dictionaries)

  • Verb tokens derived from the closed world of HTTP methods (POST→create/add/new/register, GET→list | get/read, PUT/PATCH→update/edit/set, DELETE→delete/remove/revoke) plus static POST item-action segments (…/{id}/archivearchive), added as name-level tokens at scoring time. New routes get correct tokens automatically; the mapping cannot grow.
  • Plural-insensitive exact equality (marketplace == marketplaces) for name/summary/extra matches; prefix logic untouched.
  • Exact-name short-circuit: query == operation name (case-insensitive) → +100.
  • Meaningful tiebreak: score, then fewer path params, then shorter name, alphabetical last (kept for determinism).
  • test/mcp-search-golden.test.ts is the new behavioral contract for ranking; src/mcp/README.md documents it.

Deliberately not query-side synonym dictionaries (unbounded curation, silent reordering regressions) and not embeddings (self-hosted/air-gapped deployments, per-org live catalogs, non-deterministic rankings).

After (real catalog, 170 operations, via buildMcpCatalog(openapi.json))

QUERY "create marketplace"                 QUERY "add github plugin to marketplace"
 1. 15 POST postMarketplaces   <<<<         1. 23 POST postPluginsImportMcpsFromGithubUrlPreview
 2. 13 POST postMarketplacesAccess          2. 23 POST postMarketplacesPlugins
 3. 13 POST postMarketplacesDelete          3. 21 POST postPluginsImportMcpsFromGithubUrl
 ...                                        ...

Tests run

From ee/apps/den-api (bun 1.3.10, deps via pnpm install --filter "@openwork-ee/den-api..." + workspace builds den-db, enterprise-mcp-client, connect-link, email, install-config):

Command Result
bun test test/mcp-search-golden.test.ts 8 pass
bun test test/mcp-url-guard.test.ts 53 pass
bun test test/external-capabilities-search-divergence.test.ts 23 pass
bun test test/mcp-catalog-parameter-refs.test.ts test/mcp-agent-timeouts.test.ts test/mcp-agent-config-policy.test.ts 31 pass / 1 fail — pre-existing: identical 15/1 on pristine origin/dev with the same deps built (assertion expects marketplaceId in required for the GitHub import body; the live schema has it optional)
pnpm exec tsc --noEmit clean
git stash A/B on the failing suite failure present without this diff

test/google-workspace-capabilities.test.ts was not run to green: it needs a live MySQL (DATABASE_URL), unrelated to this diff.

Validation status (honest)

Scoring-function change verified by unit contract + a live-catalog diagnostic (table above) rather than a full fraimz; this is a draft — before marking ready I'll attach the fraimz driving search_capabilities("create marketplace") against a running Den and asserting postMarketplaces is returned (pnpm fraimz --flow <id> --pr). Reviewer repro without a stack:

cd ee/apps/den-api
bun test test/mcp-search-golden.test.ts

Discovered while wiring a real org's MCP connection through the agent surface; the failing queries in the table are verbatim from that session.

…arch

Agents searching 'create marketplace' never surfaced postMarketplaces:
REST names spell creation 'post', plural resources ('marketplaces') demote
exact name hits to prefix hits, and equal-score ties broke alphabetically
so post* routes lost to delete*/get* siblings and were cut by small limits.

- derive name-level verb tokens from the closed world of HTTP methods
  (POST create/add/new/register, GET list|get/read, PUT/PATCH update/edit/set,
  DELETE delete/remove/revoke) plus static POST item-action segments
- make exact token equality plural-insensitive (prefix logic unchanged)
- short-circuit exact operation-name queries (+100)
- break score ties by fewer path params, then shorter name, alpha last
- add test/mcp-search-golden.test.ts as the ranking behavioral contract
@vercel

vercel Bot commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
openwork-app Ready Ready Preview Aug 2, 2026 2:01pm
openwork-den Ready Ready Preview Aug 2, 2026 2:01pm
openwork-den-worker-proxy Ready Ready Preview Aug 2, 2026 2:01pm
openwork-landing Ready Ready Preview, v0 Aug 2, 2026 2:01pm
1 Skipped Deployment
Project Deployment Actions Updated (UTC)
openwork-diagnostics Skipped Skipped Aug 2, 2026 2:01pm

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant