Add optimizer packages and infrastructure (non-functional)#3516
Add optimizer packages and infrastructure (non-functional)#3516
Conversation
This PR adds the core optimizer implementation without wiring it into the server startup. The optimizer provides semantic tool discovery using embeddings to reduce token usage for LLMs working with large toolsets. Changes include: - New optimizer internal packages (db, embeddings, ingestion, models, tokens) - EmbeddingOptimizer implementation with hybrid search (semantic + FTS) - Updated Optimizer interface with Close() and HandleSessionRegistration() - OptimizerHandlerProvider interface in adapter package - Config schema updates for optimizer configuration - CRD schema updates for VirtualMCPServer optimizer fields - Updated dummy_optimizer to implement new interface (backward compat) - Test updates for new schema structure This is part 1 of a 3-part PR split. Part 2 will wire the optimizer into server startup. Part 3 will add documentation and examples.
therealnb
requested review from
ChrisJBurns,
JAORMX,
amirejaz,
jerm-dro,
jhrozek and
yrobla
as code owners
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
Large PR Detected
This PR exceeds 1000 lines of changes and requires justification before it can be reviewed.
How to unblock this PR:
Add a section to your PR description with the following format:
## Large PR Justification
[Explain why this PR must be large, such as:]
- Generated code that cannot be split
- Large refactoring that must be atomic
- Multiple related changes that would break if separated
- Migration or data transformationAlternative:
Consider splitting this PR into smaller, focused changes (< 1000 lines each) for easier review and reduced risk.
See our Contributing Guidelines for more details.
This review will be automatically dismissed once you add the justification section.
|
Hey, question: how are we deploying this SQLite database in Kubernetes? SQLite's locking model relies on POSIX Note that the optimizer's SQLite DB is essentially a reconstructable cache. Tools come from discovery, embeddings from the EmbeddingServer. If the DB is lost, we just re-ingest. So, do we even need a persistent volume here? A plain emptyDir with rebuild-on-start might be enough for ~1000 tools. I'd love to see an explicit decision on the storage backend and a warning against NFS-backed StorageClasses in the docs. Thoughts? |
@JAORMX This is the plan. We will not persist the storage. Further discussion here: https://stacklok.slack.com/archives/C09L9QF47EU/p1770676947855589 |
|
Closing this PR as it will be replaced by work done in the linked tasks above. |
3 participants
Summary
This PR adds the core optimizer implementation without wiring it into the server startup. The optimizer provides semantic tool discovery using embeddings to reduce token usage for LLMs working with large toolsets.
This is part 1 of a 3-part PR split from #3440:
Changes
New Optimizer Packages (
pkg/vmcp/optimizer/internal/)db/: SQLite database with FTS5 full-text searchembeddings/: Embedding providers (Ollama, OpenAI-compatible)ingestion/: Tool ingestion servicemodels/: Data models and transport typestokens/: Token counting utilitiesCore Optimizer (
pkg/vmcp/optimizer/)EmbeddingOptimizerimplementation with hybrid search (semantic + FTS)Optimizerinterface withClose()andHandleSessionRegistration()dummy_optimizer.goto implement new interface (backward compat)Config & Schema
OptimizerHandlerProviderinterface in adapter packageTests
Test Plan