ToolHive Architecture Documentation

Welcome to the ToolHive architecture documentation. This directory contains comprehensive technical documentation about ToolHive's design, components, and implementation.

Documentation Index

Core Architecture Documents

Architecture Overview - Start here
- High-level platform overview
- Key components and concepts
- Five ways to run MCP servers
Deployment Modes
- Local Mode: CLI and UI
- Kubernetes Mode: Operator
- Mode comparison and migration paths
- Runtime abstraction and detection
Transport Architecture
- Three MCP transport types (stdio, SSE, streamable-http)
- Proxy architecture (transparent vs protocol-specific)
- Remote MCP server proxying
- Port management and sessions

Detailed Component Documentation

Core Concepts
- Nouns: Workloads, Transports, Proxy, Middleware, RunConfig, Permissions, Groups, Registry, Sessions, Skills, AI-tool Plugins
- Verbs: Deploy, Proxy, Attach, Parse, Filter, Authorize, Audit, Export, Import, Monitor
- Terminology quick reference
Secrets Management
- Provider types (encrypted, 1password, environment)
- OS keyring integration
- Fallback chain
- Security model
RunConfig and Permission Profiles
- RunConfig schema and versioning
- Permission profiles (read, write, network)
- Built-in profiles and custom profiles
- Mount declarations and resource URIs
- Security best practices
Registry System
- Built-in curated registry
- Custom registries (file and remote)
- Registry API server architecture
- MCPRegistry CRD
- Image and remote server metadata
Groups
- Group concept and use cases
- Registry groups
- Client configuration
Workloads Lifecycle Management
- Workloads API interface
- Lifecycle: deploy, stop, restart, delete, update
- State management
- Container vs remote workloads
- Async operations
Kubernetes Operator Architecture
- CRD design (MCPServer, MCPRegistry, MCPToolConfig, MCPExternalAuthConfig, VirtualMCPServer)
- Two-binary architecture (operator + proxy-runner)
- Deployment pattern
- Status management
- Design principles
Virtual MCP Server Architecture
- MCP Gateway for aggregating multiple backends
- Backend discovery and capability aggregation
- Conflict resolution strategies
- Two-boundary authentication model
- Composite tool workflows
Auth Server Storage Architecture
- Storage interface design (fosite + ToolHive extensions)
- Memory and Redis Sentinel backends
- Multi-tenancy via key prefixes
- Atomic operations with Lua scripts
- Configuration and security model
- Upstream subject identifiers (RFC 9493 sub_id) in access tokens
- Access token type (typ: JWT or at+jwt) in JWT access tokens
Skills System
- Agent Skills lifecycle (discover, build, publish, install)
- SKILL.md format and validation
- OCI-based distribution and git resolution
- Installation scopes (user, project) and multi-client support
- Security model (archive safety, SSRF prevention, supply chain)
- Skills vs MCP servers design rationale
vMCP Scalability Limits and Constraints
- Per-pod session cache cap (1,000 sessions, LRU eviction)
- Session TTL and Redis sliding-window behavior
- File descriptor constraints and estimation
- Redis sizing, eviction policy, and persistence guidance
- Stateful backend data loss on pod restart
Local vMCP CLI Mode
- thv vmcp CLI surface (serve, validate, init)
- Zero-config quick mode and config-file workflow
- Optimizer tier table (Tier 0–3: none, FTS5, TEI semantic, external service)
- TEI container lifecycle (naming, idempotent reuse, health polling, graceful shutdown)
- ARM64/Apple Silicon Rosetta 2 emulation note
- Migration guide from StacklokLabs/mcp-optimizer
vMCP Library Embedding
- Library embedding pattern and brood-box reference implementation
- pkg/vmcp/ stability table (Stable, Experimental, Internal per sub-package)
- Stability declaration convention and how to use the table as a reviewer
- Compatibility guarantees and semver-aligned deprecation policy
- Guidance for downstream embedders on pinning and upgrading
Plugins System
- AI-tool plugin lifecycle (discover, build, push, install, sync, upgrade)
- Client materialization: Claude Code settings and Codex marketplace files
- Project lock-file trust, including keyless and pinned public-key verification
- Complete component inventory and boundaries with MCP workloads
Envoy Network Proxy
- Experimental Envoy backend (TOOLHIVE_NETWORK_PROXY=envoy) replacing two Squid containers with one
- Egress forward proxy and ingress reverse proxy as separate listeners in a single process
- L7 :authority-based RBAC allow/deny and dstdomain parity (incl. Docker-gateway blocking)
- Dynamic forward proxy with per-request DNS and native HTTPS CONNECT tunnelling
- Squid-vs-Envoy comparison and known limitations (AllowPort gap, V4_ONLY DNS, deny-all on empty profile)
vMCP Multi Round-Trip Requests (MRTR)
- MRTR (SEP-2322) design for the Modern (2026-07-28) elicitation/sampling path through vMCP
- The four client×backend bridge cells: stateless Modern↔Modern pass-through, in-request Legacy-client bridge
- Per-request capability mirroring and verbatim requestState relay
- Composite-workflow suspend/resume as the sole durable-state trigger (RFC-0083 D5 handles, owner binding, TTL)
- Sampling's deprecated-pass-through-only disposition (SEP-2577) and the go-sdk/mcpcompat gap list
External Subject-Token Exchange and JWT-bearer Grants
- trustedIssuers: accepting subject tokens from an external OIDC issuer during RFC 8693 token exchange
- RFC 7523 JWT-bearer assertions, audience configuration, and replay protection
- Confused-deputy trust model and the may_act / allowlist consent signals
- Accepted limitations: client-set equivalence, disjoint subject namespaces, partial provenance
- Operational gotchas: audience/scope binding, discovery redirects, JWKS caching, diagnostics
- RFC 9396 authorization_details on token exchange: Go-only type registry, validation and echo
SPIFFE Association Declarations
- SPIFFE configuration remains rejected at startup; the feature is not yet deployable
- Auth-server TLS listener runs on the proxy runner and vMCP at fixed port 8443; operator configuration/exposure is unsupported
- File-bundle source and newServer X.509/JWT bundle wiring are present but unreachable through normal SPIFFE startup
- JWT-SVID validation logic exists; X.509-SVID verification remains unimplemented
- Fail-closed policy validation and durable, restart-safe static-client reservation
- Endpoint/Workload API loading and SPIFFE method integration with grants or discovery remain pending

Existing Documentation

For middleware architecture, see: docs/middleware.md

Complete middleware system documentation
Eight middleware components
Extending the middleware system
Error handling and performance

Architecture Map

This visual map shows how all documentation relates to the core ToolHive architecture:

graph TB
    subgraph "Start Here"
        Overview[00: Architecture Overview<br/>Platform concepts & components]
    end

    subgraph "Core Understanding"
        Concepts[02: Core Concepts<br/>Nouns & Verbs]
        Deployment[01: Deployment Modes<br/>Local CLI/UI, Kubernetes]
    end

    subgraph "Communication Layer"
        Transport[03: Transport Architecture<br/>stdio, SSE, streamable-http]
        Middleware[../middleware.md<br/>8 Middleware Components]
    end

    subgraph "Configuration & Security"
        RunConfig[05: RunConfig & Permissions<br/>Configuration format & profiles]
        Secrets[04: Secrets Management<br/>Encrypted, 1Password, env]
    end

    subgraph "Distribution & Organization"
        Registry[06: Registry System<br/>Curated catalog & API]
        Groups[07: Groups<br/>Logical collections]
    end

    subgraph "Runtime Management"
        Workloads[08: Workloads Lifecycle<br/>Deploy, stop, restart, delete]
        Operator[09: Kubernetes Operator<br/>CRDs & reconciliation]
        vMCP[10: Virtual MCP<br/>Aggregation & Gateway]
        AuthStorage[11: Auth Server Storage<br/>Memory & Redis backends]
    end

    subgraph "AI Extensions"
        Skills[12: Skills System<br/>Single instruction components]
        Plugins[14: Plugins System<br/>Multi-component AI-tool bundles]
    end

    %% Navigation paths
    Overview --> Concepts
    Overview --> Deployment

    Concepts --> Transport
    Concepts --> RunConfig
    Concepts --> Registry

    Deployment --> Operator
    Deployment --> Workloads

    Transport --> Middleware

    RunConfig --> Secrets
    RunConfig --> Workloads

    Registry --> Groups
    Registry --> Workloads

    Groups --> Workloads
    Groups --> vMCP
    Groups --> Skills
    Groups --> Plugins

    Registry --> Skills
    Registry --> Plugins

    Workloads --> Operator
    vMCP --> Operator
    AuthStorage --> Operator

    %% Styling
    style Overview fill:#e1f5fe,stroke:#01579b,stroke-width:3px
    style Concepts fill:#f3e5f5,stroke:#4a148c,stroke-width:2px
    style Deployment fill:#f3e5f5,stroke:#4a148c,stroke-width:2px
    style Transport fill:#e8f5e9,stroke:#1b5e20,stroke-width:2px
    style Middleware fill:#e8f5e9,stroke:#1b5e20,stroke-width:2px
    style RunConfig fill:#fff3e0,stroke:#e65100,stroke-width:2px
    style Secrets fill:#fff3e0,stroke:#e65100,stroke-width:2px
    style Registry fill:#fce4ec,stroke:#880e4f,stroke-width:2px
    style Groups fill:#fce4ec,stroke:#880e4f,stroke-width:2px
    style Workloads fill:#e0f2f1,stroke:#004d40,stroke-width:2px
    style Operator fill:#e0f2f1,stroke:#004d40,stroke-width:2px
    style vMCP fill:#e0f2f1,stroke:#004d40,stroke-width:2px
    style AuthStorage fill:#e0f2f1,stroke:#004d40,stroke-width:2px
    style Skills fill:#e8eaf6,stroke:#283593,stroke-width:2px
    style Plugins fill:#e8eaf6,stroke:#283593,stroke-width:2px

Color Legend:

🔵 Blue (Start Here): Entry point for all readers
🟣 Purple (Core Understanding): Foundational concepts and deployment patterns
🟢 Green (Communication Layer): How MCP servers communicate and process requests
🟠 Orange (Configuration & Security): Security model and configuration management
🔴 Pink (Distribution & Organization): How servers are cataloged and organized
🟦 Teal (Runtime Management): Lifecycle and cluster management
🔷 Indigo (AI Extensions): Skill and AI-tool plugin lifecycle and distribution

Navigation Paths:

For first-time readers: Follow the arrows from Overview → Concepts → your area of interest
For implementers: Focus on the green (Transport/Middleware) and teal (Workloads/Operator) sections
For operators: Start with Deployment → Operator, then dive into RunConfig and Registry

Quick Navigation

By Role

For Platform Developers: Start with Architecture Overview → Core Concepts → Deployment Modes

For Middleware Developers: Read Transport Architecture → Middleware

For Operators: See Deployment Modes → Kubernetes Operator

For Contributors: Review all documents in order (00 → 01 → 02 → 03 → ...)

By Topic

Understanding the Platform:

Running MCP Servers:

Configuration:

Extending ToolHive:

Middleware

Agent Skills:

Skills System

Advanced Features:

Architecture Principles

ToolHive follows these architectural principles:

1. Platform, Not Just a Runner

ToolHive is a platform for MCP server management, providing:

Proxy layer with middleware
Security and access control
Aggregation and composition
Registry and distribution

2. Abstraction and Portability

RunConfig: Portable configuration format (JSON/YAML)
Runtime Interface: Abstract container operations
Transport Interface: Abstract communication protocols
Middleware Interface: Composable request processing

3. Security by Default

Network isolation by default
Permission profiles for fine-grained control
Authentication and authorization built-in
Audit logging for compliance

4. Extensibility

Middleware system for custom processing
Custom registries
Protocol builds (uvx://, npx://, go://)
Virtual MCP composition

5. Cloud Native

Kubernetes operator for cluster deployments
Container-based isolation
StatefulSets for stateful workloads
Service discovery and load balancing

Key Architectural Decisions

Why Two Binaries for Kubernetes?

thv-operator: Watches CRDs, reconciles Kubernetes resources thv-proxyrunner: Runs in pods, creates containers, proxies traffic

This separation provides:

Clear responsibility boundaries
Operator focuses on Kubernetes resources
Proxy-runner focuses on MCP traffic
Independent scaling and lifecycle

Reference: Deployment Modes

Why Transparent Proxy for SSE/Streamable HTTP?

SSE and Streamable HTTP transports use the same transparent proxy because:

Container already speaks HTTP
No protocol translation needed
Middleware applies uniformly
Simpler implementation

Reference: Transport Architecture

Why RunConfig as API Contract?

RunConfig is part of ToolHive's API contract because:

Export/import workflows
Versioned schema with migrations
Portable across deployments
Reproducible configurations

Reference: RunConfig and Permissions

Implementation Patterns

Factory Pattern

Used extensively for runtime-specific implementations:

// Container runtime factory
runtime, err := container.NewFactory().Create(ctx)

// Transport factory
transport, err := transport.NewFactory().Create(config)

Files:

pkg/container/factory.go
pkg/transport/factory.go

Interface Segregation

Clean abstractions for:

Runtime: Container operations (pkg/container/runtime/types.go)
Transport: Communication (pkg/transport/types/transport.go)
Middleware: Request processing (pkg/transport/types/transport.go)
Workloads: Lifecycle management (pkg/workloads/manager.go)

Middleware Chain

Request processing as composable layers:

// Middleware applied in reverse order
for i := len(middlewares) - 1; i >= 0; i-- {
    handler = middlewares[i](handler)
}

Reference: Middleware

Diagrams Legend

Throughout this documentation, we use Mermaid diagrams:

Blue boxes: ToolHive components
Orange boxes: MCP servers or containers
Green boxes: Proxy components
Purple boxes: External systems
Solid arrows: Direct communication
Dashed arrows: Configuration or state

Contributing to Documentation

When adding new architecture documentation:

Use consistent numbering: XX-topic-name.md
Start with "Why": Explain design decisions
Include code references: Link to file:line where possible
Add diagrams: Use Mermaid for visual clarity
Cross-reference: Link related documents
Keep it current: Update when implementation changes

Documentation Template

# Topic Name

## Overview
Brief explanation of what this covers

## Why This Exists
Design rationale and decisions

## How It Works
Technical details with code references

## Key Components
List of main pieces

## Implementation
Code pointers and examples

## Related Documentation
Links to related docs

Getting Help

General questions: See CLAUDE.md
Operator specifics: See cmd/thv-operator/DESIGN.md
Contributing: See CONTRIBUTING.md
Middleware: See docs/middleware.md

Version: 0.1.0 (Initial architecture documentation) Last Updated: 2026-02-13 Maintainers: ToolHive Core Team

Navigation Menu

Repository navigation

README.md

README.md

ToolHive Architecture Documentation

Documentation Index

Core Architecture Documents

Detailed Component Documentation

Existing Documentation

Architecture Map

Quick Navigation

By Role

By Topic

Architecture Principles

1. Platform, Not Just a Runner

2. Abstraction and Portability

3. Security by Default

4. Extensibility

5. Cloud Native

Key Architectural Decisions

Why Two Binaries for Kubernetes?

Why Transparent Proxy for SSE/Streamable HTTP?

Why RunConfig as API Contract?

Implementation Patterns

Factory Pattern

Interface Segregation

Middleware Chain

Diagrams Legend

Contributing to Documentation

Documentation Template

Getting Help

FilesExpand file tree

README.md

Latest commit

History

README.md

File metadata and controls

ToolHive Architecture Documentation

Documentation Index

Core Architecture Documents

Detailed Component Documentation

Existing Documentation

Architecture Map

Quick Navigation

By Role

By Topic

Architecture Principles

1. Platform, Not Just a Runner

2. Abstraction and Portability

3. Security by Default

4. Extensibility

5. Cloud Native

Key Architectural Decisions

Why Two Binaries for Kubernetes?

Why Transparent Proxy for SSE/Streamable HTTP?

Why RunConfig as API Contract?

Implementation Patterns

Factory Pattern

Interface Segregation

Middleware Chain

Diagrams Legend

Contributing to Documentation

Documentation Template

Getting Help