Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

9 changes: 9 additions & 0 deletions architecture/inference-routing.md
Original file line number Diff line number Diff line change
Expand Up @@ -180,6 +180,15 @@ Before forwarding inference requests, the proxy strips sensitive and hop-by-hop
- **Request**: `authorization`, `x-api-key`, `host`, `content-length`, and hop-by-hop headers (`connection`, `keep-alive`, `proxy-authenticate`, `proxy-authorization`, `proxy-connection`, `te`, `trailer`, `transfer-encoding`, `upgrade`).
- **Response**: `content-length` and hop-by-hop headers.

### Internal debug capture

For Milestone 1 attribution work, the sandbox proxy supports an internal append-only JSONL debug log for `inference.local` traffic.

- Enable it by setting `OPENSHELL_INFERENCE_DEBUG_LOG=/absolute/path/to/inference-debug.jsonl` in the sandbox runtime environment before the proxy starts.
- Each parsed `inference.local` request appends one JSON object with process attribution (`pid`, binary path, ancestor binaries, cmdline paths), request metadata, selected route metadata, sanitized headers, and capped request/response body capture.
- Request and response bodies are captured up to 64 KiB each. The record includes total byte counts plus explicit truncation flags and captured-byte counts when the body exceeds that cap.
- Header logging is separate from forwarding logic. Auth headers are stripped and auth-like values such as cookies or token-bearing headers are redacted before they reach the JSONL file.

### Response streaming

The router supports two response modes:
Expand Down
1 change: 1 addition & 0 deletions crates/openshell-router/src/backend.rs
Original file line number Diff line number Diff line change
Expand Up @@ -411,6 +411,7 @@ mod tests {
name: "inference.local".to_string(),
endpoint: endpoint.to_string(),
model: "test-model".to_string(),
provider_type: None,
api_key: "sk-test".to_string(),
protocols: protocols.iter().map(|p| (*p).to_string()).collect(),
auth,
Expand Down
4 changes: 4 additions & 0 deletions crates/openshell-router/src/config.rs
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,7 @@ pub struct ResolvedRoute {
pub name: String,
pub endpoint: String,
pub model: String,
pub provider_type: Option<String>,
pub api_key: String,
pub protocols: Vec<String>,
/// How to inject the API key on outgoing requests.
Expand All @@ -53,6 +54,7 @@ impl std::fmt::Debug for ResolvedRoute {
.field("name", &self.name)
.field("endpoint", &self.endpoint)
.field("model", &self.model)
.field("provider_type", &self.provider_type)
.field("api_key", &"[REDACTED]")
.field("protocols", &self.protocols)
.field("auth", &self.auth)
Expand Down Expand Up @@ -125,6 +127,7 @@ impl RouteConfig {
name: self.name.clone(),
endpoint: self.endpoint.clone(),
model: self.model.clone(),
provider_type: self.provider_type.clone(),
api_key: self.resolve_api_key()?,
protocols,
auth,
Expand Down Expand Up @@ -252,6 +255,7 @@ routes:
name: "test".to_string(),
endpoint: "https://api.example.com/v1".to_string(),
model: "test-model".to_string(),
provider_type: Some("openai".to_string()),
api_key: "sk-super-secret-key-12345".to_string(),
protocols: vec!["openai_chat_completions".to_string()],
auth: AuthHeader::Bearer,
Expand Down
1 change: 1 addition & 0 deletions crates/openshell-router/src/mock.rs
Original file line number Diff line number Diff line change
Expand Up @@ -127,6 +127,7 @@ mod tests {
name: "test".to_string(),
endpoint: endpoint.to_string(),
model: model.to_string(),
provider_type: None,
api_key: "key".to_string(),
protocols: protocols.iter().map(ToString::to_string).collect(),
auth: crate::config::AuthHeader::Bearer,
Expand Down
6 changes: 6 additions & 0 deletions crates/openshell-router/tests/backend_integration.rs
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ fn mock_candidates(base_url: &str) -> Vec<ResolvedRoute> {
name: "inference.local".to_string(),
endpoint: base_url.to_string(),
model: "meta/llama-3.1-8b-instruct".to_string(),
provider_type: Some("openai".to_string()),
api_key: "test-api-key".to_string(),
protocols: vec!["openai_chat_completions".to_string()],
auth: AuthHeader::Bearer,
Expand Down Expand Up @@ -113,6 +114,7 @@ async fn proxy_no_compatible_route_returns_error() {
name: "inference.local".to_string(),
endpoint: "http://localhost:1234".to_string(),
model: "test".to_string(),
provider_type: None,
api_key: "key".to_string(),
protocols: vec!["anthropic_messages".to_string()],
auth: AuthHeader::Custom("x-api-key"),
Expand Down Expand Up @@ -174,6 +176,7 @@ async fn proxy_mock_route_returns_canned_response() {
name: "inference.local".to_string(),
endpoint: "mock://test".to_string(),
model: "mock/test-model".to_string(),
provider_type: None,
api_key: "unused".to_string(),
protocols: vec!["openai_chat_completions".to_string()],
auth: AuthHeader::Bearer,
Expand Down Expand Up @@ -308,6 +311,7 @@ async fn proxy_uses_x_api_key_for_anthropic_route() {
name: "inference.local".to_string(),
endpoint: mock_server.uri(),
model: "claude-sonnet-4-20250514".to_string(),
provider_type: Some("anthropic".to_string()),
api_key: "test-anthropic-key".to_string(),
protocols: vec!["anthropic_messages".to_string()],
auth: AuthHeader::Custom("x-api-key"),
Expand Down Expand Up @@ -366,6 +370,7 @@ async fn proxy_anthropic_does_not_send_bearer_auth() {
name: "inference.local".to_string(),
endpoint: mock_server.uri(),
model: "claude-sonnet-4-20250514".to_string(),
provider_type: Some("anthropic".to_string()),
api_key: "anthropic-key".to_string(),
protocols: vec!["anthropic_messages".to_string()],
auth: AuthHeader::Custom("x-api-key"),
Expand Down Expand Up @@ -410,6 +415,7 @@ async fn proxy_forwards_client_anthropic_version_header() {
name: "inference.local".to_string(),
endpoint: mock_server.uri(),
model: "claude-sonnet-4-20250514".to_string(),
provider_type: Some("anthropic".to_string()),
api_key: "test-anthropic-key".to_string(),
protocols: vec!["anthropic_messages".to_string()],
auth: AuthHeader::Custom("x-api-key"),
Expand Down
2 changes: 2 additions & 0 deletions crates/openshell-sandbox/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -56,8 +56,10 @@ bytes = { workspace = true }
ipnet = "2"

# Serialization
serde = { workspace = true }
serde_json = { workspace = true }
serde_yaml = { workspace = true }
base64 = { workspace = true }

# Logging
tracing = { workspace = true }
Expand Down
Loading
Loading