diff --git a/README.md b/README.md index 0077df5..9a2f9fc 100644 --- a/README.md +++ b/README.md @@ -212,6 +212,162 @@ go build -o oz-agent-worker ./oz-agent-worker --api-key "wk-abc123" --worker-id "my-worker" ``` +## Security and Sandboxing + +Oz agents run inside isolated containers on the worker host. The table below summarizes +what each backend isolates and what it does not: + +| Isolation dimension | Docker backend | Kubernetes backend | Direct backend | +|---|---|---|---| +| Process | ✅ Container per task | ✅ Pod per task | ⚠️ Host process | +| Filesystem | ✅ Ephemeral container FS | ✅ Ephemeral pod FS | ⚠️ Host filesystem | +| Resource (CPU/memory) | ✅ Limits when runner shape set | ✅ Requests/limits when runner shape set | ❌ None | +| Network egress | ⚠️ Default bridge (unrestricted) | ⚠️ Cluster default (see NetworkPolicy) | ❌ None | + +**The Direct backend provides no container boundary.** Tasks run as host processes with +access to the host network, filesystem, and process space. It is not appropriate for +environments where agents must be prevented from reaching internal infrastructure. + +### Docker: Restricting network access + +By default, each task container joins Docker's default bridge network, which gives the +container outbound connectivity to anything the worker host can reach — including internal +infrastructure. You can restrict this in two ways: + +#### 1. Isolated Docker network (`network_mode`) + +Set `backend.docker.network_mode` to attach task containers to a specific Docker network +instead of the default bridge: + +```yaml +worker_id: "my-worker" +backend: + docker: + network_mode: "none" # no networking; agent cannot make outbound connections +``` + +Or attach to a restricted network you manage: + +```yaml +backend: + docker: + network_mode: "restricted-net" # your Docker network with limited egress +``` + +Accepted values are any string accepted by Docker's `--network` flag: +- `"none"` — no networking at all (maximum isolation) +- A named network — attaches the container to that network +- `"host"` — full host network access (expands rather than restricts egress; not recommended for security-sensitive use) +- Empty string / omitted — Docker's default bridge (current behavior) + +The worker passes the value to Docker verbatim; an invalid name surfaces as a task +failure at container creation time. + +#### 2. Egress proxy (`HTTP_PROXY` / `HTTPS_PROXY`) + +Route all HTTP/HTTPS traffic through a filtering proxy you operate by injecting the +standard proxy environment variables: + +```yaml +backend: + docker: + environment: + - name: HTTP_PROXY + value: "http://proxy.internal.example.com:3128" + - name: HTTPS_PROXY + value: "http://proxy.internal.example.com:3128" + - name: NO_PROXY + value: "oz.warp.dev,app.warp.dev,localhost,127.0.0.1" +``` + +Include `oz.warp.dev` and `app.warp.dev` in `NO_PROXY` so the worker's WebSocket +connection and agent task-status calls bypass the proxy. + +**Limitations:** `HTTP_PROXY`/`HTTPS_PROXY` are honored by most HTTP client libraries +(Go, Python, Node.js) but not by raw TCP connections, UDP, or DNS resolvers. For full +protocol coverage, combine the proxy with a restricted Docker network and host-level +iptables rules. + +### Kubernetes: NetworkPolicy + +Task pods are labeled with `oz-task-id` and `oz-worker-id` by the worker. You can write +a `NetworkPolicy` that selects on these labels to restrict what task pods can reach: + +```yaml +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: restrict-oz-task-egress + namespace: agents +spec: + podSelector: + matchLabels: + oz-worker-id: my-worker + policyTypes: + - Egress + egress: + # Allow task pods to reach Warp control plane + - to: + - ipBlock: + cidr: 0.0.0.0/0 + except: + - 10.0.0.0/8 # block RFC-1918 internal ranges + - 172.16.0.0/12 + - 192.168.0.0/16 + ports: + - protocol: TCP + port: 443 +``` + +NetworkPolicy enforcement requires a CNI plugin that implements it (Calico, Cilium, +Weave, etc.). The default kubenet and flannel CNI plugins do not enforce NetworkPolicy +resources. The worker does not create NetworkPolicy resources; you manage them in your +cluster. + +You can also use `backend.kubernetes.pod_template` to set pod-level security contexts, +service accounts with restricted permissions, or custom labels for policy selectors: + +```yaml +backend: + kubernetes: + namespace: agents + pod_template: + metadata: + labels: + security-tier: restricted + containers: + - name: task + securityContext: + runAsNonRoot: true + allowPrivilegeEscalation: false + capabilities: + drop: ["ALL"] +``` + +For Kubernetes, you can also inject proxy env vars via `pod_template`: + +```yaml +backend: + kubernetes: + pod_template: + containers: + - name: task + env: + - name: HTTP_PROXY + value: "http://proxy.internal.example.com:3128" + - name: HTTPS_PROXY + value: "http://proxy.internal.example.com:3128" + - name: NO_PROXY + value: "oz.warp.dev,app.warp.dev,10.0.0.0/8" +``` + +### Warp-hosted cloud agents + +Warp-hosted runs execute in ephemeral Namespace.so microVMs that are strongly isolated +from the host and from other tenants. These sandboxes cannot reach customer internal +infrastructure by default. Enterprise customers who need agents to access internal +services use self-hosted workers. + ## Environment Variables for Task Containers Use `-e` / `--env` to pass environment variables into task containers: diff --git a/internal/config/config.go b/internal/config/config.go index 5ef1b11..677aadc 100644 --- a/internal/config/config.go +++ b/internal/config/config.go @@ -40,6 +40,11 @@ type BackendConfig struct { type DockerConfig struct { Volumes []string `yaml:"volumes"` Environment []EnvEntry `yaml:"environment" validate:"dive"` + // NetworkMode sets the Docker network the task container joins. + // Accepts any value Docker's --network flag accepts: a named network + // (e.g. "restricted-net"), "none" (no networking), "host", or empty + // string (Docker's default bridge network). + NetworkMode string `yaml:"network_mode"` } // DirectConfig holds direct-backend-specific configuration. diff --git a/internal/worker/docker.go b/internal/worker/docker.go index cab2c7a..90bf4b7 100644 --- a/internal/worker/docker.go +++ b/internal/worker/docker.go @@ -24,9 +24,14 @@ const dockerHubAuthConfigKey = "https://index.docker.io/v1/" // DockerBackendConfig holds configuration specific to the Docker backend. type DockerBackendConfig struct { - NoCleanup bool - Volumes []string - Env map[string]string + NoCleanup bool + Volumes []string + Env map[string]string + // NetworkMode sets the Docker network the task container joins. + // Accepts any value Docker's --network flag accepts: a named network + // (e.g. "restricted-net"), "none" (no networking), "host", or empty + // string (Docker's default bridge network). + NetworkMode string } func (b *DockerBackend) containerWasOOMKilled(ctx context.Context, dockerClient *client.Client, containerID string) bool { @@ -137,8 +142,9 @@ func (b *DockerBackend) ExecuteTask(ctx context.Context, params *TaskParams) err binds = append(binds, b.config.Volumes...) hostConfig := &container.HostConfig{ - Binds: binds, - Resources: dockerResourcesForShape(params.InstanceShape), + Binds: binds, + Resources: dockerResourcesForShape(params.InstanceShape), + NetworkMode: container.NetworkMode(b.config.NetworkMode), } resp, err := dockerClient.ContainerCreate(ctx, client.ContainerCreateOptions{ diff --git a/main.go b/main.go index a42f667..5c76cb0 100644 --- a/main.go +++ b/main.go @@ -319,10 +319,17 @@ func mergeConfig(fileConfig *config.FileConfig) (worker.Config, error) { } volumes = append(volumes, CLI.Volumes...) + // Resolve network_mode: only from config file (no CLI flag). + var networkMode string + if fileConfig != nil && fileConfig.Backend.Docker != nil { + networkMode = fileConfig.Backend.Docker.NetworkMode + } + wc.Docker = &worker.DockerBackendConfig{ - NoCleanup: noCleanup, - Volumes: volumes, - Env: mergedEnv, + NoCleanup: noCleanup, + Volumes: volumes, + Env: mergedEnv, + NetworkMode: networkMode, } } diff --git a/specs/security-sandboxing/PRODUCT.md b/specs/security-sandboxing/PRODUCT.md new file mode 100644 index 0000000..9665756 --- /dev/null +++ b/specs/security-sandboxing/PRODUCT.md @@ -0,0 +1,167 @@ +# Security and Sandboxing for Autonomous Oz Agents + +## Summary + +Enterprise customers running self-hosted Oz workers need clarity on the security and +sandboxing guarantees Oz provides out-of-the-box, and reliable mechanisms to restrict +what autonomous agents can reach on their internal infrastructure. Today the Docker and +Kubernetes backends provide process/filesystem isolation but expose no native controls +for network egress or filtering. This spec documents the current isolation boundary, +identifies enterprise gaps, and introduces two practical additions: a configurable Docker +network mode so task containers can be attached to a restricted or isolated network, and +first-class documentation of egress proxy configuration for both backends. + +## Background: What Oz Provides Today + +### Warp-hosted cloud agents (Namespace.so sandboxes) + +Each cloud agent run executes inside an ephemeral Namespace.so microVM. The sandbox +provides strong OS-level isolation: the agent cannot escape the VM boundary, cannot +address other tenant workloads, and cannot reach customer internal infrastructure unless +the customer explicitly exposes an endpoint. This model offers robust out-of-the-box +isolation but provides no mechanism for the agent to reach internal infrastructure at all, +which is why enterprise customers need self-hosted workers. + +### Self-hosted workers — Docker backend + +- **Process isolation:** Each task runs inside a distinct Docker container (one container + per task). The container lifecycle is tied to the task; containers are removed after + completion by default. +- **Filesystem isolation:** The container has its own ephemeral filesystem. Only volumes + explicitly mounted by the operator are accessible. +- **Resource isolation:** CPU and memory limits are applied when a runner instance shape + is configured (REMOTE-1936 / merged). Without a shape the container runs unconstrained. +- **Network isolation (current gap):** By default the container is attached to Docker's + default bridge network. This gives the agent unrestricted outbound connectivity from the + worker host, including access to internal infrastructure reachable from that host. No + egress filtering is applied by the worker; no allowlist or denylist of destinations is + enforced. + +### Self-hosted workers — Kubernetes backend + +- **Process isolation:** Each task runs as a Kubernetes Job (one pod per task). Pod and + container boundaries provide OS-level isolation. +- **Filesystem isolation:** Ephemeral `emptyDir` volumes; operator-mounted volumes only. +- **Namespace isolation:** The worker deploys task pods in a configured namespace. The + operator can separate task pods from other workloads via Kubernetes RBAC and namespace + boundaries. +- **Network isolation (operator-controlled):** Kubernetes NetworkPolicy resources can be + applied at the namespace level to restrict what task pods can reach. The worker's + `pod_template` field already accepts arbitrary PodSpec YAML, so operators can add pod + labels that a NetworkPolicy selector can match. NetworkPolicy enforcement depends on the + cluster's CNI plugin (Calico, Cilium, etc.). No NetworkPolicy is created by the worker + itself; the operator configures this entirely in their cluster. +- **Security contexts:** The Helm chart applies hardened defaults to the long-lived worker + Deployment pod: `runAsNonRoot`, `seccompProfile: RuntimeDefault`, dropped capabilities. + Task pods inherit defaults from the cluster unless the operator's `pod_template` + overrides them. + +### Self-hosted workers — Direct backend + +- **No container boundary:** Tasks run as a subprocess of the worker process on the host. + All host network access is available. No sandboxing is provided beyond process + isolation. +- **Direct backend is not appropriate for restricted-access scenarios** without the + operator layering their own OS-level controls (firewall rules, seccomp, etc.). + +## Problem + +An IMC enterprise evaluator asked whether Oz provides out-of-the-box network-level +security controls — network policies, filtering proxies — so autonomous agents cannot +freely reach internal infrastructure. The honest answer today is: we provide strong +process/filesystem isolation but not network-level egress filtering. Enterprise customers +must implement their own controls (Kubernetes NetworkPolicy, egress firewall rules, or a +filtering proxy). They need: + +1. **Clarity** on what is and is not provided today, so they can design their controls + correctly. +2. **A supported, documented egress proxy pattern** so agents can route all outbound + traffic through a filtering proxy the customer operates. +3. **Docker network isolation** — an explicit hook to attach task containers to a + restricted Docker network (instead of the default bridge) without requiring the + operator to patch entrypoints or iptables rules manually. + +## Goals + +- Document the current isolation boundary clearly in the README and the + self-hosted-worker customer guidance skill. +- Add a `network_mode` field to the Docker backend configuration so operators can specify + the Docker network name (or `none`) that task containers join. This is the practical + hook for connecting containers to a restricted network managed by the operator. +- Document the egress proxy pattern: passing `HTTP_PROXY`, `HTTPS_PROXY`, and `NO_PROXY` + to task containers via the existing environment variable injection mechanism. +- Document the Kubernetes NetworkPolicy pattern using the existing `pod_template` hook. + +## Non-goals + +- Warp does not implement a built-in filtering proxy, firewall ruleset, or content + inspection layer. These belong to customer infrastructure. +- Warp-hosted sandbox network restrictions are out of scope; customers who need to reach + internal infra already use self-hosted workers. +- No Oz server changes are required; all changes are in `oz-agent-worker`. +- The Direct backend is out of scope for network isolation; it has no container boundary + the worker can configure. + +## Behavior + +### Docker network mode + +1. The Docker backend config accepts an optional `network_mode` string. When set, the + task container's `HostConfig.NetworkMode` is set to that value. +2. Accepted values are any value Docker's `--network` flag accepts: a named network + (e.g. `restricted-net`, `host`, `none`), or the empty string (default bridge). +3. When `network_mode` is absent or empty, behavior is identical to today (Docker's + default bridge network). +4. Setting `network_mode: none` disables all networking for the task container. The + agent runtime can still reach the Warp control plane through the worker's own network + path; task-internal tool calls that require internet access will fail, which is the + intended behavior for maximum isolation. +5. Operators who want egress-filtered access (rather than no access) create a Docker + network with appropriate iptables or firewall rules and set `network_mode` to that + network's name. The worker does not manage the network itself. + +### Egress proxy + +6. Task containers respect the standard `HTTP_PROXY`, `HTTPS_PROXY`, and `NO_PROXY` + environment variables. Operators inject these via the existing `backend.docker.environment` + config entries or Kubernetes `pod_template` `env` entries. +7. A filtering proxy the operator operates receives all outbound HTTP/HTTPS traffic from + the agent and can enforce an allowlist or denylist of destinations. +8. `NO_PROXY` should include the Warp control plane endpoints (`oz.warp.dev`, + `app.warp.dev`) so the worker's own WebSocket and the agent's task-status calls do + not route through the proxy. +9. The worker does not validate or enforce proxy settings; responsibility lies with the + operator. This is documented explicitly so customers do not assume the worker + intercepts non-proxy-aware traffic. + +### Kubernetes NetworkPolicy + +10. Task pods are labeled with `oz-task-id` and `oz-worker-id` by the worker. Operators + can write NetworkPolicy resources that select on these labels to restrict task pod + egress to a set of allowed destinations. +11. No NetworkPolicy is created by the worker. Operators are responsible for creating and + maintaining NetworkPolicy resources that match their trust requirements. +12. The `pod_template` mechanism already supports adding arbitrary pod metadata (labels, + annotations) that NetworkPolicy selectors and admission controllers can consume. + Operators can also add pod-level security settings (securityContext, seccompProfile, + capabilities) via `pod_template`. + +### Documentation + +13. The README security section documents: + - The isolation model per backend (Docker, Kubernetes, Direct). + - The `network_mode` option for Docker. + - The egress proxy pattern. + - The Kubernetes NetworkPolicy guidance. + - An explicit statement that the Direct backend provides no network isolation. +14. The self-hosted-worker customer guidance skill's `secrets-and-security.md` reference + is updated to match. + +## Why not build a built-in egress filter? + +A built-in filtering layer in the worker (e.g. a sidecar transparent proxy enforced by +iptables) would require elevated capabilities (`NET_ADMIN`), is hard to keep correct +across Linux kernel versions, and duplicates tools that enterprise operators already run +(Zscaler, Squid, Cilium Egress Gateway). The proxy pattern and Kubernetes NetworkPolicy +are lower-risk, more auditable, and compose with existing enterprise controls. We document +how to use them with Oz rather than reimplementing them. diff --git a/specs/security-sandboxing/TECH.md b/specs/security-sandboxing/TECH.md new file mode 100644 index 0000000..a61d1f2 --- /dev/null +++ b/specs/security-sandboxing/TECH.md @@ -0,0 +1,154 @@ +# TECH: Security and Sandboxing for Autonomous Oz Agents + +## Context + +See `PRODUCT.md` for behavior. This spec covers the implementation changes needed to +deliver Docker network isolation and the documentation updates. + +The only code change is adding a `network_mode` field to the Docker backend config and +threading it into `ContainerCreate`. All other deliverables are documentation. + +Relevant code — `oz-agent-worker`: + +- [`internal/config/config.go (40-43)`](../internal/config/config.go) — `DockerConfig`: + `Volumes []string`, `Environment []EnvEntry`. The new `NetworkMode string` field lands + here. +- [`internal/worker/docker.go (26-30)`](../internal/worker/docker.go) — + `DockerBackendConfig`: mirrors the config struct as a runtime value; receives + `NetworkMode string`. +- [`internal/worker/docker.go (139-142)`](../internal/worker/docker.go) — + `ExecuteTask`: builds `container.HostConfig` with `Binds` and `Resources`. The + `NetworkMode` field is set here. +- [`main.go`](../main.go) — reads config file and CLI flags; `DockerConfig` is + converted into `DockerBackendConfig`. + +## Proposed changes + +### 1. Config struct (`internal/config/config.go`) + +Add `NetworkMode` to `DockerConfig`: + +```go path=null start=null +// DockerConfig holds Docker-backend-specific configuration. +type DockerConfig struct { + Volumes []string `yaml:"volumes"` + Environment []EnvEntry `yaml:"environment" validate:"dive"` + // NetworkMode sets the Docker network the task container joins. + // Accepts any value Docker's --network flag accepts: a named network + // (e.g. "restricted-net"), "none" (no networking), "host", or empty + // string (Docker's default bridge network). + NetworkMode string `yaml:"network_mode"` +} +``` + +No validation tag is needed beyond the existing string zero-value semantics: empty string +means "use Docker default", which is the correct backward-compatible behavior. + +### 2. Runtime config struct (`internal/worker/docker.go`) + +Add `NetworkMode string` to `DockerBackendConfig`: + +```go path=null start=null +// DockerBackendConfig holds configuration specific to the Docker backend. +type DockerBackendConfig struct { + NoCleanup bool + Volumes []string + Env map[string]string + NetworkMode string // Docker network name, "none", "host", or "" (default bridge) +} +``` + +### 3. Wire from config to runtime (`main.go` or equivalent mapping) + +The `DockerConfig` → `DockerBackendConfig` mapping already copies `Volumes` and +`Environment`. Add: + +```go path=null start=null +NetworkMode: cfg.Backend.Docker.NetworkMode, +``` + +### 4. Apply in `ExecuteTask` (`internal/worker/docker.go`) + +In the `HostConfig` construction inside `ExecuteTask`, set `NetworkMode` when the config +specifies one: + +```go path=null start=null +hostConfig := &container.HostConfig{ + Binds: binds, + Resources: dockerResourcesForShape(params.InstanceShape), + NetworkMode: container.NetworkMode(b.config.NetworkMode), +} +``` + +`container.NetworkMode` is a `string` typedef already in +`github.com/moby/moby/api/types/container`. When the string is empty, Docker uses its +default bridge behavior — identical to today. No conditional is needed. + +### 5. README: Security section + +Add a new "Security and Sandboxing" section to `README.md` between the existing +"Environment Variables for Task Containers" and "Docker Connectivity" sections, covering: + +- Isolation model per backend (Docker, Kubernetes, Direct). +- The `backend.docker.network_mode` config option with examples. +- The egress proxy pattern (env vars). +- Kubernetes NetworkPolicy guidance with the existing `pod_template` hook. +- Task pod labels (`oz-task-id`, `oz-worker-id`) available for NetworkPolicy selectors. +- A clear statement that the Direct backend provides no network isolation. + +### 6. Skill reference update +(`warp-server/.agents/skills/self-hosted-worker-customer-guidance/references/secrets-and-security.md`) + +Extend the "Network model" and "Practical hardening recommendations" sections to cover: + +- `network_mode` for Docker. +- Egress proxy injection pattern. +- Kubernetes NetworkPolicy pattern with task pod labels. +- Note that `NO_PROXY` should exclude Warp control plane hostnames. + +## Testing and validation + +- **Unit: `TestDockerNetworkMode`** (`internal/worker/docker_test.go`) — assert that + when `DockerBackendConfig.NetworkMode` is `"none"`, the built `HostConfig` carries + `NetworkMode: "none"`; when empty, `NetworkMode` is `""` (Docker default). Use the + existing mock Docker client pattern. +- **Config round-trip** (`internal/config/config_test.go`) — parse a YAML snippet with + `backend.docker.network_mode: restricted-net` and assert the field is preserved; parse + without the field and assert the zero value. +- **Manual smoke test (Docker backend):** Run a task with `network_mode: none` and + confirm the agent container cannot reach external addresses (e.g. `curl https://example.com` + fails); confirm the task still completes its internal work and the worker receives + task-completed over its own WebSocket. +- Run `go test ./...` in `oz-agent-worker` and `helm template` to confirm no regressions. + +## Risks and mitigations + +- **`network_mode: host`** gives the task container full host network access, which is + less restrictive than the default bridge. This is a valid user choice (e.g. for on-prem + access) but should be documented clearly. *Mitigation:* README example shows `none` and + a named network; explicitly notes that `host` expands rather than restricts access. +- **Proxy bypass by non-proxy-aware tools.** `HTTP_PROXY` / `HTTPS_PROXY` are honored by + most Go, Python, and Node.js HTTP clients but not by raw TCP connections or DNS-based + tools. A filtering proxy catches HTTP/HTTPS but not all protocols. *Mitigation:* + Document the limitation explicitly so operators layer network-level controls + (NetworkPolicy, iptables) if full protocol coverage is required. +- **Kubernetes NetworkPolicy is CNI-dependent.** Clusters running the default kubenet or + flannel CNI may not enforce NetworkPolicy resources. *Mitigation:* README notes that + NetworkPolicy enforcement requires a CNI plugin that implements it (Calico, Cilium, + Weave, etc.). +- **`container.NetworkMode` is a string typedef.** An invalid network name is rejected by + the Docker daemon at container creation time, not at config load time. The task will + fail at the `ContainerCreate` step with a clear error message. *Mitigation:* document + that the value is passed to Docker verbatim and any error surfaces as a task failure; + no worker-side validation is added to preserve flexibility. + +## Follow-ups + +- Consider exposing `network_mode` as a per-runner override (parallel to how the Docker + image is runner-scoped) so different runner configurations can apply different network + policies without restarting the worker. +- Consider a `dns` config option for the Docker backend to allow pointing task containers + at a split-horizon DNS resolver for internal service discovery. +- Evaluate Cilium Egress Gateway or similar CNI-level egress controls as a documented + pattern for Kubernetes customers who need IP-level egress filtering without an HTTP + proxy.