feat: Pyodide Python runtime via OS-isolated Deno subprocess

.changeset/python-runtime-enum.md

@@ -0,0 +1,21 @@
+---
+"sql-fs-api": major
+---
+
+**Breaking:** replace the boolean `python` sandbox capability with a nullable `python_runtime` enum (`"stdlib" | "pyodide" | null`) across the HTTP API, MCP tools, OpenAPI spec, and both SDKs.
+
+- `POST /v1/sandboxes` now accepts `python_runtime` instead of `python`; the legacy `python` key is rejected with `400 INVALID_INPUT`. Create/get/list responses echo `python_runtime` (and now consistently include `network`).
+- MCP `sandbox_create` takes `python_runtime` and `sandbox_list` echoes it.
+- TypeScript SDK (`SandboxRecord`, `CreateSandboxOptions`) and Python SDK (`SandboxRecord`, `client.create(...)`) use `python_runtime`; both `SandboxRecord` types now also expose `network`.
+
+`python_runtime: "stdlib"` is the air-gapped CPython-WASM runtime (the previous `python: true`). `python_runtime: "pyodide"` adds a numpy/pandas/scipy/openpyxl runtime in an OS-isolated Deno subprocess. The DB layer migrates rolling-deploy-safe (migration 0006 dual-writes/back-reads the legacy column).
+
+Clients must migrate `python: true` → `python_runtime: "stdlib"` and `python: false` → omit (or `null`).
+
+**Pyodide runtime memory & throughput tuning** (part of the same unreleased feature):
+
+- Cut combined runtime memory ~1.6 GB → ~1.1 GB on an 18.8 MB-CSV workload: O(n) chunk-list IPC framing (replaces an O(n²) per-chunk reallocation on both the Deno runner and the Node manager), and a streamed SHA-256 diff baseline (replaces holding every staged file's bytes for the whole run).
+- Lazy, offline package loading: only `PYODIDE_PRELOAD_PACKAGES` (default `numpy,pandas`) is resident at init; other distribution packages load on first import from the local lock. Lowers the idle floor and lets operators trade latency against RSS.
+- `matplotlib` now defaults to the headless `Agg` backend so `savefig()` works in the Deno child (previously failed resolving the DOM `webagg` backend).
+- New env vars `PYODIDE_PRELOAD_PACKAGES` and `PYODIDE_MAX_CHILD_RSS_BYTES` (optional RSS-based child retirement).
+- Raised the default IPC frame/aggregate caps (192 MiB / 256 MiB) so a monolithic drain response carrying the full `PYODIDE_MAX_TOTAL_BYTES` (128 MiB, base64-expanded) is reachable instead of being killed at the old 64 MiB frame cap, and wired the previously-documented-but-unused `PYODIDE_MAX_FRAME_BYTES` / `PYODIDE_MAX_AGGREGATE_BYTES` env overrides.

.dockerignore

@@ -8,3 +8,8 @@ dist
 tasks/
 .dockerignore
 Dockerfile
+# Vendored Pyodide assets are regenerated fresh in the builder stage; never copy
+# a (possibly stale) local copy into the build context.
+vendor
+# Design records + ~408 MB of spike scratch assets — not part of the image.
+thoughts

.env.example

@@ -21,6 +21,10 @@ PORT=8080
 SESSION_IDLE_MS=600000          # evict idle sessions after 10 min
 MAX_CONCURRENT_PYTHON=5         # cap CPython WASM workers (~80 MB each)
 MAX_CONCURRENT_JS=5             # cap QuickJS workers (~64 MB each)
+# MAX_CONCURRENT_PYODIDE=2       # cap concurrent Deno/Pyodide workers
+# MAX_RESIDENT_PYODIDE=2         # cap warm resident workers; must be >= concurrent
+# PYODIDE_PRELOAD_PACKAGES=numpy,pandas
+# PYODIDE_MAX_CHILD_RSS_BYTES=0  # retire after a run above this RSS; 0 disables
 
 # ── Optional: Redis (required for multi-replica deployments) ──────────────────
 

.gitignore

@@ -6,3 +6,9 @@ dist/
 *.tsbuildinfo
 .claude/worktrees/
 docs/
+docker-compose.local.yml
+.tmp/
+
+# Vendored Pyodide runtime assets — reproduced by scripts/fetch-pyodide-assets.mjs
+# (pinned Deno binary + Pyodide distribution + wheels + download cache). Never committed.
+vendor/

CLAUDE.md

@@ -207,6 +207,20 @@ const TABLE = Object.assign(Object.create(null) as Record<string, string>, {
 | `SESSION_IDLE_MS` | No (default: 600000) | Idle timeout before session eviction (ms) |
 | `MAX_CONCURRENT_PYTHON` | No (default: 5) | Max concurrent Python executions across all sessions. CPython WASM workers cost ~80MB each (EXIT_RUNTIME per invocation); the semaphore caps concurrency to prevent OOM. Excess scripts queue FIFO. |
 | `MAX_CONCURRENT_JS` | No (default: 5) | Max concurrent JavaScript (`js-exec`/`node`) executions across all sessions. QuickJS executions cap at 64MB each. Excess scripts queue FIFO. Note: just-bash currently serializes `js-exec` internally through a single worker, so this cap is an upper bound that may not be binding today. |
+| `MAX_CONCURRENT_PYODIDE` | No (default: 2) | Max concurrent `pyodide` execs (OS-isolated Deno subprocesses, numpy/pandas/scipy/openpyxl) across all sessions. Routed independently of `MAX_CONCURRENT_PYTHON` (which stays `stdlib`-only). Each child loads a ~2 GB-ceiling WASM heap — keep this low. Excess scripts queue FIFO. |
+| `MAX_PYODIDE_QUEUE` | No (default: 100) | Max queued `pyodide` execs waiting on the concurrency semaphore. Beyond this, new execs fail fast with `ERUNTIME_BUSY` (503). |
+| `PYODIDE_QUEUE_TIMEOUT_MS` | No (default: 60000) | Max time a `pyodide` exec waits in the semaphore queue before failing with `ERUNTIME_BUSY` (503). |
+| `MAX_RESIDENT_PYODIDE` | No (default: 2) | Cap on **resident** `pyodide` subprocesses (residency LRU), independent of `SESSION_IDLE_MS` — bounds total live Deno children so warm subprocesses don't accumulate per active session. MUST be `>= MAX_CONCURRENT_PYODIDE` (a busy worker is never evictable); the server fails to boot otherwise. Eviction/idle-kill targets only idle workers; an evicted session cold-starts a fresh child on its next exec. |
+| `PYODIDE_IDLE_MS` | No (default: 120000) | Idle window before a resident `pyodide` subprocess is idle-killed by the residency sweep (ms). Should be `< SESSION_IDLE_MS` so warm children are reclaimed well before their session is evicted. |
+| `PYODIDE_RUNTIME_TIMEOUT_MS` | No (default: 60000) | Cap on a single owned `pyodide` run (Deno spawn + Pyodide init + execution). On timeout the child is SIGKILLed (generation retired) and the exec throws (script transaction rolls back). Cold start (numpy/pandas/scipy/openpyxl load) takes several seconds — size accordingly. |
+| `PYODIDE_PRELOAD_PACKAGES` | No (default: `numpy,pandas`) | Comma-separated packages loaded at child init, trading cold-start latency against resident RSS. Packages NOT preloaded are loaded on demand from the same offline lock on first import (`loadPackagesFromImports`), so the resident floor only pays for what's preloaded. Set empty for no stock-package preload (smallest baseline). |
+| `PYODIDE_MAX_CHILD_RSS_BYTES` | No (default: 0 = disabled) | Retire a `pyodide` child after a completed run when its RSS exceeds this many bytes (Linux `/proc/<pid>/status`, else `ps`); the next exec cold-starts a fresh generation. Bounds the grow-only WASM/V8 heap's idle retention — it does NOT prevent the in-flight peak. The container memory limit remains the hard guard. |
+| `PYODIDE_MAX_FILE_BYTES` | No (default: 33554432) | Per-file cap (bytes, 32 MiB) on files staged into / drained out of a `pyodide` exec. |
+| `PYODIDE_MAX_TOTAL_BYTES` | No (default: 134217728) | Total cap (bytes, 128 MiB) across all files staged into / drained out of a single `pyodide` exec. |
+| `PYODIDE_MAX_FRAME_BYTES` | No (default: per-frame IPC cap) | Max size (bytes) of a single IPC frame on the Node↔Deno channel, measured on the base64 wire size. Oversized frames kill the child. |
+| `PYODIDE_MAX_AGGREGATE_BYTES` | No (default: aggregate IPC cap) | Max aggregate response bytes per `pyodide` run on the IPC channel. Exceeding it kills the child (slowloris guard). |
+| `PYODIDE_ASSET_DIR` | `pyodide` runtime | Absolute path to the vendored Pyodide asset dir (wasm + stdlib + wheels + custom lock). Resolved by Node and passed to the Deno child as `--allow-read=<dir>` + argv (never via the child's scrubbed env). Set in the Docker image. |
+| `DENO_BIN_PATH` | No (default: `deno`) | Path to the vendored Deno binary used to spawn the `pyodide` runner. Set in the Docker image. |
 | `REDIS_URL` | No | Redis connection string. Required for multi-replica deployments. When absent, distributed exec lock and all Redis caches are disabled — only in-process `session.mutex` protects execution. |
 | `REDIS_EXEC_LOCK_LEASE_MS` | No (default: 60000) | Distributed exec lock lease duration (ms). Lock auto-expires if the holder dies. Must be > `REDIS_EXEC_LOCK_RENEW_MS`. |
 | `REDIS_EXEC_LOCK_RENEW_MS` | No (default: 20000) | Heartbeat interval for exec lock renewal (ms). Must be strictly less than `REDIS_EXEC_LOCK_LEASE_MS` to guarantee renewal fires before expiry. |
@@ -222,6 +236,12 @@ const TABLE = Object.assign(Object.create(null) as Record<string, string>, {
 | `JUST_BASH_DEFENSE_AUDIT_MODE` | No (default: `true`) | When `JUST_BASH_DEFENSE_IN_DEPTH=true`, controls whether violations throw (`false`) or are logged only (`true`). Recommended `true` for initial rollout, then flip to `false` once logs are clean. |
 | `BLOB_GC_MIN_AGE_MS` | No (default: 10800000) | Grace window (ms, default 3h) before an orphan blob becomes collectible by `pnpm db:gc`. Orphans whose `last_referenced_at` is newer than this are kept; the `ON CONFLICT DO UPDATE` row lock on blob writes is the actual dedup re-adoption race guard — this window is churn/margin control. Rows with NULL `last_referenced_at` (legacy, pre-migration-0006) are treated as ancient and always collectible. Override per-run with `--min-age-ms`. |
 
+### Pyodide memory posture (accepted availability risk — design D5, spike S3)
+
+There is **no per-child OOM isolation**. Spike S3 proved that on `node:22-slim` as the non-root `app` user neither lever works: the cgroup v2 hierarchy is mounted read-only (`memory.max` is unwritable) and `RLIMIT_AS` is unusable because a 2 GiB-max WASM heap reserves ~10.7 GB of *virtual* address space (an `RLIMIT_AS` low enough to bound RSS makes the WASM allocation fail). The **operator-set container memory limit is the only real guard** — it covers Node plus *all* Deno children together, so a runaway child may OOM-kill the whole container.
+
+Size the container limit as **`MAX_RESIDENT_PYODIDE × per-process ceiling` + Node baseline + DB/Redis pools** (the Pyodide ~2 GB WASM cap is the per-instance heap ceiling). Use `MAX_RESIDENT_PYODIDE=1` on small hosts. On OOM-kill the child exits and the manager respawns a fresh generation on the next exec.
+
 ## File Layout
 
 ```

Dockerfile

@@ -4,6 +4,10 @@
 FROM node:22-bookworm AS builder
 WORKDIR /app
 RUN corepack enable
+# curl/unzip/bzip2 are needed by scripts/fetch-pyodide-assets.mjs (curl download,
+# unzip the Deno binary, tar -xj the Pyodide .tar.bz2).
+RUN apt-get update && apt-get install -y --no-install-recommends curl unzip bzip2 \
+  && rm -rf /var/lib/apt/lists/*
 COPY pnpm-lock.yaml package.json ./
 # Remove the `prepare` lifecycle hook before install. lefthook install panics
 # in Docker Desktop's Linux VM (Go taggedPointerPack runtime bug). Git hooks
@@ -12,16 +16,26 @@ RUN node -e "const fs=require('fs'),p=JSON.parse(fs.readFileSync('package.json',
 RUN pnpm install --frozen-lockfile
 COPY . .
 RUN pnpm build
+# Vendor the Pyodide runtime assets (pinned Deno binary + Pyodide distribution +
+# wheels) and build the supplementary custom lock. Reproduced fresh here — never
+# committed (see .gitignore / .dockerignore).
+RUN node scripts/fetch-pyodide-assets.mjs && node scripts/build-pyodide-lock.mjs
 
 # ------- Runtime -------
 FROM node:22-slim AS runtime
 ENV NODE_ENV=production
 ENV PORT=8080
+# Pyodide runtime: Node resolves these before spawning the Deno child (Phase 4).
+ENV PYODIDE_ASSET_DIR=/app/vendor/pyodide
+ENV DENO_BIN_PATH=/app/vendor/deno/deno
 WORKDIR /app
 RUN groupadd -r app && useradd -r -g app app
 COPY --from=builder --chown=app:app /app/dist ./dist
 COPY --from=builder --chown=app:app /app/node_modules ./node_modules
 COPY --from=builder --chown=app:app /app/package.json ./package.json
+# dist/pyodide-runner (raw .ts, copied by the build) ships inside ./dist above;
+# vendor/ carries the Deno binary + Pyodide distribution + custom lock.
+COPY --from=builder --chown=app:app /app/vendor ./vendor
 USER app
 EXPOSE 8080
 HEALTHCHECK --interval=15s --timeout=3s --start-period=10s \

README.md

@@ -121,6 +121,56 @@ await sandbox.delete();
 
 See [clients/typescript/README.md](clients/typescript/README.md) for the full TypeScript API.
 
+## Language runtimes
+
+Sandboxes are bash-only by default. Opt into a Python and/or JavaScript runtime at **creation** (it cannot be changed later).
+
+> ### ⚠️ Breaking change: `python` → `python_runtime`
+>
+> The boolean `python` create option has been **replaced** by the `python_runtime` enum.
+> There are now **two** Python runtimes:
+>
+> | Old (removed) | New |
+> |---|---|
+> | `python: true` | `python_runtime: "stdlib"` |
+> | `python: false` / omitted | omit `python_runtime` (no Python) |
+>
+> `SandboxRecord.python` (boolean) is likewise replaced by `SandboxRecord.python_runtime`
+> (`"stdlib" \| "pyodide" \| null`). Update SDK calls accordingly.
+
+```python
+# stdlib — lightweight CPython, stdlib only
+sb = client.sandboxes.create(python_runtime="stdlib")
+
+# pyodide — heavy, with the scientific stack
+sb = client.sandboxes.create(python_runtime="pyodide")
+
+# JavaScript (QuickJS) — unchanged
+sb = client.sandboxes.create(javascript=True)
+```
+
+### The two Python runtimes
+
+| | `stdlib` | `pyodide` |
+|---|---|---|
+| Engine | CPython → WASM (in-process worker) | Pyodide → WASM in an **OS-isolated Deno subprocess** |
+| Packages | Python **stdlib only** (no pip) | stdlib **+ offline Pyodide packages**; numpy/pandas preloaded, others loaded on first import |
+| Cold start | ~1.4 s | ~3–5 s |
+| Per-worker memory | ~80 MB | ~0.6 GB floor; ~1 GB+ with a real DataFrame |
+| File reads | **8 MiB cap** per `open()` (IPC bridge) — chunk larger files | no 8 MiB cap; bounded by `PYODIDE_MAX_FILE_BYTES` (32 MiB default) |
+| Use it for | quick scripts, glue, capability probes | CSV/Excel data analysis, plotting (charts drain back to the FS and are retrievable via `GET /files/...`) |
+
+The full Pyodide distribution (~380 packages) ships on disk. Imports are resolved from that offline lock on demand; `PYODIDE_PRELOAD_PACKAGES` controls which packages pay their memory and startup cost at child initialization. Matplotlib defaults to the headless `Agg` backend so `savefig()` works in the Deno child.
+
+### Sizing the Pyodide runtime
+
+Pyodide workers are the dominant memory cost (bash and the `stdlib` runtime are comparatively free). WASM memory does **not** shrink mid-run, so size for the worst-case single file:
+
+- **2 GB container, `MAX_CONCURRENT_PYODIDE=1`** → set `PYODIDE_MAX_FILE_BYTES=16777216` (**16 MiB**) for conservative headroom. The optimized transport measured ~0.84 GB child RSS and ~1.1 GB combined on an 18.8 MB CSV; host and workload shape still matter.
+- **`cap=2` doubles the memory** → use ~8 MiB files or a ~4 GB container.
+- A ~50 MB CSV needs the stage cap raised **and** ~3–4 GB; `chunksize`/dtype tricks do **not** help (WASM high-water + the fully-staged file dominate). Reduce columns/rows *before* upload instead.
+- The Pyodide runtime requires a vendored Deno binary + Pyodide assets (`scripts/fetch-pyodide-assets.mjs`, baked into the Docker image via `DENO_BIN_PATH` / `PYODIDE_ASSET_DIR`).
+
 ## How It Works
 
 > **Postgres is always the source of truth. Everything else is a cache or a lock.**
@@ -164,8 +214,16 @@ Key design choices:
 | `AUTH_SECRET` | Yes | — | Secret for Bearer token validation |
 | `PORT` | No | `8080` | HTTP server port |
 | `SESSION_IDLE_MS` | No | `600000` | Evict idle Bash instances after this many ms |
-| `MAX_CONCURRENT_PYTHON` | No | `5` | Cap on concurrent CPython WASM workers (~80 MB each) |
+| `MAX_CONCURRENT_PYTHON` | No | `5` | Cap on concurrent `stdlib` CPython WASM workers (~80 MB each) |
 | `MAX_CONCURRENT_JS` | No | `5` | Cap on concurrent QuickJS workers (~64 MB each) |
+| `MAX_CONCURRENT_PYODIDE` | No | `2` | Cap on concurrent `pyodide` workers (OS-isolated Deno; ~0.6–1 GB+ each). Keep low; **use `1` on a 2 GB host**. Routed separately from `MAX_CONCURRENT_PYTHON`. |
+| `MAX_RESIDENT_PYODIDE` | No | `2` | Resident `pyodide` workers (idle LRU). Must be `>=` `MAX_CONCURRENT_PYODIDE` or the server refuses to boot. |
+| `PYODIDE_PRELOAD_PACKAGES` | No | `numpy,pandas` | Comma-separated packages loaded at child initialization. Other packages from the vendored lock load on first import. Use an empty value for no stock-package preload. |
+| `PYODIDE_MAX_CHILD_RSS_BYTES` | No | `0` | Retire a child after a completed run when RSS exceeds this many bytes. `0` disables retirement; this bounds idle retention, not the in-flight peak. |
+| `PYODIDE_MAX_FILE_BYTES` | No | `33554432` | Per-file cap (32 MiB) on files staged into a `pyodide` exec. **Lower to `16777216` (16 MiB) on a 2 GB host.** |
+| `PYODIDE_MAX_TOTAL_BYTES` | No | `134217728` | Total staged bytes across one `pyodide` exec (128 MiB). |
+| `PYODIDE_ASSET_DIR` | `pyodide` runtime | — | Absolute path to vendored Pyodide assets (wasm + stdlib + wheels). Set in the Docker image. |
+| `DENO_BIN_PATH` | No | `deno` | Path to the vendored Deno binary that runs the `pyodide` runner. Set in the Docker image. |
 | `MAX_REQUEST_BODY_BYTES` | No | `268435456` | Hard cap on any HTTP request body (256 MB) — file write, bulk write, ingest. Applied before auth/handlers. Since base64 inflates content ~33%, this is usually the binding limit on ingest: ~190 MB of raw file bytes per call. |
 | `MAX_INGEST_BYTES` | No | `536870912` | Max total decoded bytes across one `ingest-files` manifest (512 MB). The request-body cap above normally trips first. |
 | `MAX_INGEST_FILES` | No | `10000` | Max number of entries (files + paths) in one `ingest-files` manifest. |

biome.json

@@ -22,6 +22,16 @@
 		"lineWidth": 120
 	},
 	"files": {
-		"ignore": ["node_modules", "dist", "*.sql", "scripts", "clients/python/.venv"]
+		"ignore": [
+			"node_modules",
+			"dist",
+			".tmp",
+			"*.sql",
+			"scripts",
+			"clients/python/.venv",
+			"thoughts",
+			"vendor",
+			"src/pyodide-runner/runner.ts"
+		]
 	}
 }

clients/python/CHANGELOG.md

@@ -5,6 +5,20 @@ All notable changes to the SQL-FS Python SDK are documented here.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.4.0] - 2026-06-08
+
+### Changed
+
+- **Breaking:** replaced the boolean `python` argument of `client.sandboxes.create()`
+  with `python_runtime` (`"stdlib" | "pyodide" | None`). `SandboxRecord.python` is now
+  `SandboxRecord.python_runtime`. Migrate `python=True` → `python_runtime="stdlib"` and
+  `python=False` → omit (or `None`).
+
+### Added
+
+- `SandboxRecord.network` is now surfaced (previously server-only).
+- Exported the `PythonRuntime` type alias.
+
 ## [0.3.1] - 2026-06-08
 
 ### Added

clients/python/README.md

@@ -22,7 +22,7 @@ pip install -e clients/python
 from sqlfs import Client
 
 with Client(base_url="https://api.example.com", auth_secret="<AUTH_SECRET>", sub="agent-001") as fs:
-    sb = fs.sandboxes.create(name="demo", python=True)
+    sb = fs.sandboxes.create(name="demo", python_runtime="stdlib")
 
     # Bash execution
     result = sb.exec("echo hello && ls /home/user")
@@ -81,7 +81,7 @@ fs = Client(base_url="...", auth_secret="...", sub="agent", max_file_size=0)
 | Method | HTTP | Notes |
 |---|---|---|
 | `client.sandboxes.list()` | `GET /v1/sandboxes` | → `list[SandboxRecord]` |
-| `client.sandboxes.create(name=, env=, files=, python=, javascript=)` | `POST /v1/sandboxes` | → `Sandbox` |
+| `client.sandboxes.create(name=, env=, files=, python_runtime=, javascript=, network=)` | `POST /v1/sandboxes` | → `Sandbox` |
 | `client.sandboxes.get(id)` | `GET /v1/sandboxes/{id}` | → `SandboxInfo` |
 | `client.sandboxes.attach(id)` | _(no network)_ | → `Sandbox` for an existing id |
 | `client.sandboxes.delete(id)` | `DELETE /v1/sandboxes/{id}` | |

clients/python/examples/quickstart.py

@@ -27,7 +27,7 @@ def main() -> int:
         token=token,
         sub="quickstart",
     ) as fs:
-        sb = fs.sandboxes.create(name="quickstart", python=False)
+        sb = fs.sandboxes.create(name="quickstart")
         print(f"created sandbox {sb.id}")
         try:
             # Bulk write — single round-trip for many files.