OpenAI-compatible server (llama-tornado --server): /v1/chat/completions + /v1/completions, streaming, zero-dep

[mikepapadim]

OpenAI-compatible server (llama-tornado --server)

Serve any GPULlama3 model behind the HTTP API OpenAI clients already speak — zero external dependencies (JDK HttpServer + a hand-rolled JSON codec), streaming and non-streaming.

llama-tornado --gpu --cuda --model model.gguf --server --port 8080

Endpoints

• POST /v1/chat/completions — chat, SSE streaming ("stream":true) or full JSON; usage token counts, finish_reason.
• POST /v1/completions — text completion (prompt as a single user turn).
• GET /v1/models · GET /health.

Any OpenAI SDK works by pointing base_url at it. Errors use the OpenAI error shape (400 bad JSON / missing field, 405 wrong method, 500 generation).

Design (reusable)

• server/InferenceService — the reuse seam: holds one State + one compile-once TornadoVMMasterPlan, serializes generation on the single GPU context, streams decoded tokens through a callback, returns full text + token counts. KV cache is overwritten from position 0 each request (state reused, no per-request reallocation; the model's start token is captured at construction and restored per request).
• server/OpenAIServer — HTTP layer: multi-threaded accept (clients queue cleanly), SSE writer, OpenAI request/response mapping.
• server/Json — dependency-free recursive-descent JSON reader/writer.

Tested — scripts/server-smoke-test.sh (11/11) on RTX 4090, Llama-3.2-1B

• chat + completions return coherent output ("Paris", "The capital of France is Paris."), correct usage.
• SSE: role chunk → content deltas → [DONE].
• Sequential 3 different prompts → Tokyo / Rome / 4 (no KV bleed between requests).
• Concurrency 5 parallel requests all succeed (serialized on the GPU).
• Errors: 400 bad JSON, 400 missing messages, 405 GET.

Scope / follow-up

Single-request-at-a-time over the proven single-token decode path — correct and drop-in for all supported models (Llama/Mistral/Qwen3/…). Wiring it onto the continuous batched-decode engine (PR #129) for concurrent-request throughput is the natural v2; the InferenceService seam is where that swaps in.

@orionpapadakis — standalone off feat/mma_cuda, review welcome.

🤖 Generated with Claude Code

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[mikepapadim]

@orionpapadakis OpenAI-compatible server off feat/mma_cuda — zero-dep JDK HttpServer, /v1/chat/completions + /v1/completions (SSE streaming + full), /v1/models, /health. Reusable InferenceService seam over one GPU context. 11/11 smoke tests (coherence, streaming, sequential no-KV-bleed, 5× concurrency, error codes) on Llama-1B. Next: wire onto the continuous batched engine (#129) for concurrent throughput.