Hybrid CUDA library tasks for single-token decode: parity, not a win — measured findings + llama.logitsLib switch

[mikepapadim]

Can the hybrid CUDA library API speed up single-token decode? — findings + reproducible switch

Explores whether the TornadoVM hybrid library tasks (cuBLAS / cuBLASLt, riding the task graph on the CUDA backend) can improve single-request, single-token decode. Companion to #127 (MMA batch prefill) and #129 (batched decode), which are where these libraries do win.

Answer: no — and this PR documents exactly why, with a switch anyone can re-run.

What's in the PR (additive, default behavior unchanged)

• -Dllama.logitsLib={jit,gemmEx,lt}: the logits vocabulary projection (biggest single matvec, ~23% of decode bytes on Llama-1B) can be swapped between the stock JIT matvec, a CuBlas::cublasGemmExFP16FP32 library task, and a CuBlasLt::ltMatmulFP16 library task (+1.2 µs FP16→FP32 copy). Default jit.
• tornado-cublas dependency + tornado.cublas module on the CUDA launch path.
• HYBRID_DECODE_LIBS.md: full numbers + analysis.

Key numbers (RTX 4090, TornadoVM 5.0.1-jdk21-dev CUDA backend)

Per-kernel (nsys, logits GEMV 128256×2048 FP16, avg of 101 calls):

kernel	µs/call	effective BW
JIT matrixVectorGeneric	553.8	~948 GB/s (94% peak)
cuBLAS gemvx (gemmEx)	550.6	~953 GB/s
cuBLASLt gemvx	550.9	~953 GB/s

End-to-end (3 runs, tok/s): Llama-1B jit 88.7–91.3 vs lt 88.3–90.5 (graphs off); 98.1–102.4 vs 99.9–102.2 (graphs on). Mistral-7B: 23.5 vs 23.7. Statistical ties; output coherent on every config.

Why (the useful part)

1. Bandwidth wall: n=1 projections have arithmetic intensity ~1; JIT already streams weights at 94% of peak — cuBLAS/cuBLASLt land on the same wall (their GEMV-specialized gemvx kernel is 0.6% faster: parity).
2. Launch-bound regime: GPU busy ~3.6 ms/token vs ~11.2 ms wall (~33% util, ~65 µs host per launch). TornadoVM's fused kernels (fusedRmsNormFFNGateUp, fusedQKVMatmulX, matvec+residual) exist to remove launches — library tasks would de-fuse them and add launches for ~13 µs/layer of GPU savings: net loss.
3. The actual single-token levers, both already in-tree: CUDA graphs (--cuda-graphs, +8–12% on every config) and kernel fusion.
4. Hybrid libraries win from n≳16, when the same projections become compute-bound tensor-core GEMMs: batch prefill (Add TornadoVM CUDA backend support and tensor-core (MMA) accelerated batch prefill (FP16 & Q8_0, Llama & Qwen3) #127) and batched/continuous decode (Static batched decode (LLaMA + Qwen3): B independent sequences per step, up to 41x aggregate throughput #129).

The library integration itself checks out: correct results, same CUstream as JIT kernels, CUDA-graph-capturable.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.