Add llama-bench-style benchmark: llama-tornado --bench (pp/tg/pg matrix, avg±stddev t/s, md/csv/json)

[mikepapadim]

llama-tornado --bench — a llama-bench-style benchmark for GPULlama3

GPULlama3 equivalent of llama.cpp's llama-bench: a reproducible, matrix-style performance benchmark for the GPU forward path.

What it does

• Test matrix over one or more models: prompt processing (pp N), token generation (tg N), combined (pg pp,tg) — every combination of the values given.
• Methodology parity with llama-bench: untimed warmup rep, -r timed repetitions, reported as avg t/s ± stddev; timings cover the forward pass only — no tokenization, no sampling, no host argmax.
• Output: markdown table (default), CSV, JSON (per-rep samples included).
• Multiple models benchmarked in one JVM (plan freed + next model loaded).

Usage

# defaults: -p 512 -n 128 -r 5, markdown
llama-tornado --gpu --cuda --model model1.gguf --bench

# multi-model matrix, JSON
llama-tornado --gpu --cuda --model model1.gguf --bench \
    --bench-args "-m model2.gguf -p 256,512 -n 64,128 -pg 512,128 -r 5 -o json"

Example (RTX 4090, TornadoVM 5.0.1-jdk21-dev CUDA backend, r=5)

model	quant	size	params	backend	test	t/s
beehive-llama-3.2-1b-instruct-fp16	FP16	2.31 GiB	1.24 B	TornadoVM CUDA	pp512	86.62 ± 2.18
beehive-llama-3.2-1b-instruct-fp16	FP16	2.31 GiB	1.24 B	TornadoVM CUDA	tg128	101.72 ± 0.43
Qwen3-1.7B-f16	FP16	3.79 GiB	2.03 B	TornadoVM CUDA	pp512	57.29 ± 0.18
Qwen3-1.7B-f16	FP16	3.79 GiB	2.03 B	TornadoVM CUDA	tg128	57.71 ± 0.10

(t/s here is higher than end-to-end achieved tok/s because tokenization/sampling/echo are excluded — same reason llama-bench numbers exceed llama-cli's.)

Files

• bench/LlamaBench.java — self-contained bench main (arg parsing, matrix, stats, 3 output formats).
• llama-tornado — --bench / --bench-args flags (swaps the main class; inference path untouched).
• README.md — new "Benchmarking" section.

Stacked on #132 (Qwen3 RMS-norm race fix) so Qwen3 rows are meaningful; the two commits before the bench commit are that fix.

🤖 Generated with Claude Code

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[mikepapadim]

@orionpapadakis could you take a look? llama-bench-style benchmark for the GPU forward path (llama-tornado --bench) — pp/tg/pg matrix, avg±stddev t/s, md/csv/json. Stacked on #132.

Maybe, we should add this to CI?

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[mikepapadim]

Feature-parity pass against upstream llama-bench (commit 1abee4e):

Added (was missing, applicable to GPULlama3):

• -d, --n-depth — context depth: untimed KV prefill of d positions before each timed test; tests named tg128@d4096. Measured signal on Llama-1B: tg16 97.4 t/s vs tg16@d512 72.7 t/s (attention over deeper KV).
• -o jsonl and -o sql output formats (now all 5 llama-bench formats: md/csv/json/jsonl/sql).
• -oe <fmt> — second output format to stderr.
• --delay <s> — pause between tests (GPU thermals).

Already had: multi-model -m, -p/-n/-pg matrix, -r, warmup/--no-warmup, avg±stddev, per-rep samples, forward-only timing.

Not applicable to GPULlama3 (llama.cpp-specific), intentionally skipped: -hf download, -b/-ub batch sizes, -ctk/-ctv KV cache types, -t/-C/--poll/--numa CPU knobs, -ngl/-sm/-mg/-ts/-dev/-nkvo/-rpc offload/multi-GPU, -fa (flash attention always on), -mmp/-dio, -embd, -ot, -fitt/-fitc.

Note: use the --bench-args="..." (equals) form — argparse rejects a separate leading-dash value.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[mikepapadim]

Model-coverage benchmark sweep + prefill-batch support (-b)

Two updates on this PR (commits f630293, 818b780):

1. Added -b — batched prompt processing (llama-bench -b parity)

When -b > 1, the pp tokens run through GPULlama3's batched-prefill tensor-core (MMA) path — b tokens per forward, compute-bound — instead of single-token forwards; generation stays single-token decode. This is the vanilla MMA batch-prefill plan (TornadoVMMasterPlanBatchPrefillDecode), the same one --with-prefill-decode --batch-prefill-size uses. Models without a batched-prefill graph raise a clean UnsupportedOperationException that the bench catches and reports (no crash).

2. Benchmarked the full local model zoo (RTX 4090, TornadoVM 5.0.1-jdk21-dev CUDA backend, r=3)

Single-token forward (pp128 / tg64, t/s):

model	quant	size	params	pp128	tg64
Llama-3.2-1B	FP16	2.31 GiB	1.24 B	107.70 ± 0.86	109.60 ± 1.55
Qwen3-1.7B	FP16	3.79 GiB	2.03 B	54.28 ± 2.56	53.30 ± 0.09
Qwen3-4B	FP16	7.50 GiB	4.02 B	33.10 ± 0.98	35.45 ± 0.06
Mistral-7B-v0.3	FP16	13.5 GiB	7.25 B	21.14 ± 0.04	24.92 ± 0.63
Llama-3.2-3B	Q8_0	3.18 GiB	3.4 B	56.45 ± 0.32	58.04 ± 2.08
Granite-3.3-2B	Q8_0	2.51 GiB	2.5 B	40.69 ± 0.15	42.71 ± 0.77

Batched prompt processing (pp512 -b, tensor-core MMA prefill):

model	test	t/s	vs single-token pp
Llama-3.2-1B	pp512 b128	4125.81 ± 42.46	~47×
Llama-3.2-1B	pp512 b32	1194.06 ± 17.59	~14×
Qwen3-1.7B	pp512 b32	719.23 ± 8.00	~13×
Qwen3-4B	pp512 b32	383.53 ± 2.01	~11×
Mistral-7B	pp512 b32	—	BATCH_PREFILL_DECODE not yet supported for MISTRAL + F16 (graceful skip)

Findings

• Single-token decode is memory-bound: t/s tracks 1/size (1B FP16 ~108, 7B FP16 ~25); pp≈tg per model (both single-token matvecs).
• Batched pp is the tensor-core win: turning prefill into a compute-bound GEMM gives ~47× on Llama-1B at b128 — this is what makes GPULlama3's prompt processing competitive, and it's now measurable in the bench.
• Batched prefill = Llama + Qwen3 only today (Mistral/Q8 fall back or are unsupported); the bench reports this cleanly rather than crashing — useful signal for where to extend the MMA path next.
• Q8_0 models run fine on the single-token path (Llama-3B-Q8 56/58, Granite-2B-Q8 41/43).

Full feature parity list vs upstream llama-bench in the earlier comment. @orionpapadakis — review welcome; happy to add more models or wire batched pp for Mistral/Q8 if useful.