feat(rocmfp4): backend for the ROCmFP4/ROCmFPx 4-bit formats on AMD RDNA3.5 APUs (Strix Halo)

[walcz-de]

What this adds

A new backend, rocmfp4, carrying the ROCmFP4 / ROCmFPx 4-bit weight-quantization formats (ggml types 100–107) for AMD RDNA3.5 APUs — gfx1150/1151/1152/1153, i.e. Strix Point, Strix Halo, Gorgon Point and Gorgon Halo. Stock llama.cpp rejects these tensor types (unknown type q4_0_rocmfp4_fast), which is precisely why this is a separate backend rather than a flag on llama-cpp.

The formats originate in charlie12345/ROCmFPX (MIT). Because that tree predates the MMQ config refactor (#24127 in llama.cpp) by several weeks, the formats were ported onto a current llama.cpp base — including re-expressing the FP4 MMQ load-tile kernels against the new ggml_cuda_mmq_config/sram_layout architecture — and are maintained at walcz-de/llama.cpp-ROCmFP4, with copyright headers and attribution intact.

Published weights this serves today: kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF.

What the format buys — honestly

Memory and bandwidth, not arithmetic. There is no FP4 matrix instruction on RDNA3.5 or RDNA4 (only CDNA4 has one), so the matmul runs as int8 dot products. The win is ~22% smaller files than Q4_K_M at perplexity parity (5.8877 vs 5.8926 on wikitext-2 per the published card) — on unified-memory APUs that can decide whether a model fits at all.

Measured end-to-end on a Radeon 8060S (gfx1151, Strix Halo), 27B model, via this backend inside LocalAI:

	result
prefill pp128	317–335 t/s
decode tg32	14.0–14.3 t/s
decode with MTP draft head (spec_type:draft-mtp, n_max 4)	30.4 t/s (2.13×)
MTP losslessness	identical routing accuracy (43/60) and coding pass rate (7/8) with and without MTP

llama-cpp stays the recommendation for everything else; the backend description in index.yaml says so explicitly (same tone as the vllm-cpp entry).

Non-displacement, by design

• Deliberately NOT in engineNamePreferenceRules — an AMD entry would make the gallery prefer FP4 variants on every AMD host. Per the adding-backends guide, leaving it out ranks it below every known engine, which is the intent for a niche format.
• Importer is preference-only (extends the existing llama-cpp drop-in list, like ik-llama-cpp/turboquant): ROCmFP4 GGUFs carry ordinary .gguf names, so there is no safe auto-detect signal. Without an explicit preference an import stays llama-cpp.
• Verified at runtime: with rocmfp4 installed, a plain Q4_K_M GGUF without backend: still auto-loads on llama-cpp (backends=[llama-cpp vllm] — the new backend never enters the candidate list).

Scope: Linux/AMD only, deliberately

The kernels are ROCm/HIP for RDNA3.5; there is neither ROCm nor the target hardware on Apple silicon, and NVIDIA already has native FP4 paths upstream. Hence no includeDarwin entry and a single hipblas matrix entry — stated here rather than omitted silently, per the checklist.

Checklist coverage

• backend/cpp/rocmfp4/ wrapper following the bonsai pattern (weight types decoded inside libllama; shared gRPC server reused verbatim, no allow-list patch needed)
• Dockerfile.rocmfp4 + .docker/rocmfp4-compile.sh (from-source and prebuilt-base stages). The Dockerfile carries a documented local-build escape hatch (LLAMA_REPO/ROCMFP4_VERSION ARGs, empty defaults) so the fork can be built from a local mirror; empty values are unset in the compile script so the Makefile pin always wins in CI
• .github/backend-matrix.yml entry (hipblas/amd64, prebuilt gRPC base, tag-suffix matches index.yaml URIs exactly)
• scripts/lib/backend-filter.mjs: path filter above the generic llama-cpp suffix + shared-input rule (make test-ci-scripts: 41 pass, 0 fail)
• bump_deps.yaml registration; pin kept in ROCMFP4_VERSION?= form the bot greps for
• backend/index.yaml meta + latest/development image entries (YAML validated)
• Makefile wiring incl. test-extra-backend-rocmfp4 loading a real ROCmFP4 GGUF (fork-only decodable, bonsai pattern; artifact is 13.75 GiB — workstation test, noted in the target comment)
• Importer preference swap + table-driven test (passes in isolation; the suite's live-HF specs for unrelated importers flake identically with and without this change)
• Docs: docs/content/features/backends.md

Happy to adjust scope, naming or the escape hatch if the maintainers prefer a different shape. We run this backend in production on Strix Halo and intend to maintain it (pin bumps via the registered bump_deps entry, rebases of the format fork against upstream llama.cpp).

🤖 Generated with Claude Code

https://claude.ai/code/session_01SeuYBDnA3qH3ngNagKDNjA

[localai-org-maint-bot]

The backend wiring needs a cleanup before this can run through CI:\n\n- In .github/workflows/bump_deps.yaml, the new walcz-de/llama.cpp-ROCmFP4 item is indented at the matrix level instead of under matrix.include. The following ds4 item is therefore left unexpectedly over-indented, making the workflow YAML structurally invalid.\n- backend/cpp/rocmfp4/patches/README.md is still the Bonsai/Prism copy: it identifies this as the bonsai backend, points at the PrismML prism fork, and describes Bonsai CI inputs. Please rewrite it for rocmfp4 and the walcz-de/llama.cpp-ROCmFP4 fork.\n\nDCO is also failing, but that attestation must be fixed by the human contributor.

[walcz-de]

Thanks for the careful review — all three addressed:

• bump_deps.yaml: the new item is re-indented under matrix.include alongside its neighbours; the ds4 item below it is back at its normal level. Workflow YAML validates again.
• backend/cpp/rocmfp4/patches/README.md: rewritten for rocmfp4 and the walcz-de/llama.cpp-ROCmFP4 fork — including why the directory is expected to stay empty (the fork is rebased onto the same LLAMA_VERSION the shared gRPC server is written against, unlike the Bonsai/Prism situation it was templated from).
• DCO: the failure was a name mismatch between the commit author (stefanwalcz) and the sign-off (Stefan Walcz), same address on both sides. The branch's author fields are normalized so author and sign-off now match exactly on every commit; the attestation itself is unchanged.

[localai-org-maint-bot]

The earlier workflow, patches README, and DCO issues are fixed. One copied Bonsai artifact remains in the same helper: backend/cpp/rocmfp4/apply-patches.sh still says it applies Bonsai patches to the PrismML prism fork, and its success message says all bonsai patches applied successfully. Please rewrite those comments and the message for rocmfp4 / walcz-de/llama.cpp-ROCmFP4. This is documentation and diagnostic output only; no behavior change is needed.

[walcz-de]

Done — apply-patches.sh header and success message rewritten for rocmfp4 / walcz-de/llama.cpp-ROCmFP4, including why patches/ is expected to stay empty here. I also swept the backend directory for further copied identity: the two remaining bonsai mentions in the Makefile are deliberate comparisons to the pattern this wrapper follows ("like bonsai, the additions are weight types decoded inside libllama"), not leftovers — happy to reword those too if you'd prefer.

[localai-org-maint-bot]

The latest commit removes the remaining Bonsai/Prism text from apply-patches.sh and keeps the helper behavior unchanged. DCO passes. Good from my side, @mudler.