Streaming: all deltas labeled 'reasoning' when the rendered prompt contains a (pre-closed) thinking marker — non-streaming parses the same output correctly

[KaletoAI]

LocalAI version: v4.7.1 (b224c96), localai/localai:latest-gpu-vulkan (docker), llama-cpp backend
Model: Gemma-4-12B-it (GGUF, Q8_0)

Summary

Gemma-4's official chat template disables thinking by ending the generation prompt with a pre-closed thought channel:

<|turn>model
<|channel>thought
<channel|>

With the channel already closed, the model emits plain answer text only — no thinking tokens at all. The non-streaming endpoint handles this correctly (message.content filled, no reasoning field). But in streaming mode, every single delta is emitted as delta.reasoning and delta.content stays null for the entire stream. The stream parser apparently enters the reasoning state because the prompt contains <|channel>thought, without checking that it is immediately closed — and never leaves that state, since the model (correctly) never emits a close marker itself.

Any correctly-templated "thinking disabled" Gemma-4 request triggers this, so OpenAI-compatible clients that read delta.content receive a completely empty stream while the server generates a full answer.

Reproduction

Model YAML (explicit template mirroring the GGUF's embedded jinja for the thinking-off case):

name: gemma-4-12b-it
backend: llama-cpp
parameters:
    model: gemma-4-12b-it-Q8_0.gguf
gpu_layers: 99
stopwords:
- "<turn|>"
template:
    chat_message: |-
        <|turn>{{if eq .RoleName "assistant"}}model{{else}}{{.RoleName}}{{end}}
        {{ if .Content }}{{.Content}}{{ end }}<turn|>
    chat: |-
        {{.Input -}}
        <|turn>model
        <|channel>thought
        <channel|>

Non-streaming — correct:

curl -s http://localhost:8080/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model":"gemma-4-12b-it","max_tokens":40,"messages":[{"role":"user","content":"Say READY."}]}'
# → "message":{"role":"assistant","content":"READY. …"}   (no reasoning field, finish_reason stop)

Streaming — broken (same request + "stream": true):

"delta":{"role":"assistant","content":null}
"delta":{"content":null,"reasoning":"READY"}
"delta":{"content":null,"reasoning":"."}
...            ← 100 % of tokens arrive as reasoning; not a single content delta

Expected

The stream parser should only enter the reasoning state when the model emits a thinking-open marker (or should recognize that the marker in the prompt is immediately followed by its close <channel|>). Streaming and non-streaming should classify identical output identically.

Workarounds tried (no effect on the stream labeling)

• request level: chat_template_kwargs: {enable_thinking: false}, thinking: false, reasoning_format: "none", disable_thinking: true
• model YAML: disable_thinking: true, thinking_start_tokens: ["<NEVER_EMITTED>"], logit_bias (appears to be ignored on the llama-cpp path)
• Removing the pre-close from the template is not viable: Gemma-4 then opens the thought channel itself (that pre-close is the model's official "thinking off" mechanism).

[localai-org-maint-bot]

I’m picking this up. I’ll reproduce the streaming/non-streaming classification mismatch in a focused parser test, trace how prompt markers seed streaming reasoning state, and make the smallest compatible fix with regression coverage.