fix(providers): clamp max_tokens to the model context window

[basil-k-aji-dev]

Refs #399 — implements the clamp half.

The problem

pconfig.BuildOptions attaches llms.WithMaxTokens(ac.MaxTokens) unconditionally (pkg/providers/pconfig/config.go:681) and nothing compares it to the model window. vLLM and other OpenAI-compatible backends reject the request when

prompt_tokens + tool_schema_tokens + max_tokens > max_model_len

with an empty-bodied HTTP 400, so callWithRetries burns its attempts and stops. The UI just looks stuck.

Where the fix goes

provider.WrapGenerateContent — every provider's CallEx and CallWithTools funnels through it (openai, openaicompat, bedrock, ollama, custom), and it already holds both the message chain and the resolved call options. One implementation, no interface change, no per-provider duplication.

The issue proposes hooking callWithRetries and performSimpleChain separately. WrapGenerateContent sits below both, so it covers the Summarizer/Adviser path in step 5 for free.

What it does

EstimatePromptTokens sizes messages and tool schemas at 4 bytes/token; ClampMaxTokens lowers max_tokens to what the window leaves, keeping a 512-token margin for the gap between a byte estimate and the backend's real tokenizer.

Deliberately conservative — it declines to act in every case where acting could do harm:

Case	Behaviour	Why
LLM_SERVER_MAX_MODEL_LEN unset	no-op	unchanged for anyone who has not opted in
request already fits	no-op	nothing to do
max_tokens unset	no-op	the provider default applies; nothing of ours to lower
prompt alone fills the window	no-op	clamping could only yield a stub reply and would hide the real problem — that case needs history compression

The estimate rounds against us throughout: tool schemas are measured as the JSON actually sent, and parts with no cheap size (binary, provider-specific) carry a flat cost rather than counting as free. Undercounting would defeat the clamp.

A clamp emits a Langfuse warning event carrying the chosen budget and the window, so it is visible rather than silent.

Scope

This is the clamp, not the whole issue. Left out: preflight summarization of older turns, hard truncation, and retry-on-overflow. Those rewrite the message chain rather than adjusting one call, and belong in csum with their own review. The clamp alone fixes the reported scenario — 32k window, large first Generator prompt, max_tokens leaving no room — which is the part that fails before any history exists to summarize.

I picked an env var over parsing max_model_len from /models deliberately: LoadModelsFromHTTP currently drops extra fields, and changing it is a larger change with its own failure modes. If you would rather the window came from /models, say so and I will rework it.

Tests

Eight tests in preflight_test.go, no network or provider needed.

ok  	pentagi/pkg/providers/provider	0.015s

The important one is TestClampMaxTokensCountsToolsTowardWindow: the messages alone fit the window and the tool schemas tip it over, so it fails if schemas are ever dropped from the estimate. I verified it bites by deleting the tool-counting loop:

--- FAIL: TestClampMaxTokensCountsToolsTowardWindow
    preflight_test.go:163: tool schemas push this request over the window and must be counted

TestClampMaxTokensShrinksOversizedBudget reproduces the issue's example — 32k window, ~20k-token prompt, max_tokens=26214 — and asserts the clamped request satisfies the inequality.

Verification

go build ./pkg/providers/..., go vet ./pkg/providers/provider/, gofmt -l and go test ./pkg/providers/provider/ all clean. No new dependencies — go.mod and go.sum are untouched; the new file uses only encoding/json, os, strconv and sync.

What I could not run: I do not have a vLLM instance, so the end-to-end path — oversized request against a real 32k backend — is unverified. The clamp arithmetic and the estimator are covered by unit tests; the integration is reasoned from config.go:681 and the call graph through WrapGenerateContent. Worth a check against a real server before merging. I also could not run golangci-lint, only go vet.

[sirozha]

Implemented in our own line and shipping with 2.2.0 — thanks for the patch. Closing this PR.

[basil-k-aji-dev]

Good to hear it is landing in 2.2.0. The empty-bodied 400 was the painful part, since the chain just retried itself out with nothing in the log, so I am glad the clamp is in regardless of whose patch carries it.

One thing worth keeping if it did not survive the port: the clamp only engages when the context window is known, so it is a no-op for anyone who has not set it. That is what keeps it from trimming a budget on a provider whose real window is larger than we guessed.