Skip to content

[Bug]: stream retries re-send already-billed requests and retry deterministic 4xx #2677

Description

@1688mengdie

Problem

execute_sse_request retries with the same request_body in two cases that
should be terminal:

  1. A TTFT (time-to-first-token) timeout re-sends an already-sent request. The
    request body is sent to the server before the first-token wait begins, so
    the retry re-sends an already-billed request (double-billing for the same
    logical turn).
  2. A deterministic 4xx (for example 400) is retried. A deterministic client
    error fails identically on every attempt, so the retry only burns the
    request budget/credits and adds latency.

Root cause

The retry decision in execute_sse_request does not distinguish terminal from
transient outcomes. The TTFT branch continues using the same request body,
and the non-2xx branch retries every status, including 400. The
SEND_MESSAGE_STREAM_ATTEMPTS counter is amplified by these re-sends.

Repro

  1. For TTFT: mock a server that never completes a response, inject a small
    ttft_timeout, and observe execute_sse_request re-send the same body
    across attempts.
  2. For 4xx: mock a server that returns 400 and observe the fixture is called
    more than once (the old bad_requests_then_success fixture even reports
    success when a later attempt returns 200).

Impact

Billing is charged twice for the same logical turn on TTFT timeout, and
deterministic client errors waste credits/latency across up to
SEND_MESSAGE_STREAM_ATTEMPTS re-sends.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions