fix(conserver): dead-letter failed storage writes instead of dropping them (CON-714)

[howethomas]

Summary

A storage write that failed was logged and forgotten. The vCon never reached the backend, nothing queued it, and the chain still reported success. On BDS a ~100s outage of one backend silently lost every vCon written during it.

Two changes:

• Failures are dead-lettered. They land on DLQ:storage:<backend> and the vCon's Redis TTL is extended from VCON_REDIS_EXPIRY (1h) to VCON_DLQ_EXPIRY (7d), so the body outlives the default retention and is still there to replay. Replay re-attempts only the storage write — routing through the ingress DLQ instead would re-run the whole chain, including transcription that already succeeded.
• Transient failures are retried in the vcon-mcp client: connection errors, read timeouts, and 429/502/503/504 with exponential backoff. Permanent 4xx are not retried; repeating a 401 only delays the dead-letter.

_process_storage still does not raise, so one failing backend does not stop the others or egress.

Notes for review

POST is in allowed_methods, which urllib3 excludes by default as non-idempotent. That is safe here specifically because vcon-mcp upserts on the vCon uuid (onConflict: 'id' in its batch-writer.ts), so a retried create converges on one row rather than duplicating. If that ever changes, this needs to change with it.

The TTL extension is load-bearing, not decorative. VCON_REDIS_EXPIRY defaults to 3600s. Without extending it, a dead-lettered vCon would be unreplayable an hour later and the DLQ would be a list of ids pointing at nothing.

Reuses conserver.dlq.count{queue_name} rather than adding a new metric, so an existing alert on that counter covers storage failures with no new rule.

/dlq/storage/reprocess stops at the first failure and puts that vCon back, so a backend that is still down does not drain the queue into nothing. Bounded by count per call, mirroring the CON-575 fix on /dlq/reprocess.

Test plan

• 17 new tests in common/tests/test_storage_dlq.py covering: DLQ naming and counter emission, storage DLQ distinct from ingress DLQ, dead-letter on failure with TTL extension, no dead-letter on success, VCON_DLQ_EXPIRY=0 skips the TTL call, Redis failure while dead-lettering does not propagate, replay endpoint (success, put-back-and-stop, empty, count bound), and the retry configuration.
• Full common/tests/ suite: 218 passed, 2 skipped. The 8 test_api.py failures are pre-existing and reproduce identically on untouched origin/main — the local Redis has no RedisJSON module (unknown command 'JSON.SET').
• Not exercised against a live backend outage. Worth a deliberate soak on BDS: stop vcon-mcp, confirm entries accumulate on DLQ:storage:vcon_mcp, restart, then drain via POST /dlq/storage/reprocess and confirm the vCons land.

Fixes CON-714.

Made with Cursor

[howethomas]

Reviewed the full diff (customer-bds / BDS ops side). The seam is right: dead-lettering the storage write alone rather than re-raising through the ingress DLQ correctly avoids replaying the whole chain (GPU transcription included), and the TTL bump to VCON_DLQ_EXPIRY is the load-bearing half that keeps the dead-lettered id from pointing at an expired body. Reusing conserver.dlq.count{queue_name} means the CON-714 alert covers this with no new rule. LGTM to merge.

Two non-blocking notes for a follow-up:

1. _dead_letter_storage's bare except Exception: also swallows a set_vcon_ttl failure. If the enqueue lands but the TTL bump fails, you get a DLQ id pointing at a body that still expires at the 1h VCON_REDIS_EXPIRY, an unreplayable ghost after an hour. Strictly better than today's silent drop, but worth logging that specific case (enqueued-but-not-extended) so it is visible rather than silent.

2. The POST-in-allowed_methods retry is safe only because vcon-mcp upserts on uuid (onConflict: 'id'). _session is correctly scoped to the vcon_mcp module today. Worth a comment/assert at the seam so a future non-idempotent storage backend cannot quietly reuse this session helper and start duplicating rows on retry.

Un-exercised path (already flagged in the description): the live-backend soak on BDS. That deliberately induces prod storage failures, so it needs a controlled window, not a Sunday fire-and-forget. Happy to run it on the next maintenance window.