fix(cli): wait out rate limits during a WebDAV transfer

[phernandez]

Why

Reported by a customer on 0.23.2: bm cloud pull --name cabinet against a Team workspace failed repeatedly with

Failed to list cloud project 'cabinet': HTTP 429
{"detail":{"code":"rate_limit_exceeded","message":"Too many requests. ...","scope":"principal"}}

A Team transfer runs over WebDAV (#1262). list_project_files walks the project with one Depth: 1 PROPFIND per directory, and webdav_project_diff runs that walk for both push and pull. The service meters every request on this transport, so a transfer of any real size meets its own rate limit as a matter of course.

All three request sites called raise_for_status() directly, so the first 429 anywhere aborted the entire transfer.

The part that made it unrecoverable rather than merely slow: each re-run restarts the walk at the first directory. Every attempt spent a fresh window re-listing directories it already had, then died at about the same depth. Waiting did not help, so a project past the per-minute ceiling could never complete. The rate-limit headers were accurate the whole time; there was simply no way to make forward progress.

The customer confirmed the diagnosis by patching _propfind() locally to honor Retry-After, after which the pull completed.

What

Route all three request sites through _request_with_rate_limit_retry:

Site	Method	Why it needs this
_propfind	PROPFIND	One per directory while enumerating
download_file	GET	One per file during a pull
upload_file	PUT	One per file during a push

Only PROPFIND was in the original report, but a pull that survives enumeration then issues one GET per file, so fixing the walk alone would have moved the failure into the transfer phase.

Retry-After is read as either delay-seconds or an HTTP-date. Waits are floored at 1s so a "retry immediately" answer cannot spin, capped at 60s so an implausible header cannot hang the CLI, and an absent or unparseable value falls back to a fixed wait, since a header we cannot read still tells us the window is closed. Attempts are bounded at 6; the last 429 is returned rather than raised so the caller's existing error path reports it.

_describe() now appends the rate-limit headers to a 429. The body alone does not say what the limit was or when it resets, which is why the customer had to edit this file to see them.

Safety of replaying

Every request here is safe to repeat. The reads carry no caller-supplied body, and the one write sends the same bytes and the same X-OC-Mtime each time, so a replay is the identical request rather than a second effect. The body is passed explicitly rather than forwarded as **kwargs, so a caller cannot later add a parameter that quietly makes a retry into a different request.

A create-only 412 is untouched: it is the answer that call asked for, not a rejection to wait out, and it is still checked before raise_for_status().

Testing

Eight new tests in tests/cli/cloud/test_webdav_client.py, using the existing MockTransport harness with _sleep patched so the waits are asserted rather than spent:

• a 429 mid-walk does not abort the transfer, and the walk completes
• the same for a throttled download, and for a throttled upload
• a retried PUT replays byte-identical content and headers, including If-None-Match: *
• a create-only 412 is not retried
• a persistent 429 stops after 6 attempts and reports limit, remaining, retry after and resets at in the error
• a 404 is not retried and gets no rate-limit detail appended
• Retry-After parsing across delay-seconds, 0 (floored), an implausible value (capped), absent, unparseable, and a past HTTP-date

Verified both guards are load-bearing:

• Setting _RATE_LIMIT_MAX_ATTEMPTS = 1 fails exactly the four survival tests
• Reverting _describe() to drop the detail fails exactly test_persistent_rate_limit_reports_the_headers

Restored after each. tests/cli/cloud/ passes in full (175 tests) and just check exits 0.

Scope

This makes a large Team transfer possible, not fast. It still costs one request per directory plus one per file, so a big project spends real time waiting out windows. Two related pieces are tracked in basicmachines-co/basic-memory-cloud#2039:

• the server charging PROPFIND as a write, which halved the throughput (fixed separately in the cloud repo)
• the server ignoring the Depth request header, so the client cannot collapse the walk into a single recursive PROPFIND. That is the change that would make large projects fast.

Credit

The diagnosis, the pointer to _propfind() and _describe(), and the confirmation that
Retry-After handling resolves it all came from a customer report, and the reporter offered
a patch. Named credit is theirs to claim rather than ours to publish; happy to add it here on
request.

🤖 Generated with Claude Code

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.