_raise_for_session_blocked_status_code records the domain delay and then falls through to
Session.is_blocked_status_code, and 429 is in _DEFAULT_BLOCKED_STATUS_CODES. So on a domain the
throttler covers, a 429 still raises SessionError, still retires the session and still spends a
max_session_rotations slot — the churn #1437 was filed about. Only ignore_http_error_status_codes
avoids it, and nothing says so.
This wants fixing together with a stall timeout, in one PR:
- Nothing bounds a permanently rate-limiting domain except the retry accounting the fix removes.
_should_retry_request caps SessionError at max_session_rotations and refuses
HttpClientStatusCodeError outright, so today such a domain drains and the crawl ends. Make a
throttled 429 free and it never terminates — run() hangs while the statistics report nothing,
which is what the TS port walked into.
- So the fix also needs a per-domain stall timeout that gives up once a domain with queued requests
has been rate-limiting for that long, and a keep_alive exemption. max_domain_stall_secs is a
constructor argument, so landing it separately ships a release where the knob does nothing.
Both halves are in apify/crawlee#3741. From #1762.
_raise_for_session_blocked_status_coderecords the domain delay and then falls through toSession.is_blocked_status_code, and 429 is in_DEFAULT_BLOCKED_STATUS_CODES. So on a domain thethrottler covers, a 429 still raises
SessionError, still retires the session and still spends amax_session_rotationsslot — the churn #1437 was filed about. Onlyignore_http_error_status_codesavoids it, and nothing says so.
This wants fixing together with a stall timeout, in one PR:
_should_retry_requestcapsSessionErroratmax_session_rotationsand refusesHttpClientStatusCodeErroroutright, so today such a domain drains and the crawl ends. Make athrottled 429 free and it never terminates —
run()hangs while the statistics report nothing,which is what the TS port walked into.
has been rate-limiting for that long, and a
keep_aliveexemption.max_domain_stall_secsis aconstructor argument, so landing it separately ships a release where the knob does nothing.
Both halves are in apify/crawlee#3741. From #1762.