Skip to content

ThrottlingRequestManager does not stop a 429 from retiring the session #2143

Description

@janbuchar

_raise_for_session_blocked_status_code records the domain delay and then falls through to
Session.is_blocked_status_code, and 429 is in _DEFAULT_BLOCKED_STATUS_CODES. So on a domain the
throttler covers, a 429 still raises SessionError, still retires the session and still spends a
max_session_rotations slot — the churn #1437 was filed about. Only ignore_http_error_status_codes
avoids it, and nothing says so.

This wants fixing together with a stall timeout, in one PR:

  • Nothing bounds a permanently rate-limiting domain except the retry accounting the fix removes.
    _should_retry_request caps SessionError at max_session_rotations and refuses
    HttpClientStatusCodeError outright, so today such a domain drains and the crawl ends. Make a
    throttled 429 free and it never terminates — run() hangs while the statistics report nothing,
    which is what the TS port walked into.
  • So the fix also needs a per-domain stall timeout that gives up once a domain with queued requests
    has been rate-limiting for that long, and a keep_alive exemption. max_domain_stall_secs is a
    constructor argument, so landing it separately ships a release where the knob does nothing.

Both halves are in apify/crawlee#3741. From #1762.

Metadata

Metadata

Assignees

Labels

bugSomething isn't working.t-toolingIssues with this label are in the ownership of the tooling team.

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions