fix: commit request queue dedup cache only after batch_add_requests succeeds

[vdusek]

Both Apify request queue clients (ApifyRequestQueueSingleClient, ApifyRequestQueueSharedClient) cached new requests locally (the single client also updates _head_requests) before calling batch_add_requests. That caused two bugs:

• A failed call left the cache entry behind, so a later retry of the same request was deduplicated locally and never re-sent. The request was silently lost.
• A concurrent producer could match the in-flight cache entry and return was_already_present before the first call finished. If that call then failed, the producer had already reported success for a request that never reached the platform.

Now the cache and head are committed only after batch_add_requests succeeds, and only for requests the platform accepted (unprocessed_requests are skipped). A failed call commits nothing.

To keep deduplicating concurrent producers (so overlapping batches do not multiply platform writes), each in-flight add is tracked by a per-request future. A concurrent producer of the same request awaits it instead of re-sending. It is told the request is present only if the original add committed it. If the original add failed, the producer reports the request unprocessed so Crawlee retries it, rather than receiving false success.

Within-batch deduplication is unaffected, since Crawlee already collapses a batch by unique_key (via _transform_requests) before it reaches these clients.

Tests (parametrized over both clients, plus integration):

• A failed batch_add_requests leaves no cached entry, and the retry reaches the platform.
• A concurrent producer of an in-flight request deduplicates against it (a single API call) and is told the request is present only once it is accepted.
• When the in-flight add fails, the concurrent producer reports the request unprocessed instead of false success.
• test_request_queue_parallel_deduplication confirms overlapping concurrent producers write each request to the platform exactly once.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 91.69%. Comparing base (0daca28) to head (fe358b5).
⚠️ Report is 50 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #975      +/-   ##
==========================================
+ Coverage   89.90%   91.69%   +1.78%     
==========================================
  Files          49       50       +1     
  Lines        3091     3203     +112     
==========================================
+ Hits         2779     2937     +158     
+ Misses        312      266      -46     

Flag	Coverage Δ
e2e	35.56% <10.76%> (-0.36%)	⬇️
integration	57.22% <98.46%> (+0.35%)	⬆️
unit	83.14% <98.46%> (+4.39%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[Copilot]

Pull request overview

This PR fixes request queue deduplication correctness for both Apify request queue client implementations by ensuring local dedup caches are only committed after a successful batch_add_requests call, and by coordinating concurrent producers via per-request in-flight futures.

Changes:

• Deferred local cache/head updates until batch_add_requests succeeds, committing only requests accepted by the platform (skipping unprocessed_requests).
• Added per-request “in-flight add” tracking to deduplicate concurrent producers without reporting false success on failures.
• Added unit tests covering failure cases, concurrent producer deduplication, and partial unprocessed behavior for both single/shared clients.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

File	Description
tests/unit/storage_clients/test_apify_request_queue_client.py	Adds unit tests for failed adds, concurrent in-flight deduplication, and partial acceptance behavior across both clients.
src/apify/storage_clients/_apify/_utils.py	Introduces helper functions for settling and awaiting in-flight per-request additions.
src/apify/storage_clients/_apify/_request_queue_single_client.py	Defers cache/head commits until API success and coordinates concurrent producers via in-flight futures.
src/apify/storage_clients/_apify/_request_queue_shared_client.py	Defers dedup-cache commits until API success and coordinates concurrent producers via in-flight futures.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.