fix(key-value store): Avoid "dictionary changed size" error in KVS autosave

[anxkhn]

KeyValueStore.get_auto_saved_value mutates the shared per-store autosave cache
(self._autosaved_values[self.id]) while holding self._autosave_lock. But the
two consumers of that cache iterated it and awaited I/O inside the loop without
taking the lock:

• persist_autosaved_values iterated cache.values() and awaited
  value.persist_state() per entry.
• _clear_cache (called from drop()) iterated cache.values() and awaited
  value.teardown() per entry, then cleared the cache.

Because both loops suspend at an await, a request handler that calls
get_auto_saved_value with a new key while a persist or drop is mid-iteration
inserts into the same dict and grows it during iteration, raising
RuntimeError: dictionary changed size during iteration and aborting the
persist/drop.

This wraps both loops in async with self._autosave_lock:, the same lock
get_auto_saved_value already uses, so the iteration and any concurrent insert
are serialized. No public API changes; _clear_cache still calls
cache.clear() (now inside the lock, which is strictly safer).

The awaited bodies (RecoverableState.persist_state() / .teardown()) bottom
out in set_value and never re-enter get_auto_saved_value,
persist_autosaved_values, or _clear_cache, so the non-reentrant
asyncio.Lock is never re-acquired on the same instance (no deadlock).

Issues

• No existing issue; found while reading the autosave/use_state code path.

Testing

Added two regression tests in tests/unit/storages/test_key_value_store.py
(both run across all four storage backends via the storage_client fixture):

• test_persist_autosaved_values_with_concurrent_new_key
• test_drop_with_concurrent_new_autosaved_key

Each parks the persist/clear loop mid-iteration (by blocking the underlying
set_value, which persist_state/teardown call) and then inserts a new
autosaved key concurrently. Without the fix they fail with RuntimeError: dictionary changed size during iteration; with the fix they pass.

• uv run pytest tests/unit/storages/test_key_value_store.py -> 296 passed, 2 skipped
• uv run pytest tests/unit/storages/ -> 887 passed, 6 skipped
• uv run poe lint -> clean
• uv run poe type-check -> no new diagnostics vs master

Checklist

• CI passed

[Copilot]

Pull request overview

This PR fixes a concurrency bug in KeyValueStore’s autosave cache by serializing cache iteration with cache mutation, preventing RuntimeError: dictionary changed size during iteration during persist_autosaved_values() and drop().

Changes:

• Guard autosave-cache iteration in persist_autosaved_values() and _clear_cache() with self._autosave_lock.
• Add regression tests that reproduce the concurrent insert-during-iteration scenario across storage backends.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File	Description
src/crawlee/storages/_key_value_store.py	Wrap autosave cache iteration/cleanup with the existing autosave lock to prevent concurrent mutation during awaited I/O.
tests/unit/storages/test_key_value_store.py	Add regression tests that simulate concurrent cache insertion during persist/drop to prevent the iteration-size-change runtime error.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[vdusek]

LGTM