You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
cluster-integration tests: exactly-once handler assertions are flaky when a lock refresh exceeds the 500 ms fence #10673
Affected:packages/platform/node/test/cluster-integration on main at 3d13d1e (tests only; no bug in effect itself).
Summary
Several cluster persistence integration tests assert that a handler ran exactly once, for example persists a non-JSON Error as a serializable defect (Persistence.test.ts:456, through assertStoredAsDefect, :143assert.strictEqual(count(tag, id), 1)). Under Effect's own contract that is not guaranteed when lock storage stalls:
Before a persisted reply is stored, confirmShardLock (packages/effect/src/cluster/Sharding.ts:1997) refreshes the shard's lock with Effect.timeoutOption(shardLockInterval). The harness sets shardLockRefreshInterval: 500 (harness.ts:142), so the limit is 500 ms.
If a lock refresh takes longer, the reply is fenced: it isn't stored and the request is reset. The regular refresh usually times out too, the runners mark lock storage unhealthy and release their shards, and after recovery the request replays and the handler runs a second time.
That replay is the designed at-least-once behaviour, so the assertion count == 1 fails, not the code. The trigger is a lock-storage write that takes over 500 ms. The test PostgreSQL container (globalSetup.ts:29, postgres:alpine) runs with the default fsync = on, so a slow or busy disk on a CI runner is enough.
Evidence
Found in pyfect, a Python port of Effect whose cluster code and integration harness follow Effect's. We have not run Effect's suite under Node.
pyfect's port of this test failed once on CI with the handler counted twice (2 == 1), on SQLite, whose test database ran with full disk syncs.
Locally, under disk load (12 processes writing and syncing 64 MB files, 4 test workers): 5 failures in 120 runs, with 957 lock refreshes over 450 ms. With syncing turned off on the test database: 0 in 120, and 45 refreshes over 450 ms. CPU load alone caused no failures in 200 runs.
On PostgreSQL under container load, a related test (rebuilds an entity after a fatal defect and replays in-flight requests) failed once in 319 runs from the same 500 ms fence.
Suggested fix
Keep the count == 1 assertions (they catch real double runs) and remove the stall source in the harness: start the test PostgreSQL with fsync=off (for example .withCommand(["postgres", "-c", "fsync=off"]), or synchronous_commit=off). A separate test can then pin the fencing contract on purpose, by holding the lock storage past shardLockInterval during a request and asserting the reply isn't stored, the request replays (two runs), and a stored reply is served to a duplicate request without running the handler again. pyfect is adding this as tests/oracle/test_cluster_reply_fence.py.
We found a second way to reach the replay above. It doesn't need a slow disk: a burst of shard releases can pass the lock-operation deadline together. Line numbers below are for packages/effect/src/cluster at 506eef1.
Mechanism
When a second runner joins, the first runner releases its handed-off shards (about 150 of 300 in the integration harness) at once. releaseShards forks one fiber per shard through FiberMap.run (Sharding.ts:497-511, :582-588), with no concurrency bound.
On PostgreSQL, every lock operation goes through the one reserved lock connection, and each release is two statements: the unlock and the pg_locks check. The connection runs them FIFO, so each release's second statement waits behind the other releases' first statements. No release finishes until about half the burst has run.
If the burst takes more than lockOperationInterval (500 ms in the harness), nearly every release times out together. withLockOperationDeadline (SqlRunnerStorage.ts:578-585) then rebuilds the lock connection. Closing the old session runs pg_advisory_unlock_all() (SqlRunnerStorage.ts:117-124), so the runner drops every advisory lock it holds, including shards it meant to keep.
retryShardRelease retries each release every 50 ms (Sharding.ts:470-494), which keeps the new connection's queue full. The next acquire and refresh time out too and rebuild again, so lock storage stays unhealthy for seconds.
A reply finished during that window fails confirmShardLock (the fence described above), and the request replays. That gives handler counts of 2 to 4 in persists a non-JSON Error as a serializable defect.
Evidence
pyfect's port follows this logic line for line (release, retry, rebuild, deadline), and its harness uses the same settings (harness.ts:138-142). Under host load (load average 30-40), its copy of the test failed 12 of 40 runs. Every failure logged hundreds of simultaneous "Could not release shards, retrying" warnings with TimeoutException(500 millis), then "Shard lock storage is unhealthy".
pyfect's statements cost more per round trip than Node's, so we hit the window more easily, and we have not reproduced this in Effect's own suite. The mechanism doesn't depend on that, though: any host where 2 x (shards handed off) statements take more than lockOperationInterval on one connection reaches it.
Possible fixes
Bound release concurrency. For example, release the shards in one batched statement per runner (the dialects already take shardIds lists elsewhere, such as the refresh query around SqlRunnerStorage.ts:566-573), or limit the number of concurrent releaseShard fibers, so a handoff can't flood the lock connection.
Don't rebuild on a deadline alone. When the statement only passed lockOperationInterval and didn't fail, rebuilding (and with it pg_advisory_unlock_all()) turns one slow release into the loss of every lock. A deadline could fail the operation without replacing a connection that is still healthy, leaving rebuilds to real SQL or connection errors.
We'd be glad to open a PR for either one, or both, if that direction works for you.
Activity
Skip disk flushes on the cluster-integration test databases
tylergibbs1 commentedon Nov 16, 2026
We found a second way to reach the replay above. It doesn't need a slow disk: a burst of shard releases can pass the lock-operation deadline together. Line numbers below are for
packages/effect/src/clusterat 506eef1.Mechanism
releaseShardsforks one fiber per shard throughFiberMap.run(Sharding.ts:497-511,:582-588), with no concurrency bound.pg_lockscheck. The connection runs them FIFO, so each release's second statement waits behind the other releases' first statements. No release finishes until about half the burst has run.lockOperationInterval(500 ms in the harness), nearly every release times out together.withLockOperationDeadline(SqlRunnerStorage.ts:578-585) then rebuilds the lock connection. Closing the old session runspg_advisory_unlock_all()(SqlRunnerStorage.ts:117-124), so the runner drops every advisory lock it holds, including shards it meant to keep.retryShardReleaseretries each release every 50 ms (Sharding.ts:470-494), which keeps the new connection's queue full. The next acquire and refresh time out too and rebuild again, so lock storage stays unhealthy for seconds.confirmShardLock(the fence described above), and the request replays. That gives handler counts of 2 to 4 inpersists a non-JSON Error as a serializable defect.Evidence
pyfect's port follows this logic line for line (release, retry, rebuild, deadline), and its harness uses the same settings (
harness.ts:138-142). Under host load (load average 30-40), its copy of the test failed 12 of 40 runs. Every failure logged hundreds of simultaneous "Could not release shards, retrying" warnings withTimeoutException(500 millis), then "Shard lock storage is unhealthy".pyfect's statements cost more per round trip than Node's, so we hit the window more easily, and we have not reproduced this in Effect's own suite. The mechanism doesn't depend on that, though: any host where 2 x (shards handed off) statements take more than
lockOperationIntervalon one connection reaches it.Possible fixes
shardIdslists elsewhere, such as the refresh query aroundSqlRunnerStorage.ts:566-573), or limit the number of concurrentreleaseShardfibers, so a handoff can't flood the lock connection.lockOperationIntervaland didn't fail, rebuilding (and with itpg_advisory_unlock_all()) turns one slow release into the loss of every lock. A deadline could fail the operation without replacing a connection that is still healthy, leaving rebuilds to real SQL or connection errors.We'd be glad to open a PR for either one, or both, if that direction works for you.