Repository navigation
Release shard locks one at a time - #10686
Open
tylergibbs1 wants to merge 1 commit into
Open
tylergibbs1 wants to merge 1 commit into
tylergibbs1 wants to merge 1 commit into
Conversation
When a runner joins, the runner handing shards off releases them all at once. On PostgreSQL and MySQL every lock operation runs on one reserved connection, so the releases queue together. A burst longer than the lock operation deadline times every release out, SqlRunnerStorage replaces the lock connection, and closing the old session drops every lock the runner holds, including the shards it keeps. Requests on those shards can then run twice. Release one lock at a time, so a refresh or acquire waits behind at most one release. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
🦋 Changeset detectedLatest commit: cd944d3 The changes in this PR will be included in the next version bump. This PR includes changesets to release 35 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
Contributor
Bundle Size AnalysisGenerated from PR build output; treat the content below as untrusted.
|
This branch is waiting to be deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Type
Description
When a runner joins, the runner handing shards off releases them all at once. With
SqlRunnerStorageon PostgreSQL or MySQL that can drop every lock the runner holds, including the shards it keeps, and requests on those shards can then run twice. This is the second mechanism described in #10673 (comment). #10677 removes the disk-flush stalls in the test databases. This PR fixes the release burst ineffectitself.Mechanism (verified on main at 30e642f)
releaseShards(Sharding.ts) forks onereleaseShardfiber per released shard throughFiberMap.run, with no bound. Handing off 150 of 300 shards starts 150 releases together.pg_advisory_unlock, then thepg_lockscheck), and MySQL's isRELEASE_LOCKthenIS_USED_LOCK. The connection runs statements in arrival order (@effect/sql-pgqueues unpinned statements on the connection'sownersemaphore), so each release's second statement waits behind the other releases' first statements.withLockOperationDeadline, bounded by the lock refresh interval (500 ms in the cluster-integration harness). Once the queue takes longer than that, nearly every release times out together and the lock connection is rebuilt. The old session's finalizer runspg_advisory_unlock_all()(orRELEASE_ALL_LOCKS()), so the runner loses every lock it holds.retryShardReleaseretries every 50 ms, so the new connection's queue fills again. The next refresh finds the kept shards lost or times out, and the runner marks lock storage unhealthy.confirmShardLock, and the request runs again once the shard is reacquired.Change
Shardingreleases one lock at a time:runShardReleaseruns the storagerelease(or a forcedreleaseAll) under a one-permit semaphore. Entity interruption, singleton shutdown and the wait for lock health still run concurrently for every shard. Only the storage call is serialized.On a single lock connection the statements already ran one after another, so this doesn't slow handoffs there. The difference is the deadline. Each release now starts its own deadline when it reaches the connection, and a refresh or acquire waits behind at most one release instead of the whole burst. Storages whose releases run on a pool (row locks on SQLite or SQL Server, custom storages) release one at a time now. Every existing handoff test passes unchanged.
Why not stop rebuilding the connection on a deadline
The issue comment offered a second option: fail a lock operation that passes its deadline without replacing a connection that is still healthy. I left that out on purpose. The deadline rebuild came in with #6516 to recover from an unresponsive reserved connection (a blackholed TCP session, a wedged statement). The
SqlRunnerStorageintegration tests depend on it ("bounds shard lock operations and rebuilds an unresponsive reserved connection", "rebuilds the reserved connection again when a rebuilt connection stops responding"). A single deadline can't tell a slow connection from a dead one. Keeping a connection that may never answer would block every later lock operation behind it. It would also take away the one thing that cleans up locks a timed-outacquiremay still take on the server.That fits how this design has handled similar cases. #10338 (closing #10330) kept the existing detector (heartbeat expiry) and avoided new judgement calls that could go wrong under a fault. This change follows the same idea: the storage keeps its recovery rule, and
Shardingstops creating the overload that set it off. So I chose bounding over adding a second, softer deadline path.Validation
Unit test (
packages/effect/test/cluster/Sharding.test.ts, "Sharding shard release burst")Two runners share in-memory lock storage.
withLockSessiongives each runner one session, asSqlRunnerStoragehas on PostgreSQL and MySQL. Statements take 5 ms and run one at a time in arrival order, and each lock operation is bounded by 500 ms. An operation that misses it replaces the session, which drops every lock the session holds. A release and a non-empty refresh are two statements, an acquire is one, and the empty liveness refresh skips the session. The config is the cluster-integration harness's (300 shards, 500 ms refresh, 1750 ms expiration).Runner A locks all 300 shards and starts 40 persisted requests whose handlers take 2 s. Then B joins and A hands off 150 shards. The test asserts:
entity 1 ran 2 timesIntegration test (
Locks.test.ts, "keeps the shards it holds while handing a burst of shards to a joining runner")The harness gets a
slowlock fault: each statement on the runner's reserved lock connection holds it for 5 ms first. The test starts one runner with all 300 shards, slows its lock connection, starts a second runner and waits for stable assignments. It records owners every 10 ms and asserts that every shard the first runner keeps never changed owner. It runs on both pg and mysql.GitHub-hosted
ubuntu-latest, 5 runs with this PR and 5 withSharding.tsreverted to main (run)Each job runs the unit test, then
pnpm test-clusterwithEFFECT_CLUSTER_TESTS=1.Sharding.tsThe holder lost shard default:6: expected [ '127.0.0.1:41667', '', …(5) ] …)ClusterCronpg: resumes without duplicate or missing ticks after the singleton owner diesEffect main itself fails 8 cluster-integration tests on these runners, every run, with or without this change (also recorded in #10677):
Entity.test.ts, pg and mysql:force-interrupts a stuck handler after the graceful termination timeout,rebalances on runner addition, graceful stop, and abrupt death,retains frozen advisory locks until the session closesLocks.test.ts, pg and mysql:converges after repeated lock-connection rebuilds and a hung releaseThe one
ClusterCronfailure ("Shard assignments did not stabilize") also happened on main'sSharding.tsin an earlier run on the same runners (run, job "before 1"), so it is an existing flake.Local
pnpm vitest run packages/effect/test/cluster: 17 files, 312 tests passed (two of three runs; the third had one timeout in "Sharding singleton cancellation", which passes on its own. Main shows similar timeouts on this host under load:RunnerServerandClusterWorkflowEngine).pnpm lint-fix,pnpm lint,pnpm checkandpnpm jsdocs --checkpass.Related
🤖 Generated with Claude Code