Skip to content

Release shard locks one at a time - #10686

Open
tylergibbs1 wants to merge 1 commit into
Effect-TS:mainfrom
tylergibbs1:fix-cluster-shard-release-burst
Open

tylergibbs1 wants to merge 1 commit into
Effect-TS:mainfrom
tylergibbs1:fix-cluster-shard-release-burst

Conversation

@tylergibbs1

Copy link
Copy Markdown
Contributor

Type

  • Refactor
  • Feature
  • Bug Fix
  • Optimization
  • Documentation Update

Description

When a runner joins, the runner handing shards off releases them all at once. With SqlRunnerStorage on PostgreSQL or MySQL that can drop every lock the runner holds, including the shards it keeps, and requests on those shards can then run twice. This is the second mechanism described in #10673 (comment). #10677 removes the disk-flush stalls in the test databases. This PR fixes the release burst in effect itself.

Mechanism (verified on main at 30e642f)

  1. releaseShards (Sharding.ts) forks one releaseShard fiber per released shard through FiberMap.run, with no bound. Handing off 150 of 300 shards starts 150 releases together.
  2. On PostgreSQL and MySQL every lock operation runs on the one reserved lock connection. A PostgreSQL release is two statements (pg_advisory_unlock, then the pg_locks check), and MySQL's is RELEASE_LOCK then IS_USED_LOCK. The connection runs statements in arrival order (@effect/sql-pg queues unpinned statements on the connection's owner semaphore), so each release's second statement waits behind the other releases' first statements.
  3. Each operation runs under withLockOperationDeadline, bounded by the lock refresh interval (500 ms in the cluster-integration harness). Once the queue takes longer than that, nearly every release times out together and the lock connection is rebuilt. The old session's finalizer runs pg_advisory_unlock_all() (or RELEASE_ALL_LOCKS()), so the runner loses every lock it holds.
  4. retryShardRelease retries every 50 ms, so the new connection's queue fills again. The next refresh finds the kept shards lost or times out, and the runner marks lock storage unhealthy.
  5. A reply that finishes meanwhile fails confirmShardLock, and the request runs again once the shard is reacquired.

Change

Sharding releases one lock at a time: runShardRelease runs the storage release (or a forced releaseAll) under a one-permit semaphore. Entity interruption, singleton shutdown and the wait for lock health still run concurrently for every shard. Only the storage call is serialized.

On a single lock connection the statements already ran one after another, so this doesn't slow handoffs there. The difference is the deadline. Each release now starts its own deadline when it reaches the connection, and a refresh or acquire waits behind at most one release instead of the whole burst. Storages whose releases run on a pool (row locks on SQLite or SQL Server, custom storages) release one at a time now. Every existing handoff test passes unchanged.

Why not stop rebuilding the connection on a deadline

The issue comment offered a second option: fail a lock operation that passes its deadline without replacing a connection that is still healthy. I left that out on purpose. The deadline rebuild came in with #6516 to recover from an unresponsive reserved connection (a blackholed TCP session, a wedged statement). The SqlRunnerStorage integration tests depend on it ("bounds shard lock operations and rebuilds an unresponsive reserved connection", "rebuilds the reserved connection again when a rebuilt connection stops responding"). A single deadline can't tell a slow connection from a dead one. Keeping a connection that may never answer would block every later lock operation behind it. It would also take away the one thing that cleans up locks a timed-out acquire may still take on the server.

That fits how this design has handled similar cases. #10338 (closing #10330) kept the existing detector (heartbeat expiry) and avoided new judgement calls that could go wrong under a fault. This change follows the same idea: the storage keeps its recovery rule, and Sharding stops creating the overload that set it off. So I chose bounding over adding a second, softer deadline path.

Validation

Unit test (packages/effect/test/cluster/Sharding.test.ts, "Sharding shard release burst")

Two runners share in-memory lock storage. withLockSession gives each runner one session, as SqlRunnerStorage has on PostgreSQL and MySQL. Statements take 5 ms and run one at a time in arrival order, and each lock operation is bounded by 500 ms. An operation that misses it replaces the session, which drops every lock the session holds. A release and a non-empty refresh are two statements, an acquire is one, and the empty liveness refresh skips the session. The config is the cluster-integration harness's (300 shards, 500 ms refresh, 1750 ms expiration).

Runner A locks all 300 shards and starts 40 persisted requests whose handlers take 2 s. Then B joins and A hands off 150 shards. The test asserts:

  • no session holding locks was replaced, and nothing was logged at error level;
  • each shard was locked by A, or by A then B, and A ends with 150;
  • every request on a shard A keeps ran exactly once.
Source Result
main Fails: A's session is replaced twice (600 dropped locks, all 300 shards), "Shard lock storage is unhealthy" is logged, and with only the request assertion kept, entity 1 ran 2 times
this PR Passes

Integration test (Locks.test.ts, "keeps the shards it holds while handing a burst of shards to a joining runner")

The harness gets a slow lock fault: each statement on the runner's reserved lock connection holds it for 5 ms first. The test starts one runner with all 300 shards, slows its lock connection, starts a second runner and waits for stable assignments. It records owners every 10 ms and asserts that every shard the first runner keeps never changed owner. It runs on both pg and mysql.

GitHub-hosted ubuntu-latest, 5 runs with this PR and 5 with Sharding.ts reverted to main (run)

Each job runs the unit test, then pnpm test-cluster with EFFECT_CLUSTER_TESTS=1.

Unit test New Locks test (pg, mysql) Other cluster-integration failures
main's Sharding.ts 5/5 fail 5/5 fail on both (The holder lost shard default:6: expected [ '127.0.0.1:41667', '', …(5) ] …) The 8 below, every run
this PR 5/5 pass 5/5 pass on both The 8 below, every run; once, ClusterCron pg: resumes without duplicate or missing ticks after the singleton owner dies

Effect main itself fails 8 cluster-integration tests on these runners, every run, with or without this change (also recorded in #10677):

  • Entity.test.ts, pg and mysql: force-interrupts a stuck handler after the graceful termination timeout, rebalances on runner addition, graceful stop, and abrupt death, retains frozen advisory locks until the session closes
  • Locks.test.ts, pg and mysql: converges after repeated lock-connection rebuilds and a hung release

The one ClusterCron failure ("Shard assignments did not stabilize") also happened on main's Sharding.ts in an earlier run on the same runners (run, job "before 1"), so it is an existing flake.

Local

  • pnpm vitest run packages/effect/test/cluster: 17 files, 312 tests passed (two of three runs; the third had one timeout in "Sharding singleton cancellation", which passes on its own. Main shows similar timeouts on this host under load: RunnerServer and ClusterWorkflowEngine).
  • pnpm lint-fix, pnpm lint, pnpm check and pnpm jsdocs --check pass.
  • The Docker integration suite was run only on the hosted runners above, not locally.

Related

🤖 Generated with Claude Code

When a runner joins, the runner handing shards off releases them all at
once. On PostgreSQL and MySQL every lock operation runs on one reserved
connection, so the releases queue together. A burst longer than the lock
operation deadline times every release out, SqlRunnerStorage replaces the
lock connection, and closing the old session drops every lock the runner
holds, including the shards it keeps. Requests on those shards can then
run twice.

Release one lock at a time, so a refresh or acquire waits behind at most
one release.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@changeset-bot

changeset-bot Bot commented Nov 16, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: cd944d3

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 35 packages
Name Type
effect Patch
@effect/opentelemetry Patch
@effect/vitest Patch
@effect/ai-amazon-bedrock Patch
@effect/ai-anthropic Patch
@effect/ai-cloudflare Patch
@effect/ai-openai-compat Patch
@effect/ai-openai Patch
@effect/ai-openrouter Patch
@effect/ai-typesafe Patch
@effect/atom-react Patch
@effect/atom-solid Patch
@effect/atom-vue Patch
@effect/platform-browser Patch
@effect/platform-bun Patch
@effect/platform-cloudflare Patch
@effect/platform-deno Patch
@effect/platform-node-shared Patch
@effect/platform-node Patch
@effect/sql-clickhouse Patch
@effect/sql-d1 Patch
@effect/sql-libsql Patch
@effect/sql-mssql Patch
@effect/sql-mysql2 Patch
@effect/sql-pg Patch
@effect/sql-pglite Patch
@effect/sql-sqlite-bun Patch
@effect/sql-sqlite-do Patch
@effect/sql-sqlite-node Patch
@effect/sql-sqlite-react-native Patch
@effect/sql-sqlite-wasm Patch
@effect/docgen Patch
@effect/doctest Patch
@effect/graphql-generator Patch
@effect/openapi-generator Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@effect-janitor effect-janitor Bot added the 4.0 label Nov 16, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Bundle Size Analysis

Generated from PR build output; treat the content below as untrusted.

File Name Current Size Previous Size Difference
arbitrary-combinators.ts 40.09 KB 40.09 KB 0.00 KB (0.00%)
basic.ts 7.31 KB 7.31 KB 0.00 KB (0.00%)
batching.ts 11.21 KB 11.21 KB 0.00 KB (0.00%)
brand.ts 7.00 KB 7.00 KB 0.00 KB (0.00%)
cache.ts 11.73 KB 11.73 KB 0.00 KB (0.00%)
config.ts 23.09 KB 23.09 KB 0.00 KB (0.00%)
differ.ts 22.20 KB 22.20 KB 0.00 KB (0.00%)
http-client.ts 27.45 KB 27.45 KB 0.00 KB (0.00%)
http-router.ts 37.59 KB 37.59 KB 0.00 KB (0.00%)
logger.ts 12.19 KB 12.19 KB 0.00 KB (0.00%)
metric.ts 9.65 KB 9.65 KB 0.00 KB (0.00%)
optic.ts 7.21 KB 7.21 KB 0.00 KB (0.00%)
pubsub.ts 16.21 KB 16.21 KB 0.00 KB (0.00%)
queue.ts 13.09 KB 13.09 KB 0.00 KB (0.00%)
schedule.ts 11.99 KB 11.99 KB 0.00 KB (0.00%)
schema-bigdecimal.ts 14.29 KB 14.29 KB 0.00 KB (0.00%)
schema-binary.ts 42.48 KB 42.48 KB 0.00 KB (0.00%)
schema-class.ts 21.96 KB 21.96 KB 0.00 KB (0.00%)
schema-codegen.ts 30.70 KB 30.70 KB 0.00 KB (0.00%)
schema-fromJsonSchemaDocument.ts 30.56 KB 30.56 KB 0.00 KB (0.00%)
schema-representation-roundtrip.ts 28.26 KB 28.26 KB 0.00 KB (0.00%)
schema-string-transformation.ts 15.54 KB 15.54 KB 0.00 KB (0.00%)
schema-string.ts 12.50 KB 12.50 KB 0.00 KB (0.00%)
schema-template-literal.ts 16.43 KB 16.43 KB 0.00 KB (0.00%)
schema-toArbitrary.ts 39.63 KB 39.63 KB 0.00 KB (0.00%)
schema-toCodecJson.ts 21.11 KB 21.11 KB 0.00 KB (0.00%)
schema-toEquivalence.ts 21.28 KB 21.28 KB 0.00 KB (0.00%)
schema-toFormatter.ts 21.37 KB 21.37 KB 0.00 KB (0.00%)
schema-toJsonSchemaDocument.ts 26.42 KB 26.42 KB 0.00 KB (0.00%)
schema-toRepresentation.ts 21.45 KB 21.45 KB 0.00 KB (0.00%)
schema.ts 21.05 KB 21.05 KB 0.00 KB (0.00%)
stm.ts 14.46 KB 14.46 KB 0.00 KB (0.00%)
stream.ts 10.56 KB 10.56 KB 0.00 KB (0.00%)

This branch is waiting to be deployed

1 waiting deployment
fork — cd944d30 Waiting Nov 16, 2026 by tylergibbs1 via approval-gate #26441
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant