Repository navigation
Conversation
A slow disk flush on a commit can hold a lock refresh past the harness's 500 ms shardLockRefreshInterval. confirmShardLock then fences the reply and the request replays by design, which breaks the suite's exactly-once handler counts. Start PostgreSQL with fsync=off and MySQL with innodb_flush_log_at_trx_commit=0 and sync_binlog=0. Smoke.test.ts now checks those settings, and ReplyFence.test.ts pins the fencing contract: the harness holds the lock check before the first reply (holdShardLockCheck) past the interval, leaving the regular lock refresh alone, so the reply is fenced, the request runs exactly twice, and a duplicate is served from the stored reply without running the handler. Closes Effect-TS#10673 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Contributor
Bundle Size AnalysisGenerated from PR build output; treat the content below as untrusted.
|
Only a persisted reply's lock check refreshes a single shard while one runner stably owns two, so the test arms the hold before sending the request and the entity fixture no longer needs a handler hook. The hold now lives on the runner's storage controller, like freeze and faultLock, instead of a cluster-wide object threaded into every controller.
1 of 5 tasks
This branch is waiting to be deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Type
Description
Tests only; no change to
effectitself.Several cluster-integration tests assert that a handler ran exactly once (for example
persists a non-JSON Error as a serializable defect, throughassertStoredAsDefect). Before a persisted reply is stored,confirmShardLock(packages/effect/src/cluster/Sharding.ts) refreshes the shard's lock underEffect.timeoutOption(shardLockInterval), which is 500 ms in this harness. A refresh that takes longer fences the reply, and the request replays. That is the designed at-least-once behaviour, so thecount == 1assertion fails rather than the code. The test databases ran with full durability, so a slow disk flush on a commit was enough to trigger it.globalSetup.tsstarts PostgreSQL withfsync=offand MySQL withinnodb_flush_log_at_trx_commit=0andsync_binlog=0. MySQL had the same exposure: both settings defaulted to1. Everycount == 1assertion, every timeout and the 500 ms interval are unchanged.Smoke.test.tschecks the database settings within its existing runner-failure scenario, without starting separate clusters just for configuration guards.Persistence.test.tscovers a reply-confirmation timeout followed by replay, reusing its existing entity unchanged. One runner owns two explicitly configured shards. Once ownership is stable the test arms a one-shot hold on that runner and sends the request. The client completes and the handler runs exactly twice. Existing stored-reply deduplication coverage remains unchanged.holdShardLockCheck(runner, shardId)hangs that runner's next lock refresh of onlyshardId. Saving a persisted reply is the only caller that refreshes a single shard on purpose, but a periodic refresh looks the same when the runner owns one shard, so the runner needs at least two shards and stable ownership while the hold is armed.No changeset, following earlier test-only cluster-integration changes (#7224, #7228).
Current revision validation
Persistence.test.tsandSmoke.test.tspassed (34/34).expected 1 to equal 2). No production changes are included.nix develop -c pnpm lint-fix,nix develop -c pnpm check, andgit diff --checkpassed.Original author validation (before test reduction)
The following historical results refer to the original separate smoke guards and
ReplyFence.test.ts, not the current test layout.globalSetup.tschange, the guard fails on both backends. pgfsyncison; MySQLinnodb_flush_log_at_trx_commitandsync_binlogare1.Smoke.test.tsandReplyFence.test.tspass (6/6).ReplyFence.test.tspassed 20 of 20 local runs on both backends.isShardOwnedtemporarily patched to skipconfirmShardLock, it fails on both backends withexpected 1 to equal 2.Persistence.test.tsrun at a time, same 25-minute window per arm):expected 4 to equal 1,expected 2 to equal 1twice).CREATE TABLEtook 2.5 to 4.6 s (0.05 s unloaded) while inserts stayed fast, so these come from per-test table creation, not from the 500 ms fence. This was a stress test, not a CI-like comparison.Residency.test.tsmysql: stays at the entity cap and keeps residents movingfails the same way with or without this change (replied: 8, unprocessed: 5after 45 s, with healthy lock storage). Locally it passed 20 of 20 runs with this branch's settings and 19 of 20 with main's settings.ubuntu-latestrunners,pnpm test-clusterfive times withglobalSetup.tsreverted tomainand five times with this branch (run):Smoke.test.tsguard fails 5/5 onmain's settings and passes 5/5 with this branch;ReplyFence.test.tspasses in all 10 runs on both backends;Entity.test.tsandLocks.test.tsthat also fail onmainon these runners, every run, with or without this change (also seen in Hand off the shards of a runner whose lock operations keep failing #10676's runs);Residency.test.tsmysql: stays at the entity capfailed in 1 of the 5main-settings runs and none of this branch's.pnpm lint-fixpasses.pnpm checkpasses.Related
Closes EFF-3852
Closes #10673
🤖 Generated with Claude Code