Skip to content

Skip disk flushes on the cluster-integration test databases - #10677

Merged
tim-smart merged 3 commits into
Effect-TS:mainfrom
tylergibbs1:test-cluster-integration-durability-off
Nov 17, 2026
Merged

tim-smart merged 3 commits into
Effect-TS:mainfrom
tylergibbs1:test-cluster-integration-durability-off

Conversation

@tylergibbs1

@tylergibbs1 tylergibbs1 commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Type

  • Refactor
  • Feature
  • Bug Fix
  • Optimization
  • Documentation Update

Description

Tests only; no change to effect itself.

Several cluster-integration tests assert that a handler ran exactly once (for example persists a non-JSON Error as a serializable defect, through assertStoredAsDefect). Before a persisted reply is stored, confirmShardLock (packages/effect/src/cluster/Sharding.ts) refreshes the shard's lock under Effect.timeoutOption(shardLockInterval), which is 500 ms in this harness. A refresh that takes longer fences the reply, and the request replays. That is the designed at-least-once behaviour, so the count == 1 assertion fails rather than the code. The test databases ran with full durability, so a slow disk flush on a commit was enough to trigger it.

  • globalSetup.ts starts PostgreSQL with fsync=off and MySQL with innodb_flush_log_at_trx_commit=0 and sync_binlog=0. MySQL had the same exposure: both settings defaulted to 1. Every count == 1 assertion, every timeout and the 500 ms interval are unchanged.
  • Smoke.test.ts checks the database settings within its existing runner-failure scenario, without starting separate clusters just for configuration guards.
  • Persistence.test.ts covers a reply-confirmation timeout followed by replay, reusing its existing entity unchanged. One runner owns two explicitly configured shards. Once ownership is stable the test arms a one-shot hold on that runner and sends the request. The client completes and the handler runs exactly twice. Existing stored-reply deduplication coverage remains unchanged.
  • holdShardLockCheck(runner, shardId) hangs that runner's next lock refresh of only shardId. Saving a persisted reply is the only caller that refreshes a single shard on purpose, but a periodic refresh looks the same when the runner owns one shard, so the runner needs at least two shards and stable ownership while the hold is armed.
  • The README documents this precondition and clarifies that database crash recovery and power-loss durability are outside this runner-failure suite.

No changeset, following earlier test-only cluster-integration changes (#7224, #7228).

Current revision validation

  • PostgreSQL and MySQL: complete Persistence.test.ts and Smoke.test.ts passed (34/34).
  • The replay test passed 5 of 5 further runs on both backends.
  • Negative mutation: temporarily bypassing reply lock confirmation made the replay test fail on both backends (expected 1 to equal 2). No production changes are included.
  • nix develop -c pnpm lint-fix, nix develop -c pnpm check, and git diff --check passed.
  • Full cluster suite, repeated stress runs, and hosted A/B comparisons were not rerun for this revision.

Original author validation (before test reduction)

The following historical results refer to the original separate smoke guards and ReplyFence.test.ts, not the current test layout.

  • Red: before the globalSetup.ts change, the guard fails on both backends. pg fsync is on; MySQL innodb_flush_log_at_trx_commit and sync_binlog are 1.
  • Green: after it, Smoke.test.ts and ReplyFence.test.ts pass (6/6).
  • ReplyFence.test.ts passed 20 of 20 local runs on both backends.
  • The fence test checks real behaviour: with isShardOwned temporarily patched to skip confirmShardLock, it fails on both backends with expected 1 to equal 2.
  • Under disk load (12 processes in the Docker VM writing and fsyncing 64 MB files, databases started and healthy before the load began, one Persistence.test.ts run at a time, same 25-minute window per arm):
    • main's settings: 1 run in the window, 26 of 30 tests failed, 224 "Shard lock storage is unhealthy" events, 3 exactly-once violations (expected 4 to equal 1, expected 2 to equal 1 twice).
    • this branch's settings: 2 runs, 0 lock-storage-unhealthy events, 0 exactly-once violations, every pg test passed. 8 MySQL tests still hit the 60 s test timeout. Under that load MySQL CREATE TABLE took 2.5 to 4.6 s (0.05 s unloaded) while inserts stayed fast, so these come from per-test table creation, not from the 500 ms fence. This was a stress test, not a CI-like comparison.
  • Residency.test.ts mysql: stays at the entity cap and keeps residents moving fails the same way with or without this change (replied: 8, unprocessed: 5 after 45 s, with healthy lock storage). Locally it passed 20 of 20 runs with this branch's settings and 19 of 20 with main's settings.
  • On GitHub-hosted ubuntu-latest runners, pnpm test-cluster five times with globalSetup.ts reverted to main and five times with this branch (run):
    • the Smoke.test.ts guard fails 5/5 on main's settings and passes 5/5 with this branch;
    • ReplyFence.test.ts passes in all 10 runs on both backends;
    • with this branch, the only failures are the same 8 tests in Entity.test.ts and Locks.test.ts that also fail on main on these runners, every run, with or without this change (also seen in Hand off the shards of a runner whose lock operations keep failing #10676's runs);
    • Residency.test.ts mysql: stays at the entity cap failed in 1 of the 5 main-settings runs and none of this branch's.
  • pnpm lint-fix passes.
  • pnpm check passes.

Related

Closes EFF-3852
Closes #10673

🤖 Generated with Claude Code

A slow disk flush on a commit can hold a lock refresh past the harness's
500 ms shardLockRefreshInterval. confirmShardLock then fences the reply and
the request replays by design, which breaks the suite's exactly-once handler
counts. Start PostgreSQL with fsync=off and MySQL with
innodb_flush_log_at_trx_commit=0 and sync_binlog=0.

Smoke.test.ts now checks those settings, and ReplyFence.test.ts pins the
fencing contract: the harness holds the lock check before the first reply
(holdShardLockCheck) past the interval, leaving the regular lock refresh
alone, so the reply is fenced, the request runs exactly twice, and a
duplicate is served from the stored reply without running the handler.

Closes Effect-TS#10673

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@changeset-bot

changeset-bot Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: 416a26c

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@github-actions

Copy link
Copy Markdown
Contributor

Bundle Size Analysis

Generated from PR build output; treat the content below as untrusted.

File Name Current Size Previous Size Difference
arbitrary-combinators.ts 40.09 KB 40.09 KB 0.00 KB (0.00%)
basic.ts 7.31 KB 7.31 KB 0.00 KB (0.00%)
batching.ts 11.21 KB 11.21 KB 0.00 KB (0.00%)
brand.ts 7.00 KB 7.00 KB 0.00 KB (0.00%)
cache.ts 11.73 KB 11.73 KB 0.00 KB (0.00%)
config.ts 23.09 KB 23.09 KB 0.00 KB (0.00%)
differ.ts 22.20 KB 22.20 KB 0.00 KB (0.00%)
http-client.ts 27.45 KB 27.45 KB 0.00 KB (0.00%)
http-router.ts 37.59 KB 37.59 KB 0.00 KB (0.00%)
logger.ts 12.19 KB 12.19 KB 0.00 KB (0.00%)
metric.ts 9.65 KB 9.65 KB 0.00 KB (0.00%)
optic.ts 7.21 KB 7.21 KB 0.00 KB (0.00%)
pubsub.ts 16.21 KB 16.21 KB 0.00 KB (0.00%)
queue.ts 13.09 KB 13.09 KB 0.00 KB (0.00%)
schedule.ts 11.99 KB 11.99 KB 0.00 KB (0.00%)
schema-bigdecimal.ts 14.29 KB 14.29 KB 0.00 KB (0.00%)
schema-binary.ts 42.48 KB 42.48 KB 0.00 KB (0.00%)
schema-class.ts 21.96 KB 21.96 KB 0.00 KB (0.00%)
schema-codegen.ts 30.70 KB 30.70 KB 0.00 KB (0.00%)
schema-fromJsonSchemaDocument.ts 30.56 KB 30.56 KB 0.00 KB (0.00%)
schema-representation-roundtrip.ts 28.26 KB 28.26 KB 0.00 KB (0.00%)
schema-string-transformation.ts 15.54 KB 15.54 KB 0.00 KB (0.00%)
schema-string.ts 12.50 KB 12.50 KB 0.00 KB (0.00%)
schema-template-literal.ts 16.43 KB 16.43 KB 0.00 KB (0.00%)
schema-toArbitrary.ts 39.63 KB 39.63 KB 0.00 KB (0.00%)
schema-toCodecJson.ts 21.11 KB 21.11 KB 0.00 KB (0.00%)
schema-toEquivalence.ts 21.28 KB 21.28 KB 0.00 KB (0.00%)
schema-toFormatter.ts 21.37 KB 21.37 KB 0.00 KB (0.00%)
schema-toJsonSchemaDocument.ts 26.42 KB 26.42 KB 0.00 KB (0.00%)
schema-toRepresentation.ts 21.45 KB 21.45 KB 0.00 KB (0.00%)
schema.ts 21.05 KB 21.05 KB 0.00 KB (0.00%)
stm.ts 14.46 KB 14.46 KB 0.00 KB (0.00%)
stream.ts 10.56 KB 10.56 KB 0.00 KB (0.00%)

Only a persisted reply's lock check refreshes a single shard while one
runner stably owns two, so the test arms the hold before sending the
request and the entity fixture no longer needs a handler hook. The hold
now lives on the runner's storage controller, like freeze and faultLock,
instead of a cluster-wide object threaded into every controller.
@tim-smart
tim-smart merged commit 508e241 into Effect-TS:main Nov 17, 2026
15 of 16 checks passed

This branch is waiting to be deployed

1 waiting deployment
fork — a82e25ad Waiting Nov 16, 2026 by effect-bot via approval-gate #26428
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

cluster-integration tests: exactly-once handler assertions are flaky when a lock refresh exceeds the 500 ms fence

3 participants