Skip to content

test(b2): pin surviving mutation behavior - #293

Merged
ffumero2003 merged 8 commits into
mainfrom
test/kill-surviving-mutants
Nov 4, 2026
Merged

ffumero2003 merged 8 commits into
mainfrom
test/kill-surviving-mutants

Conversation

@ffumero2003

@ffumero2003 ffumero2003 commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

Context

Part A of #290 closes three observable test gaps among the 27 baseline B2 survivors.
The other 24 remain LIVED in Gremlins and are classified equivalent by human
review of reachable SDK states. They are not suppressed. Their equivalence table
lives only in docs/mutation-testing.md.

Changes

  • Assert account-info cleanup errors, Windows rename retry outcomes, and the exact
    CORS request body retained during unrelated bucket updates.
  • Add completion checks for transfers, startup errors, pagination, reader-at,
    context readers, and hashing. The shared completion helper cancels and joins
    its goroutine before a timeout fails the test. Startup and transfer checks share
    a subprocess wrapper with a one-minute allowance for slow race-enabled runners;
    cancellation kills and joins even reader/writer loops that ignore the context.
  • Delete the comment-only b2/mutation_equivalence_test.go. Its annotations did
    not suppress mutants; the document is the sole equivalence record.
  • Rebase onto main after ci: adopt Gremlins mutation testing (advisory) #291 (2c1d1c8). Keep main's single .gremlins.yaml and
    its comment block; its only diff is timeout-coefficient: 20 to 40.
  • Resolve David's point 4 completely: remove only the four
    --timeout-coefficient 20 CLI overrides from both Makefile mutation targets and
    both .github/workflows/mutation.yml commands. The coefficient now lives only
    in .gremlins.yaml at 40 and governs Make/CI. The rest of those two files is
    unchanged. A human code owner must review the workflow change.
  • Document the a_ filenames' mutation-run ordering, name the published abc SHA-1
    test vector, clarify the post-replacement cleanup assertion, simplify subtest
    naming, and list the mutation audit in docs/README.md.

SDK production code and go.mod/go.sum are unchanged. skip-changelog applies.
Base survivors and Part B coverage are separate work.

Mutation results and reconciliation

The complete Gremlins v0.6.0 run used no race detector and had B2 credentials
unset: gremlins unleash --timeout-coefficient 40 --workers 2 ./b2.

Run Killed Lived Human-classified equivalent (subset of Lived) Not covered Timed out Efficacy Mutator coverage
#290 baseline (c3b00f1, coefficient 20) 754 27 — 27 17 96.54% 96.66%
Part A review rerun (286f28d, coefficient 40) 774 24 24 27 0 96.99% 96.73%
Mutation testing completed in 46 minutes 48 seconds
Killed: 774, Lived: 24, Not covered: 27
Timed out: 0, Not viable: 0, Skipped: 0
Test efficacy: 96.99%
Mutator coverage: 96.73%

Matching the new JSON report against the baseline confirms exactly 3 of the
original 27 survivors are KILLED and 24 remain LIVED
. All 24 are classified
equivalent by human review; this is not suppression. There are no additional
LIVED mutants.

reader.go:269:18 (CONDITIONALS_BOUNDARY) was reported KILLED by the earlier
scoped --diff run. Applying that exact mutation alone did not reproduce the kill
in three full-suite replays. Source review classifies its equal-value clamp as
equivalent; the scoped kill is anomalous and is not one of the three behavioral
kills. The new full pass reports it LIVED.

The 17 historical timeouts are 17 distinct mutant identities, recorded with
file, line, column, and operator in the document. At writer.go:610:26,
CONDITIONALS_BOUNDARY and CONDITIONALS_NEGATION are distinct mutants: the former
was a baseline survivor, the latter a baseline timeout. Historical timeout is
provenance, not a second current status. In the new full run the boundary mutant
is LIVED, the negation mutant is KILLED, and all 17 historical timeout
mutants are KILLED
. No mutant is currently timed out. The document lists all 17
identities with their one current status; it does not conflate operators at the
same line.

96.99% is what Part A delivers, up from 96.54%. For the unchanged production
code and the reviewed equivalents, this is the realistic ceiling:
774 / (774 + 24) = 96.99%. Equivalent LIVED mutants remain in efficacy's
denominator; suppressing or excluding them would inflate the score. The 27
uncovered mutations, including live-only limitations, affect mutator coverage
instead and remain Part B's separate scope.

The document retains the three behavioral red/green proofs and the historical
individual timeout proofs, clearly labeled as earlier output. The new full JSON
report determines the current classifications.

Verification

  • Focused race tests for the revised completion, persistence, rename, and hashing
    checks: PASS.
  • No-timeout-flag configuration probe through Gremlins v0.6.0's actual CLI
    entrypoint: /private/tmp/blazer-290-a/.gremlins.yaml, coefficient 40.
  • make mutation and make mutation-diff: PASS in dry-run smoke mode for
    b2 and base, using the real CLI. This verifies target execution without
    claiming another complete mutation pass.
  • git diff origin/main -- go.mod go.sum: empty.
  • gofmt -l .: no output; go vet ./... (also run by CI): PASS.
  • make ci: PASS with B2 credentials unset, including build, vet, format,
    tidy/verify, lint (0 issues), race tests, b2keys, cross-compiles, changelog
    check, and repository scripts.
  • Live B2 and SDK harness: NOT RUN; tests and mutation tooling change, with no
    production request behavior or harness scenario changes.

Actual make ci tail (exit 0):

CGO_ENABLED=0 GOOS=windows GOARCH=amd64 go build ./...
for t in scripts/*_test.sh; do "$t" || exit 1; done
ok   comments on the workflow's own issue
ok   ignores an issue with the same title opened by someone else
ok   ignores a longer title that contains the phrase
ok   picks the oldest matching issue
ok   opens an issue when none exists
ok   prints the summary to the log
ok   writes the summary file too
ok   prints the per-package result
ok   prints the output of a failed test
ok   prints only failed tests' output
ok   prints the timeout reason of a cut-off test
ok   skips finished tests in a failed package
ok   keeps the error line after long failure output
ok   keeps the last lines of long failure output
ok   says how many lines were omitted
ok   keeps the timeout reason above a long stack trace
ok   keeps a timeout reason that follows 150 progress lines
ok   keeps the failure of the first of three failing tests
ok   keeps the failure of the quiet test between two noisy ones
ok   keeps the failure of the last of three failing tests

Part of #290

@ffumero2003
ffumero2003 requested a review from goanpeca as a code owner November 4, 2026 19:03
@ffumero2003 ffumero2003 added the skip-changelog No changelog entry needed; skips the changelog check label Nov 4, 2026
@ffumero2003 ffumero2003 added the testing Tests, test coverage and test infrastructure label Nov 4, 2026
@ffumero2003 ffumero2003 self-assigned this Nov 4, 2026

@david-santacruz-backblaze david-santacruz-backblaze left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes. Reviewed together with #291, which this PR depends on. I read the diff and did not run tests or Gremlins, so the pasted red/green proofs are unverified on my side.

Blocking

  1. The 24 "equivalent" annotations suppress nothing. They are one comment block in b2/mutation_equivalence_test.go, away from the production lines, so Gremlins keeps reporting them as LIVED. The same table is also in docs/mutation-testing.md, and the file:line references will go stale. Keep one copy (the doc), drop the comment-only _test.go file, and say plainly that they remain LIVED.
  2. PR text vs data for reader.go:269. The scoped run killed it, then it is called equivalent because a replay did not reproduce the kill. State this in the PR body, and confirm the full-run LIVED count agrees with "3 killed, 24 equivalent".
  3. Historical timeouts. The proof table lists fewer distinct rows than the 17 claimed, and writer.go:610 appears as both an equivalent survivor and a timeout. Reconcile the two lists.
  4. .gremlins.yaml conflicts with #291 (coefficient 40 vs 20), and #291's CLI flags would override it. After #291 is fixed, rebase and drop the duplicate file, changing only the coefficient.
  5. Efficacy rises from 96.54% to 96.99%. #290 asks for "toward the high-90s". Either say that is what Part A can deliver or explain the gap.

Should fix

  • TestWriterWorkerStartupCompletes re-executes the test binary with a 5s wall-clock bound. That is fragile on slow -race runners and where subprocess spawn is restricted.
  • TestTransferOperationsComplete: on timeout t.Fatal returns while the goroutine is still running. Extract one helper for the "5s context, done channel, select" pattern used three times, and clean up the goroutine.
  • TestAccountInfoFilePersistence: the content check now runs under writeErr != nil || syncErr != nil instead of wantErr != nil. Add a comment saying why that is not a weakening.
  • The a_ filename prefix exists only to steer Gremlins' run order. Document it in docs/testing.md or drop it.
  • Name the SHA-1 constant (a9993e36..., repeated three times) and replace map[bool]string{...}[succeed].
  • docs/README.md does not list docs/mutation-testing.md.

Merge order: after #291.

@ffumero2003
ffumero2003 force-pushed the test/kill-surviving-mutants branch from ac8aa12 to cda3d7b Compare November 4, 2026 22:12
@ffumero2003

Copy link
Copy Markdown
Contributor Author

Thanks for the thorough review. All blocking and should-fix items are addressed, rebased onto main now that #291 has merged (head cda3d7b). The numbers below come from a fresh full ./b2 pass (46m48s) on the rebased test snapshot: 774 killed, 24 lived (all 24 equivalent), 0 timed out, efficacy 96.99%.

Blocking

  1. Equivalent annotations (Fixed). Deleted the comment-only b2/mutation_equivalence_test.go; the equivalence table now lives only in docs/mutation-testing.md and states plainly that these 24 remain LIVED, suppressing nothing.
  2. reader.go:269 (Fixed). The PR body now documents the anomaly: the scoped --diff run reported it KILLED, isolated replays did not reproduce the kill, and the fresh full run confirms it LIVED. It is classified equivalent, not counted among the behavioral kills.
  3. Historical timeouts (Fixed). All 17 historical timeout identities now report KILLED in the full run, reconciled by location and operator. At writer.go:610 the two operators are distinguished: boundary is LIVED (equivalent), negation is KILLED.
  4. .gremlins.yaml conflict (Fixed). Rebased; the only YAML change is timeout-coefficient: 20 -> 40, main's comments preserved. Since the Makefile and workflow passed --timeout-coefficient 20 on the CLI (which would have overridden the file), those four overrides are removed so .gremlins.yaml is the single source of truth at 40. A no-flag probe confirms the effective coefficient is 40.
  5. Efficacy (Fixed). The docs and PR body state the measured 96.99% ceiling and explain it: the remaining 24 survivors are equivalent mutants retained by design, so Part A's realistic ceiling is the high-90s rather than 100%.

Should-fix

  • TestWriterWorkerStartupCompletes / TestTransferOperationsComplete: extracted a shared subprocess-containment wrapper so the parent can terminate and join a mutated loop that ignores cancellation, with the goroutine cleaned up on timeout. (This also surfaced and fixed a real hang two reader mutations exposed under the full run.)
  • TestAccountInfoFilePersistence: added a comment explaining why the content check under writeErr != nil || syncErr != nil is not a weakening.
  • Documented the a_ filename prefix (Gremlins run-order) in the docs.
  • Named the repeated SHA-1 constant and replaced the map[bool]string{...}[succeed] idiom.
  • docs/README.md now lists docs/mutation-testing.md.

make ci passes with all B2 credential variables unset; formatting/vet clean; go.mod/go.sum unchanged.

ffumero2003 and others added 8 commits November 4, 2026 16:46
Assert session cleanup and retry failures and preserve cached CORS in unrelated updates. Annotate 24 equivalent mutants instead of forcing assertions on unreachable or behavior-identical states.

Add bounded completion checks for the 17 historical mutation timeouts and raise the advisory Gremlins timeout coefficient to 40. Record individual break proofs and the scoped remeasurement without changing SDK behavior.

Part of #290
Explain the deliberate subminimum multipart chunks used against the test fakes. Record the complete Gremlins v0.6.0 B2 run: 774 killed, 24 equivalent survivors, 27 uncovered and zero timeouts, with efficacy rising to 96.99%.

Part of #290
Join completion checks before fixture cleanup, allow slow worker subprocess startup, and keep equivalent-survivor classifications in the audit document.

Part of #290
Remove the four command-line timeout overrides so Make and workflow mutation runs inherit coefficient 40 from .gremlins.yaml.

Part of #290
The normal Close result is asserted; the deferred fallback only releases resources after an earlier test failure.

Part of #290
Share subprocess containment with startup checks so a mutated reader loop that ignores cancellation is killed and joined before the test reports failure.

Part of #290
Record the completed coefficient-40 mutation run, distinguish operator identities, keep equivalent mutants visible as LIVED, and explain the measured efficacy ceiling.

Part of #290
After rebasing onto #292, the merged TestAccountInfoFilePersistence cases
cover chmod- and close-failure paths that fail before the rename, so they
preserve the previous session. Extend the assertion to treat chmod/close
errors like write/sync errors (old session retained); cleanup errors still
occur after replacement and keep the new session.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@ffumero2003
ffumero2003 force-pushed the test/kill-surviving-mutants branch from cda3d7b to 987dbca Compare November 4, 2026 23:08
@ffumero2003
ffumero2003 merged commit 59104c6 into main Nov 4, 2026
27 of 28 checks passed
@ffumero2003
ffumero2003 deleted the test/kill-surviving-mutants branch November 4, 2026 23:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

skip-changelog No changelog entry needed; skips the changelog check testing Tests, test coverage and test infrastructure

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants