Repository navigation
test(b2): pin surviving mutation behavior - #293
Merged
Merged
Conversation
david-santacruz-backblaze
suggested changes
Nov 4, 2026
david-santacruz-backblaze
left a comment
There was a problem hiding this comment.
Requesting changes. Reviewed together with #291, which this PR depends on. I read the diff and did not run tests or Gremlins, so the pasted red/green proofs are unverified on my side.
Blocking
- The 24 "equivalent" annotations suppress nothing. They are one comment block in
b2/mutation_equivalence_test.go, away from the production lines, so Gremlins keeps reporting them as LIVED. The same table is also indocs/mutation-testing.md, and thefile:linereferences will go stale. Keep one copy (the doc), drop the comment-only_test.gofile, and say plainly that they remain LIVED. - PR text vs data for
reader.go:269. The scoped run killed it, then it is called equivalent because a replay did not reproduce the kill. State this in the PR body, and confirm the full-run LIVED count agrees with "3 killed, 24 equivalent". - Historical timeouts. The proof table lists fewer distinct rows than the 17 claimed, and
writer.go:610appears as both an equivalent survivor and a timeout. Reconcile the two lists. .gremlins.yamlconflicts with #291 (coefficient 40 vs 20), and #291's CLI flags would override it. After #291 is fixed, rebase and drop the duplicate file, changing only the coefficient.- Efficacy rises from 96.54% to 96.99%. #290 asks for "toward the high-90s". Either say that is what Part A can deliver or explain the gap.
Should fix
TestWriterWorkerStartupCompletesre-executes the test binary with a 5s wall-clock bound. That is fragile on slow-racerunners and where subprocess spawn is restricted.TestTransferOperationsComplete: on timeoutt.Fatalreturns while the goroutine is still running. Extract one helper for the "5s context, done channel, select" pattern used three times, and clean up the goroutine.TestAccountInfoFilePersistence: the content check now runs underwriteErr != nil || syncErr != nilinstead ofwantErr != nil. Add a comment saying why that is not a weakening.- The
a_filename prefix exists only to steer Gremlins' run order. Document it indocs/testing.mdor drop it. - Name the SHA-1 constant (
a9993e36..., repeated three times) and replacemap[bool]string{...}[succeed]. docs/README.mddoes not listdocs/mutation-testing.md.
Merge order: after #291.
ffumero2003
force-pushed
the
test/kill-surviving-mutants
branch
from
November 4, 2026 22:12
ac8aa12 to
cda3d7b
Compare
Contributor
Author
|
Thanks for the thorough review. All blocking and should-fix items are addressed, rebased onto main now that #291 has merged (head Blocking
Should-fix
|
david-santacruz-backblaze
approved these changes
Nov 4, 2026
Assert session cleanup and retry failures and preserve cached CORS in unrelated updates. Annotate 24 equivalent mutants instead of forcing assertions on unreachable or behavior-identical states. Add bounded completion checks for the 17 historical mutation timeouts and raise the advisory Gremlins timeout coefficient to 40. Record individual break proofs and the scoped remeasurement without changing SDK behavior. Part of #290
Explain the deliberate subminimum multipart chunks used against the test fakes. Record the complete Gremlins v0.6.0 B2 run: 774 killed, 24 equivalent survivors, 27 uncovered and zero timeouts, with efficacy rising to 96.99%. Part of #290
Join completion checks before fixture cleanup, allow slow worker subprocess startup, and keep equivalent-survivor classifications in the audit document. Part of #290
Remove the four command-line timeout overrides so Make and workflow mutation runs inherit coefficient 40 from .gremlins.yaml. Part of #290
The normal Close result is asserted; the deferred fallback only releases resources after an earlier test failure. Part of #290
Share subprocess containment with startup checks so a mutated reader loop that ignores cancellation is killed and joined before the test reports failure. Part of #290
Record the completed coefficient-40 mutation run, distinguish operator identities, keep equivalent mutants visible as LIVED, and explain the measured efficacy ceiling. Part of #290
After rebasing onto #292, the merged TestAccountInfoFilePersistence cases cover chmod- and close-failure paths that fail before the rename, so they preserve the previous session. Extend the assertion to treat chmod/close errors like write/sync errors (old session retained); cleanup errors still occur after replacement and keep the new session. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
ffumero2003
force-pushed
the
test/kill-surviving-mutants
branch
from
November 4, 2026 23:08
cda3d7b to
987dbca
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Context
Part A of #290 closes three observable test gaps among the 27 baseline B2 survivors.
The other 24 remain LIVED in Gremlins and are classified equivalent by human
review of reachable SDK states. They are not suppressed. Their equivalence table
lives only in docs/mutation-testing.md.
Changes
CORS request body retained during unrelated bucket updates.
context readers, and hashing. The shared completion helper cancels and joins
its goroutine before a timeout fails the test. Startup and transfer checks share
a subprocess wrapper with a one-minute allowance for slow race-enabled runners;
cancellation kills and joins even reader/writer loops that ignore the context.
b2/mutation_equivalence_test.go. Its annotations didnot suppress mutants; the document is the sole equivalence record.
2c1d1c8). Keep main's single.gremlins.yamlandits comment block; its only diff is
timeout-coefficient: 20to40.--timeout-coefficient 20CLI overrides from both Makefile mutation targets andboth
.github/workflows/mutation.ymlcommands. The coefficient now lives onlyin
.gremlins.yamlat 40 and governs Make/CI. The rest of those two files isunchanged. A human code owner must review the workflow change.
a_filenames' mutation-run ordering, name the published abc SHA-1test vector, clarify the post-replacement cleanup assertion, simplify subtest
naming, and list the mutation audit in
docs/README.md.SDK production code and
go.mod/go.sumare unchanged.skip-changelogapplies.Base survivors and Part B coverage are separate work.
Mutation results and reconciliation
The complete Gremlins v0.6.0 run used no race detector and had B2 credentials
unset:
gremlins unleash --timeout-coefficient 40 --workers 2 ./b2.c3b00f1, coefficient 20)286f28d, coefficient 40)Matching the new JSON report against the baseline confirms exactly 3 of the
original 27 survivors are KILLED and 24 remain LIVED. All 24 are classified
equivalent by human review; this is not suppression. There are no additional
LIVED mutants.
reader.go:269:18(CONDITIONALS_BOUNDARY) was reported KILLED by the earlierscoped
--diffrun. Applying that exact mutation alone did not reproduce the killin three full-suite replays. Source review classifies its equal-value clamp as
equivalent; the scoped kill is anomalous and is not one of the three behavioral
kills. The new full pass reports it LIVED.
The 17 historical timeouts are 17 distinct mutant identities, recorded with
file, line, column, and operator in the document. At
writer.go:610:26,CONDITIONALS_BOUNDARYandCONDITIONALS_NEGATIONare distinct mutants: the formerwas a baseline survivor, the latter a baseline timeout. Historical timeout is
provenance, not a second current status. In the new full run the boundary mutant
is LIVED, the negation mutant is KILLED, and all 17 historical timeout
mutants are KILLED. No mutant is currently timed out. The document lists all 17
identities with their one current status; it does not conflate operators at the
same line.
96.99% is what Part A delivers, up from 96.54%. For the unchanged production
code and the reviewed equivalents, this is the realistic ceiling:
774 / (774 + 24) = 96.99%. Equivalent LIVED mutants remain in efficacy'sdenominator; suppressing or excluding them would inflate the score. The 27
uncovered mutations, including live-only limitations, affect mutator coverage
instead and remain Part B's separate scope.
The document retains the three behavioral red/green proofs and the historical
individual timeout proofs, clearly labeled as earlier output. The new full JSON
report determines the current classifications.
Verification
checks: PASS.
entrypoint:
/private/tmp/blazer-290-a/.gremlins.yaml, coefficient 40.make mutationandmake mutation-diff: PASS in dry-run smoke mode forb2andbase, using the real CLI. This verifies target execution withoutclaiming another complete mutation pass.
git diff origin/main -- go.mod go.sum: empty.gofmt -l .: no output;go vet ./...(also run by CI): PASS.make ci: PASS with B2 credentials unset, including build, vet, format,tidy/verify, lint (
0 issues), race tests, b2keys, cross-compiles, changelogcheck, and repository scripts.
production request behavior or harness scenario changes.
Actual
make citail (exit 0):Part of #290