[ffumero2003]

Context

Adopt Gremlins as an advisory signal for test integrity in b2 and base, without adding library dependencies. This revision addresses the four requested workflow and documentation items from David's review. It does not complete every evaluation task in #97.

Changes

• Install Gremlins v0.6.0 as a CLI. Keep go.mod and go.sum unchanged.
• Add advisory PR diff and weekly full-suite jobs with read-only permissions, pinned action SHAs, no B2 credentials and no -race during mutation runs.
• Guard changed-code against both a Dependabot actor and a Dependabot PR author. full-suite explicitly excludes pull requests and runs only on the weekly schedule or manual dispatch.
• Upload mutation-*.json and mutation.log from both jobs, always, with missing-file warnings and 30-day retention. Both jobs also write log summaries.
• Configure timeout coefficient 20, two workers and zero advisory thresholds. Only ./b2 and ./base are mutation targets; neither contains generated non-test files. internal/b2assets, examples and bin are outside that scope, and Gremlins skips _test.go, so no unused exclude-files entry was added.
• Add make mutation and make mutation-diff, with one package per Gremlins invocation.
• Document the Gremlins, Avito go-mutesting and Ooze comparison from ci: evaluate and adopt mutation testing (gremlins, go-mutesting, ooze) #97, why Gremlins was chosen, local commands, report interpretation, and how to accept equivalent survivors or exclude generated/non-library files. The comparison is a repository/documentation review, not a measured Avito benchmark.

The previously measured baseline at c3b00f1, with coefficient 20 and two workers, is retained. This review revision did not rerun a full mutation pass:

Package	Previous efficacy	Refreshed efficacy	Refreshed mutator coverage	Timed out
base	71.88%	92.44%	78.67%	0
b2	64.05%	96.54%	96.66%	17

Survivor triage is tracked in #290. David's broader follow-up requests, including consolidating CLI/config tuning, validating the known v0.6.0 diff-filter limitations and settling a timeout-free baseline, remain open; this revision is limited to the four requested review items.

Verification

• actionlint v1.7.7: exit 0, no findings.

• zizmor: exit 0, no findings under the repository's existing configuration (offline mode, one suppressed audit).

• Gremlins v0.6.0 dry-run discovery, with all four B2 credential variables unset:

  gremlins unleash --dry-run ./base ./b2
  ERROR: accepts at most 1 arg(s), received 2
  
  gremlins unleash --dry-run ./base
  Runnable: 225, Not covered: 61
  Mutator coverage: 78.67%
  
  gremlins unleash --dry-run ./b2
  Runnable: 798, Not covered: 27
  Mutator coverage: 96.73%
  

  Both supported single-package invocations exited 0 and listed individual runnable mutants. These are discovery results, not a new efficacy measurement.

• git diff origin/main -- go.mod go.sum: empty.

• gofmt -l .: no output. go vet ./...: clean as part of make ci.

• make ci: exit 0 with all four B2 credential variables unset. Build, vet, formatting, module hygiene, lint, race tests, b2keys, cross-compilation, changelog check and script tests passed. Tail:

  CGO_ENABLED=0 GOOS=linux GOARCH=arm go build ./...
  CGO_ENABLED=0 GOOS=darwin GOARCH=arm64 go build ./...
  CGO_ENABLED=0 GOOS=windows GOARCH=amd64 go build ./...
  for t in scripts/*_test.sh; do "$t" || exit 1; done
  ok   comments on the workflow's own issue
  ok   ignores an issue with the same title opened by someone else
  ok   ignores a longer title that contains the phrase
  ok   picks the oldest matching issue
  ok   opens an issue when none exists
  ok   prints the summary to the log
  ok   writes the summary file too
  ok   prints the per-package result
  ok   prints the output of a failed test
  ok   prints only failed tests' output
  ok   prints the timeout reason of a cut-off test
  ok   skips finished tests in a failed package
  ok   keeps the error line after long failure output
  ok   keeps the last lines of long failure output
  ok   says how many lines were omitted
  ok   keeps the timeout reason above a long stack trace
  ok   keeps a timeout reason that follows 150 progress lines
  ok   keeps the failure of the first of three failing tests
  ok   keeps the failure of the quiet test between two noisy ones
  ok   keeps the failure of the last of three failing tests
  

CI/tooling and docs only, with no user-visible library behavior change: skip-changelog, no fragment. Live B2 and simulator checks were not run because no B2 request behavior changes. A human code owner must review the workflow changes.

Part of #97

[david-santacruz-backblaze]

Mate, let's check this before merging.

1. .github/workflows/mutation.yml: the changed-code job should have if: ${{ github.actor != 'dependabot[bot]' && github.event.pull_request.user.login != 'dependabot[bot]' }}. It has a pull_request trigger, and #97 R8 asks for the guard. It could be combined with the existing if using &&.
2. .gremlins.yaml: it should probably have exclude-files (#97 R6), or the PR should say why it isn't needed because only ./b2 and ./base are mutated.
3. changed-code: it should write killed/lived/not-covered counts to the job summary and upload mutation-*.json like full-suite does (#97 R5).
4. docs/testing.md: it should include the Gremlins vs go-mutesting vs Ooze comparison, why Gremlins was chosen, and how to accept or suppress a surviving mutant (#97 R7).

Note: let's not forget to update the PR description once these land, so it matches the final diff.

[david-santacruz-backblaze]

Requesting changes. Reviewed together with #293, which depends on this PR. I read the diffs and did not run Gremlins or make ci.

Blocking

1. Does not close #97. Change "Closes #97" to "Part of #97". Missing against the issue:
   • the Dependabot guard (github.actor != 'dependabot[bot]') on every job in a pull_request workflow, as changelog.yml and require-area-approvals.yml have;
   • a job summary with killed/lived/not-covered counts and a downloadable list of survivors (today it is tail -n 40 of the log, and changed-code uploads no artifact);
   • exclusion of generated and non-library code;
   • docs for the tool choice, how to suppress or accept a survivor, and thresholds near the baseline (both are 0 here).
2. timeout-coefficient and workers are repeated as CLI flags in the Makefile (2x) and the workflow (2x), and .gremlins.yaml sets them too. CLI flags override the YAML, so #293's coefficient of 40 would have no effect on CI. Keep the values only in .gremlins.yaml.
3. --diff mode is unproven. #293's docs/mutation-testing.md says Gremlins v0.6.0 builds package-relative paths that do not match Git's b2/... paths, and that grouped hunks can omit later changed lines. If that is right, changed-code mutates little or nothing and still shows green. Please show it mutating a known changed line, or document the limitation.
4. Baseline is not reproducible on main. It is measured at coefficient 20, which gives 17 timeouts. Settle on one coefficient (#293 uses 40) and re-measure once.

Should fix

• Add the make ci tail to the PR (AGENTS.md "Verifying your work"), and say why no changelog fragment is needed.
• Have the workflow call make mutation / make mutation-diff instead of repeating the package loop and install step. MUTATION_PACKAGES exists but the workflow hardcodes b2 base.
• continue-on-error is on both the job and the step, which is redundant and hides failures. Drop one and surface the result in the summary.
• Remove the unused GREMLINS ?= override and the comment-only GREMLINS_VERSION, or use it in an install target.
• Use one term: mutant-coverage in the YAML vs "Mutator coverage" in the docs.
• docs/README.md does not mention the new section.

Merge order: this PR first, then #293 rebased onto it (both add .gremlins.yaml, and a trial merge conflicts).

[ffumero2003]

Thanks for the review. All four are addressed in 41e3725:

1. Dependabot guard (Fixed). Added github.actor != 'dependabot[bot]' && github.event.pull_request.user.login != 'dependabot[bot]' to the changed-code job's if, combined with the existing condition via &&. full-suite already excludes pull requests (github.event_name != 'pull_request'), so it needs no guard.
2. .gremlins.yaml exclude-files (Fixed, documented rather than configured). Only ./b2 and ./base are mutated, neither contains generated non-test files, and Gremlins skips _test.go, so internal/b2assets, examples and bin are already out of scope. Added a comment stating this instead of an unused exclude-files entry.
3. changed-code outputs (Fixed). It now uploads mutation-*.json and mutation.log as an artifact, matching full-suite, alongside the existing job-summary counts.
4. docs/testing.md (Fixed). Added the Gremlins vs Avito go-mutesting vs Ooze comparison, why Gremlins was chosen, and a section on how to accept or suppress a surviving mutant.

The PR description is updated to match the final diff. The broader #97 follow-ups (CLI/config tuning consolidation, the v0.6.0 diff-filter limitation, a timeout-free baseline) remain listed there as open, out of scope for this revision.