Skip to content

feat(python): require full sharded mutation on pull requests - #365

Closed
swkeever wants to merge 11 commits into
mainfrom
skeever/python-full-mutation-gate
Closed

swkeever wants to merge 11 commits into
mainfrom
skeever/python-full-mutation-gate

Conversation

@swkeever

@swkeever swkeever commented Sep 24, 2026 •

Copy link
Copy Markdown
Collaborator

Python's PR mutation job previously covered changed modules and locks, leaving untouched handwritten runtime code outside the required gate. The scheduled full audit also could not start because Poe treated MUTATION_FULL=1 as an executable. This PR makes poe quality run the complete mutation set through Poe's native shell task.

CI distributes all 24 tracked handwritten runtime modules across 23 Mutmut jobs. Each job uploads native outcome metadata; the required Quality Gate verifies that every report exists, every shard succeeds, and each module appears exactly once. Surviving, uncovered, crashed, timed-out, incomplete, or empty results fail. The source inventory comes from Git at the tested commit, so new modules enter automatically. Pinned Pyrefly checks every handwritten module before running each mutant; test-killed and statically invalid mutants remain distinct outcomes.

The URL and realtime fetch worker cleanup removes equivalent mutations and covers cancellation, backpressure, callback cleanup, and query preservation with deterministic tests. The URL rewrite was compared with the previous implementation over 100,000 generated inputs. A bounded callback test avoids counting timeout machinery as a worker future on Python 3.11.

Local validation on current main: poe checks (2,262 tests; 100% of 5,193 runtime lines and 1,316 branches), Ruff, mypy, basedpyright, package builds, isolated optional-dependency installs, installed-package typing (845/845 exported symbols), shared contract dry-run, and policy inventory. The local scoped mutation run killed 307 mutants and statically rejected 444, with no failures. Exact-head CI covered all 24 handwritten runtime modules across 23 shards: 3,247 mutants killed, 5,136 statically invalid, no survivors, uncovered mutants, timeouts, or crashes. __init__.py has no mutable operators.

@swkeever

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 24, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-09-24T14:32:48.267907Z ed1a6d0 Manual request
🔒 Security Review ✅ Completed 2026-09-24T14:29:34.400149Z ed1a6d0 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review · Automatically triggered

Security review completed. No security issues were found in this pull request.

Reviewed commit: 1de182b5a4

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 1de182b5a4

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread .github/workflows/ci.yml
runs-on: ubuntu-latest
timeout-minutes: 5
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Grant the quality checkout read access

Because this workflow sets top-level permissions: {}, the new Quality Gate inherits a GITHUB_TOKEN without contents: read; unlike the existing test and mutation jobs, it does not override that permission before invoking actions/checkout. The gate can therefore fail at checkout on every trigger before validating the shard reports, so add job-level contents: read or avoid checking out the repository.

Useful? React with 👍 / 👎.

@@ -0,0 +1,118 @@
"""Require every runtime module in exactly one successful mutation shard."""

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Document why the custom shard checker is necessary

This adds a custom enforcement mechanism without documenting the native-tool gap it fills, while maintainers/mutation-testing.md still says poe quality mutates changed modules and that CI runs the mutation task. Update the maintainer guidance to describe the new full/sharded orchestration and why Mutmut/GitHub's native configuration cannot enforce cross-shard completeness; the repository explicitly requires that justification for custom checks.

AGENTS.md reference: AGENTS.md:L3-L5

Useful? React with 👍 / 👎.

@swkeever

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Another round soon, please!

Reviewed commit: 0188f70b9b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review · Automatically triggered

Security review completed. No security issues were found in this pull request.

Reviewed commit: 0188f70b9b

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

@swkeever
swkeever force-pushed the skeever/python-full-mutation-gate branch 5 times, most recently from 583e5f2 to 6c94b2f Compare September 24, 2026 13:29
@swkeever
swkeever changed the base branch from main to skeever/python-native-any-boundaries September 24, 2026 13:29
@swkeever
swkeever force-pushed the skeever/python-full-mutation-gate branch from 6c94b2f to 22c27c8 Compare September 24, 2026 13:35
@swkeever
swkeever changed the base branch from skeever/python-native-any-boundaries to main September 24, 2026 13:35
@swkeever
swkeever force-pushed the skeever/python-full-mutation-gate branch from 22c27c8 to c40e717 Compare September 24, 2026 13:38
@swkeever

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review · Automatically triggered

Security review completed. No security issues were found in this pull request.

Reviewed commit: ed1a6d0dc2

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ed1a6d0dc2

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

requires both through `Quality Gate`. The weekly `poe mutation-full` task audits
every handwritten runtime module without a debt baseline.
`uv run --locked poe quality` runs all native checks and mutates every handwritten
runtime module. CI runs `checks` and divides `mutation-full` across eight jobs;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3 Badge Correct the documented shard-job count

Update this to say 23 jobs: the CI matrix defines shards 0–22 and the Quality Gate explicitly requires 23 reports. Describing eight jobs makes the new maintainer guidance internally contradictory and gives maintainers the wrong topology when investigating capacity, timeouts, or future shard-count changes.

Useful? React with 👍 / 👎.

@swkeever
swkeever marked this pull request as ready for review September 24, 2026 14:20
@swkeever
swkeever requested a review from a team as a code owner September 24, 2026 14:20
@swkeever

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review · Automatically triggered

Security review completed. No security issues were found in this pull request.

Reviewed commit: ed1a6d0dc2

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ed1a6d0dc2

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread .github/workflows/ci.yml
strategy:
fail-fast: false
matrix:
shard: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22]

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Derive the shard count from the runtime inventory

With the current 24-module inventory, 23 shards work only because shard 0 receives both export-only __init__.py and one mutatable module. If a PR removes any handwritten runtime module, shard 0 receives only __init__.py; scripts.mutation_results then reports “No mutants were tested,” so the mandatory mutation job fails even though every remaining mutatable module was covered. Derive the shard topology from the inventory or otherwise ensure every shard includes a mutatable module.

Useful? React with 👍 / 👎.

Comment on lines +260 to 261
async with asyncio.timeout_at(deadline):
return await self._queue.get()

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve the zero-window cutoff before dequeuing

When batch_window_seconds is zero, or the deadline has otherwise elapsed, asyncio.timeout_at() does not prevent an already-ready Queue.get() from succeeding because timeout cancellation is only scheduled for the next loop iteration. If jobs are queued while an earlier fetch is blocked, the worker therefore batches those queued jobs together despite a zero batch window—for example, three jobs can produce fetches (1,) and (2, 3) instead of three separate fetches. Restore an explicit expired-deadline check before dequeuing.

Useful? React with 👍 / 👎.

@swkeever swkeever closed this Sep 24, 2026
@swkeever
swkeever deleted the skeever/python-full-mutation-gate branch September 24, 2026 14:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant