feat(swe-bench): enforce run-scoped all-or-nothing merging - #481
Open
leopck wants to merge 4 commits into
Open
Conversation
|
MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅ |
github-actions
Bot
added
the
size/normal
leopck
changed the title
leopck
force-pushed
the
swe-dist-2-merge-gate
branch
2 times, most recently
from
August 27, 2026 15:31
57ef345 to
ed80581
Compare
merge_run(wq, run_id) refuses to emit an accuracy number unless every planned unit has a terminal result, none is abandoned, every unit accounts for exactly its planned instance IDS (a set comparison, never a count), the union equals the plan with no cross-shard duplicates, every plan_digest matches, and no unit carries an infra error. Refusal is a structured MergeRefusal naming the offending units and ids; there is no force flag and no partial-credit path. There is deliberately no --all: merge_run takes a required run id and treats a foreign run id or digest as a hard error, not a skip. verify_inventory() cross-checks claims, results and the id-union as independent producers, so a blind spot shared by one instrument cannot certify itself.
The merge gate refuses a bad run, and that refusal is the most important property here. But a refusal that carries no numbers is not the end of the story: somebody still has to report *something*, and with the gate silent they compute it by hand from the artifacts -- which is exactly how a run that lost 106 of 200 instances to infrastructure came to be reported as 47.0% and compared against a complete-run reference of 70.67%. It was read as a model regression. It was attrition. `assess_run()` performs the whole of the gate's arithmetic without deciding anything, and returns a `CompletenessReport`. `merge_run()` becomes the strict all-or-nothing wrapper over it and attaches the report to both `MergeResult` and `MergeRefusal`, so a caller never has to choose between "a number" and "no information". `resolved_rate` is published only when the run is *structurally complete* -- every planned instance id accounted for exactly once -- **and** zero instances were lost to infrastructure. These are two different questions and conflating them gets both wrong. An instance the model attempted and failed is a legitimate score; one our own harness dropped never had the chance. A run short of instances has the wrong denominator; a complete run that leaned on the infrastructure has the wrong provenance. Two numbers are published either way: * `conditional_resolved_rate` -- resolved over the instances that actually completed. Honest about what it measures and not comparable to a complete-run reference. * `resolved_rate_lower_bound` -- resolved over everything planned. Infrastructure losses can only ever *add* resolutions, so this bounds the truth from below even on a badly degraded run. alongside `incomplete_instance_ids`, `infra_lost_instances`, `infra_lost_unit_ids` and a `resolved_rate_withheld_reason` that says which of the two conditions failed and by how much. Ported from the banked campaign's `wq_merge.sh:7-9`: "shard_merge.py refuses to print an accuracy unless all 20 shards account for exactly their own 10 ids, and that refusal is the single most important property in this campaign." What is added here is that the refusal now shows its working. Tests: `TestCompletenessGate` covers both decision boundaries -- complete vs incomplete, and infra-lost vs genuinely-empty (a model that resolved nothing is a score, not a casualty) -- plus an abandoned unit counting as infrastructure loss, the numbers surviving a refusal, and `assess_run` not raising on the run it is describing. Against the parent commit a refusal carries no `report` and no conditional or lower-bound figure at all.
leopck
force-pushed
the
swe-dist-2-merge-gate
branch
from
August 27, 2026 15:52
ed80581 to
1c98068
Compare
leopck
marked this pull request as ready for review
August 27, 2026 19:49
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a merge gate scoped to one run ID that requires exact instance-ID coverage before publishing the final score.
Incomplete or non-numeric accuracy results now fail explicitly, while partial results retain honest completeness metrics for diagnosis.