feat(swe-bench): add the distributed fleet scorer - #485
Open
leopck wants to merge 11 commits into
Open
Conversation
|
MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅ |
github-actions
Bot
added
the
size/large
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
leopck
changed the title
leopck
force-pushed
the
swe-dist-6-fleet-scorer
branch
from
August 27, 2026 15:25
8419d67 to
fc38b5a
Compare
leopck
force-pushed
the
swe-dist-6-fleet-scorer
branch
from
August 27, 2026 15:31
fc38b5a to
eebd13e
Compare
leopck
force-pushed
the
swe-dist-6-fleet-scorer
branch
from
August 27, 2026 15:52
eebd13e to
d6bf133
Compare
… fleet
Wires the pieces into a scorer registered as eval_method: swe_bench_fleet.
It is a scheduler in front of the existing SWE-bench service protocol, not a
new runtime: a unit is one RunRequest over a shard, so exact instance
binding, per-instance containers, artifact allow-listing and cancellation
are reused rather than reimplemented.
preflight() runs the gates against the inference endpoints and /health
against every service, raising SetupError before a single instance is
dispatched. score() plans, dispatches with one in-flight run per service,
classifies every unit, requeues any unit with infra_error_count > 0 even
when the service reported succeeded, and takes the accuracy number only
past the merge gate - self.complete comes from the gate, never a count
heuristic.
Stall quarantine verifies effect rather than status: a service that is
/health-OK but has completed no unit within stall_timeout_s is quarantined
and its in-flight unit requeued.
Also adds scripts/swe_bench_wq.py {status,merge,requeue,reap} for
operators. reap is dry-run by default and requeue prints exactly which
result, claim and attempt records it removed, because the failure mode in
the field was an operator believing a delete had requeued something.
score() reads the run's settings back from the report directory's config.yaml,
which yaml.safe_load() returns as plain dictionaries. It then handed that
mapping to SWEBenchScorer._generation_params(), which calls .model_dump() on
it, so the fleet scorer raised
AttributeError: 'dict' object has no attribute 'model_dump'
on every run, after the plan and the work queue had been written but before a
single unit was dispatched.
Re-validate the mapping into ModelParams instead of re-implementing the field
selection here, so the fleet path and the single-service path stay in
agreement about which generation settings are forwarded to the service.
Every unit was submitted with endpoint_urls[:1], so a fleet configured with N engines sent all of its work to the first one and left the other N-1 idle. The comment justified this by noting that the service accepts exactly one endpoint per run and that the fleet's parallelism comes from running many units -- true, but it does not follow that every unit must pick the same one. Two consequences. The obvious one is a throughput ceiling: concurrency is bounded by one engine no matter how much hardware the run was given. The serious one is a measurement hazard -- a single engine's behaviour decides the whole run's accuracy, so one degraded engine is indistinguishable from a degraded model, which is precisely the confusion the endpoint fingerprint exists to prevent. Bind unit -> endpoint by shard index instead. The mapping is deterministic, so a retried unit lands on the endpoint it was originally measured against and stays comparable to its first attempt, and a run with one endpoint behaves exactly as before.
…e retry Also surface the completeness report on the refusal paths that already existed: `SWEBenchFleetScorer` writes it into the run's merge artifacts and `swe_bench_wq.py merge` prints it. A refusal is not an absence of information, and leaving the conditional rate and the lower bound unpublished is what makes somebody recompute a headline by hand from the artifacts -- which is how a run that lost 106 of 200 instances came to be reported as 47.0% accuracy.
leopck
force-pushed
the
swe-dist-6-fleet-scorer
branch
from
August 27, 2026 15:58
d6bf133 to
440c6b5
Compare
leopck
marked this pull request as ready for review
August 27, 2026 19:50
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Wires the distributed SWE-bench fleet scorer, deterministic endpoint-to-shard binding, operator work-queue CLI, configuration validation, and documentation.
The scorer fans units across multiple services, polls the durable queue, and merges only complete results belonging to the active run.