feat(swe-bench): classify evaluation infrastructure failures - #483
Open
leopck wants to merge 5 commits into
Open
feat(swe-bench): classify evaluation infrastructure failures#483leopck wants to merge 5 commits into
leopck wants to merge 5 commits into
Conversation
|
MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅ |
github-actions
Bot
added
the
size/normal
leopck
changed the title
leopck
force-pushed
the
swe-dist-4-eval-classifier
branch
from
August 27, 2026 15:25
1f42e6f to
97f19cb
Compare
leopck
force-pushed
the
swe-dist-4-eval-classifier
branch
from
August 27, 2026 15:31
97f19cb to
ddc8278
Compare
…t behind One worker's infrastructure failure discarded the entire eval phase. The agent phase runs `--workers` trajectories concurrently. `_run_agent` re-raises whatever the phase raised, so a single worker that could not start its container -- or whose `srun` step never launched -- took the exception all the way out of `_run()`. That happens *before* `preds.json` is ever looked at, so the predictions every other worker had already written were never scored. Observed: a 200-instance run with 137 predictions on disk reported as a total loss, exit non-zero, no accuracy number, and had to be re-scored by hand from the retained artifacts. The GPU allocation that produced those 137 predictions was gone by then. Eval is now robust to individual worker failure: whatever predictions exist are always scored. The failure is not hidden -- * it is written to the new `agent_phase_error.txt` run artifact, with secrets redacted, and served through the existing artifact route; * it is logged at ERROR; * it is chained as `__cause__` onto the `preds.json` failure when the phase genuinely produced nothing, so an empty run still fails loudly. `RunCancelled` is explicitly not tolerated: a cancelled run is not a degraded run and must not proceed to eval. Tests: `test_run_scores_predictions_left_behind_by_a_failed_agent_phase` (the eval phase runs and the artifact is written), `test_run_still_fails_when_the_agent_phase_produced_nothing` (no false pass, cause chained), `test_run_redacts_secrets_from_the_agent_phase_error`, `test_run_does_not_tolerate_cancellation` and `test_agent_phase_error_is_a_retrievable_artifact`. The existing cleanup-after-failure test asserted that the agent error propagated verbatim and is updated to assert the chained failure instead.
The Pyxis sentinel only covers the agent phase; eval-phase error_ids were counted as real outcomes and never retried, which is what produced 24 of 25 permanently-bad runs on the source cluster. classify.py reads the SWE-bench report's error_ids and each instance's run_instance.log and classifies them through an ORDERED rule list, first match wins. The order is load-bearing: BuildImageError is checked before everything because its message embeds the other rules' needles, CONMON_EAGAIN and TEST_TIMEOUT precede WEDGE_EVAL, and PATCH_APPLY_FAILED is last. Anything unclassifiable is UNKNOWN and UNKNOWN is GENUINE, asserted by a membership test: a false bad-run costs one redo, a false retry biases the measurement toward optimism. Memory-kill markers are consumed by phase - an eval-phase kill is a genuine failure (an unbounded allocation is a failing patch), an agent-phase kill is recorded for audit only, since the agent merely gets an error observation and the instance still reaches a real outcome.
…hole report The same defect as the agent phase, one phase later. `pyxis_worker` grades each prediction in its own container concurrently, collected the per-instance failures, and then raised `RunnerError` *before* `make_run_report()`. So one wedged evaluation container threw away every other instance's grade -- the work was done, the reports were on disk, and nothing was ever written. The eval phase is now robust to individual instance failure. Whatever was graded is reported: an instance with no `report.json` is counted as an error by `make_run_report`, which is the correct and visible outcome, and is exactly what an operator needs to see. A run in which *no* instance could be evaluated still fails -- but only after the report has been written, so the run can be diagnosed from its own artifacts rather than from nothing. The losses are not hidden and, more importantly, not left as prose in a log: `eval_infra_failures.txt` lists `instance_id<TAB>error` per lost instance, is copied beside `swe_bench_results.json` and is served through the existing artifact route. That distinction is load bearing. An instance the harness dropped is not an instance the model failed, and a consumer that cannot separate them reads attrition as an accuracy regression -- which is precisely the misreport the completeness gate exists to prevent. Leaving the evidence only in a human-readable log would leave the gate unable to see it. Found while auditing which cluster-side runtime patches the package had made redundant: this one had not been, and it would have compromised the very run intended to validate the package. Tests: `test_pyxis_worker_reports_the_instances_one_bad_container_did_not_kill` (the report is produced, only the failed instance is listed), `test_pyxis_worker_still_fails_when_no_instance_could_be_evaluated` (no false pass, and the report is written first), `test_pyxis_worker_records_no_failure_file_for_a_clean_eval`, `test_eval_infra_failures_are_published_beside_the_results` and `test_eval_infra_failures_is_a_retrievable_artifact`. Four fail against the parent commit. The existing propagation test still holds: its single instance is also every instance.
leopck
force-pushed
the
swe-dist-4-eval-classifier
branch
from
August 27, 2026 15:52
ddc8278 to
4cec80e
Compare
leopck
marked this pull request as ready for review
August 27, 2026 19:50
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Distinguishes evaluation infrastructure failures from genuine SWE-bench outcomes so recoverable units can be retried safely.
Predictions and reports are retained when agent or evaluation containers fail, preventing useful artifacts from being discarded.