feat(swe-bench): add a durable distributed work queue - #480
Open
leopck wants to merge 4 commits into
Open
Conversation
|
MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅ |
github-actions
Bot
added
the
size/large
leopck
changed the title
leopck
force-pushed
the
swe-dist-1-units-queue
branch
from
August 27, 2026 15:25
c3cbdb8 to
0eb2299
Compare
…he step Pyxis creates the container *inside* the `srun` step, so Enroot reads `ENROOT_TEMP_PATH` and `ENROOT_CONFIG_PATH` there and not in the service process. Neither was on the step environment allow-list, so both were silently dropped. The consequence is not a failed step, which is why it took so long to see: an operator points `ENROOT_TEMP_PATH` at a large device precisely so that unpacking a ~2.5 GB rootfs with ~16.8k hardlinks does not compete for space with the unpacked rootfs itself, the override never arrives, and the create-time temp lands back on the very device it was meant to spare. On a 20-node run this is how `/raid` reached 4.1 GB free of 527 GB, after which every subsequent container creation failed for want of space -- reported as an ordinary infrastructure failure with no mention of the setting that was discarded. This is the same class as the proxy variables already on the list and for the same structural reason: work that looks like it happens in the service actually happens in the step, and configuration that does not cross that boundary is configuration that does nothing. No other `SLURM_*` variable is added; inheriting `SLURM_JOB_ID` / `SLURM_STEP_ID` is what breaks a nested `srun` and is why the allow-list exists. Kept deliberately separate from the `SLURM_CONF` + proxy commit rather than folded into it. That commit is also PR #452 upstream; if #452 merges on its own and the stack drops its commit, this fix has to survive that, and it only does if it stands alone. Tests: the existing allow-list parametrisation gains both variables (each fails against the previous list), and `test_pyxis_srun_environment_withholds_inherited_step_identity` continues to pin the other half of the contract. The service README documents the allow-list as a table with the reason each load-bearing entry is on it.
…reated `PyxisEnvironment.cleanup()` never reclaimed anything. Pyxis namespaces named containers by the allocation, so `--container-name=X` inside job `N` is the Enroot container `pyxis_N_X`. cleanup() asked for `pyxis_X`, which does not exist. `enroot remove` exited non-zero, and `check=False` with `capture_output=True` discarded both the status and the message, so the failure was invisible. Symptom: nothing is reclaimed for the life of an allocation. Measured on a 20-node run -- 199 trajectory rootfs coexisting on one node and `/raid` down to 4.1 GB free of 527 GB, after which every subsequent container creation failed for want of space. It is also the origin of the "scancel doesn't reap enroot containers" folklore: `scancel` genuinely does not remove Enroot containers, but the containers here were never asked to go away in the first place, so the blame landed on SLURM. Two changes: * `enroot_container_name(job_id, name)` builds the name Pyxis actually created, and cleanup() uses it. * A non-zero `enroot remove` is logged with its stderr instead of being swallowed, so the next time this path breaks it says so. Tests: `test_pyxis_cleanup_removes_the_container_pyxis_actually_created` models an Enroot container set and asserts the created container is the one removed (it fails against the old name, which removes nothing); `test_pyxis_cleanup_reports_a_removal_that_did_not_happen` asserts the warning; `test_enroot_container_name_is_namespaced_by_job` pins the naming rule. The existing container-reuse test asserted the unnamespaced form and is corrected.
…ork queue Adds the two foundations of the distributed SWE-bench harness: - units.py: shards an instance-id list into immutable, content-addressed units. The sha256 digest covers the ordered id list, so a plan cannot be silently reused across a different run, instance list, or ordering. - queue.py: a filesystem work queue whose claim is a bare os.mkdir (never makedirs(exist_ok=True), which hands a unit to every caller). available() is plan - claims - results, so deleting a result alone does NOT requeue a unit; requeue() is the only supported path and removes the result, the claim and the attempt records together. Env faults are ledgered separately from counted attempts, and abandoning a unit publishes a terminal result AND releases the claim so claims/ and results/ never disagree.
leopck
force-pushed
the
swe-dist-1-units-queue
branch
from
August 27, 2026 15:31
0eb2299 to
33279b8
Compare
leopck
marked this pull request as ready for review
August 27, 2026 19:49
This was referenced Aug 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds content-addressed work units and a mkdir-atomic durable queue so distributed SWE-bench runs can resume safely after client failures.
Also makes Pyxis launch configuration, environment forwarding, and container cleanup deterministic, with coverage for unit planning and queue lifecycle.
Dependency base: swe-dist-dependencies contains the canonical patches from open PRs #453, #454, and #456. This base will collapse back to main after those PRs merge; merged PR #452 is already inherited from main.