fix(agent): save the kernel registry from a snapshot, not the live map - #14746
Open
daemyung-lablup wants to merge 6 commits into
Open
daemyung-lablup wants to merge 6 commits into
daemyung-lablup wants to merge 6 commits into
Conversation
The save awaits once per kernel while iterating the registry itself, and a create on the same node registers its kernel in between -- `dictionary changed size during iteration`, raised inside the creating call, which then fails the kernel it was creating. Measured with 27 two-node sessions started at once on three nodes: one of 27 failed on exactly this. Iterate over a copy; a kernel that arrives mid-save is written by the save its own create runs. (cherry picked from commit 99b04f557afa987db00a66d9c3a66652fd4f4110)
A kernel is registered a step before its REPL ports exist, so a save that runs in between cannot record it yet -- ordinary under concurrent creates, and its own start writes it a moment later. It was logged as an exception with a full traceback, which reads as a fault and buries the real ones: one line per kernel per save, at warning, says what happened.
… a save The other side of the same race: the save's snapshot still names a kernel whose scratch directory a destroy has just removed, and writing its recovery record re-created the directory -- an empty scratch left behind for a kernel that is gone. The config directory is the kernel's own, made when it was created and removed with it, so the save writes into it and treats its absence as the kernel having left.
…ecord The parse error covers every missing REPL key, and treating all of them as the start-up race hid a broken invariant behind a warning: a kernel that is PREPARING has no ports yet by design, a RUNNING one has them by contract. Only the first is skipped quietly; the second keeps the exception log.
…that kernel A restart destroys and re-creates a kernel under the same id and keeps its scratch directory, so a save that snapshotted the old object could land the old REPL ports in the new incarnation's directory, and the next agent restart would recover it unreachable. The record is now staged beside `recovery.json` and renamed into place only after the registry still maps the id to the snapshot's object -- one atomic rename, no await between the check and it. The rename also means a crash mid-write cannot leave a torn record. Staging never creates the config directory, which replaces the `create_config_dir` flag: the directory is the kernel's own, and its absence means the kernel is gone.
Staging opened its temporary file before serializing, so a serialization or write error -- or a cancelled save -- left `recovery.json.<uuid>.tmp` in the kernel's directory with no handle to discard it. Serialize first; if opening or writing fails for any reason, remove the file and re-raise. The existing `ScratchConfig` tests still called the removed `save_json_recovery_data`; they now cover the stage/commit contract, which also takes in the scratch tests added under a separate directory.
HyeockJinKim
approved these changes
Sep 18, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
save_kernel_registryiterates the livekernel_registrymap and awaits once per kernel. A kernel created on the same node in between registers itself into that map, and the save raisesRuntimeError: dictionary changed size during iterationinside the creating call -- which then fails the kernel it was creating. Measured with 27 two-node sessions started at once on three nodes: 1 of 27 failed on exactly this.recovery.jsonand renamed into place in one step, with no await between the check and the rename -- which also means a crash mid-write cannot leave a torn record.Independent of #14551, which fixes the periodic reconciler destroying a just-created kernel; no file overlap.
Tests:
tests/unit/agent/kernel_registry/test_writer_container.py(mid-save registration, same-id replacement, destroyed-while-saving, PREPARING vs RUNNING),tests/unit/agent/test_scratch_config.py(stage/commit/discard, no directory creation, cleanup on failed serialization and on cancellation)Checklist: (if applicable)
fix:PR goes to every version in.github/maintained-versions.ymlautomatically.A
Backport:line here names the targets instead — some of them, other ones, ornone.ai.backend.testdocsdirectory