nsenter: fix FUSE deadlock from synchronous child waits#121
Open
benjamincburns wants to merge 2 commits into
Open
nsenter: fix FUSE deadlock from synchronous child waits#121benjamincburns wants to merge 2 commits into
benjamincburns wants to merge 2 commits into
Conversation
Replace mu.Lock() with mu.TryLock() in the zombie reaper goroutine. Go's sync.RWMutex implements writer-preference: a pending Lock() blocks all subsequent RLock() callers. When the reaper's Lock() is pending while an nsenter child holds RLock, all new FUSE Lookup handlers are blocked from acquiring RLock to start their own nsenter processes. If the nsenter child's operation triggers a FUSE request back to sysbox-fs, the FUSE handler cannot proceed, creating a deadlock. TryLock() does not register a pending writer, so FUSE handlers can still acquire RLock. If TryLock fails, the reaper sleep a second to retry. Fixes: nestybox/sysbox#1002 Signed-off-by: okhowang(王沛文) <okhowang@tencent.com>
Author
|
cc @okhowang for visibility |
SendRequest() held the reaper RLock while synchronously waiting for nsenter child processes via Process.Wait(). If a child's exit was delayed by the kernel's fuse_flush on inherited FUSE file descriptors, the RLock was held indefinitely, preventing both the reaper and new FUSE handlers from making progress. Move the pipe read before the PARENT wait (the PARENT writes the pid before exiting) and replace synchronous Process.Wait() calls with background goroutines so the caller releases the RLock promptly. Fixes: nestybox/sysbox#998 Signed-off-by: Ben Burns <803016+benjamincburns@users.noreply.github.com>
benjamincburns
force-pushed
the
fix/fuse-deadlock
branch
from
09c6630 to
8ddc7c6
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes nestybox/sysbox#998. Builds on #119.
SendRequest()held the reaperRLockwhile synchronously waiting fornsenterchild processes. If a child's exit was delayed byfuse_flushon inherited FUSE file descriptors, theRLockwas held indefinitely, blocking both the reaper and new FUSE handlers.This PR includes the
TryLockfix from #119 and adds:PARENTwait (PARENTwrites pid before exiting)Process.Wait()calls with background goroutinesreapProcessAsyncfor the INIT process to trigger a reaper sweep on exitJust for transparency, I am an experienced software engineer with some systems experience, but I'm not a go dev. I used AI tooling (Claude Code and Claude Opus 4.6) to find the underlying deadlock and author this fix. I'm mentioning that here in case this project has any policies against AI contributions. I don't want to subvert any such policy.
I have tested this fix on my own system. See notes in nestybox/sysbox#998 (comment).