[Bug]: SIGTRAP "Deinited NIOAsyncWriter without calling finish()" when Vminitd.close() runs before runConnections() starts (e.g. a cancelled LinuxContainer.statistics()) #970

I have done the following

• I have searched the existing issues
• If possible, I've reproduced the issue using the 'main' branch of this project
  (Reproduced on the 0.51.0 tag, c411138. On main (cc8d321), the only later change to Vminitd.swift is the statistics field mapping, and init/close() are unchanged.)

Steps to reproduce

Minimal Swift package (vminitd-close-trap): https://gist.github.com/dandmetcalf-stack/704e77f8a5b3a368a3b8d14572dece05. It depends only on Containerization 0.51.0, swift-nio and grpc-swift; the gist README shows how to restore the package layout.

A. No VM (deterministic, uses only the public Vminitd API):

import Containerization
import Foundation
import NIOPosix

let group = MultiThreadedEventLoopGroup(numberOfThreads: 1)
var fds: [Int32] = [0, 0]
precondition(socketpair(AF_UNIX, SOCK_STREAM, 0, &fds) == 0)
// The peer never answers, so nothing is in flight.
let agent = try await Vminitd(connection: FileHandle(fileDescriptor: fds[0], closeOnDealloc: false), group: group)
try? await agent.close()   // close() runs before the Task created in init has called runConnections()
// When `agent` is released, the process traps (SIGTRAP).

1. swift build, then run vminitd-close-trap no-vm 20.
2. The process exits with SIGTRAP (exit status 133) in the first round.
3. Control: vminitd-close-trap no-vm 20 10 sleeps 10 ms between init and close(), and it does not trap.

B. Real VM (the way we hit it in production):

1. Boot a LinuxContainer with VZVirtualMachineManager. We used the Kata 3.17.0 arm64 kernel, the vminit 0.51.0 initfs and a stock debian:bookworm-slim rootfs, running /bin/sleep 600.
2. Repeatedly call container.statistics(categories: [.cpu]) from a Task, and cancel that task after a random 0–3 ms delay:

for _ in 0..<1000 {
    let call = Task { _ = try? await container.statistics(categories: [.cpu]) }
    usleep(UInt32.random(in: 0...3000))
    call.cancel()
    await call.value
}

3. The process traps with SIGTRAP, usually within the first few hundred iterations (0.07–0.52 s after the container starts).

Current behavior

The process terminates with EXC_BREAKPOINT / SIGTRAP:

NIOCore/NIOAsyncWriter.swift:177: Fatal error: Deinited NIOAsyncWriter without calling finish()

Measured on the machine below. Each "process" is one run of the reproducer.

The trap shows up at two sites, both releasing the same unconsumed NIOAsyncChannel.

1. When the Vminitd value is destroyed in withAgent (3 of 5 VM crash reports, and every no-VM report):

0  libswiftCore.dylib  _assertionFailure(_:_:file:line:flags:)
1  NIOAsyncWriter.InternalClass.deinit (NIOAsyncWriter.swift:177)
5  outlined destroy of HTTP2ClientTransport.WrappedChannel.Configured?
6  HTTP2ClientTransport.WrappedChannel.deinit (WrappedChannel.swift)
10 GRPCClient.deinit (GRPCClient.swift)
14 destroy for Vminitd
15 VirtualMachineInstance.withAgent<A>(fn:) (LinuxContainer.swift:1636)
16 closure #1 in LinuxContainer.statistics(categories:) (LinuxContainer.swift:1279)
18 AsyncMutex.withLock<A>(_:) (AsyncMutex.swift:57)
19 LinuxContainer.statistics(categories:) (LinuxContainer.swift:1276)

2. On the NIO event loop, when the channel becomes active after the Vminitd has already been closed and released (2 of 5 VM crash reports):

0  libswiftCore.dylib  _assertionFailure(_:_:file:line:flags:)
1  NIOAsyncWriter.InternalClass.deinit (NIOAsyncWriter.swift:177)
5  NIOAsyncChannelHandler.channelActive(context:) (AsyncChannelHandler.swift:139)
13 NIOHTTP2Handler.channelActive(context:) (HTTP2ChannelHandler.swift:548)
21 ChannelPipeline.SynchronousOperations.fireChannelActive() (ChannelPipeline.swift:1475)
23 BaseSocketChannel.becomeActive0(promise:) (BaseSocketChannel.swift:1420)
24 BaseSocketChannel.registerAlreadyConfigured0(promise:) (BaseSocketChannel.swift:1020)
25 closure #2 in setupChannel #1 in ClientBootstrap.withConnectedSocket(_:) (Bootstrap.swift:1158)

Expected behavior

Calling close() on a Vminitd at any point after init returns should be safe, including before its connection task has started and after a cancelled RPC. Cancelling a task that calls a LinuxContainer API should surface as an error such as CancellationError or RPCError, not crash the host process.

Root cause (as far as we can tell)

The references below are to Containerization 0.51.0 and to the dependency versions that resolved in our build (grpc-swift-2 2.4.4, grpc-swift-nio-transport 2.11.0, swift-nio 2.104.0). The same code is present in grpc-swift-2 2.4.2 and grpc-swift-nio-transport 2.9.0.

1. Vminitd.init (Sources/Containerization/Vminitd.swift:36-62) builds the transport with HTTP2ClientTransport.WrappedChannel.wrapping(config:serviceConfig:makeChannel:). That call creates a pre-configured NIOAsyncChannel whose writer must be finished. init then starts the client from an unstructured task (:59-61, Task { try await grpcClient.runConnections() }). init returns without waiting for that task to start. It also resumes its continuation from inside channelInitializer (:44), before the channel is registered or active.
2. Vminitd.close() (:65-68) calls grpcClient.beginGracefulShutdown() and then awaits the task.
3. If close() gets there first, GRPCClient moves straight from .notStarted to .stopped. It does not notify the transport (grpc-swift-2 Sources/GRPCCore/GRPCClient.swift:124-129 and :261-266). runConnections() then throws clientIsStopped (:237-238, :98-122) and never calls transport.connect().
4. WrappedChannel consumes and finishes its pre-configured NIOAsyncChannel only inside connect(), through connection.executeThenClose (grpc-swift-nio-transport Sources/GRPCNIOTransportCore/Client/WrappedChannel/WrappedChannel.swift:146-216, :172). Without connect(), nothing finishes the outbound writer. When the last reference goes away, NIOAsyncWriter.InternalClass.deinit hits preconditionFailure (swift-nio Sources/NIOCore/AsyncSequences/NIOAsyncWriter.swift:177). That release happens either in Vminitd's destructor or in NIOAsyncChannelHandler.channelActive (Sources/NIOCore/AsyncChannel/AsyncChannelHandler.swift:139), whichever comes last.
5. There is a second, rarer path (1 of 40 no-VM processes). The client is already .running, but the transport is still .idle or .configuring when beginGracefulShutdown() arrives. WrappedChannel+State.swift:188-199 then moves it to .shutDown or .shuttingDown. After that, connect() either returns early (WrappedChannel.swift:150) or closes the channel (:203-204) without finishing the pre-configured NIOAsyncChannel. In this case close() returns normally and the process still traps on release.

A cancelled caller makes this likely because the RPC inside withAgent (LinuxContainer.swift:1628-1638) fails immediately. close() then runs microseconds after init. Any withAgent / dialAgent() caller can hit it: statistics, filesystemOperation, copy, exec and so on.

Callers can't fully avoid this themselves. Running the SDK call in Task.detached hides the caller's cancellation, but cancellation that starts inside the SDK can still reach an RPC. One example (not reproduced here) is the withThrowingTaskGroup in LinuxProcess.start (LinuxProcess.swift:336-361): if setupIO fails first, the group cancels the sibling agent.createProcess task.

Possible fix

Containerization-side options:

• Make close() safe before start. In our tests (patch below, with no other changes), cancelling the connection task instead of calling beginGracefulShutdown() removed the trap: 0/5 VM processes and 0/10 no-VM processes trapped. runConnections() still runs transport.connect(), and executeThenClose finishes the channel. A gRPC-only probe that forces close() to run first trapped 10/10 with the current close() and 0/10 with the cancel version (500 closes). The trade-off is that RPCs still in flight on this client are cancelled instead of drained. That seems acceptable for per-call clients (withAgent), but you would need to judge it for longer-lived agents.

       public func close() async throws {
  -        self.grpcClient.beginGracefulShutdown()
  -        try await self.connectionTask.value
  +        self.connectionTask.cancel()
  +        _ = await self.connectionTask.result
       }

• Alternatively, keep graceful shutdown but only after runConnections() has reached the transport. For example, init could wait until the connection task has started, or close() could fall back to cancellation when the client never started. init currently returns before the channel is even active, so the rarer race in step 5 would still need handling.

Possibly upstream (grpc-swift-nio-transport / grpc-swift-2): WrappedChannel could finish or close its pre-configured NIOAsyncChannel when it is shut down or deinitialized without connect() having consumed it. GRPCClient.beginGracefulShutdown() in .notStarted could also tell the transport. Either change would fix this for every user of wrapping(config:serviceConfig:makeChannel:). We're happy to file it there as well if you think that's the better place.

Environment

- OS: macOS 27.0.1 (26A434), Apple M6 (Mac18,5)
- Xcode: none; Command Line Tools 27.0.0.0.1788430756
- Swift: Apple Swift version 6.4 (swiftlang-6.4.0.34.1 clang-2100.3.34.1)
- Containerization: 0.51.0 (c411138bab442564f65d5e2bebd4a807d5a3f990)
- Resolved: grpc-swift-2 2.4.4, grpc-swift-nio-transport 2.11.0, swift-nio 2.104.0, swift-nio-http2 1.46.0
  (the same trap was first seen in an app embedding Containerization, built with grpc-swift-2 2.4.2,
  grpc-swift-nio-transport 2.9.0, swift-nio 2.101.3)
- Guest: Kata 3.17.0 arm64 kernel, vminit 0.51.0 initfs, debian:bookworm-slim arm64 rootfs
- Binary: debug build, ad-hoc signed with com.apple.security.virtualization only

Relevant log output

$ vminitd-close-trap no-vm 1
[...] round 1: close() threw clientIsStopped: "Can't call 'runConnections()' as the client is stopped (or is stopping). This can happen if the you call 'runConnections()' after shutting the client down or if you used 'withGRPCClient' with an empty body."
NIOCore/NIOAsyncWriter.swift:177: Fatal error: Deinited NIOAsyncWriter without calling finish()
(exit status 133, SIGTRAP)

$ run-vm.sh … 1000 0 unshielded.log
[...] container started in 313 ms; shield=false rounds=1000
[...] uncancelled statistics() ok
[...] round 1 CancellationError
[...] round 2 CancellationError
...
[...] round 31 CancellationError
NIOCore/NIOAsyncWriter.swift:177: Fatal error: Deinited NIOAsyncWriter without calling finish()
exit=133

$ run-vm.sh … 1000 1 shielded.log
[...] SURVIVED rounds=1000 shield=true outcomes=["ok": 1000]

Code of Conduct

• I agree to follow this project's Code of Conduct