Skip to content

Commit fcc92dd

Browse files
authored
Add a per-disk direct I/O option for the cloud-hypervisor backend (#954)
The cloud-hypervisor backend never set `DiskConfig.direct`, so every virtio-blk disk image went through the host page cache, with no way to change that. A virtio-blk mount can now carry the flag `directio` in its runtime options. `Mount.chDiskConfig(id:)` then sets `DiskConfig.direct` to `true`, and cloud-hypervisor opens the image with `O_DIRECT`, so guest I/O to that disk bypasses the host page cache. A mount without the flag sends no `direct` field, so the VM config does not change. The option is per disk. For example, many VMs can boot from the same read-only rootfs and share one cached copy of it, while the read-write volume of each VM uses `O_DIRECT`: ```swift let rootfs = Mount.block(format: "ext4", source: "/images/shared.ext4", destination: "/", options: ["ro"]) let volume = Mount.block( format: "ext4", source: "/volumes/vm1.ext4", destination: "/data", runtimeOptions: ["directio"] ) ``` - The flag takes no value. Only the exact option `directio` turns it on. Any other runtime option, including `directio=true` or `directio=false`, is ignored, in the same way as the `vz` keys. - `chDiskConfig(id:)` has nothing to parse, so it does not throw. - The flag applies to boot disks and to hotplugged disks. Hotplug now builds its `DiskConfig` with `chDiskConfig(id:)`, so a hotplugged disk gets the same options as a boot disk. - The Virtualization backend skips runtime options without `=`, so a mount with `directio` can also be given to that backend. ### How it was tested - Unit tests in `MountCHTests` check `chDiskConfig(id:)`: `directio` gives `direct: true`; `directio=true`, `directio=false`, `directio=`, `DirectIO` and the `vz` keys give no `direct`. - Unit tests in `CHDirectDiskIOTests` check the VM config at boot and the `vm.add-disk` request on hotplug: - a disk without the flag has no `direct` field; - in one VM, a read-only rootfs without the flag stays cached and a volume with `directio` gets `direct: true`; - `directio=true` and `directio=false` are not the flag, at boot and on hotplug; - `vz` keys are ignored. - Linux (arm64): `swift format lint --strict`, `swift build --build-tests -Xswiftc -warnings-as-errors`, and `swift test`. All 798 tests pass. - macOS: the same build, and `swift test` for `MountCHTests`. All tests pass.
1 parent d12d6f6 commit fcc92dd

5 files changed

Lines changed: 334 additions & 8 deletions

File tree

‎Sources/Containerization/CHHotplugProvider.swift‎

Lines changed: 5 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -101,15 +101,14 @@ final class CHHotplugProvider: HotplugProvider {
101101
case .virtioblk:
102102
let letter = try allocator.allocate()
103103
let chId = "blk-\(id)-\(letter)"
104-
let disk = CloudHypervisor.DiskConfig(
105-
path: rootfs.source,
106-
readonly: rootfs.options.contains("ro"),
107-
id: chId,
108-
imageType: .raw
109-
)
110104

105+
// chDiskConfig applies the per-device runtime options of the mount, such as
106+
// `directio`, the same as for boot disks.
111107
let pci: CloudHypervisor.PciDeviceInfo
112108
do {
109+
guard let disk = rootfs.chDiskConfig(id: chId) else {
110+
throw ContainerizationError(.invalidArgument, message: "mount \(rootfs.source) is not a virtio-blk device")
111+
}
113112
pci = try await chCall { try await self.client.vmAddDisk(disk) }
114113
} catch {
115114
try? allocator.release(letter)

‎Sources/Containerization/CHVirtualMachineInstance.swift‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -514,6 +514,8 @@ extension CHVirtualMachineInstance {
514514

515515
// Disks: rootfs forced read-only at the device level; container disks
516516
// honor their `ro` option through chDiskConfig.
517+
// Every disk also gets the per-device runtime options of its mount, such
518+
// as `directio`, through chDiskConfig.
517519
var disks: [CloudHypervisor.DiskConfig] = []
518520
for bd in bootDisks {
519521
let chId = bd.containerId.map { "blk-\($0)-\(bd.letter)" } ?? "rootfs"

‎Sources/Containerization/Mount+CH.swift‎

Lines changed: 19 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -31,14 +31,31 @@ extension Mount {
3131
/// back to `Unknown` and silently rejects all writes — see CH's
3232
/// `virtio-devices/src/block.rs` "Attempting to write to sector 0 on a
3333
/// disk without specifying image_type" warning.
34+
///
35+
/// If the runtime options of the mount (`Mount.block(runtimeOptions:)`) contain
36+
/// the flag `directio`, this disk gets `direct = true`.
37+
/// cloud-hypervisor then opens the image with `O_DIRECT`, so guest I/O to the
38+
/// disk bypasses the host page cache.
39+
/// Without the flag, `direct` is not set and cloud-hypervisor keeps its default.
40+
/// The flag takes no value: only the exact option `directio` turns it on.
41+
/// The option is per device because the right choice depends on the disk.
42+
/// For example, many VMs can boot from the same read-only rootfs image without
43+
/// the flag, so the host page cache holds one copy of it for all of them.
44+
/// Each VM can then attach its own read-write volume with `directio`, so the
45+
/// writes of the guest do not also fill the host page cache.
46+
///
47+
/// Other runtime options, such as the `vz` keys of the Virtualization backend,
48+
/// are ignored.
49+
/// The Virtualization backend skips runtime options that have no `=`, so a
50+
/// mount with `directio` can also be given to that backend.
3451
public func chDiskConfig(id: String) -> CloudHypervisor.DiskConfig? {
35-
guard case .virtioblk = self.runtimeOptions else {
52+
guard case .virtioblk(let runtimeOptions) = self.runtimeOptions else {
3653
return nil
3754
}
3855
return CloudHypervisor.DiskConfig(
3956
path: self.source,
4057
readonly: self.options.contains("ro"),
41-
direct: nil,
58+
direct: runtimeOptions.contains("directio") ? true : nil,
4259
iommu: nil,
4360
id: id,
4461
pciSegment: nil,
Lines changed: 283 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,283 @@
1+
//===----------------------------------------------------------------------===//
2+
// Copyright © 2026 Apple Inc. and the Containerization project authors.
3+
//
4+
// Licensed under the Apache License, Version 2.0 (the "License");
5+
// you may not use this file except in compliance with the License.
6+
// You may obtain a copy of the License at
7+
//
8+
// https://www.apache.org/licenses/LICENSE-2.0
9+
//
10+
// Unless required by applicable law or agreed to in writing, software
11+
// distributed under the License is distributed on an "AS IS" BASIS,
12+
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
// See the License for the specific language governing permissions and
14+
// limitations under the License.
15+
//===----------------------------------------------------------------------===//
16+
17+
#if os(Linux)
18+
import CloudHypervisor
19+
import ContainerizationExtras
20+
import Foundation
21+
import NIO
22+
import NIOHTTP1
23+
import Synchronization
24+
import Testing
25+
26+
@testable import Containerization
27+
28+
/// A virtio-blk disk gets `direct` in the cloud-hypervisor config only when its own
29+
/// mount carries the `directio` runtime option, at boot and on hotplug.
30+
/// The boot tests record the VM config in the `configureCH` of an extension and
31+
/// then fail the start, so cloud-hypervisor never runs.
32+
/// The hotplug tests send `vm.add-disk` to a stub API server on a Unix socket.
33+
struct CHDirectDiskIOTests {
34+
private struct Captured: Error {}
35+
36+
private final class CaptureConfig: CHInstanceExtension {
37+
let config = Mutex<CloudHypervisor.VmConfig?>(nil)
38+
39+
func configureCH(_ config: inout CloudHypervisor.VmConfig) throws {
40+
let value = config
41+
self.config.withLock { $0 = value }
42+
throw Captured()
43+
}
44+
}
45+
46+
/// Starts a VM with `mounts` as the mounts of container `c1` and returns the
47+
/// disks of its VM config, the VM rootfs first.
48+
private func bootDisks(_ mounts: [Mount]) async throws -> [CloudHypervisor.DiskConfig] {
49+
let runtimeRoot = FileManager.default.temporaryDirectory.appendingPathComponent("ch-direct-\(UUID().uuidString)")
50+
try FileManager.default.createDirectory(at: runtimeRoot, withIntermediateDirectories: true)
51+
defer { try? FileManager.default.removeItem(at: runtimeRoot) }
52+
let capture = CaptureConfig()
53+
let instance = try CHVirtualMachineInstance(
54+
runtimeRoot: runtimeRoot,
55+
chBinary: URL(fileURLWithPath: "/bin/false"),
56+
virtiofsdBinary: nil
57+
) { config in
58+
config.kernel = Kernel(path: URL(fileURLWithPath: "/dev/null"), platform: .linuxArm)
59+
config.initialFilesystem = .block(format: "ext4", source: "/tmp/rootfs.ext4", destination: "/")
60+
config.mountsByID = ["c1": mounts]
61+
config.extensions = [capture]
62+
}
63+
do {
64+
try await instance.start()
65+
} catch is Captured {
66+
return try #require(capture.config.withLock { $0 }?.disks)
67+
}
68+
Issue.record("start did not reach configureCH")
69+
return []
70+
}
71+
72+
/// Returns the JSON object that is sent to cloud-hypervisor for `disk`.
73+
private func encoded(_ disk: CloudHypervisor.DiskConfig) throws -> [String: Any] {
74+
try jsonObject(JSONEncoder().encode(disk))
75+
}
76+
77+
private func jsonObject(_ data: Data) throws -> [String: Any] {
78+
try #require(try JSONSerialization.jsonObject(with: data) as? [String: Any])
79+
}
80+
81+
@Test func rootfsWithoutFlagHasNoDirect() async throws {
82+
let containerRootfs = Mount.block(format: "ext4", source: "/tmp/c1.ext4", destination: "/", options: ["ro"])
83+
let disks = try await bootDisks([containerRootfs])
84+
#expect(disks.count == 2)
85+
for disk in disks {
86+
#expect(disk.direct == nil)
87+
#expect(try encoded(disk)["direct"] == nil)
88+
}
89+
}
90+
91+
@Test func volumeWithFlagGetsDirect() async throws {
92+
let volume = Mount.block(
93+
format: "ext4",
94+
source: "/tmp/vol.ext4",
95+
destination: "/data",
96+
runtimeOptions: ["directio"]
97+
)
98+
let disks = try await bootDisks([volume])
99+
#expect(disks.count == 2)
100+
#expect(disks[1].path == "/tmp/vol.ext4")
101+
#expect(disks[1].readonly == false)
102+
#expect(disks[1].direct == true)
103+
#expect(try encoded(disks[1])["direct"] as? Bool == true)
104+
}
105+
106+
/// The container rootfs is shared by many VMs and stays cached, and the
107+
/// read-write volume of this VM uses O_DIRECT.
108+
@Test func sharedRootfsAndDirectVolumeInOneVM() async throws {
109+
let containerRootfs = Mount.block(format: "ext4", source: "/tmp/shared.ext4", destination: "/", options: ["ro"])
110+
let volume = Mount.block(
111+
format: "ext4",
112+
source: "/tmp/vol.ext4",
113+
destination: "/data",
114+
runtimeOptions: ["directio"]
115+
)
116+
let disks = try await bootDisks([containerRootfs, volume])
117+
#expect(disks.map(\.path) == ["/tmp/rootfs.ext4", "/tmp/shared.ext4", "/tmp/vol.ext4"])
118+
#expect(disks.map(\.direct) == [nil, nil, true])
119+
}
120+
121+
/// The flag takes no value, so a `directio=<value>` option is not the flag and
122+
/// does not start the VM with `direct` set.
123+
@Test func valueFormsAreNotTheFlag() async throws {
124+
let withTrue = Mount.block(
125+
format: "ext4",
126+
source: "/tmp/true.ext4",
127+
destination: "/true",
128+
runtimeOptions: ["directio=true"]
129+
)
130+
let withFalse = Mount.block(
131+
format: "ext4",
132+
source: "/tmp/false.ext4",
133+
destination: "/false",
134+
runtimeOptions: ["directio=false"]
135+
)
136+
let disks = try await bootDisks([withTrue, withFalse])
137+
#expect(disks.map(\.direct) == [nil, nil, nil])
138+
for disk in disks {
139+
#expect(try encoded(disk)["direct"] == nil)
140+
}
141+
}
142+
143+
@Test func virtualizationKeysAreIgnored() async throws {
144+
let vzOnly = Mount.block(
145+
format: "ext4",
146+
source: "/tmp/vz.ext4",
147+
destination: "/vz",
148+
runtimeOptions: ["vzDiskImageCachingMode=uncached", "vzDiskImageSynchronizationMode=none"]
149+
)
150+
let both = Mount.block(
151+
format: "ext4",
152+
source: "/tmp/both.ext4",
153+
destination: "/both",
154+
runtimeOptions: ["vzDiskImageCachingMode=cached", "directio", "vzTimeout=5"]
155+
)
156+
let disks = try await bootDisks([vzOnly, both])
157+
#expect(disks.map(\.direct) == [nil, nil, true])
158+
}
159+
160+
// MARK: - Hotplug
161+
162+
fileprivate final class Bodies: Sendable {
163+
let value = Mutex<[Data]>([])
164+
}
165+
166+
/// Hotplugs `mount` for container `c1` through a stub cloud-hypervisor API
167+
/// server, and returns the result and the request bodies that the server got.
168+
private func hotplug(_ mount: Mount) async throws -> (result: Result<AttachedFilesystem, any Error>, bodies: [Data]) {
169+
let group = MultiThreadedEventLoopGroup.singleton
170+
let socketPath = FileManager.default.temporaryDirectory
171+
.appendingPathComponent("ch-direct-\(UUID().uuidString.prefix(8)).sock").path
172+
let bodies = Bodies()
173+
let server = try await ServerBootstrap(group: group)
174+
.childChannelInitializer { channel in
175+
channel.eventLoop.makeCompletedFuture {
176+
try channel.pipeline.syncOperations.configureHTTPServerPipeline(withPipeliningAssistance: false)
177+
try channel.pipeline.syncOperations.addHandler(AddDiskHandler(bodies: bodies))
178+
}
179+
}
180+
.bind(unixDomainSocketPath: socketPath, cleanupExistingSocketFile: true)
181+
.get()
182+
183+
let client = try CloudHypervisor.Client(socketPath: URL(fileURLWithPath: socketPath), eventLoopGroup: group)
184+
let provider = CHHotplugProvider(
185+
client: client,
186+
workDir: FileManager.default.temporaryDirectory,
187+
virtiofsdBinary: nil,
188+
allocator: Character.blockDeviceTagAllocator(),
189+
initialMounts: [:],
190+
logger: nil
191+
)
192+
let result: Result<AttachedFilesystem, any Error>
193+
do {
194+
result = .success(try await provider.hotplug(mount, id: "c1"))
195+
} catch {
196+
result = .failure(error)
197+
}
198+
199+
try await client.shutdown()
200+
try await server.close()
201+
try? FileManager.default.removeItem(atPath: socketPath)
202+
return (result, bodies.value.withLock { $0 })
203+
}
204+
205+
@Test func hotplugHonorsTheFlag() async throws {
206+
let volume = Mount.block(
207+
format: "ext4",
208+
source: "/tmp/vol.ext4",
209+
destination: "/data",
210+
runtimeOptions: ["directio"]
211+
)
212+
let (result, bodies) = try await hotplug(volume)
213+
#expect(try result.get().source == "/dev/vda")
214+
#expect(bodies.count == 1)
215+
let disk = try JSONDecoder().decode(CloudHypervisor.DiskConfig.self, from: try #require(bodies.first))
216+
#expect(disk.path == "/tmp/vol.ext4")
217+
#expect(disk.id == "blk-c1-a")
218+
#expect(disk.imageType == .raw)
219+
#expect(disk.direct == true)
220+
}
221+
222+
@Test func hotplugWithoutFlagSendsNoDirect() async throws {
223+
let rootfs = Mount.block(format: "ext4", source: "/tmp/c1.ext4", destination: "/", options: ["ro"])
224+
let (result, bodies) = try await hotplug(rootfs)
225+
_ = try result.get()
226+
#expect(bodies.count == 1)
227+
let object = try jsonObject(try #require(bodies.first))
228+
#expect(object["direct"] == nil)
229+
#expect(object["readonly"] as? Bool == true)
230+
}
231+
232+
@Test func hotplugIgnoresValueForms() async throws {
233+
let volume = Mount.block(
234+
format: "ext4",
235+
source: "/tmp/vol.ext4",
236+
destination: "/data",
237+
runtimeOptions: ["directio=true"]
238+
)
239+
let (result, bodies) = try await hotplug(volume)
240+
_ = try result.get()
241+
#expect(bodies.count == 1)
242+
let object = try jsonObject(try #require(bodies.first))
243+
#expect(object["direct"] == nil)
244+
}
245+
}
246+
247+
/// Records each request body and answers with the `PciDeviceInfo` of an added disk.
248+
private final class AddDiskHandler: ChannelInboundHandler, @unchecked Sendable {
249+
typealias InboundIn = HTTPServerRequestPart
250+
typealias OutboundOut = HTTPServerResponsePart
251+
252+
private let bodies: CHDirectDiskIOTests.Bodies
253+
private var body: [UInt8] = []
254+
255+
init(bodies: CHDirectDiskIOTests.Bodies) {
256+
self.bodies = bodies
257+
}
258+
259+
func channelRead(context: ChannelHandlerContext, data: NIOAny) {
260+
switch unwrapInboundIn(data) {
261+
case .head:
262+
body = []
263+
case .body(var buffer):
264+
body.append(contentsOf: buffer.readBytes(length: buffer.readableBytes) ?? [])
265+
case .end:
266+
let request = Data(body)
267+
bodies.value.withLock { $0.append(request) }
268+
let response = Array(#"{"id":"_disk0","bdf":"0000:00:05.0"}"#.utf8)
269+
var headers = HTTPHeaders()
270+
headers.add(name: "Content-Type", value: "application/json")
271+
headers.add(name: "Content-Length", value: "\(response.count)")
272+
headers.add(name: "Connection", value: "close")
273+
let head = HTTPResponseHead(version: .http1_1, status: .ok, headers: headers)
274+
context.write(wrapOutboundOut(.head(head)), promise: nil)
275+
context.write(wrapOutboundOut(.body(.byteBuffer(context.channel.allocator.buffer(bytes: response)))), promise: nil)
276+
let bound = NIOLoopBound(context, eventLoop: context.eventLoop)
277+
context.writeAndFlush(wrapOutboundOut(.end(nil))).whenComplete { _ in
278+
bound.value.close(promise: nil)
279+
}
280+
}
281+
}
282+
}
283+
#endif

‎Tests/ContainerizationTests/MountCHTests.swift‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -49,6 +49,31 @@ struct MountCHTests {
4949
#expect(any.chDiskConfig(id: "x") == nil)
5050
}
5151

52+
@Test("block mount with the 'directio' runtime option produces DiskConfig with direct=true")
53+
func blockDirectIO() {
54+
let mount = Mount.block(format: "ext4", source: "/foo.img", destination: "/data", runtimeOptions: ["directio"])
55+
let cfg = mount.chDiskConfig(id: "blk-0")
56+
#expect(cfg?.direct == true)
57+
#expect(cfg?.readonly == false)
58+
}
59+
60+
@Test(
61+
"only the exact runtime option 'directio' sets direct",
62+
arguments: [
63+
["directio=true"],
64+
["directio=false"],
65+
["directio="],
66+
["DirectIO"],
67+
["vzDiskImageCachingMode=uncached", "vzDiskImageSynchronizationMode=none"],
68+
]
69+
)
70+
func otherRuntimeOptionsAreIgnored(runtimeOptions: [String]) {
71+
let mount = Mount.block(format: "ext4", source: "/foo.img", destination: "/data", runtimeOptions: runtimeOptions)
72+
let cfg = mount.chDiskConfig(id: "blk-0")
73+
#expect(cfg != nil)
74+
#expect(cfg?.direct == nil)
75+
}
76+
5277
@Test("share mount produces FsConfig with tag and socket")
5378
func shareMount() {
5479
let mount = Mount.share(source: "/host/dir", destination: "/guest/dir")

0 commit comments

Comments
 (0)