feat: add fanotify pre-content on-demand service for native EROFS mounts by Zephyrcf · Pull Request #1982 · dragonflyoss/nydus

Zephyrcf · 2026-07-14T20:20:37Z

Overview

Implement the nydus fanotify daemon behind #[cfg(feature = "fanotify")].
It serves a multi-device EROFS image on demand through the kernel fanotify
pre-content interface (FAN_CLASS_PRE_CONTENT + FAN_PRE_ACCESS, Linux 6.15+).

The bootstrap is mounted directly as a local EROFS file; each blob's sparse
cache file is a separate device= backing file, marked for FAN_PRE_ACCESS.
Cold blob-data fault reads, the daemon fetches and decodes the range into
the cache, and answers FAN_ALLOW — the filled cache file is the same file
the kernel reads, so no separate copy step is needed.

Includes a bounded event coordinator with per-event deadlines, exactly-once
response ownership, a range_ready fast-path that avoids backend I/O when
the authoritative groupmap already covers the faulted range, and a
single-stage graceful shutdown.

Verified end-to-end on kernel 6.15.0-061500-generic with a registry backend.

Also fixes an existing GroupFlight bug where a leader panic left follower
threads permanently blocked (LeaderGuard drop guard).

Related Issues

Please link to the relevant issue. For example: Fix #123 or Related #456.

Change Details

Please describe your changes in detail:

Test Results

test-fanotify.log

Change Type

Please select the type of change your pull request relates to:

Self-Checklist

Before submitting a pull request, please ensure you have completed the following:

I have run a code style check and addressed any warnings/errors.
I have added appropriate comments to my code (if applicable).
I have updated the documentation (if applicable).
I have written appropriate unit tests.

When the leader in ensure_group() panics, flight.complete() is never called and every follower in flight.wait() hangs permanently. Make GroupFlight::complete() idempotent and add a LeaderGuard that calls complete(Err) on Drop, so followers are always unblocked. Signed-off-by: zephyrcf <zinsist777@gmail.com>

joy-allen · 2026-07-15T03:23:46Z

Can we introduce an optimization to skip marking full ready blob files.

Copilot

Pull request overview

This PR introduces an optional fanotify-gated Nydus daemon that serves native multi-device EROFS mounts on-demand using Linux fanotify pre-content permission events (FAN_CLASS_PRE_CONTENT + FAN_PRE_ACCESS), and includes an end-to-end verification harness plus a fix for a GroupFlight deadlock on leader panic.

Changes:

Add nydus fanotify CLI subcommand and the underlying fanotify service modules (event parsing, response ownership, coordinator, mounting).
Extend nydus-accessor with a non-fetching ready_ranges() probe and fix GroupFlight so follower threads don’t block forever if a leader panics.
Add documentation and an E2E test script for validating correctness and event-path behavior against a local registry backend.

Reviewed changes

Copilot reviewed 16 out of 16 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
scripts/fanotify-e2e.sh	New end-to-end harness for fanotify pre-content service validation (registry-backed).
nydus/src/lib.rs	Exposes `fanotify` module behind feature gate.
nydus/src/fanotify/mod.rs	Fanotify module entry + exports.
nydus/src/fanotify/coordinator.rs	Bounded pending-event accounting and late-completion handling.
nydus/src/fanotify/core.rs	Device enumeration + range decision/alignment + fetch entrypoints.
nydus/src/fanotify/event.rs	Kernel-independent parser for pre-content events and RANGE record.
nydus/src/fanotify/mount.rs	File-backed EROFS mount/unmount helpers and option encoding.
nydus/src/fanotify/response.rs	Exactly-once fanotify permission response ownership/submission.
nydus/src/fanotify/service.rs	Tokio-based event loop coordinating admission, deadlines, and fetch completion.
nydus/src/bin/nydus/main.rs	Adds `nydus fanotify` subcommand behind feature gate.
nydus/src/bin/nydus/fanotify.rs	Implements the fanotify command: runtime, signals, mount lifecycle.
nydus/Cargo.toml	Adds `fanotify` feature (tokio + tokio/time) and related comment.
nydus-accessor/src/storage/cache/local.rs	Reworks GroupFlight to prevent follower deadlock on leader panic.
nydus-accessor/src/accessor.rs	Adds `BlobAccessor::ready_ranges()` to query ready intervals without fetching.
docs/nydus.md	Documents the new `nydus fanotify` CLI subcommand and options.
docs/fanotify.md	New design/usage document for the fanotify pre-content service.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Zephyrcf · 2026-07-15T04:08:44Z

Can we introduce an optimization to skip marking full ready blob files.

This is already implemented in FanotifyService::setup. Fully ready blob files are not marked with fanotify_mark.

imeoer

Overall this is a high-quality PR: the byte-level event parser is pure and thoroughly tested, PendingPermission gives clean exactly-once response ownership, the O_NOFOLLOW + fd-identity check closes the path/mark race, and the data path fully reuses nydus-accessor (the daemon itself never copies blob bytes — the kernel reads the same cache file the accessor fills).

Requesting changes for: (1) the incorrect "kernel denies on close" claim and the resulting fail-open shutdown window, (2) the silent drop of packed events after a mid-batch parse error, and (3) at minimum documenting the deny-on-overload behavior. Details in the inline comments.

Verified locally: cargo clippy -p nydus --features fanotify --all-targets clean, 38 unit tests pass.

imeoer · 2026-07-15T06:15:27Z

+5. **Shutdown.** On `SIGTERM`/`SIGINT`/`SIGHUP`, deny every still-undecided
+   event, then close the group fd; the kernel denies any remaining permission
+   events on its own. The mount is unmounted after the fd is closed, so no read


[major] The kernel ALLOWS (not denies) outstanding permission events when the group fd is closed.

fanotify_release() in fs/notify/fanotify/fanotify_user.c simulates a userspace reply of FAN_ALLOW for every unanswered permission event on both the access list and the notification queue — a deliberate fail-open design so that a crashed listener never wedges readers.

Consequence for this daemon: between "group fd closed" and "unmount", any cold read still queued in the kernel (or arriving in that window) is allowed and reads zero pages from the sparse cache file — silent data corruption rather than fail-closed. deny_undecided() only covers events already read into userspace; it cannot cover events still sitting in the kernel queue. The same applies to a daemon crash.

The same claim appears in nydus/src/fanotify/service.rs (L166-169, L511-513) and nydus/src/bin/nydus/fanotify.rs (L154-158).

Suggestions:

Fix the comment/doc claims in all four places.

Reorder shutdown to unmount first, then close the group fd, so no new reads can reach the marked device files once the fd stops being serviced.

Document the crash-window semantics explicitly (fanotify is inherently fail-open on listener death).

imeoer · 2026-07-15T06:15:27Z

+
+#[allow(clippy::too_many_arguments)]
+async fn admit_batch(
+    fan: &AsyncFd<OwnedFd>,
+    core: &Arc<FanotifyCore>,
+    options: &FanotifyOptions,
+    semaphore: &Arc<Semaphore>,
+    writer: &Arc<dyn ResponseWriter>,
+    completion_tx: &mpsc::UnboundedSender<Completion>,


[major] A mid-batch parse error silently drops all remaining events in the same read(2) buffer → fd leaks + readers blocked forever.

EventIter sets done = true on any error, so after this branch denies the one fd and continues, the iterator yields nothing more — but the rest of the batch may contain valid events whose fds were already dup'd into this process at read() time. Those fds are never closed (leak) and their readers never get a response (they block indefinitely; permission events have no kernel-side timeout).

Two acceptable fixes:

Treat any mid-batch parse error as fatal (return Err and enter the fail-closed shutdown path, same as the no-fd "batch corruption" branch below), or

For error kinds where the event boundary is still trustworthy (event_len already validated, e.g. InvalidVersion, MissingRange), let the iterator skip the bad event and continue.

The current "deny one, silently drop the rest" is the worst of both options.

imeoer · 2026-07-15T06:15:27Z

+            "fanotify: event range {count} exceeds max_event_bytes {}",
+            options.max_event_bytes
+        );
+        return respond(fan, &mut permission, Response::Deny).await;
+    }
+    match core.range_ready(&device.id, offset, count) {
+        Ok(true) => {
+            debug!(
+                "fanotify: range [{}, +{}) already ready for blob {}; allowing immediately",
+                offset, count, device.id


[major] Denying on transient overload turns bursts into application-visible read errors (EPERM).

A burst of more than queue_depth/max_pending_events concurrent cold reads (common during bulk container startup) makes innocent read() calls fail outright.

A more natural backpressure model: when at capacity, stop draining the fanotify fd and let events queue in the kernel — readers then wait instead of failing. That interacts with the "queue overflow is fatal" policy, so it may need FAN_UNLIMITED_QUEUE or a raised max_queued_events.

If fail-fast is intentional, please at least document the trade-off in docs/fanotify.md (currently only the deny reason is listed, not the operational consequence).

imeoer · 2026-07-15T06:15:27Z

  "dep:tokio",
 ]
+# Serve an EROFS multi-device image on demand through fanotify pre-content
+# hooks (FAN_CLASS_PRE_CONTENT + FAN_PRE_ACCESS). Requires Linux >= 6.15,


[minor] Truncated comment: Requires Linux >= 6.14, — the sentence ends with a dangling comma. Also the minimum kernel version is inconsistent across the PR: this says 6.14, docs/fanotify.md L7 says 6.15+, L215 says 6.14 for the mask / 6.15 for RANGE, and event.rs L14/L20 splits them too. Since the daemon hard-requires the RANGE record (missing RANGE ⇒ deny), the effective minimum is whichever kernel ships FAN_EVENT_INFO_TYPE_RANGE — please pin one number everywhere.

imeoer · 2026-07-15T06:15:27Z

+unsafe impl Send for FdResponseWriter {}
+unsafe impl Sync for FdResponseWriter {}


[minor] The unsafe impl Send/Sync safety argument ("the service keeps the fanotify AsyncFd alive until all PendingPermission values have been drained or dropped") currently holds only through implicit local-variable drop order in run() (pending_events is declared after fan, so it drops first). That's correct but fragile — a refactor that reorders declarations turns the Drop best-effort deny into a write to a closed (or worse, reused) fd number. Suggest either a comment in run() pinning the ordering, or having the writer hold an Arc<OwnedFd> so the invariant is structural.

imeoer · 2026-07-15T06:15:27Z

+    pub fn fetch(
+        &self,
+        id: &BlobID,
+        cache_size: u64,
+        offset: u64,
+        count: u64,
+    ) -> Result<(), FetchError> {
+        let (aligned_off, aligned_len) =
+            align_fetch_range(offset, count, cache_size).map_err(FetchError::Range)?;


[minor] admit_event already calls align_fetch_range (service.rs L359) and then this fetch re-aligns the same range. Idempotent so harmless, but it obscures who owns the invariant. Consider making fetch take pre-aligned arguments and debug_assert! alignment instead.

imeoer · 2026-07-15T06:15:27Z

+
+use super::event::{PreContentEvent, Range};
+
+const EROFS_BLOCK_SIZE: u64 = 4096;


[nit] EROFS_BLOCK_SIZE is redefined here; nydus-accessor already has it. Re-export from the accessor to avoid drift.

imeoer · 2026-07-15T06:15:28Z

+    pub fn device_for(&self, dev: u64, ino: u64) -> Option<&BlobDevice> {
+        self.devices.iter().find(|d| d.dev == dev && d.ino == ino)
+    }


[nit] device_for is a linear scan per event. Fine for typical blob counts; a HashMap<(dev, ino), usize> would make it O(1) if images with many layers are expected.

imeoer · 2026-07-15T06:15:28Z

+/// event by `job_id` alone — no generation counter is needed.
+pub struct PendingTable {
+    entries: HashMap<u64, Entry>,
+    next_job_id: u64,
+    max_pending: usize,
+}
+


[design note, non-blocking] PendingTable and the pending_events: HashMap<u64, PendingEvent> in service.rs are two parallel maps keyed by the same job_id, where this table only adds a decided bool. The split buys pure unit-testable logic, which is a fair trade — just noting that merging them (entry holds Option<PendingPermission> + decided flag) would remove the "completion has no permission event" impossible-state error paths.

imeoer · 2026-07-15T06:15:28Z

+        let _guard = LeaderGuard {
+            flight: flight.clone(),
+            group_index,
+            inflight: &self.inflight_groups,
+        };

-        let mut buffers = self.buffers.lock().unwrap();
-        let decoded = fetch_decode_validate_group_into(
-            &self.blob_id,
-            &self.blob_meta,
-            &self.backend,
-            group,
-            &mut buffers,
-            RequestSource::OnDemand,
-        )?;
-        write_all_at(cache_file, group.uncompressed_byte_offset(), decoded)?;
-        self.groupmap.set_ready(group_index)
+        let result = (|| {
+            if self.groupmap.is_ready(group_index)? {
+                crate::metrics::inc_cache_hit_group();
+                return Ok(());
+            }
+            if let Some(recorder) = self.trace_recorder.as_ref() {
+                recorder.record_group_access(self.blob_index, group_index as u32);
+            } else {
+                crate::metrics::trace::record_group_access(self.blob_index, group_index as u32);
+            }
+
+            let mut buffers = BlobCacheBuffers::default();


The LeaderGuard fix is correct and necessary — the complete-before-remove ordering avoids the new-leader race, and idempotent complete() keeps the normal and panic paths from clobbering each other. Nice.

Two notes worth capturing in the commit message:

This is also a behavior change: the old fetch_lock serialized all group fetches per blob; now distinct groups fetch concurrently. Peak memory becomes up to queue_depth × BlobCacheBuffers and backend concurrency rises accordingly.

BlobCacheBuffers::default() is now allocated per cold-group fetch instead of reused via the old mutex-guarded buffer — a reasonable concurrency-for-allocation trade, just making it explicit.

imeoer · 2026-07-15T06:18:35Z

@Zephyrcf Thanks for the PR, could we add a performance comparison between fanotify and fuse in the doc? Using wordpress or openclaw image as examples?

imeoer · 2026-07-15T06:44:33Z

I have suggestions for the 4 options of the nydus fanotify CLI to simplify user decisions and reduce complexity:

The four FanotifyOptions knobs can all be removed or derived, which also simplifies the coordinator considerably:

--queue-depth → fold into --threads. Fetches run on the spawn_blocking pool, so the semaphore duplicates what runtime.max_blocking_threads(N) already provides. Drop the semaphore and let the blocking pool bound fetch concurrency; when the pool is busy, jobs queue instead of being denied — which is exactly the backpressure semantics we want. (Note: fetches are network-bound, so the default should be something like max(ncpu, 64), not the CPU count.)

--max-pending-events → internal constant + pause-draining. Replace "table full ⇒ FAN_DENY" with "table full ⇒ stop reading the fanotify fd". Unread events stay queued in the kernel and readers simply wait — no application-visible EPERM. Truly unlimited isn't safe (each pending event pins a dup'd fd, so RLIMIT_NOFILE is the real bound), but with pause-draining the limit never needs user tuning.

--max-event-bytes → delete the check entirely. align_fetch_range already clamps to the device size, which is the true upper bound; kernel readahead produces small ranges in practice, so this check only rejects mid-size ranges that are perfectly legal. (Clamping-then-allowing is not an option — allowing with a partially-filled range would let the reader see zero pages.)

--event-timeout-secs → rely on backend timeouts. The registry backend config already has connect/read timeouts and bounded retries, so a fetch always returns in bounded time and the event is answered through the normal completion path. This removes the per-event deadline, and with it the entire late-completion machinery (PendingTable's Decide/Late/Unknown states, expire_events, the deadline timer). One caveat: the cache-file write (write_all_at) has no timeout, so a wedged disk would leave readers hanging — but that matches normal POSIX semantics for hung local I/O, and a generous hard-coded safety deadline could be kept internally if desired. Requires backend timeout defaults to be enforced (not "unset = infinite").

Net result: the CLI shrinks to --bootstrap/--config/--mountpoint (+ logging), overload never surfaces as EPERM to applications, and coordinator.rs mostly disappears.

imeoer · 2026-07-15T06:51:37Z

Also, I recommend switching scripts/fanotify-e2e.sh to Go’s e2e implementation, such as tests/integration/fanotify_test.go.

Implement the subcommand behind . The daemon serves a multi-device EROFS image through the kernel fanotify pre-content interface (FAN_CLASS_PRE_CONTENT + FAN_PRE_ACCESS, Linux 6.15+): - event.rs: pure byte parser for fanotify_event_metadata and FAN_EVENT_INFO_TYPE_RANGE records - core.rs: decide, align_fetch_range, fd_identity, device_for, range_ready fast-path, fetch bridge to accessor - coordinator.rs: fixed-capacity pending table, monotonic job ids, on_completion Decide/Late/Unknown dispatch - response.rs: PendingPermission — exactly-once event fd ownership with best-effort deny on drop - mount.rs: file-backed EROFS mount with device= options - service.rs: FAN_CLASS_PRE_CONTENT group setup, AsyncFd event loop with biased select (stop > deadline > completion > readable), bounded admission with Semaphore + spawn_blocking fetch, per-event deadline and graceful single-stage shutdown implementations on LocalBlobCache, plus BlobAccessor::entries(), fetch(), and ready_ranges() bridge points. Signed-off-by: zephyrcf <zinsist777@gmail.com>

Add docs/fanotify.md covering the multi-device model, event ABI, processing pipeline, response protocol, lifecycle, requirements and constraints. Update docs/nydus.md to reflect the implemented command. Add tests/integration/fanotify_test.go for automated end-to-end smoke testing. Signed-off-by: Zephyrcf <zinsist777@gmail.com>

gaius-qi requested a review from Copilot July 15, 2026 03:35

Copilot started reviewing on behalf of gaius-qi July 15, 2026 03:36 View session

Copilot AI reviewed Jul 15, 2026

View reviewed changes

Comment thread scripts/fanotify-e2e.sh Outdated

Comment thread scripts/fanotify-e2e.sh Outdated

Comment thread scripts/fanotify-e2e.sh Outdated

Comment thread nydus/Cargo.toml

Zephyrcf force-pushed the feat/fanotify-pre-content branch 2 times, most recently from 39e18b1 to 35ffa2f Compare July 15, 2026 04:07

imeoer requested changes Jul 15, 2026

View reviewed changes

Zephyrcf force-pushed the feat/fanotify-pre-content branch 4 times, most recently from d94d340 to 6673737 Compare July 16, 2026 05:04

Zephyrcf force-pushed the feat/fanotify-pre-content branch from 6673737 to ca3f0f5 Compare July 16, 2026 05:08

		unsafe impl Send for FdResponseWriter {}
		unsafe impl Sync for FdResponseWriter {}


		use super::event::{PreContentEvent, Range};

		const EROFS_BLOCK_SIZE: u64 = 4096;

Uh oh!

Conversation

Zephyrcf commented Jul 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

Related Issues

Change Details

Test Results

Change Type

Self-Checklist

Uh oh!

joy-allen commented Jul 15, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Zephyrcf commented Jul 15, 2026

Uh oh!

imeoer left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

imeoer commented Jul 15, 2026

Uh oh!

imeoer commented Jul 15, 2026

Uh oh!

imeoer commented Jul 15, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Zephyrcf commented Jul 14, 2026 •

edited

Loading