DAOS-19526 dfs: Enable Hardlink support for dfuse by sherintg · Pull Request #19065 · daos-stack/daos

sherintg · 2026-09-15T07:24:43Z

This commit implements the following changes:

Introduced a callback handler for hardlink (link) in dfuse.
Modified the dfuse unlink handler to treat the unlink of a hardlink differently from the removal of the last link.
Introduced two new dfs functions for unlink and rename that tell the caller whether the target file was really removed/deleted. These apis are called by dfuse rename and unlink handlers.
Added support for tracking multiple dentries per inode entry. The entries are updated as they are encountered during link, unlink, rename and lookup/readdir operations, and every name is invalidated when the inode is dropped.
NLT functional dfuse tests and DFS unit tests for hardlinks.

Allow-unstable-test: true

Steps for the author:

Commit message follows the guidelines.
Appropriate Features or Test-tag pragmas were used.
Appropriate Functional Test Stages were run.
At least two positive code reviews including at least one code owner from each category referenced in the PR.
Testing is complete. If necessary, forced-landing label added and a reason added in a comment.

After all prior steps are complete:

Gatekeeper requested (daos-gatekeeper added as a reviewer).

github-actions · 2026-09-15T07:25:01Z

Ticket title is ' Enable Hardlink support for dfuse'
Status is 'In Review'
https://daosio.atlassian.net/browse/DAOS-19526

daosbuild3 · 2026-09-15T08:10:34Z

Test stage NLT completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-19065/1/testReport/

daosbuild3 · 2026-09-15T10:38:04Z

Test stage Functional on EL 9 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19065/1/execution/node/1286/log

daosbuild3 · 2026-09-15T11:21:38Z

Test stage NLT completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-19065/2/testReport/

This commit implements the following changes: 1. Introduced a callback handler for hardlink (link) in dfuse. 2. Modified the dfuse unlink handler to treat the unlink of a hardlink differently from the removal of the last link. 3. Introduced two new dfs functions for unlink and rename that tell the caller whether the target file was really removed/deleted. These apis are called by dfuse rename and unlink handlers. 4. Added support for tracking multiple dentries per inode entry. The entries are updated as they are encountered during link, unlink, rename and lookup/readdir operations, and every name is invalidated when the inode is dropped. 5. NLT functional dfuse tests and DFS unit tests for hardlinks. Allow-unstable-test: true Signed-off-by: Sherin T George <sherin-t.george@hpe.com>

daosbuild3 · 2026-09-15T15:23:22Z

Test stage Functional on EL 9 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-19065/3/testReport/

mchaarawi

it's going to take me a while to review this large PR.
for now, please checkout the copilot review in:
/home/chaarawi/reviews/PR-19065-dfuse-hardlinks-review.md
this is in the nfs from any of our internal hsw / brd nodes. let me know if you can't access it.

mchaarawi · 2026-09-15T19:27:23Z

also please update the feature branch to latest master and merge this PR.
the branch is quite stale now.

daosbuild3 · 2026-09-16T02:39:19Z

Test stage Functional Hardware Medium MD on SSD completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-19065/3/testReport/

…9526 Signed-off-by: Sherin T George <sherin-t.george@hpe.com>

sherintg · 2026-09-25T05:45:56Z

@mchaarawi @kanard38 I have addressed the review comments including the merge.

mchaarawi · 2026-09-25T12:28:34Z

can you please repush with commit pragma:

Features: dfs dfuse pil4dfs

mchaarawi · 2026-09-25T12:46:32Z

a few nits:

the last commit message looks inaccurate:

dfs: set the 'deleted' out-param in dfs_move_internal() when a rename
resolves the same object id

dfs_move_internal looks untouched to me. The actual DFS change is the remove_hardlink() restart fix.

there is no EXDEV test for the new check in df_ll_link(). so that has no test coverage now.
should be simple to add a pool-level mount (DFuse(..., container=None)) with two containers and os.link across them expecting errno.EXDEV
couple things to address and require tickets to follow on:

ln -P symlink hl (hardlink to a symlink)
pil4dfs interception

mchaarawi · 2026-09-25T12:47:14Z

                                     test_script: 'ci/unit/test_nlt.sh' +
                                                  ' --system-ram-reserved 4' +
-                                                  ' --max-log-size 1950MiB' +
+                                                  ' --max-log-size 2500MiB' +


why is this needed?

With the new set of tests added, I had seen failure related to log-size not sufficient. Hence increased the size.

…link Rework for the dfuse hardlink support. - dfs: in remove_hardlink(), set the 'deleted' out-param on the branch that drops the final link (link_cnt reaches 0), so the value is set on every path and stays correct across a transaction restart instead of retaining a stale result from an earlier attempt; previously it was only cleared to false when a link survived. - dfuse/unlink: evict the surviving inode's metadata cache in dfuse_hardlink_removed() so a peer-name unlink/rename no longer leaves stale st_nlink/ctime for the kernel's follow-up GETATTR to read. - dfuse/fuseops: reject cross-container hardlinks in df_ll_link() with EXDEV before calling into DFS, instead of relying on an oid lookup that fails with a misleading ENOENT (oids are container-scoped). - dfuse/core: in dfuse_ie_dentry_replace(), de-duplicate when the rename destination name is already tracked (primary or secondary) so the secondary list never gains a duplicate entry. Defer freeing a removed dentry in both dfuse_ie_dentry_replace() and dfuse_ie_dentry_set_single() until after the spinlock is released, matching the other dentry helpers. - tests: add test_hardlink_stale_notify_delete (out-of-band removal then a local final unlink to exercise notify_delete on stale same-parent dentries), add test_hardlink_stale_name, add test_hardlink_cross_container (pool-level mount with two containers, os.link across them expects EXDEV), and assert the refreshed link count after a peer-name unlink in test_hardlink_caching. Features: dfs dfuse pil4dfs Allow-unstable-test: true Signed-off-by: Sherin T George <sherin-t.george@hpe.com>

daosbuild3 · 2026-09-25T14:47:03Z

Test stage Functional on EL 9 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19065/5/execution/node/1381/log

sherintg · 2026-09-25T14:56:46Z

@mchaarawi I have incorporated the comments.

I have added additional test for cross container link() op.
repushed with the amended commit string.
Created a ticket DAOS-19701 for pil4dfs changes.
hardlinks on symlinks is not supported by the current implementation. As discussed over chat, I will add it as limitation in the user documentation.

Addressed a regression seen in CI/CD where open_stat() in DFS was not sanitizing the mode passed as argument. The fix strips the internal bits (DFS_EXTERNAL_MODE) from caller-supplied modes at the DFS boundary so an untrusted mode can never forge internal state: - open_stat(): mask the mode before it is stored; legitimate hardlinks are still flagged from fetched entry state after creation. - dfs_mkdir(): mask the mode before building the directory entry. - dfs_osetattr(): return the sanitized external mode in the out stat instead of echoing the caller's raw mode. Features: dfs dfuse pil4dfs Allow-unstable-test: true Signed-off-by: Sherin T George <sherin-t.george@hpe.com>

daosbuild3 · 2026-09-26T10:00:29Z

Test stage Functional Hardware Medium MD on SSD completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-19065/7/testReport/

Addressed a regression seen in CI/CD where simul tests expected hardlink related tests to fail. Now hardlink related tests are removed from the faillist. Features: dfs dfuse pil4dfs Allow-unstable-test: true Signed-off-by: Sherin T George <sherin-t.george@hpe.com>

knard38

First round of the comment

knard38 · 2026-09-28T10:12:24Z

+dfuse_ie_dentry_inval(struct dfuse_info *dfuse_info, struct dfuse_dentry *released);
+
+/* Queue every name in released for invalidation on the invalidation thread, consuming released.
+ * ie_drop, if set, is released after the final name is invalidated.


Should be worth of it to mention that there is some actions which are always taken even in case of error.
For example removing the entries and calling dfuse_inode_decref()

knard38 · 2026-09-28T12:33:08Z

+}
+
+void
+dfuse_ie_inode_delete(struct dfuse_info *dfuse_info, struct dfuse_inode_entry *ie,


Would it be possible to reuse the dfuse_queue_inval_dentries() / ival_drain_queue() pattern here, so that the fuse_lowlevel_notify_delete() calls are issued from the invalidation thread rather than synchronously on the FUSE worker?

From my investigation with copilot, in the kernel, fuse_reverse_inval_entry() takes the parent directory's i_rwsem , which may be held by another in-flight operation itself waiting for a dfuse reply. With several stale names in other directories this could stall the worker noticeably and reduce the reactivity of the dfuse daemon.
Delaying the delete should be safe since the kernel checks the child nodeid before d_delete(). Adding an ino field to struct dfuse_inval_item plus a branch in ival_drain_queue() looks sufficient.

From my side, it is a nice to have but could be worth considering for a follow-up PR.

sherintg force-pushed the sherintg/dfs_hardlinks/DAOS-19526 branch from 14c35b2 to aff5ff5 Compare September 15, 2026 10:36

sherintg force-pushed the sherintg/dfs_hardlinks/DAOS-19526 branch from aff5ff5 to d0d244e Compare September 15, 2026 11:31

sherintg requested review from knard38 and mchaarawi September 15, 2026 12:30

sherintg marked this pull request as ready for review September 15, 2026 12:30

sherintg requested review from a team as code owners September 15, 2026 12:30

mchaarawi reviewed Sep 15, 2026

View reviewed changes

Merge branch feature/dfs_hardlinks into sherintg/dfs_hardlinks/DAOS-1…

f934184

…9526 Signed-off-by: Sherin T George <sherin-t.george@hpe.com>

sherintg requested a review from mchaarawi September 25, 2026 05:46

sherintg force-pushed the sherintg/dfs_hardlinks/DAOS-19526 branch from 7ed58df to 5f7332a Compare September 25, 2026 05:53

mchaarawi reviewed Sep 25, 2026

View reviewed changes

sherintg force-pushed the sherintg/dfs_hardlinks/DAOS-19526 branch from 5f7332a to 3b0b5ab Compare September 25, 2026 14:45

sherintg requested review from a team as code owners September 28, 2026 05:56

knard38 reviewed Sep 28, 2026

View reviewed changes

Conversation

sherintg commented Sep 15, 2026

Steps for the author:

After all prior steps are complete:

Uh oh!

github-actions Bot commented Sep 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

daosbuild3 commented Sep 15, 2026

Uh oh!

daosbuild3 commented Sep 15, 2026

Uh oh!

daosbuild3 commented Sep 15, 2026

Uh oh!

daosbuild3 commented Sep 15, 2026

Uh oh!

mchaarawi left a comment

Choose a reason for hiding this comment

Uh oh!

mchaarawi commented Sep 15, 2026

Uh oh!

daosbuild3 commented Sep 16, 2026

Uh oh!

sherintg commented Sep 25, 2026

Uh oh!

mchaarawi commented Sep 25, 2026

Uh oh!

mchaarawi commented Sep 25, 2026

Uh oh!

mchaarawi Sep 25, 2026

Choose a reason for hiding this comment

Uh oh!

sherintg Sep 25, 2026

Choose a reason for hiding this comment

Uh oh!

daosbuild3 commented Sep 25, 2026

Uh oh!

sherintg commented Sep 25, 2026

Uh oh!

daosbuild3 commented Sep 26, 2026

Uh oh!

knard38 left a comment

Choose a reason for hiding this comment

Uh oh!

knard38 Sep 28, 2026

Choose a reason for hiding this comment

Uh oh!

knard38 Sep 28, 2026

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

4 participants

github-actions Bot commented Sep 15, 2026 •

edited

Loading