De-duplicate filesystem emplace - #2396
Merged
MarcusSorealheis merged 2 commits intoJun 12, 2026
Merged
Conversation
|
@palfrey is attempting to deploy a commit to the native-link-web-assets Team on Vercel. A member of the Team first needs to authorize it. |
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
palfrey
force-pushed
the
add-hardlink-failure-log
branch
6 times, most recently
from
June 8, 2026 10:12
86abbd3 to
e1e05ad
Compare
palfrey
force-pushed
the
add-hardlink-failure-log
branch
from
June 10, 2026 10:17
dd61b5f to
e58cc5a
Compare
palfrey
commented
Jun 10, 2026
|
|
||
| case $nativelink_output in | ||
| *"ERROR"* ) | ||
| *\"ERROR\"* ) |
Member
Author
There was a problem hiding this comment.
Because the new logs now contain the word error in some file paths, which is fine. We're looking for a log message error, which has quoted ERROR. Bash case uses non-escaped quotes to handle the case for matches with spaces, which is not what we want.
palfrey
marked this pull request as ready for review
June 10, 2026 11:36
palfrey
force-pushed
the
add-hardlink-failure-log
branch
from
June 10, 2026 11:40
e58cc5a to
ac25a3f
Compare
palfrey
force-pushed
the
add-hardlink-failure-log
branch
from
June 10, 2026 11:42
ac25a3f to
692e206
Compare
amankrx
approved these changes
Jun 12, 2026
MarcusSorealheis
approved these changes
Jun 12, 2026
MarcusSorealheis
enabled auto-merge (squash)
June 12, 2026 02:47
rejuvenile
pushed a commit
to rejuvenile/nativelink
that referenced
this pull request
Jul 13, 2026
Co-authored-by: Marcus Eagan <marcuseagan@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
We've seen intermittent hardlink failures on the buildstream runs (e.g. https://github.com/TraceMachina/nativelink/actions/runs/26881965504/job/79284041143?pr=2381) but the failure case was unclear. After some experimentation, I've been able to figure it out, both by adding logging here, and then debugging based on that.
The hardlink code first gets a file entry from the filesystem cache, then hardlinks to it. Most of the time, this succeeds, and the primary previous failure case was "your filesystem cache isn't big enough for a single build run, so it's evicting between getting and linking", which was a config fix. I added a logging helper that dumps the cache status, and established this wasn't the case for the buildstream case (was hitting ~1% of cache usage).
The problem for buildstream is that for some reason a single file (usually a Python library file in the case of buildstream) is being uploaded multiple times, and so we evict the previous instance and put a new one in. Sometimes, if we're very unlucky (~1/3rd of the time on buildstream runs right now), we're doing hardlinking in the middle of that.
This PR checks for duplicate file entries i.e. we're trying to emplace a file with identical contents to the current one, and so therefore we can throw away the new entry. This means the file is always correctly in place on the filesystem, and so the hardlink should no longer fail in the previous way.
Type of change
Please delete options that aren't relevant.
How Has This Been Tested?
bazel test //...Checklist
bazel test //...passes locallygit amendsee some docsThis change is