Skip to content

du: match --exclude against trailing path components - #16699

Open
costajohnt wants to merge 4 commits into
uutils:mainfrom
costajohnt:fix/du-exclude-dot-operand
Open

costajohnt wants to merge 4 commits into
uutils:mainfrom
costajohnt:fix/du-exclude-dot-operand

Conversation

@costajohnt

Copy link
Copy Markdown
Contributor

--exclude with a pattern that has a / in it only matched the whole path or the basename, so du --exclude=target/debug . still counted ./target/debug. GNU matches a pattern against the full path and against every trailing part that starts after a /, so target/debug matches ./target/debug, x/target/debug and /abs/target/debug, while get/debug doesn't. du now does the same through one is_excluded(pattern, path) used at all three places that check excludes (the operands, safe_du and du_regular). That also removes an into_string().unwrap() on the basename that could panic on a non-UTF-8 name.

GNU also strips trailing slashes from an operand before walking it (du .// prints ./deep/target, and /, /// and //// all print as / while // stays //). uutils kept them, so child paths read .//deep/target, and with suffix matching a /-anchored pattern like --exclude=/target .// would have matched where GNU doesn't. Operands from the command line and --files0-from now have trailing separators collapsed to one before the walk and before -D handling, so a symlink given as la// is still followed.

Tests: test_du_exclude_trailing_components runs --exclude=a/b against ., a and ./a and checks xa/b is kept; test_du_operand_trailing_slashes covers a/, a//, a/// and excludes on those operands. Both fail on main. I compared output with GNU du 9.7 (LC_ALL=C) over 34 patterns, 20 operands (including .//, target//, symlinks with trailing slashes, ..//w// and /tmp//dir/w) and none, -L, -a, -D and -H: all 3,400 runs print the same as GNU, where main differed in 1,527. --files0-from with trailing-slash entries matches too.

Fixes #16691

Exclude patterns were only tried against the full path and the base
name, so --exclude=target/debug missed ./target/debug when the operand
was . or ./target. Also try every part of the path that starts after a
separator, for operands and traversed entries alike, matching GNU du.

Fixes uutils#16691
An operand like a// was kept as is, so entries showed as a//x, unlike
GNU's a/x, and the exclude suffix walk could match /x against them.
Collapse trailing separators to one before traversal, keeping / and //
as GNU does.
@sylvestre

Copy link
Copy Markdown
Contributor

thanks for the patch!
pre-commit.ci is failing, please fix it.
Could you also check that the per-component matching doesn't slow down du on a big tree with excludes (see inline)?

Comment thread src/uu/du/src/du.rs Outdated
pattern.matches(&path)
|| path
.char_indices()
.any(|(i, c)| std::path::is_separator(c) && pattern.matches(&path[i + 1..]))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is now one glob match per component for every entry and every pattern, any perf impact on a big tree?
could we only check the full path and the basename when the pattern has no /?

Comment thread src/uu/du/src/du.rs Outdated
if end == bytes.len() {
return path;
}
uucore::os_str_from_bytes(&bytes[..end]).map_or_else(|_| path.clone(), PathBuf::from)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a prefix of valid bytes can't fail here, do we need the path.clone() fallback?

A pattern starting with * already matched the whole path, and a pattern
without wildcards or separators can only match the last component.
Convert the path once per entry instead of once per pattern, and drop
the unreachable fallback in collapse_trailing_separators.
@costajohnt

Copy link
Copy Markdown
Contributor Author

Thanks for the review!

pre-commit.ci: that run never got to the hooks, it stopped at "mergeable check: timeout waiting for merge ref". Running the hooks locally on the changed files passes (cspell, fmt, clippy -D warnings on uu_du), so a re-run should be green.

perf: you were right, it was measurable. On a generated tree with 200k files (paths about 7 components deep), du -s, median of 15 interleaved runs:

excludes main PR before PR now
none 492 ms 488 ms 489 ms
*.o node_modules target/debug *.tmp .git 548 ms 746 ms 535 ms
target/debug x?y foo[0-9] (all need the scan) 488 ms 524 ms 502 ms

I didn't go with "full path and basename when the pattern has no /", because GNU's ?, * and [...] also match / here, so a pattern without / can still match more than the basename: GNU du --exclude='a?b' x drops x/a/b, which basename only matching would keep. I added a test for that case.

Instead the scan is skipped only where it can't change the result:

  • a pattern starting with * that matches a trailing part also matches the whole path, so the full path check is enough
  • a pattern with no wildcard and no separator (node_modules, .git) can only match the last component, so only that is checked
  • the path is converted to a string once per entry instead of once per pattern

I compared the old and new binaries against GNU du 9.11 with -a over 13 patterns x 4 operands on a small tree with these corner cases, and all three give the same output.

du.rs:873: right, the bytes came from os_str_as_bytes so the prefix can't fail. Removed the path.clone() fallback.

@github-actions

Copy link
Copy Markdown

Binary size comparison:

Individual binary size comparison VS main (threshold: >=5% AND >=4 KB).

Total size of compared binaries: 155.48 MB (+100 KB, +0.06%)

Significant per-binary changes:
  env     1.20 MB ->    1.28 MB  (+80 KB, +6.51%)

@codspeed

codspeed Bot commented Nov 19, 2026

Copy link
Copy Markdown
Contributor

Merging this PR will degrade performance by 5.19%

1 warning

⚠️ 4 benchmarks are dominated by system calls

System calls cannot be simulated consistently and are left out of the measure. The reported value understates the real cost. Switch to the Walltime instrument to measure them.

Learn more about system calls

See affected benchmarks

❌ 3 regressed benchmarks
✅ 5 untouched benchmarks
⏩ 476 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
❌ CPU Simulation du_deep_tree[(100, 3)] 1.8 ms 1.9 ms -5.53%
❌ CPU Simulation du_summarize_balanced_tree[(5, 4, 10)] 11.3 ms 11.9 ms -5.25%
❌ CPU Simulation du_wide_tree[(5000, 500)] 15.8 ms 16.5 ms -4.79%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing costajohnt:fix/du-exclude-dot-operand (63f925c) with main (293a724)2

Open in CodSpeed

Footnotes

  1. 476 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

  2. No successful run was found on main (91f89c4) during the generation of this report, so 293a724 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

@github-actions

Copy link
Copy Markdown

GNU testsuite comparison:

Congrats! The gnu test tests/tail/follow-name is no longer failing!
Congrats! The gnu test tests/tail/pid-pipe is no longer failing!

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

du: adding ./ to the operand silently defeats --exclude=target/debug

2 participants