Skip to content

fix(indexer): skip unreadable files instead of aborting the whole index - #297

Merged
Helweg merged 2 commits into
Helweg:mainfrom
dkhokhlov:fix/skip-unreadable-files
Aug 17, 2026
Merged

fix(indexer): skip unreadable files instead of aborting the whole index#297
Helweg merged 2 commits into
Helweg:mainfrom
dkhokhlov:fix/skip-unreadable-files

Conversation

@dkhokhlov

Copy link
Copy Markdown
Contributor

A file that is unreadable at the OS level (e.g., an LSM denial returning EPERM
despite readable mode bits, or a permissions EACCES) aborted the entire indexing
fixes #296

Fix

Wrap both hashFile calls in try/catch:

  • Main loop: record the file in stats.skippedFiles with a new "unreadable"
    reason and continue, so the remaining files index.
  • Freshness check: return reason: "unreadable" (reuses the existing
    IndexFreshnessReason) so the next index_codebase rebuilds — which skips the
    unreadable file — instead of aborting.

Add "unreadable" to the SkippedFile.reason union in src/utils/files.ts.

Repro

~/SOURCE/Matlab2/lib/@medusa7/calc_spreads.m is LSM-protected (open() returns
EPERM errno 1, lsattr EPERMs, despite 0644 owner-owned perms — expected by
design). Its readdir position in lib/@medusa7 (219 .m files) is 110
(0-indexed), so it was skipped under the default maxFilesPerDirectory: 100 but
included once the global config raised it to 500, aborting ci -p with
Operation not permitted (os error 1) and 0 files processed. With this fix the
run completes and the file is reported as skipped; 973 of 974 .m files index.

Test

tests/skip-unreadable-file.test.ts indexes a project containing a readable file
and a chmod 000 file (EACCES — same catch path as EPERM); asserts the run
succeeds, the readable file is indexed, and the unreadable file appears in
skippedFiles with reason: "unreadable". Existing host-paths /
host-mode-paths suites stay green (41/41).

A file that is unreadable at the OS level (e.g., an LSM denial that returns
EPERM despite readable mode bits, or a permissions EACCES) aborted the entire
indexing run. The native hasher (native/src/hasher.rs fs::File::open) throws on
such a file, and the two TS call sites propagated it unguarded:

- src/indexer/index.ts main loop (hashFile per file)
- src/indexer/index.ts getIndexFreshness loop

Wrap both hashFile calls in try/catch. In the main loop, record the file in
stats.skippedFiles with a new "unreadable" reason and continue, so the remaining
files index. In the freshness check, return reason "unreadable" (reusing the
existing IndexFreshnessReason) so the next index_codebase rebuilds, which skips
the unreadable file, instead of aborting.

Add "unreadable" to the SkippedFile.reason union in src/utils/files.ts.

Repro: ~/SOURCE/Matlab2/lib/@medusa7/calc_spreads.m is LSM-protected (open()
returns EPERM errno 1, lsattr EPERMs, despite 0644 owner-owned perms). Its
readdir position in lib/@Medusa7 (219 .m files) is 110 (0-indexed), so it was
skipped under the default maxFilesPerDirectory:100 but included once the global
config raised it to 500, aborting ci -p with "Operation not permitted (os error
1)" and 0 files processed. With this fix the run completes and the file is
reported as skipped.

Test: tests/skip-unreadable-file.test.ts indexes a project containing a readable
file and a chmod 000 file (EACCES, same catch path as EPERM); asserts the run
succeeds, the readable file is indexed, and the unreadable file appears in
skippedFiles with reason "unreadable".
@Helweg

Helweg commented Aug 16, 2026

Copy link
Copy Markdown
Owner

Thanks for the focused fix. The two guarded hash paths look right. Could you add one regression test that exercises the freshness-check path as well? The issue affects both the main indexing loop and getIndexFreshness, while the current test covers only the indexing loop. Once that is covered, this should be ready for CI approval and merge review.

PR Helweg#297 guarded hashFile in two places: the main indexing loop and
getIndexFreshness. The existing regression test exercised only the
indexing loop, so the freshness-check guard had no coverage.

Add a test that builds an index (so getIndexFreshness passes the
"missing" early return), then calls getIndexFreshness with the
unreadable source file still present. Assert it returns
{ readable: false, current: false, reason: "unreadable" } rather than
rejecting with the hashFile Permission-denied error.

Verified the test fails when the freshness guard is reverted (the
pre-fix throw aborts the check) and passes with it restored.
@dkhokhlov

Copy link
Copy Markdown
Contributor Author

Added a regression test covering the freshness-check path.

The test builds an index first so getIndexFreshness passes the "missing" early return and reaches the hash loop, then calls getIndexFreshness() with the unreadable source file still present. It asserts { readable: false, current: false, reason: "unreadable" } rather than rejecting with the hashFile permission error.

Verified the test fails when the freshness guard is reverted (the pre-fix Permission denied throw aborts the check) and passes with it restored. Both tests in the suite are green.

@Helweg Helweg added the bug Bug fix label Aug 17, 2026
@Helweg
Helweg merged commit d23f2a9 into Helweg:main Aug 17, 2026
7 of 8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Bug fix

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] One unreadable file (EPERM) aborts the entire index run

2 participants