fix(indexer): skip unreadable files instead of aborting the whole index - #297
Conversation
A file that is unreadable at the OS level (e.g., an LSM denial that returns EPERM despite readable mode bits, or a permissions EACCES) aborted the entire indexing run. The native hasher (native/src/hasher.rs fs::File::open) throws on such a file, and the two TS call sites propagated it unguarded: - src/indexer/index.ts main loop (hashFile per file) - src/indexer/index.ts getIndexFreshness loop Wrap both hashFile calls in try/catch. In the main loop, record the file in stats.skippedFiles with a new "unreadable" reason and continue, so the remaining files index. In the freshness check, return reason "unreadable" (reusing the existing IndexFreshnessReason) so the next index_codebase rebuilds, which skips the unreadable file, instead of aborting. Add "unreadable" to the SkippedFile.reason union in src/utils/files.ts. Repro: ~/SOURCE/Matlab2/lib/@medusa7/calc_spreads.m is LSM-protected (open() returns EPERM errno 1, lsattr EPERMs, despite 0644 owner-owned perms). Its readdir position in lib/@Medusa7 (219 .m files) is 110 (0-indexed), so it was skipped under the default maxFilesPerDirectory:100 but included once the global config raised it to 500, aborting ci -p with "Operation not permitted (os error 1)" and 0 files processed. With this fix the run completes and the file is reported as skipped. Test: tests/skip-unreadable-file.test.ts indexes a project containing a readable file and a chmod 000 file (EACCES, same catch path as EPERM); asserts the run succeeds, the readable file is indexed, and the unreadable file appears in skippedFiles with reason "unreadable".
|
Thanks for the focused fix. The two guarded hash paths look right. Could you add one regression test that exercises the freshness-check path as well? The issue affects both the main indexing loop and |
PR Helweg#297 guarded hashFile in two places: the main indexing loop and getIndexFreshness. The existing regression test exercised only the indexing loop, so the freshness-check guard had no coverage. Add a test that builds an index (so getIndexFreshness passes the "missing" early return), then calls getIndexFreshness with the unreadable source file still present. Assert it returns { readable: false, current: false, reason: "unreadable" } rather than rejecting with the hashFile Permission-denied error. Verified the test fails when the freshness guard is reverted (the pre-fix throw aborts the check) and passes with it restored.
|
Added a regression test covering the freshness-check path. The test builds an index first so Verified the test fails when the freshness guard is reverted (the pre-fix Permission denied throw aborts the check) and passes with it restored. Both tests in the suite are green. |
A file that is unreadable at the OS level (e.g., an LSM denial returning EPERM
despite readable mode bits, or a permissions EACCES) aborted the entire indexing
fixes #296
Fix
Wrap both
hashFilecalls in try/catch:stats.skippedFileswith a new"unreadable"reason and
continue, so the remaining files index.reason: "unreadable"(reuses the existingIndexFreshnessReason) so the nextindex_codebaserebuilds — which skips theunreadable file — instead of aborting.
Add
"unreadable"to theSkippedFile.reasonunion insrc/utils/files.ts.Repro
~/SOURCE/Matlab2/lib/@medusa7/calc_spreads.mis LSM-protected (open()returnsEPERM errno 1,
lsattrEPERMs, despite 0644 owner-owned perms — expected bydesign). Its readdir position in
lib/@medusa7(219.mfiles) is 110(0-indexed), so it was skipped under the default
maxFilesPerDirectory: 100butincluded once the global config raised it to 500, aborting
ci -pwithOperation not permitted (os error 1)and 0 files processed. With this fix therun completes and the file is reported as skipped; 973 of 974
.mfiles index.Test
tests/skip-unreadable-file.test.tsindexes a project containing a readable fileand a
chmod 000file (EACCES — same catch path as EPERM); asserts the runsucceeds, the readable file is indexed, and the unreadable file appears in
skippedFileswithreason: "unreadable". Existinghost-paths/host-mode-pathssuites stay green (41/41).