Skip to content

Improve loading performance of large folders and network mounts - #3825

Open
paulvandenburg wants to merge 3 commits into
linuxmint:masterfrom
paulvandenburg:feature/performance-large-folders
Open

Improve loading performance of large folders and network mounts#3825
paulvandenburg wants to merge 3 commits into
linuxmint:masterfrom
paulvandenburg:feature/performance-large-folders

Conversation

@paulvandenburg

Copy link
Copy Markdown

Context:
I noticed that opening folders with lots of files, especially from a network mount (SMB), was very slow and while loading the files shown would jump around as newly loaded files were inserted between the previously shown files. Opening the same folder mounted in Windows was fast, so I wanted to fix this.

For context I used these folders for my tests:

  • My backed up camera roll from my phone, 5807 files, 22,3GB in size, all images/jpg
  • A similar folder but with images without EXIF data, mix of png/jpg
  • A small background images folder with only a handful of files for regression checking

SMB notes, I use mostly default settings for the mounted folder. Of note is that rsize was at 4MB (default).
The relevant network setup is that the server is at home (wired connection), that I'm trying to access from my laptop over WIFI (stable/strong signal). So near optimal conditions.

Nemo settings are mostly default. Thumbnail generation is set to Yes so those are also generated for network mounts.

Steps taken:

  • Split the loading of the folder contents in 2 parts, loading the simple folder contents, and loading additional file details to enrich the view and file handling. I noticed a similar performance hit when using ls in the CLI. Where time ls /mnt/lots-o-files shows a real time of 0,3s but a time ls -l /mnt/lots-o-files gives 11,8s. Explained by the second needing to get file stats for each, needing a network round trip per file, getting slow even on a fast connection with large file numbers.
  • Thumbnail generation starts after the list is complete, this achieves 2 things: Prevent fighting over network resources between thumbnail generation and list loading. Secondly only the files you actually see get their thumbnail generated instead of a bunch that quickly drop out of your view window due to later loaded files.
  • Some images (camera/phone pictures) have a thumbnail embedded in the EXIF data, using that instead of loading the full image to generate a thumbnail is much faster, if available. A check is there to ensure only included thumbnails that are equal or better in quality than the thumbnail generator would do are used.
    To avoid the rsize value causing a 4MB load regardless, posix_fadvise is used to only get the necessary bytes.

Results:

  • The camera roll now displays listed files in about 0,19 seconds, compared to 15,1 seconds previously.
  • The EXIF thumbnail in an example photo of 5,4MB now only needed to load 0,27MB.

Considerations:

  • Due to the split loading, at first this solution doesn't have precise knowledge about file types other than what can be derived from the filename. This derived information is still used until the precise knowledge comes in, but there is a gap there. I think this is fine as I think it is better to have a snappy UI, that is eventually consistent, than a slow UI but always correct about file types. Due to the much faster loading, this gap should be negligible in practice.

Notes:

  • I've tested this to the best of my abilities on my Linux Mint 22.3 laptop against the listed folder setups.
  • I've used Claude to help make these changes. I'm a Java developer by profession so I'm not that familiar with C. I've checked and refined the output as best I could, but a critical look couldn't hurt.
  • I found the mint docs mention some editor standards, but missed a .editorconfig in the project, added one that I think reflects those standards, which I included in a separate commit. Can be removed. Did notice the existing code contained quite a few whitespace at the end of lines, tried to not include the cleanup of those here.

…rantly.

free_location_change() null-checks slot->new_content_view, then calls
nemo_view_stop_loading() on it. That removes the view's directory
monitor, which runs the directory's async state machine and can emit
signals that come back round to free_location_change() for the same
slot. The reentrant call clears and unrefs slot->new_content_view, so
when the outer call resumes it hands NULL to
nemo_window_disconnect_content_view(), which asserts on it and aborts.
@paulvandenburg

Copy link
Copy Markdown
Author

Latest commit also fixes a crash I encountered, under certain conditions I'm not fully sure of with back/forth navigation and view refreshing. Not sure if it was introduced here, but this hopefully fixes it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant