Workaround for Windows being unable to remove tmp directories when installing GGUF files #8699
+59
−4
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Due to a core issue involving torch memory mapping of GGUF, attempts to install GGUF models on Windows platforms fail with an error that
rmtree()cannot remove the temporary directory created to hold the downloaded model. This PR fixes the issue in several ways:torch.from_numpy(tensor.data)in the GGUF reader withtorch.from_numpy(tensor.data.copy()). This makes a copy of the torch data so that the memory-mapped file is not opened. This is the core issue and should fix the problem. But just in case...rmtree()and move() calls in loops that call garbage collection and insert short delays over multiple tries.Related Issues / Discussions
See Discord discussion starting around message https://discord.com/channels/1020123559063990373/1049495067846524939/1453426867733401663.
QA Instructions
On a Windows system:
Failed to remove temporary directory XXXXXX: Permission Denied.... It will be removed on next server startRemoving dangling temporary directory XXXXXXMerge Plan
Small change. Should be a simple merge.
Checklist
What's Newcopy (if doing a release after this PR)