Skip to content

Conversation

rfletchr
Copy link

@rfletchr rfletchr commented Sep 6, 2024

This change optimizes the hashing of checkpoint files using memory mapping.

Essentially this makes file I/O far more efficient and allows the hasher to run extremely quickly.

A SDXL checkpoint was taking roughly 30 seconds to hash using the original implementation, this is reduced to about 6 seconds with memory mapping.

This could be accelerated further by switching to a non cryptographic hash like blake3 which is extremely fast, and supports multithreading.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant