This repo includes .gguf built for HuggingFace/Candle.
.gguf
Refer to the original repo for more details.
4-bit
5-bit
8-bit
16-bit