minosu

/

godot_dodo_4x_60k_llama_7b

like 3

Hey, thanks for your interest!

You should be able to follow the instructions in the llama.cpp repo to quantize this model: https://github.com/ggerganov/llama.cpp

There's detailed documentation in the readme there, but in short the process should look something like this:

• Clone llama.cpp
• make
• Download weights from HF and put in subfolder, like models/dodo
• python3 -m pip install -r requirements.txt
• python3 convert.py models/dodo/
• ./quantize ./models/dodo/ggml-model-f16.gguf ./models/dodo/ggml-model-q4_0.gguf q4_0

Thanks for the detailed description, now I’ll try to follow your steps! 😊