|
--- |
|
license: mit |
|
--- |
|
|
|
## How to convert |
|
|
|
First, you need git clone [llama.cpp](https://github.com/ggerganov/llama.cpp) and make it. |
|
|
|
Then follow the instrution to generate gguf files. |
|
|
|
``` |
|
# convert Qwen HF models to gguf fp16 format |
|
python convert-hf-to-gguf.py --outfile qwen7b-chat-f16.gguf --outtype f16 Qwen-7B-Chat |
|
|
|
# quantize the model to 4-bits (using q4_0 method) |
|
./quantize qwen7b-chat-f16.gguf qwen7b-chat-q4_0.gguf q4_0 |
|
|
|
# chat with Qwen models |
|
./main -m qwen7b-chat-q4_0.gguf -n 512 --color -i -cml -f prompts/chat-with-qwen.txt |
|
``` |
|
|
|
|
|
## Files are split and require joining |
|
|
|
**Note:** HF does not support uploading files larger than 50GB but upload a 41GB file is too hard for me. Therefore I have uploaded the Q4_0 by splitting it of 5GB per file. |
|
|
|
To join the files, do the following: |
|
|
|
Linux and macOS: |
|
|
|
``` |
|
cat qwen72b-chat-q4_0.gguf-split-* >qwen72b-chat-q4_0.gguf && rm qwen72b-chat-q4_0.gguf-split-* |
|
``` |
|
|
|
Windows: |
|
|
|
``` |
|
copy /B qwen72b-chat-q4_0.gguf-split-aa + qwen72b-chat-q4_0.gguf-split-ab + qwen72b-chat-q4_0.gguf-split-ac + qwen72b-chat-q4_0.gguf-split-ad + qwen72b-chat-q4_0.gguf-split-ae + qwen72b-chat-q4_0.gguf-split-af + qwen72b-chat-q4_0.gguf-split-ag + qwen72b-chat-q4_0.gguf-split-ah qwen72b-chat-q4_0.gguf |
|
``` |
|
|
