Thank you for quantizing! :)

#1
by Wladastic - opened

Hello I feel honored to have you quantize my first model! :)

Does your script for any chance test longer context ranges?
When I use llama.cpp for quantizing the long context ability just breaks above 5000 tokens, but fp16 and bf16 are fine

No, we generally don't test the models, we just quantize them. If your quant breaks down after that, ours will likely, too (but of course you can try it out yourself).

mradermacher changed discussion status to closed
Your need to confirm your account before you can post a new comment.

Sign up or log in to comment