Thank you for quantizing! :)
#1
by
Wladastic
- opened
Hello I feel honored to have you quantize my first model! :)
Does your script for any chance test longer context ranges?
When I use llama.cpp for quantizing the long context ability just breaks above 5000 tokens, but fp16 and bf16 are fine
No, we generally don't test the models, we just quantize them. If your quant breaks down after that, ours will likely, too (but of course you can try it out yourself).
mradermacher
changed discussion status to
closed