iMatrix effect on other languages?
Hi,
could it be that an iMatrix quant severly hurts Miqu's German capabilities? As I understood you determine quantization parameters using text samples from wikipedia? Maybe English text only?
Tried miqu-1-70b-Requant-b2431-iMat-c512_ch600-IQ3_S.gguf and miqu-1-70b-Requant-b2035-iMat-c32_ch400-Q4_K_S.gguf
and both subjectively fall far behind the "original" miqu-1-70b.q5_K_M.gguf.
Sorry if this might be a noob question, I am pretty new to LLMs and quantizations in general and was just playing around with them. Wonder what resources I'd need to try making my own quants.
The higher quantization loss might explain it.
Also, I didn't notice massive regression on French due to English iMatrix. Maybe because French is "the second language" of Mistral Quantized..
You could help (a bit) German language with a iMatrix made from a German wiki training dataset.
Cut 10-12mo of it, make it wiki.train.de.raw, and use that file to make an iMatrix.