Hugging Face's logo

NeverSleep
/
MiquMaid-v2-2x70B-DPO-GGUF

GGUF
Not-For-All-Audiences
nsfw
Model card Files Community
2

The impact of quantization on compliance…

#2
by Tokie - opened
Discussion
Tokie

…is huge.

Specifically, unlike the other quantizations I tested, MiquMaid-v2-2x70B-DPO.IQ3_XXS.gguf is pretty much completely unrestrained: In a comparison between 7 different GGUF quantizations of MiquMaid-v2-2x70B-DPO, the IQ3_XXS model is a clear outlier, not declining any of my test questions, and with hardly any moralizing (99 % compliance). The other 6 models, in contrast, refuse to help with most of the prompts (9–15 % compliance).

This extends to MiquMaid-v2-70B-DPO: Kooten/MiquMaid-v2-70B-DPO-Imatrix-GGUF/MiquMaid-v2-70B-DPO-IQ3_XXS.gguf scores 90 % compliance; NeverSleep/MiquMaid-v2-70B-DPO-GGUF/MiquMaid-v2-70B-DPO.q5_k_m.gguf only 9 %.

And to MiquMaid-v2-70B: Nexesenex/NeverSleep_MiquMaid-v2-70B-iMat.GGUF/NeverSleep_MiquMaid-v2-70B-b2093-iMat-c32_ch1000-IQ3_XXS.gguf scores 88 % compliance; NeverSleep/MiquMaid-v2-70B-GGUF/MiquMaid-v2-70B.q5_k_m.gguf only 11 %.

But not to Miqu-70B-DPO: Both Nexesenex/Undi95_Miqu-70B-Alpaca-DPO-iMat.GGUF/Undi95_Miqu-70B-Alpaca-DPO-b2101-iMat-c32_ch1000-IQ3_XXS.gguf and Undi95/Miqu-70B-Alpaca-DPO-GGUF/Miqu-70B-DPO.q5_k_m.gguf score 7 % compliance.

What is going on here? 🤔

Undi95
 NeverSleep org

No idea but this is very interesting to see, thank you for this!

Sign up or log in to comment