| # Initial Testing 2024-04-25 | |
| Some folks are claiming there's something funky going on with GGUF quanting for Llama 3 models. I don't disagree. | |
| Some of those people are speculating that it has something to do with converting the raw weights from bf16 to fp16 instead | |
| of converting to fp32 as an intermediate step. I think that's bollocks. There is no logical or mathmatical justification for | |
| how that could possibly matter. | |
| So to test this crazy theory, I downloaded Undi95/Meta-Llama-3-8B-Instruct-hf and converted it to GGUF three ways: | |
| - fp16 specifically with `--outtype f16` | |
| - fp32 specifically with `--outtype f32` | |
| - "Auto" with no outtype specified | |
| I then quantized each of these conversions to Q4_K_M and ran perplexity tests on everything using my abbreviated wiki.short.raw | |
| text file. The results: | |
| ```` | |
| FP16 specified: size 14.9GB PPL @ fp16 9.5158 +/- 0.15418 PPL @ Q4km 9.6414 +/- 0.15494 | |
| FP32 specified: size 29.9GB PPL @ fp32 9.5158 +/- 0.15418 PPL @ Q4km 9.6278 +/- 0.15466 | |
| None specified: size 29.9GB PPL @ ???? 9.5158 +/- 0.15418 PPL @ Q4km 9.6278 +/- 0.15466 | |
| ```` | |
| As you can see, converting to fp32 has no meaningful effect on PPL compared to converting to fp16. PPL is identical at full weight, | |
| and the miniscule loss shown at Q4km is will within the margin of error. There will no doubt be some people who will claim | |
| "PpL iSn'T gOoD eNoUgH!!1!". For those people, I have uploaded all GGUFs used in this test. Feel free to use those files to do | |
| more extensive testing on your own time. I consider the matter resolved until somebody can conclusively demonstrate otherwise. | |
| # Continued Experiments 2024-05-11 | |
| As an imatrix enjoyer, it has been bugging me whether the precision of the quant used to generate the imatrix actually | |
| matters. Scuttlebut says "yes, but only a little". Logically, I don't think it should matter to a meaningful extent. PPL | |
| scales, so a value that is relatively important at fp16 should also register as relatively important at Q8 or even Q4. | |
| To test this theory properly, I took failspy/Llama-3-8B-Instruct-abliterated and converted it to GGUF in both fp16 and fp32 | |
| formats. I then quantized each of those GGUFs to both Q8_0 and Q4_0. I then generated imatrices for each of those six | |
| GGUFs. Then I created eight GGUFs quantized at Q4_k_m: | |
| - fp32 GGUF, fp32 imatrix | |
| - fp16 GGUF, fp16 imatrix | |
| - fp32 GGUF, fp32->Q8 imatrix | |
| - fp16 GGUF, fp16->Q8 imatrix | |
| - fp32 GGUF, fp32->Q4 imatrix | |
| - fp16 GGUF, fp16->Q4 imatrix | |
| - fp32 GGUF, no imatrix | |
| - fp16 GGUF, no imatrix | |
| I ran PPL against all 8 quants, as well as the full fp16 and fp32 GGUFs. All iMatrices were created using Kalomaze's | |
| groups_merged.txt. All PPL calcs were run using wiki.short.raw. Results: | |
| ```` | |
| <in progress> | |
| ```` | |
| Conclusion: | |
| ```` | |
| <in progress> | |
| ```` | |