Some folks are claiming there's something funky going on with GGUF quanting for Llama 3 models.  I don't disagree.

Some of those people are speculating that it has something to do with converting the raw weights from bf16 to fp16 instead
of converting to fp32 as an intermediate step.  I think that's bollocks.  There is no logical or mathmatical justification for
how that could possibly matter.

So to test this crazy theory, I downloaded Undi95/Meta-Llama-3-8B-Instruct-hf and converted it to GGUF three ways:
- fp16 specifically with `--outtype f16`
- fp32 specifically with `--outtype f32`
- "Auto" with no outtype specified

I then quantized each of these conversions to Q4_K_M and ran perplexity tests on everything using my abbreviated wiki.short.raw

The results:




As you can see, converting to fp32 has no meaningful effect on PPL.  There will no doubt be some people who will claim 
"PpL iSn'T gOoD eNoUgH!!1!".  For those people, I have uploaded all GGUFs used in this test.  Feel free to do more extensive
testing on your own time.  I consider the matter resolved until somebody can conclusively demonstrate otherwise.