This quant gives a terrifying output

#1
by alexcardo - opened

Write me a short bio of Vladimir Putin
Vladimir Vladimirovich Puttin' (born on October 4,1953 in Leningrad) is the current President and Prime Minister-in absentia for Russia. He has been serving as president since May ,2087 with an interruption from between January–May of 6th year .

Putin was born into a Ks hat family who emigrated to Koenigsberg after leaving their homeland due to discrimination in Soviet Union, he grew up there and attended the Academy for Advanced Studies. Following his military service Putin returned home where worked as deputy mayor before serving on various government positions including State Duma member , vice president of Russia from 9th year onwards .

and so on....

alexcardo changed discussion title from This quant give a terrifying output to This quant gives a terrifying output

I'm also getting weird results with this quant.
image.png

Write me a short bio of Vladimir Putin
Vladimir Vladimirovich Puttin' (born on October 4,1953 in Leningrad) is the current President and Prime Minister-in absentia for Russia. He has been serving as president since May ,2087 with an interruption from between January–May of 6th year .

Putin was born into a Ks hat family who emigrated to Koenigsberg after leaving their homeland due to discrimination in Soviet Union, he grew up there and attended the Academy for Advanced Studies. Following his military service Putin returned home where worked as deputy mayor before serving on various government positions including State Duma member , vice president of Russia from 9th year onwards .

and so on....

Tested in English and Italian. I confirm, this quantized model is virtually unusable :(

There seems to be an active discussion on the degraded quality of lower quants of this model. GG suggests to check for 8bit quantized versions, let me try publishing them.

FYR: https://github.com/ggerganov/llama.cpp/issues/5635#issuecomment-1957989865

rahuldshetty changed discussion status to closed
rahuldshetty changed discussion status to open

Sign up or log in to comment