It doesn't work with the latest GPTQ-for-LLaMa
#4
by
BGLuck
- opened
Hi there, GPTQ-for-LLaMa was updated recently (to add group size) and now neither alpaca 13b-int4 or alpaca 30b-int 4 works
Similar issue to https://github.com/oobabooga/text-generation-webui/issues/584#issuecomment-1484178924
I will work on getting updated versions for 30B and 13B tomorrow.
Here is the 13b model. The 30b model is uploading. https://huggingface.co/elinas/alpaca-13b-lora-int4/blob/main/alpaca-13b-4bit-128g.safetensors
It's using group-size 128 as recommended by GPTQ for models larger than 7b.
Evals for 13b
c4-new
6.92567491531372
ptb-new
9.23875904083252
wikitext2
5.219980716705322
elinas
changed discussion status to
closed