3bpw spam random words on default settings with oobabooga

#2
by Ainonake - opened

I've tried changing the system prompt/changing the instructions, but nothing helps. Model keeps spamming random words starting from the first message.

Screenshot_12.png
Screenshot_13.png
Screenshot_14.png
Screenshot_15.png

I just tested the 4.25bpw version and it seems to work fine, which torch or exllamav2 version are you using? I will redownload the 3bpw (had to delete it because storage) and test it again.

Here are some images

imagen.png

imagen.png

imagen.png

Wait, my bad, this is for the normal version (not RP). Which pytorch/exllamav2 are you using?

Rp version works normally. I was using latest pytorch&exllamav2 on a fresh Ubuntu vps install yesterday (unfortunately I deleted it already and it's pain to setup everything again, but you probably can check latest available versions).

I noticed a closed topic where was a similar problem and it was solved by removing the BOS token. Unfortunately, I was not able to check, because i already deleted this and installed the RP version.

But I noticed that if the BOS token is removed from the RP version, the model becomes noticeably dumber and does not follow the instructions.

I see, it seems to be that issue. Gonna add the info in the main page in the meanwhile. Not sure, maybe it is related to the wikitext dataset. Closing it for now!

Panchovix changed discussion status to closed

Sign up or log in to comment