oobabooga starting without errors, but output is gibberish

#29
by markenzwieback - opened

Hello y'all,

being new to the whole topic i thought to give it a go with this model, but encountered an error which i am unable to solve.
Using oobabooga i am able to run the provided default models without issues and with appropriate responses.

This model also boots up without any (shown) issues in the console:
oobabooga_output_example_2.jpg
but the output is complete garbage and looks like this:

oobabooga_output_example_1.jpg

I appended the following parameters to the server.py in start-webui.bat for this specific model, the default models obviously won't start with these settings:
server.py --chat --auto-devices --wbits 4 --groupsize 128 --verbose

Not using wbits and groupsize results in errors starting this model, which is to be expected.
I did notice that the VRAM usage instantly drops from 4G to around 350M after entering any prompt.

Can anyone point me in a direction of what this may be caused by?
Thanks in advance!

System specs:
i7 12700KF
RTX 3080 (10G)
32GB RAM
CUDA 11.7

I was having the same issue as you. I was able to get it working by forcing it to use the cuda model.

Your screen shot it's loading the non cuda model pt file.

image.png

I was able to fix it by renaming the pt file without -cuda at the end to .pt.bkup so it wouldn't pick it up and instead load the correct model file. while keeping the other model just in case.

image.png

image.png

hope that helps!

Thank you very much for pointing that out, it now works like a charm! Time to explore the possibilities.

markenzwieback changed discussion status to closed

Sign up or log in to comment