Recommended settings?
#1
by
sdalemorrey
- opened
Hello!
First off, thanks so much for this quant it's everything I've ever wanted in a frankenmerge.
Do you have any recommended settings for running this though? I tried it with Layla https://discord.com/channels/1127838352016678932/1198916915997978724 and try as I might I could not get it produce anything resembling coherent text.
I tried every temp and top p that would normally work and nothing seems to help. I also tried all the various chat templates all to no avail.
I would love to have this working.
Incidentally, can you make q4 quant? I have the resources to run this at q5 but my understanding is q4 produces almost the same quality and I've had a lot better luck with them.
Thanks again!