How to load in text generation web ui?
I'm really keen to try this model, but I can't seem to get the GGUF version to load. I upgraded Llama.cpp to version 2.23 but i just get an assert error. Same issue trying to load through transformers. Are there any other components that need to be upgraded? Thanks...
Last Llama.cpp or Ooba or Kobold load it fine for me, are you sure you're on last version?
Hey, thanks for getting back to me, big fan of your models. I upgraded/recompiled Llama.cpp to version 2.23 which should be the most recent. Haven't tried do a full update on Ooba yet so I'll give that a try. It's 2am here so probably a tomorrow thing :). I'll let you know. Thanks very much...
Try reducing the n_ctx to below 10,000 or to 4096. The latest version of Ooba (Text gen webui) has this issue where it fails to load models if the value is too high.
It worked for me at 32k context loaded with Transformer on an A100 on runpod
A full Ooba update did the trick, so it must have been more than just the llama component. Only just started using it, but I love the way this model writes. I recently found and started using a min-P preset, and so far it has provided the best output I have seen from all of the models I've been playing with. Highly recommended if you haven't tried it out. Thanks for your help - and for the models. I check the new releases every day, always particularly hoping to see one of yours......
A full Ooba update did the trick, so it must have been more than just the llama component. Only just started using it, but I love the way this model writes. I recently found and started using a min-P preset, and so far it has provided the best output I have seen from all of the models I've been playing with. Highly recommended if you haven't tried it out. Thanks for your help - and for the models. I check the new releases every day, always particularly hoping to see one of yours......
Just min_p? Share if possible, ty.
Try the settings here:
"https://www.reddit.com/r/LocalLLaMA/comments/17vonjo/your_settings_are_probably_hurting_your_model_why/"
I found they give a nice writing quality boost to pretty much everything. Would be interested to hear if you get the same impression.