Can't load in ollama

by joushe - opened Aug 26

Aug 26

[root@ubuntu]$ ollama run magnum-v2:12b-fp16
Error: exception error loading model hyperparameters: iinvalid n_rot: 160, expected 128

Can someone explain why this is happening?
It is the first time models do not run on my ollama v0.3.5
The gguf quantized versions of these two models can run properly.

lucyknada

Anthracite org Aug 26

We can't provide support for third party engines, but I'd try updating to v0.3.6 in hopes they merged / fixed nemo inferencing, if that still doesn't work I'd open an issue on their github: https://github.com/ollama/ollama/issues hope that helps

lucyknada changed discussion status to closed Aug 26

joushe

Aug 26

•

edited Aug 26

Do you have any recommended engines that can run on Ubuntu?
The models your team has developed are really impressive.

lucyknada

Anthracite org Aug 26

•

edited Aug 26

tabbyAPI works great with it (exl2), but you could try koboldcpp too (gguf) or if you want to load fp16 you can give vllm or aphrodite (a little more experimental) a go. (though afaik tabby can load fp16 too)

personally I only run it in tabby so I can not vouch for any of the other options.

joushe

Aug 26

I'll try loading fp16 with tabby, as I've been wanting to experience it. I’ve tried other quantized models and they’ve been great. Thank you for your efforts and sharing!

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment