Conversion of MoE version of 1.6
#2
by
Jack-771a
- opened
https://huggingface.co/LanguageBind/MoE-LLaVA-Phi2-2.7B-4e
Can you please convert this model as well into .gguf file format?
Or tell the way how to do this. All (2) scripts I found doesn't work and can't convert models into .GGUF
have you followed the convert and quantize steps similar to what's in this PR? https://github.com/ggerganov/llama.cpp/pull/4406
this should work for MoE
@Jack-771a
llama cpp doesnt have support for that model yet
the steps that cjpais gave is for normal moe but that one is llava moe.
so you either ask for support in llama cpp or try to create it by yourself