allura-org/MoE-Girl-1BA-7BT
hi! i made a thing :3
it's based on olmoe which was only merged into lcpp master a few weeks ago, i'm not sure how new your copy is so it might not work
https://huggingface.co/allura-org/MoE-Girl-1BA-7BT
usually not older than a few weeks, but it's always good to mention this, as I indeed don't update automatically. anyways, let's quant your thing :) you can watch its progress at http://hf.tst.eu/status.html, but it won't be more than an hour, likely.
unfortunately, it fails even with current llama.cpp:
ValueError: Duplicated key name 'olmoe.attention.layer_norm_rms_epsilon'
I have looked into it. It might be a regression in newer llama.cpp (basically, llama.cpp didn't check for duplicate keys until recently), and I have applied a workaround. Please take the result with a grain of salt.
hmm. weird that it randomly broke...
thanks, though!
That change broke lots and lots of models. It's one of those changes where somebody changes something vital, and then does not care that it breaks lots of lesser-used models.