152334H

/

miqu-1-70b-sf

like 218

1. build llama.cpp, run ./quantize --pure --allow-requantize ./miqu-1-70b.q5_K_M.gguf 1
2. apply a.patch, run python3 convert.py --dump ./miqu-1-70b.q5_K_M.gguf
3. run https://gist.github.com/152334H/27d4181ce3641cec335131b971584ddd

behaviour is hardcoded for llama2-70b shape so be weary of that much

also, the result of d = {v[1]:k for k,v in tm.get_tensor_name_map(tm.MODEL_ARCH.LLAMA, LAYERS).mapping.items()} is an unstable hack that will change in future llama.cpp versions

Interesting. Now I'm praying that I can train/merge qlora and quant to exl2 without issues lol.