Plz quants for Firefly
Please could you add https://huggingface.co/invisietch/MiS-Firefly-v0.1-22B to your queue?
It may need tokenizer.model from Mistral Small repo (it did for me locally).
Great, I hate hacks like that :(
It's added, and I will replace the file when needed.
Cheers!
I assume you mean https://huggingface.co/mistralai/Mistral-Small-Instruct-2409 - I don't have access to that repo at the moment, any alternative?
Do you have a Discord? I can send the file. Mine is the same as username here.
gguf-my-repo seems to quant it without the file but my local llama.cpp wouldn't.
Ah, ok, I'll try without any hacks first.
If you give the model a try, let me know what you think. I get 70B vibes from it especially in creative writing.
Worked fine, should be done... very soon.
Already added to model card!
Too fast :) But seriously, you should link the imatrix quants instead once finished.
Too fast :) But seriously, you should link the imatrix quants instead once finished.
I linked both ;)
Hi, apologies. The model worked well at Q8 but there was a bug with quantized models (not your fault, happened when I tried making Q6 locally too).
It's fixed in https://huggingface.co/invisietch/MiS-Firefly-v0.2-22B -- would you mind doing GGUF for this when its uploaded in a couple of hours?
Sure, just notify me when it's fully uploaded :)
It's up, and thanks!
Aaand its queued :)