My GGUF-IQ-Imatrix quants for NeverSleep/Lumimaid-v0.2-8B.
I recommend checking their page for feedback and support.
Quantization process:
Imatrix data was generated from the FP16-GGUF and conversions directly from the BF16-GGUF.
This is a bit more disk and compute intensive but hopefully avoids any losses during conversion.
To run this model, please use the latest version of KoboldCpp.
If you noticed any issues let me know in the discussions.
Presets:
- Llama-3.
Some compatible SillyTavern presets can be found here (Virt's Roleplay Presets - v1.9).
Check discussions such as this one and this one for other presets and samplers recommendations.
Lower temperatures are recommended by the authors, so make sure to experiment.General usage with KoboldCpp:
For 8GB VRAM GPUs, I recommend the Q4_K_M-imat (4.89 BPW) quant for up to 12288 context sizes without the use of--quantkv.
Using--quantkv 1(≈Q8) or even--quantkv 2(≈Q4) can get you to 32K context sizes with the caveat of not being compatible with Context Shifting, only relevant if you can manage to fill up that much context.
Read more about it in the release here.
Recommended read:
"Which GGUF is right for me? (Opinionated)" by Artefact2
Click the image to view full size.
Personal-support:
I apologize for disrupting your experience.
Eventually I may be able to use a dedicated server for this, but for now hopefully these quants are helpful.
If you want and you are able to...
You can spare some change over here (Ko-fi).Author-support:
You can support the authors at their pages/here.
Wandb: https://wandb.ai/undis95/Lumi-Llama-3-1-8B?nw=nwuserundis95
Lumimaid 0.1 -> 0.2 is a HUGE step up dataset wise.
As some people have told us our models are sloppy, Ikari decided to say fuck it and literally nuke all chats out with most slop.
Our dataset stayed the same since day one, we added data over time, cleaned them, and repeat. After not releasing model for a while because we were never satisfied, we think it's time to come back!
<|begin_of_text|><|start_header_id|>system<|end_header_id|>
{system_prompt}<|eot_id|><|start_header_id|>user<|end_header_id|>
{input}<|eot_id|><|start_header_id|>assistant<|end_header_id|>
{output}<|eot_id|>
We sadly didn't find the sources of the following, DM us if you recognize your set !
Undi: If you want to support us, you can here.
IkariDev: Visit my retro/neocities style website please kek
Base model
NeverSleep/Lumimaid-v0.2-8B