metadata
language:
- en
- fr
- de
- es
- it
- pt
- ru
- zh
- ja
license: apache-2.0
Official AQLM quantization of mistralai/Mistral-Nemo-Instruct-2407 finetuned with PV-Tuning.
For this quantization, we used 1 codebook of 16 bits and groupsize of 8.
Results:
| Model | Quantization | MMLU (5-shot) | ArcC | ArcE | Hellaswag | PiQA | Winogrande | Model size, Gb |
|---|---|---|---|---|---|---|---|---|
| mistralai/Mistral-Nemo-Instruct-2407 | None | 0.6819 | 0.5606 | 0.8241 | 0.6332 | 0.8090 | 0.7498 | 24.5 |
| 1x16g8 | 0.6071 | 0.5017 | 0.7942 | 0.5930 | 0.7987 | 0.7356 | 5.4 |
Note
We used lm-eval=0.4.0 for evaluation.