Create README.md

README.md

@@ -0,0 +1,14 @@
+Official quantization of [Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1) using [PV-Tuning](https://arxiv.org/abs/2405.14852) on top of [AQLM](https://arxiv.org/abs/2401.06118).
+
+For this quantization, we used 1 codebook of 16 bits for groups of 8 weights.
+
+Results (0-shot `acc`): 
+| Model      | Quantization | WinoGrande | PiQA | HellaSwag | ArcE | ArcC | Model size, Gb |
+|------|------|------|-------|-------|-------|------|------|
+|Mistral-7B-v0.1| None | 0.7364	| 0.8047 | 0.6115 | 0.7887 | 0.4923 | 14.5 |
+|  |1x16 (this) | 0.7151 | 0.7976 | 0.5880 | 0.7698 | 0.4514 | 2.51 |
+
+The 1x16g16 (1-bit) models are on the way, as soon as we update the inference lib with their respective kernels.
+
+To learn more about the inference, as well as the information on how to quantize models yourself, please refer to the [official GitHub repo](https://github.com/Vahe1994/AQLM).
+The original code for PV-Tuning can be found in the [AQLM@pv-tuning](https://github.com/Vahe1994/AQLM/tree/pv-tuning) branch.