File size: 3,424 Bytes
510cb83 3b54e17 818df15 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 |
---
license: apache-2.0
---
# GreenBit LLMs
This is GreenBitAI's pretrained **low-bit** LLMs with extreme compression yet still strong performance.
Please refer to our [Github page](https://github.com/GreenBitAI/green-bit-llm) for the code to run the model and more information.
### zero-shot evaluation
| **Repository (Phi Family)** | **Avg Acc.** | **OpenBQ** | **ARC-E** | **Winogr.** | **HellaS.** | **ARC-C** | **PIQA** | **BoolQ** | **RACE** | **ANLI-R1** | **ANLI-R2** | **ANLI-R3** | **WiC** |
|:---------------------------------------------|:------------:|:-----------:|:-----------:|:-----------:|:-----------:|:-----------:|:-----------:|:---------:|:--------:|:-----------:|:-----------:|:-----------:|:-------:|
| `Phi-3-mini-128k-instruct-layer-mix-bpw-2.2` | 0.510 | 0.270 | 0.706 | 0.648 | 0.479 | 0.411 | 0.736 | 0.783 | 0.381 | 0.393 | 0.38 | 0.399 | 0.536 |
| `Phi-3-mini-128k-instruct-layer-mix-bpw-2.5` | 0.514 | 0.290 | 0.719 | 0.656 | 0.488 | 0.401 | 0.750 | 0.778 | 0.401 | 0.392 | 0.410 | 0.407 | 0.493 |
| `Phi-3-mini-128k-instruct-layer-mix-bpw-3.0` | 0.548 | 0.318 | 0.761 | 0.663 | 0.519 | 0.453 | 0.777 | 0.798 | 0.393 | 0.473 | 0.404 | 0.442 | 0.579 |
| `Phi-3-mini-128k-instruct-layer-mix-bpw-4.0` | 0.582 | 0.346 | 0.779 | 0.708 | 0.582 | 0.495 | 0.787 | 0.840 | 0.412 | 0.529 | 0.459 | 0.448 | 0.606 |
| `Phi-3-mini-128k-instruct ` | 0.586 | 0.342 | 0.785 | 0.731 | 0.596 | 0.512 | 0.782 | 0.851 | 0.401 | 0.547 | 0.464 | 0.432 | 0.594 |
### 5-shot evaluation
| **Repository (Phi Family)** | **Avg Acc.** | **OpenBQ** | **ARC-E** | **Winogr.** | **HellaS.** | **ARC-C** | **PIQA** | **BoolQ** | **RACE** | **ANLI-R1** | **ANLI-R2** | **ANLI-R3** | **WiC** |
|:---------------------------------------------|:------------:|:-----------:|:-----------:|:-----------:|:-----------:|:-----------:|:-----------:|:---------:|:--------:|:-----------:|:-----------:|:-----------:|:-------:|
| `Phi-3-mini-128k-instruct-layer-mix-bpw-2.2` | 0.534 | 0.302 | 0.738 | 0.659 | 0.487/0.636 | 0.438 | 0.744 | 0.793 | 0.408 | 0.421 | 0.404 | 0.439 | 0.583 |
| `Phi-3-mini-128k-instruct-layer-mix-bpw-2.5` | 0.543 | 0.310 | 0.771 | 0.671 | 0.501/0.657 | 0.441 | 0.763 | 0.799 | 0.405 | 0.453 | 0.427 | 0.443 | 0.534 |
| `Phi-3-mini-128k-instruct-layer-mix-bpw-3.0` | 0.563 | 0.346 | 0.796 | 0.687 | 0.528/0.694 | 0.500 | 0.782 | 0.809 | 0.410 | 0.473 | 0.394 | 0.474 | 0.565 |
| `Phi-3-mini-128k-instruct-layer-mix-bpw-4.0` | 0.602 | 0.374 | 0.817 | 0.725 | 0.598/0.768 | 0.542 | 0.766 | 0.864 | 0.428 | 0.523 | 0.456 | 0.497 | 0.658 |
| `Phi-3-mini-128k-instruct ` | 0.608 | 0.408 | 0.825 | 0.725 | 0.608/0.781 | 0.534 | 0.768 | 0.866 | | 0.538 | 0.483 | 0.515 | 0.627 |
|