Update README.md
Browse files
README.md
CHANGED
|
@@ -2,7 +2,52 @@
|
|
| 2 |
license: apache-2.0
|
| 3 |
---
|
| 4 |
# K2-Chat: a fully-reproducible large language model outperforming Llama 2 70B Chat using 35% less compute
|
| 5 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 6 |
K2 Chat is finetuned from [K2-65B](https://huggingface.co/LLM360/K2). K2 Chat outperforms Llama 2-70B-Chat on all evaluations conducted. The model also outperforms Llama 3-70B-Instruct on coding tasks.
|
| 7 |
|
| 8 |
<center><img src="k2_chat_eval_table.png" alt="k2 eval table" /></center>
|
|
|
|
| 2 |
license: apache-2.0
|
| 3 |
---
|
| 4 |
# K2-Chat: a fully-reproducible large language model outperforming Llama 2 70B Chat using 35% less compute
|
| 5 |
+
K2 Chat is finetuned from [K2-65B](https://huggingface.co/LLM360/K2). The most recent model update 10/31/24.
|
| 6 |
+
|
| 7 |
+
In this release, we introduce function calling features and target improvements across math, coding, and safety.
|
| 8 |
+
|
| 9 |
+
We utilized the following datasets:
|
| 10 |
+
|
| 11 |
+
[Infinity-Instruct](https://huggingface.co/datasets/BAAI/Infinity-Instruct)
|
| 12 |
+
[JiuZhang3.0-Corpus-SFT](https://huggingface.co/datasets/ToheartZhang/JiuZhang3.0-Corpus-SFT)
|
| 13 |
+
[glaive-function-calling-v2-sharegpt](https://huggingface.co/datasets/hiyouga/glaive-function-calling-v2-sharegpt)
|
| 14 |
+
|
| 15 |
+
## Results
|
| 16 |
+
|
| 17 |
+
| | K2-Chat-060124 | K2-Chat |
|
| 18 |
+
|-------------------------|---------|----------|
|
| 19 |
+
| **Natural Language Benchmarks** | | |
|
| 20 |
+
| MMLU (0-shot) | 63.5 | 69.14 |
|
| 21 |
+
| RACE (0-shot) | 46.1 | 46.60 |
|
| 22 |
+
| HellaSwag (10-shot) | 81.7 | 80.80 |
|
| 23 |
+
| PIQA (5-shot) | 82.3 | 81.34 |
|
| 24 |
+
| ARC-easy (5-shot) | 84.6 | 79.00 |
|
| 25 |
+
| ARC-challenge (25-shot) | 61.3 | 61.09 |
|
| 26 |
+
| OpenBookQA (5-shot) | 48.0 | 47.00 |
|
| 27 |
+
| Winogrande (5-shot) | 79.5 | 78.30 |
|
| 28 |
+
| TruthfulQA (0-shot) | 44.7 | 57.32 |
|
| 29 |
+
| CrowS-Pairs (0-shot) | 64.2 | 65.32 |
|
| 30 |
+
| GSM8K (5-shot) | 60.7 | 77.10 |
|
| 31 |
+
| MathQA (5-shot) | 44.8 | 43.12 |
|
| 32 |
+
| LogiQA2.0 (0-shot) | 38.0 | 36.83 |
|
| 33 |
+
| BBH CoT (0-shot) | 64.9 | 70.37 |
|
| 34 |
+
| **Code Benchmarks** | | |
|
| 35 |
+
| HumanEval (pass@1) | 47.9 | 71.20 |
|
| 36 |
+
| **Domain Specific (Medical)** | | |
|
| 37 |
+
| MedQA (0-shot) | 53.6 | 52.87 |
|
| 38 |
+
| MedMCQA (5-shot) | 51.3 | 50.71 |
|
| 39 |
+
| PubMedQA (0-shot) | 75.0 | 71.20 |
|
| 40 |
+
| **Other** | | |
|
| 41 |
+
| MT-Bench | 6.87 | 7.55 |
|
| 42 |
+
| JSON-Mode-Eval | 77.21 | 90.09 |
|
| 43 |
+
| **Overall Average Score**| | |
|
| 44 |
+
| Avg Score | 58.88 | 61.30 |
|
| 45 |
+
|
| 46 |
+
|
| 47 |
+
|
| 48 |
+
|
| 49 |
+
|
| 50 |
+
## K2-Chat-060124
|
| 51 |
K2 Chat is finetuned from [K2-65B](https://huggingface.co/LLM360/K2). K2 Chat outperforms Llama 2-70B-Chat on all evaluations conducted. The model also outperforms Llama 3-70B-Instruct on coding tasks.
|
| 52 |
|
| 53 |
<center><img src="k2_chat_eval_table.png" alt="k2 eval table" /></center>
|