Commit
•
7af1936
1
Parent(s):
8fd5614
Update README.md
Browse files
README.md
CHANGED
@@ -36,6 +36,7 @@ This official repository unveils the TransNormerLLM3 model along with its open-s
|
|
36 |
> --23.12.25-- startup: [WeChat - 预训练启航](https://mp.weixin.qq.com/s/YjUY-uy89WkF75_-rBTuKw) <<<>>> [Twitter - Pre-training Commences ](https://twitter.com/opennlplab/status/1739568669502611825) <<<>>> [YouTube Recording](https://t.co/wk7svS4o5r) <<<>>> [bilibili 回放](https://www.bilibili.com/video/BV11j411J7Dy)
|
37 |
> --24.01.02-- first week review: [WeChat - 第一周概览](https://mp.weixin.qq.com/s/zwGnZZI3itNPoxzzXkuU2w) <<<>>> [Twitter - First Week Review](https://twitter.com/opennlplab/status/1742187694078501038)
|
38 |
> --24.01.09-- second week review: [WeChat - 第二周概览](https://mp.weixin.qq.com/s/6D0qi-0aBier05OKuHfPEA) <<<>>> [Twitter - Second Week Review](https://twitter.com/opennlplab/status/1744720007299523063)
|
|
|
39 |
|
40 |
|
41 |
# Released Weights
|
@@ -44,6 +45,8 @@ This official repository unveils the TransNormerLLM3 model along with its open-s
|
|
44 |
| :-----: | :---: | :--------------------------------------------------------------------------------------------------------------------: | :---------: | :-------: |
|
45 |
| **15B** | 50B | 🤗[step13000](https://huggingface.co/OpenNLPLab/TransNormerLLM3-15B-Intermediate-Checkpoints/tree/step13000-50Btokens) | 🤖 | 🐯 |
|
46 |
| **15B** | 100B | 🤗[step26000](https://huggingface.co/OpenNLPLab/TransNormerLLM3-15B-Intermediate-Checkpoints/tree/step26000-100Btokens) | 🤖 | 🐯 |
|
|
|
|
|
47 |
|
48 |
|
49 |
```python
|
@@ -56,16 +59,23 @@ model = AutoModelForCausalLM.from_pretrained("OpenNLPLab/TransNormerLLM3-15B-Int
|
|
56 |
# Benchmark Results
|
57 |
The evaluations of all models are conducted using the official settings and the [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) framework.
|
58 |
|
59 |
-
| Model | P | T | BoolQ | PIQA | HS | WG | ARC-e | ARC-c | OBQA |
|
60 |
-
| ----------------------- | --- | ---- | ----- | ----- | ----- | ----- | ----- | ----- | ----- |
|
61 |
-
| **TransNormerLLM3-15B** | 15 | 0.05 | 62.08 | 72.52 | 55.55 | 57.14 | 62.12 | 31.14 | 32.40 | 27.50 |
|
62 |
-
| **TransNormerLLM3-15B** | 15 | 0.10 | 63.98 | 74.70 | 61.09 | 61.33 | 65.95 | 34.64 | 35.60 | 25.38
|
63 |
-
| **TransNormerLLM3-15B** | 15 | 0.15 | 60.34 | 75.08 | 63.99 | 62.04 | 64.56 | 34.90 | 35.20 | 22.64
|
64 |
-
|
65 |
-
|
|
|
66 |
|
67 |
> **P**: parameter size (billion). **T**: tokens (trillion). **BoolQ**: acc. **PIQA**: acc. **HellaSwag**: acc_norm. **WinoGrande**: acc. **ARC-easy**: acc. **ARC-challenge**: acc_norm. **OpenBookQA**: acc_norm. **MMLU**: 5-shot acc. **C-Eval**: 5-shot acc.
|
68 |
|
|
|
|
|
|
|
|
|
|
|
|
|
69 |
|
70 |
# Acknowledgments and Citation
|
71 |
|
|
|
36 |
> --23.12.25-- startup: [WeChat - 预训练启航](https://mp.weixin.qq.com/s/YjUY-uy89WkF75_-rBTuKw) <<<>>> [Twitter - Pre-training Commences ](https://twitter.com/opennlplab/status/1739568669502611825) <<<>>> [YouTube Recording](https://t.co/wk7svS4o5r) <<<>>> [bilibili 回放](https://www.bilibili.com/video/BV11j411J7Dy)
|
37 |
> --24.01.02-- first week review: [WeChat - 第一周概览](https://mp.weixin.qq.com/s/zwGnZZI3itNPoxzzXkuU2w) <<<>>> [Twitter - First Week Review](https://twitter.com/opennlplab/status/1742187694078501038)
|
38 |
> --24.01.09-- second week review: [WeChat - 第二周概览](https://mp.weixin.qq.com/s/6D0qi-0aBier05OKuHfPEA) <<<>>> [Twitter - Second Week Review](https://twitter.com/opennlplab/status/1744720007299523063)
|
39 |
+
> --24.01.15-- third week review: [WeChat - 第三周概览](https://mp.weixin.qq.com/s/EQg8evZ2cNtAk4HruwCXPA) <<<>>> [Twitter - Third Week Review](https://twitter.com/opennlplab/status/1746920293069910190)
|
40 |
|
41 |
|
42 |
# Released Weights
|
|
|
45 |
| :-----: | :---: | :--------------------------------------------------------------------------------------------------------------------: | :---------: | :-------: |
|
46 |
| **15B** | 50B | 🤗[step13000](https://huggingface.co/OpenNLPLab/TransNormerLLM3-15B-Intermediate-Checkpoints/tree/step13000-50Btokens) | 🤖 | 🐯 |
|
47 |
| **15B** | 100B | 🤗[step26000](https://huggingface.co/OpenNLPLab/TransNormerLLM3-15B-Intermediate-Checkpoints/tree/step26000-100Btokens) | 🤖 | 🐯 |
|
48 |
+
| **15B** | 150B | 🤗[step39000](https://huggingface.co/OpenNLPLab/TransNormerLLM3-15B-Intermediate-Checkpoints/tree/step39000-150Btokens) | 🤖 | 🐯 |
|
49 |
+
| **15B** | 200B | 🤗[step52000](https://huggingface.co/OpenNLPLab/TransNormerLLM3-15B-Intermediate-Checkpoints/tree/step52000-200Btokens) | 🤖 | 🐯 |
|
50 |
|
51 |
|
52 |
```python
|
|
|
59 |
# Benchmark Results
|
60 |
The evaluations of all models are conducted using the official settings and the [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) framework.
|
61 |
|
62 |
+
| Model | P | T | BoolQ | PIQA | HS | WG | ARC-e | ARC-c | OBQA | C-Eval | MMLU |
|
63 |
+
| ----------------------- | --- | ---- | ----- | ----- | ----- | ----- | ----- | ----- | ----- | ------ | ----- |
|
64 |
+
| **TransNormerLLM3-15B** | 15 | 0.05 | 62.08 | 72.52 | 55.55 | 57.14 | 62.12 | 31.14 | 32.40 | 26.18 | 27.50 |
|
65 |
+
| **TransNormerLLM3-15B** | 15 | 0.10 | 63.98 | 74.70 | 61.09 | 61.33 | 65.95 | 34.64 | 35.60 | 25.38 | 27.40 |
|
66 |
+
| **TransNormerLLM3-15B** | 15 | 0.15 | 60.34 | 75.08 | 63.99 | 62.04 | 64.56 | 34.90 | 35.20 | 22.64 | 26.60 |
|
67 |
+
| **TransNormerLLM3-15B** | 15 | 0.20 | 52.05 | 74.48 | 64.72 | 62.75 | 66.16 | 35.15 | 36.80 | 27.25 | 30.80 |
|
68 |
+
| **TransNormerLLM3-15B** | 15 | 0.25 | 66.70 | 76.50 | 66.51 | 64.80 | 66.84 | 36.18 | 39.40 | 30.87 | 36.10 |
|
69 |
+
| **TransNormerLLM3-15B** | 15 | 0.30 | 67.00 | 76.50 | 67.17 | 64.40 | 66.29 | 36.77 | 38.80 | 33.99 | 37.60 |
|
70 |
|
71 |
> **P**: parameter size (billion). **T**: tokens (trillion). **BoolQ**: acc. **PIQA**: acc. **HellaSwag**: acc_norm. **WinoGrande**: acc. **ARC-easy**: acc. **ARC-challenge**: acc_norm. **OpenBookQA**: acc_norm. **MMLU**: 5-shot acc. **C-Eval**: 5-shot acc.
|
72 |
|
73 |
+
```bash
|
74 |
+
# Please configure the following settings when do evaluation
|
75 |
+
export do_eval=True
|
76 |
+
export use_triton=False
|
77 |
+
```
|
78 |
+
|
79 |
|
80 |
# Acknowledgments and Citation
|
81 |
|