Update README.md
Browse files
README.md
CHANGED
|
@@ -52,22 +52,6 @@ and is comparable with Mistral-7B-Instruct-v0.1 on MMLU and MT-Bench in English.
|
|
| 52 |
| Mistral-7B-v0.1 | 33.01 | 42.23 | 35.86 | 37.63 |
|
| 53 |
|
| 54 |
|
| 55 |
-
## Inference Performance
|
| 56 |
-
In this test, we use the first 1500 characters of one of 201802最高法院民事裁判書 as input and ask the model to rewrite the article.
|
| 57 |
-
The max_new_tokens is set to 1000 (except Qwen/Qwen-7B, which is set to 400). All models were inferenced with `vllm` on 2 A6000 (TP=2 ).
|
| 58 |
-
|
| 59 |
-
| Models | Speed (char/sec) |Estimated Max Input Length (TC Char)|
|
| 60 |
-
|--------------------------------------------------------------------|-------------------|--------------------------|
|
| 61 |
-
| Yi-6B | 62.08 | 4.4k |
|
| 62 |
-
| **Breeze-7B-Base-v0.1** | 59.57 | 10.1k |
|
| 63 |
-
| Qwen-7B | 55.00 | 9.7k |
|
| 64 |
-
| Qwen-14B | 51.12 | 9.7k |
|
| 65 |
-
| Mistral-7B-v0.1 | 45.31 | 6.3k |
|
| 66 |
-
| Taiwan-LLM-13B-v2.0-base | 19.61 | 2.6k |
|
| 67 |
-
| Taiwan-LLM-7B-v2.1-base | 16.23 | 2.6k |
|
| 68 |
-
| Yi-34B | 15.18 | 4.4k |
|
| 69 |
-
|
| 70 |
-
|
| 71 |
## Chat Model Performance
|
| 72 |
|
| 73 |
| Models | | TMMLU+ (ACC) | TMMLU+ (ACC) | DRCD (EM) | MT-Bench-tw (Score) | MMLU (ACC) | MMLU (ACC) | MT-Bench (Score) |
|
|
@@ -100,6 +84,23 @@ The max_new_tokens is set to 1000 (except Qwen/Qwen-7B, which is set to 400). Al
|
|
| 100 |
| Taiwan-LLM-7B-v2.1-chat | 25.58 | 31.76 | 27.36 | 27.61 |
|
| 101 |
|
| 102 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 103 |
## Examples
|
| 104 |
|
| 105 |
|
|
|
|
| 52 |
| Mistral-7B-v0.1 | 33.01 | 42.23 | 35.86 | 37.63 |
|
| 53 |
|
| 54 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 55 |
## Chat Model Performance
|
| 56 |
|
| 57 |
| Models | | TMMLU+ (ACC) | TMMLU+ (ACC) | DRCD (EM) | MT-Bench-tw (Score) | MMLU (ACC) | MMLU (ACC) | MT-Bench (Score) |
|
|
|
|
| 84 |
| Taiwan-LLM-7B-v2.1-chat | 25.58 | 31.76 | 27.36 | 27.61 |
|
| 85 |
|
| 86 |
|
| 87 |
+
## Inference Performance
|
| 88 |
+
In this test, we use the first 1500 characters of one of 201802最高法院民事裁判書 as input and ask the model to rewrite the article.
|
| 89 |
+
The max_new_tokens is set to 1000 (except Qwen/Qwen-7B, which is set to 400). All models were inferenced with `vllm` on 2 A6000 (TP=2 ).
|
| 90 |
+
|
| 91 |
+
| Models | Speed (char/sec) |Estimated Max Input Length (TC Char)|
|
| 92 |
+
|--------------------------------------------------------------------|-------------------|--------------------------|
|
| 93 |
+
| Yi-6B | 62.08 | 4.4k |
|
| 94 |
+
| **Breeze-7B-Base-v0.1** | 59.57 | 10.1k |
|
| 95 |
+
| **Breeze-7B-Instruct-64k-v0.1** | | |
|
| 96 |
+
| Qwen-7B | 55.00 | 9.7k |
|
| 97 |
+
| Qwen-14B | 51.12 | 9.7k |
|
| 98 |
+
| Mistral-7B-v0.1 | 45.31 | 6.3k |
|
| 99 |
+
| Taiwan-LLM-13B-v2.0-base | 19.61 | 2.6k |
|
| 100 |
+
| Taiwan-LLM-7B-v2.1-base | 16.23 | 2.6k |
|
| 101 |
+
| Yi-34B | 15.18 | 4.4k |
|
| 102 |
+
|
| 103 |
+
|
| 104 |
## Examples
|
| 105 |
|
| 106 |
|