Update README.md
Browse files
README.md
CHANGED
@@ -20,12 +20,12 @@ language:
|
|
20 |
- **Intended Use Cases:** Intended for commercial and research use in multiple languages. Similarly to [Meta-Llama-3.1-8B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3.1-8B-Instruct), this models is intended for assistant-like chat.
|
21 |
- **Out-of-scope:** Use in any manner that violates applicable laws or regulations (including trade compliance laws). Use in languages other than English.
|
22 |
- **Release Date:** 7/24/2024
|
23 |
-
- **Version:** 1.
|
24 |
- **License(s):** [llama3.1](https://huggingface.co/meta-llama/Meta-Llama-3.1-8B/blob/main/LICENSE)
|
25 |
- **Model Developers:** Neural Magic
|
26 |
|
27 |
Quantized version of [Meta-Llama-3.1-405B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3.1-405B-Instruct).
|
28 |
-
It achieves an average score of 86.
|
29 |
|
30 |
### Model Optimizations
|
31 |
|
@@ -164,7 +164,7 @@ oneshot(
|
|
164 |
|
165 |
The model was evaluated on MMLU, ARC-Challenge, GSM-8K, Hellaswag, Winogrande and TruthfulQA.
|
166 |
Evaluation was conducted using the Neural Magic fork of [lm-evaluation-harness](https://github.com/neuralmagic/lm-evaluation-harness/tree/llama_3.1_instruct) (branch llama_3.1_instruct) and the [vLLM](https://docs.vllm.ai/en/stable/) engine.
|
167 |
-
This version of the lm-evaluation-harness includes versions of ARC-Challenge
|
168 |
|
169 |
### Accuracy
|
170 |
|
@@ -183,41 +183,41 @@ This version of the lm-evaluation-harness includes versions of ARC-Challenge and
|
|
183 |
<tr>
|
184 |
<td>MMLU (5-shot)
|
185 |
</td>
|
186 |
-
<td>
|
187 |
</td>
|
188 |
-
<td>
|
189 |
</td>
|
190 |
-
<td>99.
|
191 |
</td>
|
192 |
</tr>
|
193 |
<tr>
|
194 |
<td>ARC Challenge (0-shot)
|
195 |
</td>
|
196 |
-
<td>
|
197 |
</td>
|
198 |
-
<td>
|
199 |
</td>
|
200 |
-
<td>
|
201 |
</td>
|
202 |
</tr>
|
203 |
<tr>
|
204 |
<td>GSM-8K-cot (8-shot, strict-match)
|
205 |
</td>
|
206 |
-
<td>
|
207 |
</td>
|
208 |
-
<td>95.
|
209 |
</td>
|
210 |
-
<td>99.
|
211 |
</td>
|
212 |
</tr>
|
213 |
<tr>
|
214 |
<td>Hellaswag (10-shot)
|
215 |
</td>
|
216 |
-
<td>88.
|
217 |
</td>
|
218 |
-
<td>88.
|
219 |
</td>
|
220 |
-
<td>99.
|
221 |
</td>
|
222 |
</tr>
|
223 |
<tr>
|
@@ -225,29 +225,29 @@ This version of the lm-evaluation-harness includes versions of ARC-Challenge and
|
|
225 |
</td>
|
226 |
<td>87.21
|
227 |
</td>
|
228 |
-
<td>
|
229 |
</td>
|
230 |
-
<td>
|
231 |
</td>
|
232 |
</tr>
|
233 |
<tr>
|
234 |
<td>TruthfulQA (0-shot, mc2)
|
235 |
</td>
|
236 |
-
<td>
|
237 |
</td>
|
238 |
-
<td>64.
|
239 |
</td>
|
240 |
-
<td>
|
241 |
</td>
|
242 |
</tr>
|
243 |
<tr>
|
244 |
<td><strong>Average</strong>
|
245 |
</td>
|
246 |
-
<td><strong>86.
|
247 |
</td>
|
248 |
-
<td><strong>86.
|
249 |
</td>
|
250 |
-
<td><strong>99.
|
251 |
</td>
|
252 |
</tr>
|
253 |
</table>
|
@@ -261,8 +261,10 @@ The results were obtained using the following commands:
|
|
261 |
```
|
262 |
lm_eval \
|
263 |
--model vllm \
|
264 |
-
--model_args pretrained="neuralmagic/Meta-Llama-3.1-405B-Instruct-FP8",dtype=auto,add_bos_token=True,max_model_len=4096,tensor_parallel_size=8 \
|
265 |
-
--tasks
|
|
|
|
|
266 |
--num_fewshot 5 \
|
267 |
--batch_size auto
|
268 |
```
|
|
|
20 |
- **Intended Use Cases:** Intended for commercial and research use in multiple languages. Similarly to [Meta-Llama-3.1-8B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3.1-8B-Instruct), this models is intended for assistant-like chat.
|
21 |
- **Out-of-scope:** Use in any manner that violates applicable laws or regulations (including trade compliance laws). Use in languages other than English.
|
22 |
- **Release Date:** 7/24/2024
|
23 |
+
- **Version:** 1.1
|
24 |
- **License(s):** [llama3.1](https://huggingface.co/meta-llama/Meta-Llama-3.1-8B/blob/main/LICENSE)
|
25 |
- **Model Developers:** Neural Magic
|
26 |
|
27 |
Quantized version of [Meta-Llama-3.1-405B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3.1-405B-Instruct).
|
28 |
+
It achieves an average score of 86.39 on the [OpenLLM](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard) benchmark (version 1), whereas the unquantized model achieves 86.57.
|
29 |
|
30 |
### Model Optimizations
|
31 |
|
|
|
164 |
|
165 |
The model was evaluated on MMLU, ARC-Challenge, GSM-8K, Hellaswag, Winogrande and TruthfulQA.
|
166 |
Evaluation was conducted using the Neural Magic fork of [lm-evaluation-harness](https://github.com/neuralmagic/lm-evaluation-harness/tree/llama_3.1_instruct) (branch llama_3.1_instruct) and the [vLLM](https://docs.vllm.ai/en/stable/) engine.
|
167 |
+
This version of the lm-evaluation-harness includes versions of ARC-Challenge, GSM-8K, and MMLU that match the prompting style of [Meta-Llama-3.1-Instruct-evals](https://huggingface.co/datasets/meta-llama/Meta-Llama-3.1-8B-Instruct-evals).
|
168 |
|
169 |
### Accuracy
|
170 |
|
|
|
183 |
<tr>
|
184 |
<td>MMLU (5-shot)
|
185 |
</td>
|
186 |
+
<td>87.41
|
187 |
</td>
|
188 |
+
<td>87.05
|
189 |
</td>
|
190 |
+
<td>99.59%
|
191 |
</td>
|
192 |
</tr>
|
193 |
<tr>
|
194 |
<td>ARC Challenge (0-shot)
|
195 |
</td>
|
196 |
+
<td>94.97
|
197 |
</td>
|
198 |
+
<td>94.97
|
199 |
</td>
|
200 |
+
<td>100.0%
|
201 |
</td>
|
202 |
</tr>
|
203 |
<tr>
|
204 |
<td>GSM-8K-cot (8-shot, strict-match)
|
205 |
</td>
|
206 |
+
<td>95.98
|
207 |
</td>
|
208 |
+
<td>95.83
|
209 |
</td>
|
210 |
+
<td>99.84%
|
211 |
</td>
|
212 |
</tr>
|
213 |
<tr>
|
214 |
<td>Hellaswag (10-shot)
|
215 |
</td>
|
216 |
+
<td>88.54
|
217 |
</td>
|
218 |
+
<td>88.11
|
219 |
</td>
|
220 |
+
<td>99.51%
|
221 |
</td>
|
222 |
</tr>
|
223 |
<tr>
|
|
|
225 |
</td>
|
226 |
<td>87.21
|
227 |
</td>
|
228 |
+
<td>87.77
|
229 |
</td>
|
230 |
+
<td>100.6%
|
231 |
</td>
|
232 |
</tr>
|
233 |
<tr>
|
234 |
<td>TruthfulQA (0-shot, mc2)
|
235 |
</td>
|
236 |
+
<td>65.31
|
237 |
</td>
|
238 |
+
<td>64.58
|
239 |
</td>
|
240 |
+
<td>98.88%
|
241 |
</td>
|
242 |
</tr>
|
243 |
<tr>
|
244 |
<td><strong>Average</strong>
|
245 |
</td>
|
246 |
+
<td><strong>86.57</strong>
|
247 |
</td>
|
248 |
+
<td><strong>86.39</strong>
|
249 |
</td>
|
250 |
+
<td><strong>99.75%</strong>
|
251 |
</td>
|
252 |
</tr>
|
253 |
</table>
|
|
|
261 |
```
|
262 |
lm_eval \
|
263 |
--model vllm \
|
264 |
+
--model_args pretrained="neuralmagic/Meta-Llama-3.1-405B-Instruct-FP8",dtype=auto,add_bos_token=True,max_model_len=4096,max_gen_toks=10,tensor_parallel_size=8 \
|
265 |
+
--tasks mmlu_llama_3.1_instruct \
|
266 |
+
--apply_chat_template \
|
267 |
+
--fewshot_as_multiturn \
|
268 |
--num_fewshot 5 \
|
269 |
--batch_size auto
|
270 |
```
|