NeMo
PyTorch
nemotron
srvm commited on
Commit
5e7fcb9
·
1 Parent(s): d249aff

Add results preview

Browse files
Files changed (1) hide show
  1. README.md +23 -0
README.md CHANGED
@@ -53,6 +53,29 @@ print(output_text)
53
 
54
  Minitron is released under the [NVIDIA Open Model License Agreement](https://developer.download.nvidia.com/licenses/nvidia-open-model-license-agreement-june-2024.pdf).
55
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
56
  ## Citation
57
 
58
  If you find our work helpful, please consider citing our paper:
 
53
 
54
  Minitron is released under the [NVIDIA Open Model License Agreement](https://developer.download.nvidia.com/licenses/nvidia-open-model-license-agreement-june-2024.pdf).
55
 
56
+ ## Evaluation Results
57
+
58
+ *5-shot performance.* Language Understanding evaluated using [Massive Multitask Language Understanding](https://arxiv.org/abs/2009.03300):
59
+
60
+ | Average |
61
+ | :---- |
62
+ | 58.6 |
63
+
64
+ *Zero-shot performance.* Evaluated using select datasets from the [LM Evaluation Harness](https://github.com/EleutherAI/lm-evaluation-harness) with additions:
65
+
66
+ | HellaSwag | Winogrande | GSM8K| ARC-C | XLSum |
67
+ | :------------- | :------------- | :------------- | :------------- | :------------- |
68
+ | 75.0 | 74.0 | 24.1 | 50.9 | 29.5
69
+
70
+
71
+ *Code generation performance*. Evaluated using [HumanEval](https://github.com/openai/human-eval):
72
+
73
+ | p@1, 0-Shot |
74
+ | :------------- |
75
+ | 23.3 |
76
+
77
+ Please refer to our [paper](https://arxiv.org/abs/2407.14679) for the full set of results.
78
+
79
  ## Citation
80
 
81
  If you find our work helpful, please consider citing our paper: