TeeZee/NEBULA-XB-v1.03
Experiment, can DUS be taken one or more steps further?
Technical notes:
- pretrained model v03 finetuned on 50k entries from SlimOrca dataset
- 18 layers removed from both models of finetuned GALAXY-XB-v03
- model has 108 layers (((48-12)*2)-18)*2 = 108
- second step in scaling DUS procedure
To evaluate
- model performance after merge, should be a little lover that GALAXY finetuned on 50k of slimorca
Open LLM Leaderboard Evaluation Results
Detailed results can be found here
| Metric | Value |
|---|---|
| Avg. | 53.52 |
| AI2 Reasoning Challenge (25-Shot) | 56.66 |
| HellaSwag (10-Shot) | 81.78 |
| MMLU (5-Shot) | 60.98 |
| TruthfulQA (0-shot) | 44.03 |
| Winogrande (5-shot) | 77.66 |
| GSM8k (5-shot) | 0.00 |
- Downloads last month
- 748
Dataset used to train TeeZee/NEBULA-XB-v1.0
Evaluation results
- normalized accuracy on AI2 Reasoning Challenge (25-Shot)test set Open LLM Leaderboard56.660
- normalized accuracy on HellaSwag (10-Shot)validation set Open LLM Leaderboard81.780
- accuracy on MMLU (5-Shot)test set Open LLM Leaderboard60.980
- mc2 on TruthfulQA (0-shot)validation set Open LLM Leaderboard44.030
- accuracy on Winogrande (5-shot)validation set Open LLM Leaderboard77.660
- accuracy on GSM8k (5-shot)test set Open LLM Leaderboard0.000