openaccess-ai-collective/mighty-llama-1b

Single Pass

hf-causal-experimental (pretrained=openaccess-ai-collective/mighty-llama-1b,use_accelerate=True,dtype=bfloat16,trust_remote_code=True), limit: None, provide_description: False, num_fewshot: 0, batch_size: 32
|    Task     |Version| Metric |Value |   |Stderr|
|-------------|------:|--------|-----:|---|-----:|
|arc_challenge|      0|acc     |0.2355|_  |0.0124|
|             |       |acc_norm|0.2671|_  |0.0129|
|arc_easy     |      0|acc     |0.4444|_  |0.0102|
|             |       |acc_norm|0.4276|_  |0.0102|
|boolq        |      1|acc     |0.5358|_  |0.0087|
|hellaswag    |      0|acc     |0.3784|_  |0.0048|
|             |       |acc_norm|0.5034|_  |0.0050|
|openbookqa   |      0|acc     |0.1580|_  |0.0163|
|             |       |acc_norm|0.2840|_  |0.0202|
|piqa         |      0|acc     |0.6518|_  |0.0111|
|             |       |acc_norm|0.6464|_  |0.0112|
|winogrande   |      0|acc     |0.5422|_  |0.0140|

16x Passees

hf-causal-experimental (pretrained=openaccess-ai-collective/mighty-llama-1b,use_accelerate=True,dtype=bfloat16,trust_remote_code=True), limit: None, provide_description: False, num_fewshot: 0, batch_size: 64
|    Task     |Version| Metric |Value |   |Stderr|
|-------------|------:|--------|-----:|---|-----:|
|arc_challenge|      0|acc     |0.2466|_  |0.0126|
|             |       |acc_norm|0.2824|_  |0.0132|
|arc_easy     |      0|acc     |0.3649|_  |0.0099|
|             |       |acc_norm|0.3582|_  |0.0098|
|boolq        |      1|acc     |0.6214|_  |0.0085|
|hellaswag    |      0|acc     |0.3085|_  |0.0046|
|             |       |acc_norm|0.3614|_  |0.0048|
|openbookqa   |      0|acc     |0.1900|_  |0.0176|
|             |       |acc_norm|0.2800|_  |0.0201|
|piqa         |      0|acc     |0.5702|_  |0.0116|
|             |       |acc_norm|0.5729|_  |0.0115|
|winogrande   |      0|acc     |0.5399|_  |0.0140|