| ## Perplexity (ctx 8192) | |
| BF16: | |
| ``` | |
| I1005 10:12:54.337877 967332 eval_ppl.py:66] wikitext2 perplexity: 6.49992036819458 | |
| I1005 10:17:19.718732 967332 eval_ppl.py:66] c4 perplexity: 8.022844314575195 | |
| ``` | |
| 4 Bit QTIP (this model): | |
| ``` | |
| I1005 10:13:22.272425 967335 eval_ppl.py:66] wikitext2 perplexity: 6.610454082489014 | |
| I1005 10:19:33.215519 967335 eval_ppl.py:66] c4 perplexity: 8.128130912780762 | |
| ``` | |
| ## Zeroshot Results with lm_eval 0.3.0 | |
| BF16: | |
| ``` | |
| |-------------|------:|--------|-----:|---|-----:| | |
| |arc_challenge| 0|acc |0.5196|± |0.0146| | |
| | | |acc_norm|0.5512|± |0.0145| | |
| |arc_easy | 0|acc |0.8182|± |0.0079| | |
| | | |acc_norm|0.7980|± |0.0082| | |
| |boolq | 1|acc |0.8410|± |0.0064| | |
| |piqa | 0|acc |0.8003|± |0.0093| | |
| | | |acc_norm|0.8090|± |0.0092| | |
| |winogrande | 0|acc |0.7380|± |0.0124| | |
| ``` | |
| 4 Bit QTIP (this model): | |
| ``` | |
| |-------------|------:|--------|-----:|---|-----:| | |
| |arc_challenge| 0|acc |0.5179|± |0.0146| | |
| | | |acc_norm|0.5486|± |0.0145| | |
| |arc_easy | 0|acc |0.8157|± |0.0080| | |
| | | |acc_norm|0.7942|± |0.0083| | |
| |boolq | 1|acc |0.8425|± |0.0064| | |
| |piqa | 0|acc |0.8025|± |0.0093| | |
| | | |acc_norm|0.8079|± |0.0092| | |
| |winogrande | 0|acc |0.7419|± |0.0123| | |
| ``` |