Hugging Face's logo

neuralmagic
/
granite-3.1-8b-instruct-quantized.w8a8

Text Generation
Transformers
Safetensors
English
granite
w8a8
int8
vllm
conversational
Inference Endpoints
8-bit precision
compressed-tensors
Model card Files Community
nm-research commited on
Commit
133efb6
·
verified ·
1 Parent(s): 9ba2b30

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +8 -8
README.md CHANGED
@@ -197,17 +197,17 @@ evalplus.evaluate \
197
 
198
  | Metric | ibm-granite/granite-3.1-8b-instruct | neuralmagic-ent/granite-3.1-8b-instruct-quantized.w8a8 |
199
  |-----------------------------------------|:---------------------------------:|:-------------------------------------------:|
200
- | ARC-Challenge (Acc-Norm, 25-shot) | 66.81 | 66.81 |
201
- | GSM8K (Strict-Match, 5-shot) | 64.52 | 64.37 |
202
- | HellaSwag (Acc-Norm, 10-shot) | 84.18 | 83.91 |
203
- | MMLU (Acc, 5-shot) | 65.52 | 65.00 |
204
- | TruthfulQA (MC2, 0-shot) | 60.57 | 60.29 |
205
  | Winogrande (Acc, 5-shot) | 80.19 | 79.87 |
206
- | **Average Score** | **70.30** | **70.04** |
207
- | **Recovery** | **100.00** | **99.64** |
208
 
209
  #### HumanEval pass@1 scores
210
  | Metric | ibm-granite/granite-3.1-8b-instruct | neuralmagic-ent/granite-3.1-8b-instruct-quantized.w8a8 |
211
  |-----------------------------------------|:---------------------------------:|:-------------------------------------------:|
212
- | HumanEval Pass@1 | 71.00 | 72.00 |
213
 
 
197
 
198
  | Metric | ibm-granite/granite-3.1-8b-instruct | neuralmagic-ent/granite-3.1-8b-instruct-quantized.w8a8 |
199
  |-----------------------------------------|:---------------------------------:|:-------------------------------------------:|
200
+ | ARC-Challenge (Acc-Norm, 25-shot) | 66.81 | 67.06 |
201
+ | GSM8K (Strict-Match, 5-shot) | 64.52 | 65.66 |
202
+ | HellaSwag (Acc-Norm, 10-shot) | 84.18 | 83.93 |
203
+ | MMLU (Acc, 5-shot) | 65.52 | 65.03 |
204
+ | TruthfulQA (MC2, 0-shot) | 60.57 | 60.02 |
205
  | Winogrande (Acc, 5-shot) | 80.19 | 79.87 |
206
+ | **Average Score** | **70.30** | **70.26** |
207
+ | **Recovery** | **100.00** | **99.95** |
208
 
209
  #### HumanEval pass@1 scores
210
  | Metric | ibm-granite/granite-3.1-8b-instruct | neuralmagic-ent/granite-3.1-8b-instruct-quantized.w8a8 |
211
  |-----------------------------------------|:---------------------------------:|:-------------------------------------------:|
212
+ | HumanEval Pass@1 | 71.00 | 70.50 |
213