noneUsername/SAINEMO-reMIX-W8A8

vllm (pretrained=/root/autodl-tmp/SAINEMO-reMIX,add_bos_token=true,tensor_parallel_size=2,max_model_len=2048,dtype=bfloat16), gen_kwargs: (None), limit: 500.0, num_fewshot: 5, batch_size: auto

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
gsm8k	3	flexible-extract	5	exact_match	↑	0.774	±	0.0187
		strict-match	5	exact_match	↑	0.712	±	0.0203

vllm (pretrained=/root/autodl-tmp/SAINEMO-reMIX,add_bos_token=true,tensor_parallel_size=2,max_model_len=2048,dtype=bfloat16), gen_kwargs: (None), limit: 250.0, num_fewshot: 5, batch_size: auto

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
gsm8k	3	flexible-extract	5	exact_match	↑	0.772	±	0.0266
		strict-match	5	exact_match	↑	0.728	±	0.0282

vllm (pretrained=/root/autodl-tmp/SAINEMO-reMIX-86,add_bos_token=true,tensor_parallel_size=2,max_model_len=2048,dtype=bfloat16), gen_kwargs: (None), limit: 250.0, num_fewshot: 5, batch_size: auto

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
gsm8k	3	flexible-extract	5	exact_match	↑	0.804	±	0.0252
		strict-match	5	exact_match	↑	0.784	±	0.0261

vllm (pretrained=/root/autodl-tmp/SAINEMO-reMIX-86,add_bos_token=true,tensor_parallel_size=2,max_model_len=2048,dtype=bfloat16), gen_kwargs: (None), limit: 500.0, num_fewshot: 5, batch_size: auto

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
gsm8k	3	flexible-extract	5	exact_match	↑	0.788	±	0.0183
		strict-match	5	exact_match	↑	0.748	±	0.0194