InternLM-Math's picture
Upload folder using huggingface_hub
44cc4a3 verified
metadata
pipeline_tag: text-generation
license: other
language:
  - en
  - zh
tags:
  - math

InternLM-Math-Plus

InternLM-Math Plus

State-of-the-art bilingual open-sourced Math reasoning LLMs. A solver, prover, verifier, augmentor.

💻 Github 🤗 Demo

News

  • [2024.05.24] We release updated version InternLM2-Math-Plus with 4 sizes and state-of-the-art performances including 1.8B, 7B, 20B, and 8x22B. We improve informal math reasoning performance (chain-of-thought and code-intepreter) and formal math reasoning performance (LEAN 4 translation and LEAN 4 theorem proving) significantly.
  • [2024.02.10] We add tech reports and citation reference.
  • [2024.01.31] We add MiniF2F results with evaluation codes!
  • [2024.01.29] We add checkpoints from ModelScope. Update results about majority voting and Code Intepreter. Tech report is on the way!
  • [2024.01.26] We add checkpoints from OpenXLab, which ease Chinese users to download!

Performance

Formal Math Reasoning

We evaluate the performance of InternLM2-Math-Plus on formal math reasoning benchmark MiniF2F-test. The evaluation setting is same as Llemma with LEAN 4.

Models MiniF2F-test
ReProver 26.5
LLMStep 27.9
GPT-F 36.6
HTPS 41.0
Llemma-7B 26.2
Llemma-34B 25.8
InternLM2-Math-7B-Base 30.3
InternLM2-Math-20B-Base 29.5
InternLM2-Math-Plus-1.8B 38.9
InternLM2-Math-Plus-7B 43.4
InternLM2-Math-Plus-20B 42.6
InternLM2-Math-Plus-Mixtral8x22B 37.3

Informal Math Reasoning

We evaluate the performance of InternLM2-Math-Plus on informal math reasoning benchmark MATH and GSM8K. InternLM2-Math-Plus-1.8B outperforms MiniCPM-2B in the smallest size setting. InternLM2-Math-Plus-7B outperforms Deepseek-Math-7B-RL which is the state-of-the-art math reasoning open source model. InternLM2-Math-Plus-Mixtral8x22B achieves 68.5 on MATH (with Python) and 91.8 on GSM8K.

Model MATH MATH-Python GSM8K
MiniCPM-2B 10.2 - 53.8
InternLM2-Math-Plus-1.8B 37.0 41.5 58.8
InternLM2-Math-7B 34.6 50.9 78.1
Deepseek-Math-7B-RL 51.7 58.8 88.2
InternLM2-Math-Plus-7B 53.0 59.7 85.8
InternLM2-Math-20B 37.7 54.3 82.6
InternLM2-Math-Plus-20B 53.8 61.8 87.7
Mixtral8x22B-Instruct-v0.1 41.8 - 78.6
Eurux-8x22B-NCA 49.0 - -
InternLM2-Math-Plus-Mixtral8x22B 58.1 68.5 91.8

We also evaluate models on MathBench-A. InternLM2-Math-Plus-Mixtral8x22B has comparable performance compared to Claude 3 Opus.

Model Arithmetic Primary Middle High College Average
GPT-4o-0513 77.7 87.7 76.3 59.0 54.0 70.9
Claude 3 Opus 85.7 85.0 58.0 42.7 43.7 63.0
Qwen-Max-0428 72.3 86.3 65.0 45.0 27.3 59.2
Qwen-1.5-110B 70.3 82.3 64.0 47.3 28.0 58.4
Deepseek-V2 82.7 89.3 59.0 39.3 29.3 59.9
Llama-3-70B-Instruct 70.3 86.0 53.0 38.7 34.7 56.5
InternLM2-Math-Plus-Mixtral8x22B 77.5 82.0 63.6 50.3 36.8 62.0
InternLM2-Math-20B 58.7 70.0 43.7 24.7 12.7 42.0
InternLM2-Math-Plus-20B 65.8 79.7 59.5 47.6 24.8 55.5
Llama3-8B-Instruct 54.7 71.0 25.0 19.0 14.0 36.7
InternLM2-Math-7B 53.7 67.0 41.3 18.3 8.0 37.7
Deepseek-Math-7B-RL 68.0 83.3 44.3 33.0 23.0 50.3
InternLM2-Math-Plus-7B 61.4 78.3 52.5 40.5 21.7 50.9
MiniCPM-2B 49.3 51.7 18.0 8.7 3.7 26.3
InternLM2-Math-Plus-1.8B 43.0 43.3 25.4 18.9 4.7 27.1

Citation and Tech Report

@misc{ying2024internlmmath,
      title={InternLM-Math: Open Math Large Language Models Toward Verifiable Reasoning}, 
      author={Huaiyuan Ying and Shuo Zhang and Linyang Li and Zhejian Zhou and Yunfan Shao and Zhaoye Fei and Yichuan Ma and Jiawei Hong and Kuikun Liu and Ziyi Wang and Yudong Wang and Zijian Wu and Shuaibin Li and Fengzhe Zhou and Hongwei Liu and Songyang Zhang and Wenwei Zhang and Hang Yan and Xipeng Qiu and Jiayu Wang and Kai Chen and Dahua Lin},
      year={2024},
      eprint={2402.06332},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}