EvalPlus: Rigorous Evaluation of LLMs for Code Generation

About

EvalPlus evaluates LLM-generated code on:

• Code Correctness: HumanEval+ and MBPP+
• Code Efficiency: EvalPerf

Resources

• 💻 GitHub Repo: evalplus/evalplus
• 🏆 Leader Board: evalplus.github.io
• 📜 Papers: EvalPlus@NeurIPS'23, EvalPerf@COLM'24
• 🐍 Python Package: PyPI

Citations

@inproceedings{evalplus,
  title = {Is Your Code Generated by Chat{GPT} Really Correct? Rigorous Evaluation of Large Language Models for Code Generation},
  author = {Liu, Jiawei and Xia, Chunqiu Steven and Wang, Yuyao and Zhang, Lingming},
  booktitle = {Thirty-seventh Conference on Neural Information Processing Systems},
  year = {2023},
  url = {https://openreview.net/forum?id=1qvx610Cu7},
}

@inproceedings{evalperf,
  title = {Evaluating Language Models for Efficient Code Generation},
  author = {Liu, Jiawei and Xie, Songrun and Wang, Junhao and Wei, Yuxiang and Ding, Yifeng and Zhang, Lingming},
  booktitle = {First Conference on Language Modeling},
  year = {2024},
  url = {https://openreview.net/forum?id=IBCBMeAhmC},
}