| # Model Card for CodeFuse-13B-4K | |
|  | |
| [[中文]](#chinese) [[English]](#english) | |
| <a id="english"></a> | |
| ## Model Description | |
| CodeFuse-13B is a 13 billion parameter code generation model trained on the GPT-NeoX framework, capable of handling code sequences of up to 4096 characters. This model was pretrained on a dataset consisting of 1000B token code, Chinese, and English data, covering over 40 programming languages. To further enhance the effectiveness and quality of the generated code, the model was fine-tuned on the CodeFuse-Evol-instruction-66k dataset, enabling it to produce more accurate, efficient, and compliant code. Pass@1 achieved 37.1% on the HumanEval evaluation set(BeamSearch strategy, BeamSize=3). | |
| ## Requirements | |
| * Python 3.8 or above. | |
| * PyTorch 1.12 or above, with a recommendation for 2.0 or above. | |
| * Transformers 4.24.0 or above. | |
| * It is advisable to use CUDA 11.4 or above (for GPU users and flash-attention users, this option should be considered). | |
| ## Quickstart | |
| ``` | |
| import torch | |
| from transformers import AutoModelForCausalLM, AutoTokenizer | |
| tokenizer = AutoTokenizer.from_pretrained("CodeFuse-13B") | |
| model = AutoModelForCausalLM.from_pretrained("CodeFuse-13B", torch_dtype="auto", device_map="auto") | |
| input_ids = encode("def quick_sort(array):\n", return_tensors="pt") | |
| output_ids = model.generate(input_ids, max_new_tokens=200, num_beams=3, num_return_sequences=1, repetition_penalty=1.2) | |
| print(tokenizer.decode(output_idss[0])) | |
| ``` | |
| <a id="chinese"></a> | |
| ## 简介 | |
| CodeFuse-13B是基于GPT-NeoX框架训练的13B参数代码生成模型，能够处理4096个字符的代码序列。该模型在1000B Token的代码、中文、英文数据数据集上进行预训练，覆盖超过40种编程语言。为了进一步提升生成代码的效果和质量，该模型还在CodeFuse-Evol-instruction-66k数据集上进行了微调，使得该模型能够生成更加准确、高效、符合要求的代码。在HumanEval评测集上Pass@1达到37.1%(采用BeamSearch解码，其中BeamSize=3)。 | |
| ## 要求 | |
| * python 3.8及以上版本 | |
| * pytorch 1.12及以上版本，推荐2.0及以上版本 | |
| * transformers 4.24.0及以上版本 | |
| * 建议使用CUDA 11.4及以上（GPU用户、flash-attention用户等需考虑此选 | |
| ## 快速使用 | |
| ``` | |
| import torch | |
| from transformers import AutoModelForCausalLM, AutoTokenizer | |
| tokenizer = AutoTokenizer.from_pretrained("CodeFuse-13B") | |
| model = AutoModelForCausalLM.from_pretrained("CodeFuse-13B", torch_dtype="auto", device_map="auto") | |
| input_ids = encode("def quick_sort(array):\n", return_tensors="pt") | |
| output_ids = model.generate(input_ids, max_new_tokens=200, num_beams=3, num_return_sequences=1, repetition_penalty=1.2) | |
| print(tokenizer.decode(output_idss[0])) | |
| ``` |