metadata
license: apache-2.0
We introduced a new model designed for the Code generation task. Its test accuracy on the HumanEval base dataset surpasses that of GPT-4 Turbo (April 2024). (90.9% vs 90.2%).
Additionally, compared to previous open-source models, AutoCoder offers a new feature: it can automatically install the required packages and attempt to run the code until it deems there are no issues, whenever the user wishes to execute the code.
See details on the AutoCoder GitHub.
Simple test script:
model_path = ""
tokenizer = AutoTokenizer.from_pretrained(model_path)
model = AutoModelForCausalLM.from_pretrained(model_path,
device_map="auto")
HumanEval = load_dataset("evalplus/humanevalplus")
Input = "" # input your question here
messages=[
{ 'role': 'user', 'content': Input}
]
inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True,
return_tensors="pt").to(model.device)
outputs = model.generate(inputs,
max_new_tokens=1024,
do_sample=False,
temperature=0.0,
top_p=1.0,
num_return_sequences=1,
eos_token_id=tokenizer.eos_token_id)
answer = tokenizer.decode(outputs[0][len(inputs[0]):], skip_special_tokens=True)