ymcki/Llama-3.1-8B-SFT-GRPO-Instruct

Original model: https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct

Prompt format

<|begin_of_text|><|start_header_id|>system<|end_header_id|>
Cutting Knowledge Date: December 2023
Today Date: 26 July 2024
{system_prompt}<|eot_id|><|start_header_id|>user<|end_header_id|>
{prompt}<|eot_id|><|start_header_id|>assistant<|end_header_id|>

By following the same procedure of Deepseek R1, SFT with Cognitive Computations' dolphin-r1 was performed first and then followed by Group Relative Policy Optimization (GRPO) with OpenAI gsm8k dataset. Two adapters are obtained and were applied to Llama-3.1-8B-Instruct to see if Reasoning and Math can be further improved.

One epoch was run for the GRPO run. High reward average score for the last 53 steps was recorded at 0.96 epoch. The adapter is then applied to Llama-3.1-8B-Instruct.

Epoch	reward/format	reward/correct	reward/total
0.52	0.469783	1.27358	1.74337
0.96	0.750012	1.10613	1.85614
1.00	0.747508	1.05425	1.80175

This model is uploaded here to be evaluated by the Open LLM Leaderboard. Further GRPO fine tuning is currently underway to see further improvement is possible.

Benchmark (100.0*raw scores only)

Click on the model name go to the raw score json generated by Open LLM Leaderboard.

Model	Average	IFEval	BHH	Math Lv5	GPQA	MUSR	MMLU-PRO
Llama-3.1-8B-Instruct	42.24	80.48	50.62	19.34	26.76	38.62	37.62
Llama-3.1-8B-GRPO-Instruct	42.00	75.61	51.21	20.24	29.45	38.10	37.38
Llama-3.1-8B-SFT-GRPO-Instruct

How to run this model

from transformers import AutoTokenizer, AutoModelForCausalLM
import transformers
import torch

model_id = "Llama-3.1-8B-SFT-GRPO-Instruct"
dtype = torch.bfloat16

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map="cuda",
    torch_dtype=dtype,)

chat = [
    { "role": "user", "content": "Write a hello world program" },
]
prompt = tokenizer.apply_chat_template(chat, tokenize=False, add_generation_prompt=True)

Downloading using huggingface-cli

First, make sure you have hugginface-cli installed:

pip install -U "huggingface_hub[cli]"

Then, you can target the specific file you want:

huggingface-cli download ymcki/Llama-3.1-8B-SFT-GRPO-Instruct --include "*" --local-dir ./

Credits

Thanks Deepseek to develop the original GRPO method.