mistral_dpo

This model is a fine-tuned version of mistralai/Mistral-7B-v0.1 on the None dataset. It achieves the following results on the evaluation set:

• Loss: 0.6891
• Rewards/chosen: 0.0621
• Rewards/rejected: 0.0538
• Rewards/accuracies: 0.6213
• Rewards/margins: 0.0083
• Logps/rejected: -52.0979
• Logps/chosen: -55.7624
• Logits/rejected: -0.2991
• Logits/chosen: -0.3269

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-07
• train_batch_size: 4
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 8
• total_train_batch_size: 32
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 20
• training_steps: 100

Training results

Training Loss	Epoch	Step	Validation Loss	Rewards/chosen	Rewards/rejected	Rewards/accuracies	Rewards/margins	Logps/rejected	Logps/chosen	Logits/rejected	Logits/chosen
0.6929	0.0882	20	0.6923	0.0239	0.0221	0.6109	0.0018	-52.4151	-56.1451	-0.2993	-0.3271
0.6919	0.1763	40	0.6908	0.0490	0.0443	0.6187	0.0047	-52.1927	-55.8934	-0.2991	-0.3268
0.6903	0.2645	60	0.6898	0.0587	0.0518	0.6157	0.0068	-52.1173	-55.7971	-0.2991	-0.3269
0.6899	0.3526	80	0.6892	0.0595	0.0515	0.6135	0.0081	-52.1210	-55.7885	-0.2991	-0.3269
0.6898	0.4408	100	0.6891	0.0621	0.0538	0.6213	0.0083	-52.0979	-55.7624	-0.2991	-0.3269

Framework versions

• PEFT 0.11.1
• Transformers 4.41.2
• Pytorch 2.3.1+cu121
• Datasets 2.20.0
• Tokenizers 0.19.1