phi-3-mini-LoRA

This model is a fine-tuned version of microsoft/Phi-3.5-mini-instruct on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 0.3840

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0002
• train_batch_size: 4
• eval_batch_size: 4
• seed: 3407
• gradient_accumulation_steps: 4
• total_train_batch_size: 16
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 5
• training_steps: 120

Training results

Training Loss	Epoch	Step	Validation Loss
1.5071	0.5882	5	1.4674
1.1659	1.1765	10	1.0849
0.894	1.7647	15	0.8655
0.7243	2.3529	20	0.6989
0.5752	2.9412	25	0.5856
0.5724	3.5294	30	0.5257
0.4834	4.1176	35	0.4875
0.3861	4.7059	40	0.4588
0.35	5.2941	45	0.4368
0.3126	5.8824	50	0.4251
0.367	6.4706	55	0.4080
0.2792	7.0588	60	0.3955
0.3952	7.6471	65	0.3914
0.2854	8.2353	70	0.3784
0.3224	8.8235	75	0.3867
0.3187	9.4118	80	0.3765
0.1675	10.0	85	0.3799
0.1888	10.5882	90	0.3858
0.2021	11.1765	95	0.3759
0.1518	11.7647	100	0.3868
0.2075	12.3529	105	0.3915
0.1497	12.9412	110	0.3814
0.1797	13.5294	115	0.3821
0.1606	14.1176	120	0.3840

Framework versions

• PEFT 0.13.2
• Transformers 4.45.2
• Pytorch 2.4.0
• Datasets 3.0.1
• Tokenizers 0.20.0