Llama-31-8B_task-3_120-samples_config-4

This model is a fine-tuned version of meta-llama/Meta-Llama-3.1-8B-Instruct on the GaetanMichelet/chat-60_ft_task-3 and the GaetanMichelet/chat-120_ft_task-3 datasets. It achieves the following results on the evaluation set:

• Loss: 0.4427

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 1e-05
• train_batch_size: 1
• eval_batch_size: 1
• seed: 42
• distributed_type: multi-GPU
• gradient_accumulation_steps: 16
• total_train_batch_size: 16
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 150

Training results

Training Loss	Epoch	Step	Validation Loss
2.4707	0.9091	5	2.5024
2.3847	2.0	11	2.4794
2.5822	2.9091	16	2.4506
2.2635	4.0	22	2.3903
2.446	4.9091	27	2.3184
2.3172	6.0	33	2.1938
2.0582	6.9091	38	2.0560
1.9038	8.0	44	1.8385
1.7291	8.9091	49	1.6252
1.3996	10.0	55	1.3456
1.1127	10.9091	60	1.1189
0.8648	12.0	66	0.8627
0.8247	12.9091	71	0.7228
0.5681	14.0	77	0.6453
0.4968	14.9091	82	0.6020
0.589	16.0	88	0.5632
0.3902	16.9091	93	0.5394
0.4795	18.0	99	0.5253
0.3937	18.9091	104	0.5188
0.3482	20.0	110	0.5149
0.4633	20.9091	115	0.5076
0.4324	22.0	121	0.5073
0.5268	22.9091	126	0.5014
0.3829	24.0	132	0.4946
0.3884	24.9091	137	0.4875
0.3955	26.0	143	0.4859
0.3296	26.9091	148	0.4827
0.364	28.0	154	0.4805
0.3218	28.9091	159	0.4765
0.2995	30.0	165	0.4727
0.3728	30.9091	170	0.4668
0.2413	32.0	176	0.4653
0.4141	32.9091	181	0.4626
0.3693	34.0	187	0.4562
0.3666	34.9091	192	0.4526
0.3605	36.0	198	0.4506
0.2923	36.9091	203	0.4474
0.3422	38.0	209	0.4443
0.3489	38.9091	214	0.4427
0.3737	40.0	220	0.4435
0.2804	40.9091	225	0.4437
0.3212	42.0	231	0.4469
0.2322	42.9091	236	0.4480
0.2569	44.0	242	0.4507
0.2501	44.9091	247	0.4544
0.2247	46.0	253	0.4641

Framework versions

• PEFT 0.12.0
• Transformers 4.44.0
• Pytorch 2.1.2+cu121
• Datasets 2.20.0
• Tokenizers 0.19.1