Llama-31-8B_task-2_60-samples_config-3_full

This model is a fine-tuned version of meta-llama/Meta-Llama-3.1-8B-Instruct on the GaetanMichelet/chat-60_ft_task-2 dataset. It achieves the following results on the evaluation set:

• Loss: 1.0710

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 1e-05
• train_batch_size: 1
• eval_batch_size: 1
• seed: 42
• distributed_type: multi-GPU
• gradient_accumulation_steps: 8
• total_train_batch_size: 8
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 150

Training results

Training Loss	Epoch	Step	Validation Loss
1.5659	0.8696	5	1.5846
1.5947	1.9130	11	1.5811
1.6305	2.9565	17	1.5758
1.5682	4.0	23	1.5673
1.5687	4.8696	28	1.5571
1.5556	5.9130	34	1.5406
1.4699	6.9565	40	1.5190
1.5027	8.0	46	1.4958
1.5203	8.8696	51	1.4719
1.4872	9.9130	57	1.4445
1.4184	10.9565	63	1.4151
1.3817	12.0	69	1.3868
1.3397	12.8696	74	1.3648
1.3234	13.9130	80	1.3390
1.2893	14.9565	86	1.3122
1.2999	16.0	92	1.2852
1.2212	16.8696	97	1.2628
1.234	17.9130	103	1.2358
1.1704	18.9565	109	1.2078
1.1499	20.0	115	1.1796
1.1265	20.8696	120	1.1570
1.0716	21.9130	126	1.1357
1.0332	22.9565	132	1.1223
1.0631	24.0	138	1.1155
1.0659	24.8696	143	1.1111
1.0637	25.9130	149	1.1068
0.9979	26.9565	155	1.1031
1.0495	28.0	161	1.0993
1.0126	28.8696	166	1.0966
0.9884	29.9130	172	1.0938
1.0366	30.9565	178	1.0909
1.0434	32.0	184	1.0886
1.0222	32.8696	189	1.0862
0.9978	33.9130	195	1.0842
0.9593	34.9565	201	1.0824
1.0383	36.0	207	1.0804
0.9958	36.8696	212	1.0792
0.9774	37.9130	218	1.0778
0.9853	38.9565	224	1.0763
0.9241	40.0	230	1.0747
1.0387	40.8696	235	1.0743
0.9616	41.9130	241	1.0733
0.9909	42.9565	247	1.0724
0.9055	44.0	253	1.0720
1.0025	44.8696	258	1.0722
0.9325	45.9130	264	1.0711
0.8921	46.9565	270	1.0723
0.9079	48.0	276	1.0710
0.9615	48.8696	281	1.0729
0.9517	49.9130	287	1.0718
0.8619	50.9565	293	1.0730
0.8894	52.0	299	1.0739
0.8389	52.8696	304	1.0742
0.9032	53.9130	310	1.0750
0.9015	54.9565	316	1.0760

Framework versions

• PEFT 0.12.0
• Transformers 4.44.0
• Pytorch 2.1.2+cu121
• Datasets 2.20.0
• Tokenizers 0.19.1