Llama-2-7b-chat-hf-finetune_90_10_MIX

This model is a fine-tuned version of meta-llama/Llama-2-7b-chat-hf on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 1.3119

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0001
• train_batch_size: 3
• eval_batch_size: 3
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: reduce_lr_on_plateau
• num_epochs: 50
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss
0.6795	0.9968	316	0.7737
0.2756	1.9937	632	0.8534
0.166	2.9905	948	0.9507
0.1135	3.9874	1264	1.0163
0.086	4.9842	1580	1.0497
0.0788	5.9811	1896	1.0818
0.1423	6.9779	2212	1.1176
0.0778	7.9748	2528	1.1538
0.0792	8.9716	2844	1.1963
0.0657	9.9685	3160	1.1900
0.0639	10.9653	3476	1.2259
0.0681	11.9621	3792	1.2195
0.0522	12.9590	4108	1.2163
0.0492	13.9558	4424	1.2259
0.048	14.9527	4740	1.2378
0.0441	15.9495	5056	1.2492
0.0629	16.9464	5372	1.2564
0.0622	17.9432	5688	1.2606
0.0589	18.9401	6004	1.2662
0.0592	19.9369	6320	1.2712
0.0586	20.9338	6636	1.2780
0.0594	21.9306	6952	1.2807
0.0616	22.9274	7268	1.2874
0.0554	23.9243	7584	1.2904
0.0562	24.9211	7900	1.2934
0.0543	25.9180	8216	1.2961
0.0553	26.9148	8532	1.2986
0.0547	27.9117	8848	1.3009
0.0543	28.9085	9164	1.3025
0.0535	29.9054	9480	1.3040
0.0535	30.9022	9796	1.3053
0.0533	31.8991	10112	1.3068
0.053	32.8959	10428	1.3078
0.0528	33.8927	10744	1.3096
0.0526	34.8896	11060	1.3098
0.0523	35.8864	11376	1.3100
0.052	36.8833	11692	1.3102
0.0516	37.8801	12008	1.3104
0.0513	38.8770	12324	1.3105
0.0504	39.8738	12640	1.3107
0.0508	40.8707	12956	1.3109
0.0503	41.8675	13272	1.3111
0.0501	42.8644	13588	1.3114
0.0502	43.8612	13904	1.3116
0.05	44.8580	14220	1.3118
0.0498	45.8549	14536	1.3118
0.0517	46.8517	14852	1.3118
0.0496	47.8486	15168	1.3118
0.0486	48.8454	15484	1.3118
0.0475	49.8423	15800	1.3119

Framework versions

• PEFT 0.11.1
• Transformers 4.40.2
• Pytorch 2.3.0+cu121
• Datasets 2.20.0
• Tokenizers 0.19.1