BachelorThesis/LLama-3-8b_V03_BRONCO_CARDIO_SUMMARY_CATALOG

LLama3_V03_BRONCO_CARDIO_SUMMARY_CATALOG

This model is a fine-tuned version of meta-llama/Meta-Llama-3-8B on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 0.2899
• Num Input Tokens Seen: 19629947

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 1
• eval_batch_size: 1
• seed: 42
• gradient_accumulation_steps: 4
• total_train_batch_size: 4
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 50
• num_epochs: 5

Training results

Training Loss	Epoch	Step	Validation Loss	Input Tokens Seen
0.5525	0.2500	1370	0.5227	996500
0.462	0.5000	2740	0.4506	1964768
0.3773	0.7501	4110	0.4044	2929826
0.39	1.0001	5480	0.3481	3926663
0.3027	1.2501	6850	0.3403	4961583
0.262	1.5001	8220	0.3151	5936811
0.261	1.7502	9590	0.2882	6889700
0.2196	2.0002	10960	0.2684	7853240
0.1674	2.2502	12330	0.2706	8812931
0.1651	2.5002	13700	0.2685	9784307
0.1653	2.7503	15070	0.2589	10759461
0.1489	3.0003	16440	0.2516	11779925
0.1185	3.2503	17810	0.2745	12770206
0.1206	3.5003	19180	0.2732	13738387
0.1147	3.7503	20550	0.2745	14717093
0.1184	4.0004	21920	0.2726	15706481
0.1019	4.2504	23290	0.2883	16717365
0.1037	4.5004	24660	0.2897	17715535
0.1039	4.7504	26030	0.2899	18685743

Framework versions

• Transformers 4.40.2
• Pytorch 2.2.2+cu121
• Datasets 2.19.0
• Tokenizers 0.19.1