G0514HMA4H

This model is a fine-tuned version of google/gemma-2b on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: -17.7428

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0003
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 16
• total_train_batch_size: 128
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine_with_restarts
• lr_scheduler_warmup_steps: 80
• num_epochs: 3
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss
0.9766	0.09	10	0.1938
-0.7095	0.18	20	-2.1543
-3.5129	0.27	30	-5.3325
-6.6654	0.36	40	-8.4889
-9.7041	0.45	50	-11.4077
-12.4428	0.54	60	-13.8134
-14.5638	0.63	70	-15.4354
-15.9411	0.73	80	-16.4692
-16.7235	0.82	90	-17.0250
-17.1404	0.91	100	-17.2723
-17.3354	1.0	110	-17.4128
-17.4522	1.09	120	-17.4974
-17.5272	1.18	130	-17.5592
-17.5718	1.27	140	-17.6055
-17.6154	1.36	150	-17.6336
-17.6416	1.45	160	-17.6542
-17.656	1.54	170	-17.6683
-17.6769	1.63	180	-17.6832
-17.6876	1.72	190	-17.6950
-17.6975	1.81	200	-17.7028
-17.7043	1.9	210	-17.7114
-17.7137	1.99	220	-17.7188
-17.7217	2.08	230	-17.7245
-17.7306	2.18	240	-17.7293
-17.7293	2.27	250	-17.7330
-17.7348	2.36	260	-17.7365
-17.741	2.45	270	-17.7386
-17.7415	2.54	280	-17.7403
-17.7442	2.63	290	-17.7410
-17.7456	2.72	300	-17.7422
-17.7426	2.81	310	-17.7425
-17.7471	2.9	320	-17.7427
-17.7473	2.99	330	-17.7428

Framework versions

• Transformers 4.36.0.dev0
• Pytorch 2.1.2+cu121
• Datasets 2.14.6
• Tokenizers 0.14.0