G0514HMA22H

This model is a fine-tuned version of google/gemma-2b on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: -17.8940

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0003
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 16
• total_train_batch_size: 128
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine_with_restarts
• lr_scheduler_warmup_steps: 80
• num_epochs: 3
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss
0.8474	0.09	10	-0.1051
-1.133	0.18	20	-2.6746
-4.0726	0.27	30	-5.9791
-7.4096	0.36	40	-9.2948
-10.5075	0.45	50	-12.1604
-13.1872	0.54	60	-14.5712
-15.3198	0.63	70	-16.1577
-16.5601	0.73	80	-17.0062
-17.1749	0.82	90	-17.3669
-17.4459	0.91	100	-17.5280
-17.5636	1.0	110	-17.6099
-17.6344	1.09	120	-17.6593
-17.6708	1.18	130	-17.6865
-17.6958	1.27	140	-17.7099
-17.7175	1.36	150	-17.7283
-17.7369	1.45	160	-17.7437
-17.7549	1.54	170	-17.7646
-17.7752	1.63	180	-17.7824
-17.785	1.72	190	-17.7920
-17.8012	1.81	200	-17.8080
-17.8109	1.9	210	-17.8184
-17.8264	1.99	220	-17.8386
-17.85	2.08	230	-17.8633
-17.8652	2.18	240	-17.8736
-17.8735	2.27	250	-17.8818
-17.8791	2.36	260	-17.8860
-17.8821	2.45	270	-17.8882
-17.8883	2.54	280	-17.8912
-17.891	2.63	290	-17.8924
-17.8909	2.72	300	-17.8933
-17.8886	2.81	310	-17.8938
-17.8926	2.9	320	-17.8939
-17.8924	2.99	330	-17.8940

Framework versions

• Transformers 4.36.0.dev0
• Pytorch 2.1.2+cu121
• Datasets 2.14.6
• Tokenizers 0.14.0