G0514HMA9H

This model is a fine-tuned version of google/gemma-2b on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: -17.8302

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0003
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 16
• total_train_batch_size: 128
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine_with_restarts
• lr_scheduler_warmup_steps: 100
• num_epochs: 3
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss
0.9517	0.09	10	0.1638
-0.6528	0.18	20	-1.9084
-3.1267	0.27	30	-4.7902
-6.0596	0.36	40	-7.8054
-9.0118	0.45	50	-10.6366
-11.6312	0.54	60	-13.0868
-13.9766	0.63	70	-15.0345
-15.6124	0.73	80	-16.3032
-16.6378	0.82	90	-17.0207
-17.1714	0.91	100	-17.3497
-17.4216	1.0	110	-17.5126
-17.5543	1.09	120	-17.5999
-17.6217	1.18	130	-17.6473
-17.6609	1.27	140	-17.6818
-17.6899	1.36	150	-17.7041
-17.7101	1.45	160	-17.7147
-17.7215	1.54	170	-17.7306
-17.7352	1.63	180	-17.7425
-17.7484	1.72	190	-17.7559
-17.7629	1.81	200	-17.7670
-17.77	1.9	210	-17.7756
-17.7798	1.99	220	-17.7847
-17.7898	2.08	230	-17.7911
-17.799	2.18	240	-17.7988
-17.8001	2.27	250	-17.8040
-17.807	2.36	260	-17.8101
-17.8173	2.45	270	-17.8156
-17.8186	2.54	280	-17.8207
-17.8237	2.63	290	-17.8248
-17.8266	2.72	300	-17.8279
-17.8269	2.81	310	-17.8294
-17.8338	2.9	320	-17.8301
-17.831	2.99	330	-17.8302

Framework versions

• Transformers 4.36.0.dev0
• Pytorch 2.1.2+cu121
• Datasets 2.14.6
• Tokenizers 0.14.0