collapse_gemma-2-2b_hs2_replace_iter18_sftsd0

This model is a fine-tuned version of google/gemma-2-2b on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 2.6152
• Num Input Tokens Seen: 4612568

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 8e-06
• train_batch_size: 8
• eval_batch_size: 16
• seed: 0
• gradient_accumulation_steps: 16
• total_train_batch_size: 128
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: constant_with_warmup
• lr_scheduler_warmup_ratio: 0.05
• num_epochs: 1

Training results

Training Loss	Epoch	Step	Validation Loss	Input Tokens Seen
No log	0	0	1.3909	0
1.4791	0.0511	5	1.2823	234336
0.7755	0.1021	10	1.3555	468312
0.5206	0.1532	15	1.5591	705040
0.2677	0.2042	20	1.8280	942456
0.1169	0.2553	25	2.1030	1175928
0.0577	0.3063	30	2.3359	1415480
0.0419	0.3574	35	2.4555	1658512
0.0294	0.4084	40	2.5475	1894936
0.0247	0.4595	45	2.5924	2129624
0.0226	0.5105	50	2.6222	2365768
0.0234	0.5616	55	2.6268	2602808
0.0241	0.6126	60	2.6131	2839776
0.0235	0.6637	65	2.6022	3083424
0.0211	0.7147	70	2.5919	3314048
0.024	0.7658	75	2.5998	3557112
0.0232	0.8168	80	2.5986	3792832
0.0231	0.8679	85	2.5935	4040848
0.0237	0.9190	90	2.6008	4281816
0.0221	0.9700	95	2.6136	4515280

Framework versions

• Transformers 4.44.0
• Pytorch 2.4.0+cu121
• Datasets 2.20.0
• Tokenizers 0.19.1