collapse_gemma-2-2b_hs2_replace_iter19_sftsd1

This model is a fine-tuned version of google/gemma-2-2b on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 2.6320
• Num Input Tokens Seen: 4433616

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 8e-06
• train_batch_size: 8
• eval_batch_size: 16
• seed: 1
• gradient_accumulation_steps: 16
• total_train_batch_size: 128
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: constant_with_warmup
• lr_scheduler_warmup_ratio: 0.05
• num_epochs: 1

Training results

Training Loss	Epoch	Step	Validation Loss	Input Tokens Seen
No log	0	0	1.3909	0
1.5442	0.0511	5	1.2777	235784
0.9407	0.1021	10	1.2947	465856
0.5093	0.1532	15	1.4877	690512
0.3055	0.2042	20	1.6980	924856
0.1536	0.2553	25	1.9076	1156432
0.0891	0.3063	30	2.1777	1381992
0.0462	0.3574	35	2.3302	1607640
0.0319	0.4084	40	2.4326	1837216
0.028	0.4595	45	2.5273	2069720
0.0262	0.5105	50	2.6012	2284640
0.0263	0.5616	55	2.6232	2522216
0.0234	0.6126	60	2.6381	2747792
0.0236	0.6637	65	2.6291	2974368
0.025	0.7147	70	2.6370	3205008
0.0237	0.7658	75	2.6392	3430816
0.0243	0.8168	80	2.6377	3665264
0.0227	0.8679	85	2.6507	3892528
0.0228	0.9190	90	2.6512	4123312
0.0251	0.9700	95	2.6477	4342976

Framework versions

• Transformers 4.44.0
• Pytorch 2.4.0+cu121
• Datasets 2.20.0
• Tokenizers 0.19.1