collapse_gemma-2-2b_hs2_accumulatesubsample_iter7_sftsd0

This model is a fine-tuned version of google/gemma-2-2b on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 1.1806
• Num Input Tokens Seen: 5046464

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 8e-06
• train_batch_size: 8
• eval_batch_size: 16
• seed: 0
• gradient_accumulation_steps: 16
• total_train_batch_size: 128
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: constant_with_warmup
• lr_scheduler_warmup_ratio: 0.05
• num_epochs: 1

Training results

Training Loss	Epoch	Step	Validation Loss	Input Tokens Seen
No log	0	0	1.3909	0
1.3617	0.0544	5	1.2733	278136
1.199	0.1088	10	1.2001	559424
1.0696	0.1632	15	1.1783	845472
1.01	0.2175	20	1.1976	1120848
0.9704	0.2719	25	1.1931	1401104
0.7928	0.3263	30	1.1923	1680784
0.8869	0.3807	35	1.1986	1956960
0.7529	0.4351	40	1.2010	2233408
0.6472	0.4895	45	1.2048	2509944
0.6316	0.5438	50	1.2074	2783928
0.6944	0.5982	55	1.1871	3057160
0.5639	0.6526	60	1.1970	3334664
0.5933	0.7070	65	1.1907	3613112
0.5994	0.7614	70	1.1841	3885304
0.5655	0.8158	75	1.1783	4165304
0.5658	0.8702	80	1.1869	4444120
0.5515	0.9245	85	1.1819	4716040
0.4924	0.9789	90	1.1824	4993184

Framework versions

• Transformers 4.44.0
• Pytorch 2.4.0+cu121
• Datasets 2.20.0
• Tokenizers 0.19.1