O0507TESTW2

This model is a fine-tuned version of allenai/OLMo-1B on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: -15.8217

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0003
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 16
• total_train_batch_size: 128
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine_with_restarts
• lr_scheduler_warmup_steps: 100
• num_epochs: 3
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss
-0.1647	0.09	10	-3.1185
-4.2812	0.18	20	-5.4241
-6.9822	0.27	30	-9.2575
-11.0895	0.36	40	-13.0391
-13.9419	0.45	50	-14.5066
-14.9177	0.54	60	-15.0055
-15.2124	0.63	70	-15.2714
-15.4062	0.73	80	-15.5051
-15.5194	0.82	90	-15.5283
-6.9681	0.91	100	-3.6542
-6.2069	1.0	110	-13.6707
-14.5293	1.09	120	-15.4137
-15.5064	1.18	130	-15.5754
-15.6102	1.27	140	-15.6458
-15.6795	1.36	150	-15.7236
-15.7487	1.45	160	-15.7689
-15.7781	1.54	170	-15.7856
-15.786	1.63	180	-15.7905
-15.7915	1.72	190	-15.7937
-15.8052	1.81	200	-15.7934
-15.8	1.9	210	-15.7950
-15.796	1.99	220	-15.7954
-15.8014	2.08	230	-15.7982
-15.7996	2.18	240	-15.8007
-15.8055	2.27	250	-15.8022
-15.8089	2.36	260	-15.8037
-15.8139	2.45	270	-15.8082
-15.8142	2.54	280	-15.8109
-15.821	2.63	290	-15.8151
-15.8227	2.72	300	-15.8183
-15.8206	2.81	310	-15.8206
-15.8252	2.9	320	-15.8216
-15.8238	2.99	330	-15.8217

Framework versions

• Transformers 4.36.0.dev0
• Pytorch 2.1.2+cu121
• Datasets 2.14.6
• Tokenizers 0.14.0