codeparrot-ds

This model is a fine-tuned version of gpt2 on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 5.1436

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0005
• train_batch_size: 32
• eval_batch_size: 32
• seed: 42
• gradient_accumulation_steps: 8
• total_train_batch_size: 256
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 10
• num_epochs: 5
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss
9.5238	0.2062	10	8.0787
7.1728	0.4124	20	7.0714
6.6796	0.6186	30	6.8378
6.4165	0.8247	40	6.5842
6.1172	1.0309	50	6.3309
5.838	1.2371	60	6.1147
5.6075	1.4433	70	5.9416
5.4355	1.6495	80	5.8080
5.3001	1.8557	90	5.7022
5.1595	2.0619	100	5.6176
5.0118	2.2680	110	5.5504
4.9696	2.4742	120	5.4852
4.8653	2.6804	130	5.4179
4.8474	2.8866	140	5.3556
4.706	3.0928	150	5.3083
4.6329	3.2990	160	5.2737
4.5585	3.5052	170	5.2405
4.5445	3.7113	180	5.2051
4.497	3.9175	190	5.1791
4.4332	4.1237	200	5.1661
4.3867	4.3299	210	5.1535
4.4075	4.5361	220	5.1473
4.4265	4.7423	230	5.1445
4.4294	4.9485	240	5.1436

Framework versions

• Transformers 4.44.2
• Pytorch 2.4.1+cu121
• Datasets 3.0.0
• Tokenizers 0.19.1