2d3b03ac-9b1a-44ae-8e85-d190e78b25a8

This model is a fine-tuned version of NousResearch/CodeLlama-13b-hf on the None dataset. It achieves the following results on the evaluation set:

• Loss: 1.1021

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.000206
• train_batch_size: 4
• eval_batch_size: 4
• seed: 42
• gradient_accumulation_steps: 2
• total_train_batch_size: 8
• optimizer: Use OptimizerNames.ADAMW_BNB with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 50
• training_steps: 500

Training results

Training Loss	Epoch	Step	Validation Loss
No log	0.0001	1	1.5558
2.464	0.0064	50	1.1849
1.7612	0.0128	100	1.3235
2.4932	0.0192	150	1.1773
1.6114	0.0256	200	1.4369
2.2952	0.0320	250	1.1528
1.7323	0.0383	300	1.2193
2.2685	0.0447	350	1.1176
1.6619	0.0511	400	1.1314
2.236	0.0575	450	1.1032
1.6742	0.0639	500	1.1021

Framework versions

• PEFT 0.13.2
• Transformers 4.46.0
• Pytorch 2.5.0+cu124
• Datasets 3.0.1
• Tokenizers 0.20.1