peft-starcoder-lora-cutlass

This model is a fine-tuned version of bigcode/starcoderbase-7b on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 0.4322

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0005
• train_batch_size: 16
• eval_batch_size: 16
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 30
• training_steps: 4000

Training results

Training Loss	Epoch	Step	Validation Loss
0.5409	0.025	100	0.4729
0.3672	0.05	200	0.4051
0.3432	0.075	300	0.3876
0.312	0.1	400	0.3548
0.2594	0.125	500	0.3507
0.2821	0.15	600	0.3428
0.1958	0.175	700	0.3391
0.2362	0.2	800	0.3405
0.209	0.225	900	0.3421
0.2114	0.25	1000	0.3481
0.1877	0.275	1100	0.3447
0.1903	0.3	1200	0.3533
0.1723	0.325	1300	0.3578
0.1465	0.35	1400	0.3643
0.1751	0.375	1500	0.3639
0.1213	0.4	1600	0.3749
0.119	0.425	1700	0.3692
0.1199	0.45	1800	0.3820
0.0986	0.475	1900	0.3816
0.1232	0.5	2000	0.3868
0.0819	0.525	2100	0.3923
0.096	0.55	2200	0.3964
0.0873	0.575	2300	0.3977
0.0892	0.6	2400	0.4007
0.0802	0.625	2500	0.4078
0.066	0.65	2600	0.4119
0.0863	0.675	2700	0.4125
0.0722	0.7	2800	0.4189
0.0888	0.725	2900	0.4201
0.0757	0.75	3000	0.4263
0.0666	0.775	3100	0.4234
0.0757	0.8	3200	0.4275
0.053	0.825	3300	0.4289
0.0772	0.85	3400	0.4290
0.0546	0.875	3500	0.4297
0.0601	0.9	3600	0.4311
0.0606	0.925	3700	0.4317
0.0544	0.95	3800	0.4314
0.065	0.975	3900	0.4320
0.0503	1.0	4000	0.4322

Framework versions

• PEFT 0.11.1
• Transformers 4.42.4
• Pytorch 2.4.0a0+07cecf4168.nv24.05
• Datasets 2.20.0
• Tokenizers 0.19.1