qwen_ce_entropy

This model is a fine-tuned version of trl-lib/qwen1.5-0.5b-sft on the yakazimir/ultrafeedback_binarized dataset. It achieves the following results on the evaluation set:

• Loss: 1.2625
• Rewards/chosen: -1.2622
• Rewards/rejected: -1.3864
• Rewards/accuracies: 0.5475
• Rewards/margins: 0.1242
• Logps/rejected: -1.3864
• Logps/chosen: -1.2622
• Logits/rejected: 0.1431
• Logits/chosen: 0.0760

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 1e-06
• train_batch_size: 2
• eval_batch_size: 4
• seed: 42
• distributed_type: multi-GPU
• gradient_accumulation_steps: 16
• total_train_batch_size: 32
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 3.0

Training results

Training Loss	Epoch	Step	Validation Loss	Rewards/chosen	Rewards/rejected	Rewards/accuracies	Rewards/margins	Logps/rejected	Logps/chosen	Logits/rejected	Logits/chosen
1.2903	0.2141	400	1.3234	-1.3231	-1.4418	0.5556	0.1187	-1.4418	-1.3231	0.3478	0.2657
1.2586	0.4282	800	1.2926	-1.2924	-1.4167	0.5482	0.1243	-1.4167	-1.2924	0.3140	0.2391
1.217	0.6422	1200	1.2836	-1.2833	-1.4047	0.5475	0.1213	-1.4047	-1.2833	0.2906	0.2178
1.299	0.8563	1600	1.2774	-1.2772	-1.3985	0.5467	0.1213	-1.3985	-1.2772	0.2371	0.1683
1.2617	1.0704	2000	1.2726	-1.2724	-1.3958	0.5482	0.1234	-1.3958	-1.2724	0.1842	0.1180
1.1894	1.2845	2400	1.2689	-1.2687	-1.3924	0.5460	0.1238	-1.3924	-1.2687	0.1212	0.0586
1.2779	1.4986	2800	1.2662	-1.2659	-1.3880	0.5453	0.1221	-1.3880	-1.2659	0.1199	0.0573
1.225	1.7127	3200	1.2650	-1.2647	-1.3872	0.5490	0.1225	-1.3872	-1.2647	0.1854	0.1171
1.1621	1.9267	3600	1.2636	-1.2634	-1.3853	0.5475	0.1219	-1.3853	-1.2634	0.1551	0.0880
1.1565	2.1408	4000	1.2633	-1.2631	-1.3880	0.5482	0.1250	-1.3880	-1.2631	0.0952	0.0325
1.1515	2.3549	4400	1.2629	-1.2626	-1.3868	0.5467	0.1242	-1.3868	-1.2626	0.0880	0.0251
1.1364	2.5690	4800	1.2625	-1.2623	-1.3865	0.5467	0.1242	-1.3865	-1.2623	0.1292	0.0630
1.1256	2.7831	5200	1.2626	-1.2623	-1.3864	0.5475	0.1241	-1.3864	-1.2623	0.1208	0.0553
1.1655	2.9972	5600	1.2625	-1.2622	-1.3864	0.5475	0.1242	-1.3864	-1.2622	0.1431	0.0760

Framework versions

• Transformers 4.44.2
• Pytorch 2.2.2+cu121
• Datasets 2.18.0
• Tokenizers 0.19.1