--- license: llama3 library_name: peft tags: - alignment-handbook - trl - orpo - generated_from_trainer base_model: meta-llama/Meta-Llama-3-8B-Instruct datasets: - HuggingFaceH4/ultrafeedback_binarized model-index: - name: Meta-Llama-3-8B-Instruct-ORPO-QLoRA results: [] --- [Visualize in Weights & Biases](https://wandb.ai/statking/huggingface/runs/5h649ptl) # Meta-Llama-3-8B-Instruct-ORPO-QLoRA This model is a fine-tuned version of [meta-llama/Meta-Llama-3-8B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct) on the HuggingFaceH4/ultrafeedback_binarized dataset. It achieves the following results on the evaluation set: - Loss: 0.5734 - Rewards/chosen: -0.0085 - Rewards/rejected: -0.0105 - Rewards/accuracies: 0.6070 - Rewards/margins: 0.0020 - Logps/rejected: -1.0492 - Logps/chosen: -0.8470 - Logits/rejected: -0.2321 - Logits/chosen: -0.2275 - Nll Loss: 0.5669 - Log Odds Ratio: -0.6615 - Log Odds Chosen: 0.3163 ## Model description More information needed ## Intended uses & limitations More information needed ## Training and evaluation data More information needed ## Training procedure ### Training hyperparameters The following hyperparameters were used during training: - learning_rate: 7e-06 - train_batch_size: 4 - eval_batch_size: 8 - seed: 42 - distributed_type: multi-GPU - num_devices: 2 - gradient_accumulation_steps: 4 - total_train_batch_size: 32 - total_eval_batch_size: 16 - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08 - lr_scheduler_type: cosine - lr_scheduler_warmup_ratio: 0.1 - num_epochs: 1 ### Training results | Training Loss | Epoch | Step | Validation Loss | Rewards/chosen | Rewards/rejected | Rewards/accuracies | Rewards/margins | Logps/rejected | Logps/chosen | Logits/rejected | Logits/chosen | Nll Loss | Log Odds Ratio | Log Odds Chosen | |:-------------:|:------:|:----:|:---------------:|:--------------:|:----------------:|:------------------:|:---------------:|:--------------:|:------------:|:---------------:|:-------------:|:--------:|:--------------:|:---------------:| | 0.8633 | 0.0524 | 100 | 0.7181 | -0.0135 | -0.0158 | 0.6060 | 0.0023 | -1.5779 | -1.3476 | -0.4503 | -0.4466 | 0.7126 | -0.6965 | 0.2913 | | 0.7831 | 0.1048 | 200 | 0.6487 | -0.0105 | -0.0125 | 0.6140 | 0.0020 | -1.2499 | -1.0520 | -0.3621 | -0.3619 | 0.6432 | -0.6627 | 0.2691 | | 0.7146 | 0.1572 | 300 | 0.6238 | -0.0102 | -0.0122 | 0.6140 | 0.0020 | -1.2194 | -1.0173 | -0.3196 | -0.3169 | 0.6181 | -0.6594 | 0.2790 | | 0.7361 | 0.2096 | 400 | 0.6137 | -0.0100 | -0.0120 | 0.6140 | 0.0020 | -1.2012 | -1.0014 | -0.2841 | -0.2811 | 0.6078 | -0.6618 | 0.2770 | | 0.7382 | 0.2620 | 500 | 0.6066 | -0.0099 | -0.0119 | 0.6120 | 0.0020 | -1.1884 | -0.9868 | -0.3023 | -0.2982 | 0.6006 | -0.6603 | 0.2812 | | 0.7339 | 0.3143 | 600 | 0.6009 | -0.0097 | -0.0118 | 0.6100 | 0.0020 | -1.1751 | -0.9714 | -0.2544 | -0.2490 | 0.5948 | -0.6587 | 0.2859 | | 0.7133 | 0.3667 | 700 | 0.5968 | -0.0096 | -0.0116 | 0.6070 | 0.0020 | -1.1590 | -0.9588 | -0.2830 | -0.2764 | 0.5906 | -0.6590 | 0.2828 | | 0.6988 | 0.4191 | 800 | 0.5926 | -0.0095 | -0.0115 | 0.6070 | 0.0020 | -1.1491 | -0.9451 | -0.2817 | -0.2745 | 0.5864 | -0.6576 | 0.2898 | | 0.7493 | 0.4715 | 900 | 0.5882 | -0.0093 | -0.0114 | 0.6080 | 0.0021 | -1.1357 | -0.9301 | -0.2547 | -0.2476 | 0.5820 | -0.6552 | 0.2952 | | 0.7022 | 0.5239 | 1000 | 0.5842 | -0.0091 | -0.0111 | 0.6070 | 0.0020 | -1.1110 | -0.9090 | -0.2588 | -0.2514 | 0.5780 | -0.6569 | 0.2962 | | 0.6805 | 0.5763 | 1100 | 0.5807 | -0.0089 | -0.0108 | 0.6020 | 0.0020 | -1.0833 | -0.8865 | -0.2590 | -0.2519 | 0.5744 | -0.6608 | 0.2937 | | 0.6427 | 0.6287 | 1200 | 0.5780 | -0.0087 | -0.0107 | 0.6070 | 0.0020 | -1.0670 | -0.8682 | -0.2483 | -0.2430 | 0.5717 | -0.6609 | 0.3024 | | 0.6762 | 0.6811 | 1300 | 0.5762 | -0.0086 | -0.0106 | 0.6070 | 0.0020 | -1.0576 | -0.8586 | -0.2376 | -0.2322 | 0.5698 | -0.6618 | 0.3069 | | 0.6944 | 0.7335 | 1400 | 0.5750 | -0.0085 | -0.0105 | 0.6070 | 0.0020 | -1.0548 | -0.8542 | -0.2468 | -0.2420 | 0.5686 | -0.6609 | 0.3102 | | 0.6695 | 0.7859 | 1500 | 0.5742 | -0.0085 | -0.0105 | 0.6080 | 0.0020 | -1.0505 | -0.8493 | -0.2426 | -0.2372 | 0.5678 | -0.6616 | 0.3135 | | 0.7258 | 0.8382 | 1600 | 0.5738 | -0.0085 | -0.0105 | 0.6080 | 0.0020 | -1.0497 | -0.8485 | -0.2418 | -0.2371 | 0.5673 | -0.6619 | 0.3140 | | 0.7193 | 0.8906 | 1700 | 0.5735 | -0.0085 | -0.0105 | 0.6050 | 0.0020 | -1.0499 | -0.8477 | -0.2403 | -0.2352 | 0.5671 | -0.6610 | 0.3162 | | 0.7038 | 0.9430 | 1800 | 0.5734 | -0.0085 | -0.0105 | 0.6090 | 0.0020 | -1.0493 | -0.8471 | -0.2360 | -0.2311 | 0.5670 | -0.6615 | 0.3164 | | 0.6723 | 0.9954 | 1900 | 0.5734 | -0.0085 | -0.0105 | 0.6070 | 0.0020 | -1.0493 | -0.8470 | -0.2369 | -0.2320 | 0.5669 | -0.6615 | 0.3168 | ### Framework versions - PEFT 0.11.1 - Transformers 4.41.0 - Pytorch 2.3.0+cu121 - Datasets 2.19.1 - Tokenizers 0.19.1