Update README.md

README.md

@@ -7,49 +7,46 @@ tags:
 - openchat
 - RLAIF
 - reward model
+language:
+- en
+base_model: openchat/openchat-3.5-0106
+datasets:
+- berkeley-nest/Nectar
 ---
 
 # Storm-7B
 - **Developed by**: [Jie Liu](https://jieliu.site/) \\(^{*1,2}\\), [Zhanhui Zhou](https://scholar.google.com/citations?user=SbACfYQAAAAJ&hl=zh-CN) \\(^{*2}\\), [Jiaheng Liu](https://liujiaheng.github.io/) \\(^{2}\\), [Xingyuan Bu](https://scholar.google.com.hk/citations?user=cqYaRhUAAAAJ&hl=zh-CN) \\(^{2}\\), [Chao Yang](https://scholar.google.com/citations?user=5KRbHPMAAAAJ&hl=zh-CN) \\(^{2}\\), [Han-Sen Zhong](https://scholar.google.com.hk/citations?user=X_ZfX8sAAAAJ&hl=zh-CN) \\(^{\dag 2}\\), [Wanli Ouyang](https://wlouyang.github.io/) \\(^{1,2}\\).
 - \\(^{1}\\)MMLab, The Chinese University of Hong Kong    \\(^{2}\\)Shanghai AI Laboratory
+- Paper: [Iterative Length-Regularized Direct Preference Optimization: A Case Study on Improving 7B Language Models to GPT-4 Level]()
+- Finetuned from model: [openchat-3.5-0106](https://huggingface.co/openchat/openchat-3.5-0106)
+- Dataset: [berkeley-nest/Nectar](https://huggingface.co/datasets/berkeley-nest/Nectar)
+- Reward Model: [Starling-RM-34B](https://huggingface.co/Nexusflow/Starling-RM-34B)
+
+Please see our paper for more details.
 
 ## Introduction
 
 We released Storm-7B, the first open-source language model comparable to the GPT-4 series on the [AlpacaEval 2.0](https://tatsu-lab.github.io/alpaca_eval/) leaderboard.
 
-Recent studies show that DPO benefits from iterative training with online preferences labeled by a trained reward model. In this work, we identify a pitfall of vanilla iterative DPO - improved response quality can lead to increased verbosity. To address this, we introduce iterative length-regularized DPO (iLR-DPO) to penalize response length. Our empirical results show that iLR-DPO can enhance a 7B model to perform on par with GPT-4 without increasing verbosity. 
-
-A snapshot of the AlpacaEval 2.0 leaderboard (Single Model, 2024/6/18) is listed below: 
-
-|                          | **LC Win Rate** | **Win Rate** |
-| :----------------------: | :-------------: | :----------: |
-|   GPT-4 Turbo (04/09)    |      55.0%      |    46.1%     |
-|   GPT-4 Turbo (04/09)    |      55.0%      |    46.1%     |
-|   GPT-4 Turbo (04/09)    |      55.0%      |    46.1%     |
-|   GPT-4 Turbo (04/09)    |      55.0%      |    46.1%     |
-|  GPT-4 Preview (11/06)   |      50.0%      |    50.0%     |
-|       **Storm-7B**       |      48.9%      |    52.5%     |
-| Nanbeige Plus Chat v0.1  |      44.5%      |    56.7%     |
-|    Qwen1.5 110B Chat     |      43.9%      |    33.8%     |
-| Aligner 2B+Claude 3 Opus |      41.8%      |    34.5%     |
-|  Claude 3 Opus (02/29)   |      40.5%      |    29.1%     |
-|          GPT-4           |      38.1%      |    23.6%     |
-|    openchat-3.5-0106     |      15.4%      |    10.1%     |
-
-Please refer to the [leaderboard webpage](https://tatsu-lab.github.io/alpaca_eval/) for up-to-date results. 
-
-We also conducted preliminary evaluations on other benchmarks and observed no significant degradation.
-
-|                   | ARC   | HellaSwag | MMLU  | TruthfulQA | Winogrande | Avg.  |
-| ----------------- | ----- | --------- | ----- | ---------- | ---------- | ----- |
-| **Storm-7B**      | 67.58 | 80.97     | 62.21 | 57.24      | 80.51      | 69.70 |
-| openchat-3.5-0106 | 66.38 | 83.00     | 63.47 | 52.55      | 81.06      | 69.29 |
-| internlm2-7b      | 58.02 | 81.24     | 65.24 | 48.73      | 83.82      | 67.41 |
-| gemma-7B          | 61.09 | 82.20     | 64.56 | 44.79      | 79.01      | 66.33 |
-| Yi-9B             | 61.18 | 78.82     | 70.06 | 42.45      | 77.51      | 66.00 |
-| Meta-Llama-3-8B   | 59.47 | 82.09     | 66.69 | 43.90      | 77.35      | 65.90 |
-| Mistral-7B-v0.1   | 59.98 | 83.31     | 64.16 | 42.15      | 78.37      | 65.59 |
-| Qwen-7b           | 51.37 | 78.47     | 59.84 | 47.79      | 72.69      | 62.03 |
+Recent studies show that DPO benefits from iterative training with online preferences labeled by a trained reward model. In this work, we identify a pitfall of vanilla iterative DPO - improved response quality can lead to increased verbosity. To address this, we introduce iterative length-regularized DPO (iLR-DPO) to penalize response length. Our empirical results show that iLR-DPO can enhance a 7B model to perform on par with GPT-4 **without increasing verbosity**. 
+
+## Performance
+Our 7B model achieves a **50.5%** length-controlled win rate against GPT-4 Preview on AlpacaEval 2.0.
+<p align="center">
+    <img src="https://cdn-uploads.huggingface.co/production/uploads/639be86b59473c6ae02ef9c4/Tj_a1QntAxkhy2SXbOdmT.png" width="30%">
+</p>
+Our model's LC win rate improves over iterations without significantly changing the response length, indicating better alignment with human values without length bias. The final trained model (iteration 3) achieves a 50.5% LC win rate, making it the first open-source model to surpass the baseline model GPT-4 Preview.
+
+In addition to regular decoding, we also test beam search and best-of-n sampling on top of our trained model. Beam search over our trained model shows a 5% improvement over regular decoding, Best-of-n sampling with Starling-RM-34B achieves 61.6% LC Win rate and outperforms GPT-4 Omni.
+<p align="center">
+    <img src="https://cdn-uploads.huggingface.co/production/uploads/639be86b59473c6ae02ef9c4/GGa28vaREaVq099MPdqcP.png" width="50%">
+</p>
+
+We observe no significant degradation in traditional NLP tasks from the Huggingface Open LLM Leaderboard.
+<p align="center">
+    <img src="https://cdn-uploads.huggingface.co/production/uploads/639be86b59473c6ae02ef9c4/8KEm_Ladg7Kqko8mC63SN.png" width="50%">
+</p>
+
 
 ## Uses
 
@@ -96,16 +93,25 @@ alpaca_eval evaluate_from_model --model_configs 'Storm-7B'
 
 ## Limitations
 
-Storm-7B is a quick demonstration that a language model, fine-tuned with AI feedback, can easily surpass or match state-of-the-art models, as assessed by the same AI feedback. However, this improvement on the automatic leaderboard may not necessarily indicate better alignment with human intentions. Our model therefore represents a critical, preliminary reevaluation of the RLAIF paradigm, questioning how much learning from and being evaluated by AI feedback aligns with actual human preferences.
+Our work has several limitations:
+(1) We focus on aligning with human preferences but only use GPT-4 as a proxy for human judgment to evaluate language models. 
+(2) We reduce verbosity with a length penalty, though verbosity and length are not necessarily correlated. Future work could train a specific reward model to directly penalize verbosity, replacing the length margin with a verbosity margin, following the standard [MODPO pipeline](https://github.com/ZHZisZZ/modpo).
 
 ## Citation
 
 ```
 @misc{liu2024storm,
-    title = {Storm-7B: An Empirical Study of Iterative Direct Preference Optimization},
+    title = {Iterative Length-Regularized Direct Preference Optimization: A Case Study on Improving 7B Language Models to GPT-4 Level},
     url = {},
-    author = {Jie Liu and Zhanhui Zhou and Chao Yang and Han-Sen Zhong and Wanli Ouyang},
-    month = {April},
+    author = {Liu, Jie and Zhou, Zhanhui and Liu, Jiaheng and Bu, Xingyuan and Yang, Chao and Zhong Han-Sen and Ouyang, Wanli},
+    month = {June},
     year = {2024}
 }
-```
+
+@article{zhou2023beyond,
+  title={Beyond one-preference-for-all: Multi-objective direct preference optimization},
+  author={Zhou, Zhanhui and Liu, Jie and Yang, Chao and Shao, Jing and Liu, Yu and Yue, Xiangyu and Ouyang, Wanli and Qiao, Yu},
+  journal={arXiv preprint arXiv:2310.03708},
+  year={2023}
+}
+```