Edit model card
YAML Metadata Warning: empty or missing yaml metadata in repo card (https://huggingface.co/docs/hub/model-cards#model-card-metadata)

Quantization made by Richard Erkhov.

Github

Discord

Request more models

SFR-Iterative-DPO-LLaMA-3-8B-R - GGUF

Original model description:

Original Salesforce Readme

I think they had the wrong license, which is why they deleted the original repos. The correct license would be the llama 3 license


license: cc-by-nc-nd-3.0

SFR-Iterative-DPO-Llama-3-8B-R

Introduction

We release a state-of-the-art instruct model of its class, SFR-Iterative-DPO-LLaMA-3-8B-R. On all three widely-used instruct model benchmarks: Alpaca-Eval-V2, MT-Bench, Chat-Arena-Hard, our model outperforms all models of similar size (e.g., LLaMA-3-8B-it), most large open-sourced models (e.g., Mixtral-8x7B-it), and strong proprietary models (e.g., GPT-3.5-turbo-0613). The model is trained with open-sourced datasets without any additional human-/GPT4-labeling.

Model Releases

Training methods

We have developed a simple and efficient online RLHF recipe for LLM instruct training. Our recipe is DPO-based and thus much cheaper and simpler to train and tune compared to PPO-based approaches. Unlike widely-used offline DPO, the online component of our approach effectively mitigates distribution shifts during policy optimization. For a detailed exposition, please refer to our accompanying technical report.

Chat Benchmarks

Model Size Method LC Alpaca-Eval-V2 MT-Bench Chat-Arena-Hard
Small Open-Sourced Models
Gemma-7B-it 7B SFT 10.4 6.38 7.5
Zephyr-7B-beta 7B Vanilla DPO 13.1 7.34 -
Mistral-7B-v0.2-it 7B SFT 17.1 7.51 12.6
Open-Chat-0106 7B SFT 15.6 7.8 -
Starling-7B-beta 7B PPO 25.8 8.12 23.0
LLaMA-3-8B-it 8B RS+DPO+PPO 22.9 8.16 20.6
Ours
Ours (SFT baseline) 8B SFT 10.2 7.69 5.6
Ours (DPO baseline) 8B Vanilla DPO 22.5 8.17 22.4
Ours (Online RLHF) 8B Iterative DPO 37.2 8.46 29.1
Large Open-Sourced Models
Vicuna-33b-v1.3 33B SFT 17.6 7.12 8.6
Yi-34B-Chat 34B SFT 27.2 - 23.1
Mixtral-8x7B-it 45B* SFT 23.7 8.30 23.4
Tulu-2-DPO-70B 70B Vanilla DPO 21.2 7.89 15.0
LLaMA-3-70B-it 70B RS+DPO+PPO 34.4 8.95 41.1
Mixtral-8x22B-it 141B* SFT 30.9 8.66 36.4
Proprietary Models
GPT-3.5-turbo-1106 - - 19.3 8.35 18.9
GPT-3.5-turbo-0613 - - 22.7 8.39 24.8
GPT-4-0613 - - 30.2 9.18 37.9
Claude-3-Opus - - 40.5 9.00 60.4
GPT-4 Turbo (04/09) - - 55.0 - 82.6

Academic Benchmarks

Model Size Method GSM-8K MMLU HumanEval TruthfulQA ARC MBPP
LLaMA-3-8B-it 8B RS+DPO+PPO 79.6 66.0 61.6 43.9 59.5 61.1
Ours (SFT baseline) 8B SFT 74.2 64.7 65.2 53.4 61.4 62.3
Ours (DPO baseline) 8B Vanilla DPO 79.8 64.5 63.4 61.8 65.2 60.3
Ours (Iterative RLHF) 8B Iterative DPO 80.7 65.3 64.6 60.4 64.3 60.8

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

device = "cuda" 

model = AutoModelForCausalLM.from_pretrained("Salesforce/SFR-Iterative-DPO-LLaMA-3-8B-R")
tokenizer = AutoTokenizer.from_pretrained("Salesforce/SFR-Iterative-DPO-LLaMA-3-8B-R")

messages = [
    {"role": "user", "content": "I'm trying to teach myself to have nicer handwriting. Can you help?"},
]

model_inputs = tokenizer.apply_chat_template(messages, return_tensors="pt")

model_inputs = model_inputs.to(device)
model.to(device)

output_tokens = model.generate(model_inputs, max_new_tokens=1024, do_sample=True)
model_outputs = tokenizer.batch_decode(output_tokens)
print(model_outputs[0])

Limitations

SFR-Iterative-DPO-LLaMA-3-8B-R is a research model developed as part of our RLHF initiative at Salesforce. While safety and ethical considerations are integral to our alignment process, there remains the possibility that the model could generate offensive or unethical content, particularly under adversarial conditions. We are committed to continuous improvement in our models to minimize such risks and encourage responsible usage.

Citation

Please cite our papers if you find our models are useful.

@misc{dong2024rlhf,
      title={RLHF Workflow: From Reward Modeling to Online RLHF}, 
      author={Hanze Dong and Wei Xiong and Bo Pang and Haoxiang Wang and Han Zhao and Yingbo Zhou and Nan Jiang and Doyen Sahoo and Caiming Xiong and Tong Zhang},
      year={2024},
      eprint={2405.07863},
      archivePrefix={arXiv},
      primaryClass={cs.LG}
}

@misc{xiong2024iterative,
      title={Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint}, 
      author={Wei Xiong and Hanze Dong and Chenlu Ye and Ziqi Wang and Han Zhong and Heng Ji and Nan Jiang and Tong Zhang},
      year={2024},
      eprint={2312.11456},
      archivePrefix={arXiv},
      primaryClass={cs.LG}
}
Downloads last month
171
GGUF
Model size
8.03B params
Architecture
llama

2-bit

3-bit

4-bit

5-bit

6-bit

8-bit

Inference API
Unable to determine this model's library. Check the docs .