File size: 8,937 Bytes

---
license: apache-2.0
datasets:
- openai/summarize_from_feedback
- openai/webgpt_comparisons
- berkeley-nest/Nectar
- Dahoas/instruct-synthetic-prompt-responses
- Anthropic/hh-rlhf
- lmsys/chatbot_arena_conversations
- openbmb/UltraFeedback
- argilla/ultrafeedback-binarized-preferences-cleaned
metrics:
- accuracy
tags:
- reward_model
- reward-model
- RLHF
- evaluation
- llm
- instruction
- reranking
language:
- en
---
# Better Implementation of [*PairRM*](https://huggingface.co/llm-blender/PairRM)

## Introduction

This version of PairRM have some fixes on training process, which improve model's performance by **15%**.

### Minor Fixes

- Longer Context Length (2048 -> 3370)

Thanks to deberta's tokenzer, original PairRM model had enough Context Length.

But, the longer the better :>

---

### Major Fixes

- Change Prompt Format

Why use something like
```
<Response i + 1> {response}
```

So, I changed to a format based on Vicuna 1.1.

---

- Change Truncate side

The original process was using right side truncate even on Input. This can cause serious problem when Input exceeds model's context length.

---

- Dataset Filter

There was decent amount of empty assistant response on original dataset. So, I dropped them.

---

## Example Code

**The code below is modified from** (**PairRM-hf Repo**)[https://huggingface.co/llm-blender/PairRM-hf]

```python
import os
os.environ["CUDA_VISIBLE_DEVICES"] = "0"
from llm_blender.pair_ranker.pairrm import DebertaV2PairRM
from transformers import AutoTokenizer
from typing import List
pairrm = DebertaV2PairRM.from_pretrained("maywell/Better-PairRM", device_map="cuda:0").eval()
tokenizer = AutoTokenizer.from_pretrained("maywell/Better-PairRM")
source_prefix = "<|source|>"
cand1_prefix = "<|candidate1|>"
cand2_prefix = "<|candidate2|>"
inputs = ["hello!", "I love you!"]
candidates_A = ["hi!", "I hate you!"]
candidates_B = ["f**k off!", "I love you, too!"]
def tokenize_pair(sources:List[str], candidate1s:List[str], candidate2s:List[str], source_max_length=2030, candidate_max_length=670):
    ids = []
    assert len(sources) == len(candidate1s) == len(candidate2s)
    max_length = source_max_length + 2 * candidate_max_length
    for i in range(len(sources)):
        source_ids = tokenizer.encode(source_prefix + sources[i], max_length=source_max_length, truncation=True)
        candidate_max_length = (max_length - len(source_ids)) // 2
        candidate1_ids = tokenizer.encode(cand1_prefix + candidate1s[i], max_length=candidate_max_length, truncation=True)
        candidate2_ids = tokenizer.encode(cand2_prefix + candidate2s[i], max_length=candidate_max_length, truncation=True)
        ids.append(source_ids + candidate1_ids + candidate2_ids)
    encodings = tokenizer.pad({"input_ids": ids}, return_tensors="pt", padding="max_length", max_length=max_length)
    return encodings

encodings = tokenize_pair(inputs, candidates_A, candidates_B)
encodings = {k:v.to(pairrm.device) for k,v in encodings.items()}
outputs = pairrm(**encodings)
logits = outputs.logits.tolist()
comparison_results = outputs.logits > 0
print(logits)
print(comparison_results)
```

You can also easily compare two conversations like the followings:
```python
import jinja2
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("microsoft/deberta-v3-large")

def truncate_texts(text, max_length, truncate_side):
    tokenizer.truncation_side = truncate_side
    tokens = tokenizer.encode(text, add_special_tokens=False, max_length=max_length)
    truncated_text = tokenizer.decode(tokens, skip_special_tokens=True)
    return truncated_text

MY_JINJA_TEMPLATE = """{% for message in messages -%}
{% if message['role'] == 'user' -%}
USER: {{ message['content']|trim -}}
{% if not loop.last -%}


{% endif %}
{% elif message['role'] == 'assistant' -%}
ASSISTANT: {{ message['content']|trim -}}
{% if not loop.last -%}


{% endif %}
{% elif message['role'] == 'user_context' -%}
USER: {{ message['content']|trim -}}
{% if not loop.last -%}


{% endif %}
{% elif message['role'] == 'system' -%}
SYSTEM MESSAGE: {{ message['content']|trim -}}
{% if not loop.last -%}


{% endif %}
{% endif %}
{% endfor -%}
{% if add_generation_prompt and messages[-1]['role'] != 'assistant' -%}
ASSISTANT: {% endif -%}"""

my_jinja2_env = jinja2.Environment()
my_jinja2_template = my_jinja2_env.from_string(MY_JINJA_TEMPLATE)

def tokenize_conv_pair(convAs: List[str], convBs: List[str]):

    # check conversations correctness
    assert len(convAs) == len(convBs), "Number of conversations must be the same"
    for c_a, c_b in zip(convAs, convBs):
        assert len(c_a) == len(c_b), "Number of turns in each conversation must be the same"
        assert all([c_a[i]['content'] == c_b[i]['content'] for i in range(0, len(c_a), 2)]), "USER turns must be the same"
    
    inputs = [
        truncate_texts(my_jinja2_template.render(messages=x[:-1], add_generation_prompt=True), 2030, "left") for x in convAs
    ]
    cand1_texts = [
        truncate_texts(x[-1]['content'], 670, "right") for x in convAs
    ]
    cand2_texts = [
        truncate_texts(x[-1]['content'], 670, "right") for x in convBs
    ]
    encodings = tokenize_pair(inputs, cand1_texts, cand2_texts)
    return encodings
```

## Statistics

### Context length
|  PairRanker type  | Source max length | Candidate max length | Total max length |
|:-----------------:|:-----------------:|----------------------|------------------|
| [pair-ranker](https://huggingface.co/llm-blender/pair-ranker)             | 128               | 128                  | 384              |
| [PairRM](https://huggingface.co/llm-blender/pair-reward-model/) | 1224              | 412                  | 2048             |
| [Better-PairRM](https://huggingface.co/maywell/Better-PairRM/) (This model) | 2030              | 670                  | 3370             |

### Performance

#### Reward-Bench by AllenAI

| Metric                     | llm-blender/PairRM-hf  | maywell/Better-PairRM  |
|----------------------------|------------------------|------------------------|
| model                      | llm-blender/PairRM-hf  | maywell/Better-PairRM  |
| model_type                 | Custom Classifier      | Custom Classifier      |
| alpacaeval-length          | 0.758                  | **0.863**                |
| alpacaeval-hard            | 0.979                  | **1.000**                  |
| alpacaeval-easy            | 0.970                  | **0.990**                  |
| donotanswer                | 0.360                  | **0.522**                  |
| hep-cpp                    | 0.628                  | **0.646**                  |
| hep-go                     | 0.689                  | **0.713**                  |
| hep-java                   | 0.628                  | **0.713**                  |
| hep-js                     | 0.604                  | **0.707**                  |
| hep-python                 | 0.646                  | **0.713**                  |
| hep-rust                   | 0.652                  | **0.726**                  |
| llmbar-adver-GPTInst       | **0.304**                  | 0.141                  |
| llmbar-adver-GPTOut        | **0.596**                  | 0.447                  |
| llmbar-adver-manual        | **0.500**                  | 0.261                  |
| llmbar-adver-neighbor      | **0.433**                  | 0.276                  |
| llmbar-natural             | **0.800**                  | 0.720                  |
| math-prm                   | **0.333**                  | 0.295                  |
| mt-bench-hard              | 0.649                  | **0.703**                  |
| mt-bench-med               | 0.900                  | **1.000**                  |
| mt-bench-easy              | **0.964**                  | 0.929                  |
| refusals-dangerous         | 0.080                  | **0.730**                  |
| refusals-offensive         | 0.010                  | **0.940**                  |
| xstest-should-refuse       | 0.370                  | **0.968**                  |
| xstest-should-respond      | **0.952**                  | 0.876                  |
| average                    | 0.600                  | **0.690**                  |

> *Note - llmbar test score is bit weird across all models on [Reward-Bench](https://huggingface.co/spaces/allenai/reward-bench)*

## Thanks to

- [Sionic AI](https://sionic.ai/) for providing the A100 cluster.

## Contact

- [Discord Server Link](https://discord.gg/MrBt3PXdXc)

## Original Paper 
```
@inproceedings{llm-blender-2023,
    title = "LLM-Blender: Ensembling Large Language Models with Pairwise Comparison and Generative Fusion",
    author = "Jiang, Dongfu and Ren, Xiang and Lin, Bill Yuchen",
    booktitle = "Proceedings of the 61th Annual Meeting of the Association for Computational Linguistics (ACL 2023)",
    year = "2023"
}
```