---
datasets:
  - PKU-Alignment/PKU-SafeRLHF
language:
  - en
tags:
  - reinforcement-learning-from-human-feedback
  - reinforcement-learning
  - beaver
  - safety
  - llama
  - ai-safety
  - deepspeed
  - rlhf
  - alpaca
library_name: safe-rlhf
---

# 🦫 Beaver's Cost Model

## Model Details

The Beaver Cost model is a preference model trained using the [PKU-SafeRLHF](https://huggingface.co/datasets/PKU-Alignment/PKU-SafeRLHF) dataset.
It can play a role in the safe RLHF algorithm, helping the Beaver model become more safe and harmless.

- **Developed by:** the [PKU-Alignment](https://github.com/PKU-Alignment) Team.
- **Model Type:** An auto-regressive language model based on the transformer architecture.
- **License:** Non-commercial license.
- **Fine-tuned from model:** [LLaMA](https://arxiv.org/abs/2302.13971), [Alpaca](https://github.com/tatsu-lab/stanford_alpaca).

## Model Sources

- **Repository:** <https://github.com/PKU-Alignment/safe-rlhf>
- **Beaver:** <https://huggingface.co/PKU-Alignment/beaver-7b-v1.0>
- **Dataset:** <https://huggingface.co/datasets/PKU-Alignment/PKU-SafeRLHF>
- **Reward Model:** <https://huggingface.co/PKU-Alignment/beaver-7b-v1.0-reward>
- **Cost Model:** <https://huggingface.co/PKU-Alignment/beaver-7b-v1.0-cost>
- **Dataset Paper:** <https://arxiv.org/abs/2307.04657>
- **Paper:** *Coming soon...*

## How to Use the Cost Model

```python
from transformers import AutoTokenizer
from safe_rlhf.models import AutoModelForScore

model = AutoModelForScore.from_pretrained('PKU-Alignment/beaver-7b-v1.0-cost', device_map='auto')
tokenizer = AutoTokenizer.from_pretrained('PKU-Alignment/beaver-7b-v1.0-cost', use_fast=False)

input = 'BEGINNING OF CONVERSATION: USER: hello ASSISTANT:Hello! How can I help you today?'

input_ids = tokenizer(input, return_tensors='pt')
output = model(**input_ids)
print(output)

# ScoreModelOutput(
#     scores=tensor([[[-19.6476],
#         [-20.2238],
#         [-21.4228],
#         [-19.2506],
#         [-20.2728],
#         [-23.8799],
#         [-22.6898],
#         [-21.5825],
#         [-21.0855],
#         [-20.2068],
#         [-23.8296],
#         [-21.4940],
#         [-21.9484],
#         [-13.1220],
#         [ -6.4499],
#         [ -8.1982],
#         [ -7.2492],
#         [ -9.3377],
#         [-13.5010],
#         [-10.4932],
#         [ -9.7837],
#         [ -6.4540],
#         [ -6.0084],
#         [ -5.8093],
#         [ -6.6134],
#         [ -5.8995],
#         [ -9.1505],
#         [-11.3254]]], grad_fn=<ToCopyBackward0>),
#     end_scores=tensor([[-11.3254]], grad_fn=<ToCopyBackward0>)
# )
```