Llama2-13B-SteerLM-RM

License

The use of this model is governed by the Llama 2 Community License Agreement

Description:

Llama2-13B-SteerLM-RM is a 13 billion parameter language model (with context of up to 4,096 tokens) used as the Attribute Prediction Model in training Llama2-70B-SteerLM-Chat

Attribute Prediction Model is a multi-aspect Reward Model that rates model responses on various aspects that makes a response desirable instead of a singular score in a conventional Reward Model.

Given a conversation with multiple turns between user and assistant, it rates the following attributes (between 0 and 4) for every assistant turn.

1. Quality: Perceived goodness of response
2. Toxicity: Undesirable elements such as vulgar, harmful or potentially biased response
3. Humor: Sense of humor within response
4. Creativity: Willingness to generate non-conventional response
5. Helpfulness: Overall helpfulness of the response to the prompt.
6. Correctness: Inclusion of all pertinent facts without errors.
7. Coherence: Consistency and clarity of expression.
8. Complexity: Intellectual depth required to write response (i.e. whether the response can be written by anyone with basic language competency or requires deep domain expertise).
9. Verbosity: Amount of detail included in the response, relative to what is asked for in the prompt.

The first four attributes are taken from the Open Assistant dataset while the others are taken from HelpSteer dataset

HelpSteer Paper : HelpSteer: Multi-attribute Helpfulness Dataset for SteerLM

SteerLM Paper: SteerLM: Attribute Conditioned SFT as an (User-Steerable) Alternative to RLHF

Llama2-13B-SteerLM-RM is trained with NVIDIA NeMo-Aligner, a scalable toolkit for performant and efficient model alignment. NeMo-Aligner is built using the NeMo Framework which allows for scaling training up to 1000s of GPUs using tensor, data and pipeline parallelism for all components of alignment. All of our checkpoints are cross compatible with the NeMo ecosystem, allowing for inference deployment and further customization.

Usage:

You can use the model with NeMo Aligner following SteerLM training user guide.

This model can be useful to train a model like Llama2-70B-SteerLM-Chat or annotate the attributes for any conversation.

1. Spin up an inference server within the NeMo Aligner container

python /opt/NeMo-Aligner/examples/nlp/gpt/serve_reward_model.py \
      rm_model_file=Llama2-13B-SteerLM-RM.nemo \
      trainer.num_nodes=1 \
      trainer.devices=8 \
      ++model.tensor_model_parallel_size=4 \
      ++model.pipeline_model_parallel_size=1 \
      inference.micro_batch_size=2 \
      inference.port=1424

2. Annotate data files using the served reward model. If you are seeking to reproduce training of Llama2-70B-SteerLM-Chat, this will be the Open Assistant train/val files. Then follow the next step to train a SteerLM model based on SteerLM training user guide .

python /opt/NeMo-Aligner/examples/nlp/data/steerlm/preprocess_openassistant_data.py --output_directory=data/oasst

python /opt/NeMo-Aligner/examples/nlp/data/steerlm/attribute_annotate.py \
      --input-file=data/oasst/train.jsonl \
      --output-file=data/oasst/train_labeled.jsonl \
      --port=1424

3. Alternatively, this can be any conversational data file (in .jsonl) in the following format, where each line looks like

{
    "conversations": [
              {"value": <user_turn_1>, "from": "User", "label": None},
              {"value": <assistant_turn_1>, "from": "Assistant", "label": <formatted_label_1>},
              {"value": <user_turn_2>, "from": "User", "label": None},
              {"value": <assistant_turn_2>, "from": "Assistant", "label": <formatted_label_2>},
          ],
    "mask": "User"
}

Ideally, each <formatted_label_n> refers to the ground truth label for the assistant turn but if they are not available, we can also use quality:4,toxicity:0,humor:0,creativity:0,helpfulness:4,correctness:4,coherence:4,complexity:4,verbosity:4

Contact

E-Mail: Zhilin Wang

Citation

If you find this dataset useful, please cite the following works

@misc{wang2023helpsteer,
      title={HelpSteer: Multi-attribute Helpfulness Dataset for SteerLM}, 
      author={Zhilin Wang and Yi Dong and Jiaqi Zeng and Virginia Adams and Makesh Narsimhan Sreedhar and Daniel Egert and Olivier Delalleau and Jane Polak Scowcroft and Neel Kant and Aidan Swope and Oleksii Kuchaiev},
      year={2023},
      eprint={2311.09528},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

@misc{dong2023steerlm,
      title={SteerLM: Attribute Conditioned SFT as an (User-Steerable) Alternative to RLHF}, 
      author={Yi Dong and Zhilin Wang and Makesh Narsimhan Sreedhar and Xianchao Wu and Oleksii Kuchaiev},
      year={2023},
      eprint={2310.05344},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}