Transformers
PyTorch
English
trl
rlhf
natolambert meg HF staff commited on
Commit
dc028f3
1 Parent(s): 90ee929

Updating to more formal model card. (#6)

Browse files

- Updating to more formal model card. (2b299a537fa87ab5e7acc530abec4ad88409c372)


Co-authored-by: Margaret Mitchell <meg@users.noreply.huggingface.co>

Files changed (1) hide show
  1. README.md +72 -24
README.md CHANGED
@@ -14,40 +14,85 @@ datasets:
14
  ![pull_figure](https://huggingface.co/datasets/trl-internal-testing/example-images/resolve/main/images/stack-llama.png)
15
 
16
  # Llama-se-rl-peft
17
- Adapter weights of an RL fine-tuned model based on LLaMA (see Meta's LLaMA release for the original LLaMA model).
 
18
  For more info check out the [blog post](https://huggingface.co/blog/stackllama) and [github example](https://github.com/lvwerra/trl/tree/main/examples/stack_llama/scripts).
19
- The reward model used to train this model can be found [here](https://huggingface.co/trl-lib/llama-7b-se-rm-peft).
20
 
21
- ## Model Description
22
- **Llama-se-rl** is a Llama-based model that has been first fine-tuned on the Stack Exchange dataset and then RL fine-tuned using a Stack Exchange Reward Model.
23
- This dataset consists of questions and answers from various domains in Stack Exchange, such as programming, mathematics, physics, and more.
24
- The model is designed to generate human-like responses to questions in these domains.
25
- The model has been training to respond to prompts with the following template:
26
 
27
- ```
28
- Question: <Query>
29
 
30
- Answer: <Response>
31
- ```
 
 
 
 
 
 
 
32
 
33
- ## Intended Uses & Limitations
34
- The **Llama-se-rl** model was trained for long form QA using [Stack Exchange](https://stackexchange.com) data wich is released under a [CC-BY-SA 4.0](https://creativecommons.org/licenses/by-sa/4.0/), and covers topics such as programming, mathematics, and physics.
35
- It is intended to demonstrate a Large Language Model's ability to follow a target behavior (in this case, generating answers to a question that would have been rated more highly on SE).
36
- It is not intended to replace human expertise, and answers should be validated through the use of external sources.
37
- Further research is also needed to attribute model generations to sources in the training data, especially in cases where the model may copy answers from the training data *verbatim*.
38
 
39
- ## Limitations and Bias
40
- The **Llama-se-rl** model inherits limitations and biases from the Llama model and also those contained in the Stack Exchange dataset.
41
- In particular, per the [latest developer survey for Stack Overflow](https://survey.stackoverflow.co/2022/),
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
42
  which constitutes a significant part of the StackExchange data,
43
  most users who answered the survey identified themselves as [White or European, men, between 25 and 34 years old, and based in the US (with a significant part of responders from India).](https://survey.stackoverflow.co/2022/#developer-profile-demographics)
44
- While this demographic information likely varies by topic, disparities between the data contributors and the direct and indirect users of the technology should inform developers in assessing what constitutes an appropriate use case.
 
 
45
 
46
- Additionally, the model may generate answers that are incorrect or misleading due to the inherent limitations of the Llama architecture.
 
 
 
47
 
48
- ## BibTeX entry and citation info
49
 
50
- ```bibtex
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
51
  @misc {beeching2023stackllama,
52
  author = { Edward Beeching and
53
  Younes Belkada and
@@ -63,4 +108,7 @@ Additionally, the model may generate answers that are incorrect or misleading du
63
  doi = { 10.57967/hf/0513 },
64
  publisher = { Hugging Face Blog }
65
  }
66
- ```
 
 
 
 
14
  ![pull_figure](https://huggingface.co/datasets/trl-internal-testing/example-images/resolve/main/images/stack-llama.png)
15
 
16
  # Llama-se-rl-peft
17
+ Adapter weights of a Reinforcement Learning fine-tuned model based on the LLaMA model (see [Meta's LLaMA release](https://ai.facebook.com/blog/large-language-model-llama-meta-ai) for the original LLaMA model).
18
+ The model is designed to generate human-like responses to questions in Stack Exchange domains of programming, mathematics, physics, and more.
19
  For more info check out the [blog post](https://huggingface.co/blog/stackllama) and [github example](https://github.com/lvwerra/trl/tree/main/examples/stack_llama/scripts).
 
20
 
21
+ ## Model Details
 
 
 
 
22
 
23
+ ### Model Description
24
+ **Developed by:** Hugging Face
25
 
26
+ **Model type:** An auto-regressive language model based on the transformer architecture, and fine-tuned with [Stack Exchange datasets](https://huggingface.co/datasets/lvwerra/stack-exchange-paired).
27
+
28
+ **Languages:** Predominantly English, with additional data from languages with the following ISO codes:
29
+
30
+ | bg | ca | cs | da | de | es | fr | hr | hu | it | nl | pl | pt | ro | ru | sl | sr | sv | uk |
31
+ | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
32
+
33
+
34
+ **License:** [bigscience-openrail-m](https://drive.google.com/file/d/16NqKiAkzyZ55NClubCIFup8pT2jnyVIo/view?usp=sharing)
35
 
36
+ **Finetuned from:** [LLaMA](https://github.com/facebookresearch/llama/blob/main/MODEL_CARD.md)
 
 
 
 
37
 
38
+ ### Model Sources
39
+ **Repository:** [https://huggingface.co/trl-lib/llama-7b-se-rl-peft/tree/main](https://huggingface.co/trl-lib/llama-7b-se-rl-peft/tree/main)
40
+
41
+ **Base Model Repository:** [https://github.com/facebookresearch/llama](https://github.com/facebookresearch/llama)
42
+
43
+ **Demo:** [https://huggingface.co/spaces/trl-lib/stack-llama](https://huggingface.co/spaces/trl-lib/stack-llama)
44
+
45
+ ## Uses
46
+
47
+ ### Direct Use
48
+ - Long-form question-answering on topics of programming, mathematics, and physics
49
+ - Demonstrating a Large Language Model's ability to follow target behavior of generating answers to a question that would be highly rated on [Stack Exchange](https://stackexchange.com).
50
+
51
+ ### Out of Scope Use
52
+ - Replacing human expertise
53
+
54
+
55
+ ## Bias, Risks, and Limitations
56
+ - Inherits bias, risks, and limitations from the LLaMA model, as described in the [LLaMA Model Card Bias Evaluation](https://github.com/facebookresearch/llama/blob/main/MODEL_CARD.md#quantitative-analysis) and [Ethical Considerations](https://github.com/facebookresearch/llama/blob/main/MODEL_CARD.md#ethical-considerations).
57
+ - Retains biases present in the Stack Exchange dataset. Per the [latest developer survey for Stack Overflow](https://survey.stackoverflow.co/2022/),
58
  which constitutes a significant part of the StackExchange data,
59
  most users who answered the survey identified themselves as [White or European, men, between 25 and 34 years old, and based in the US (with a significant part of responders from India).](https://survey.stackoverflow.co/2022/#developer-profile-demographics)
60
+ - May generate answers that are incorrect or misleading.
61
+ - May copy answers from the training data verbatim.
62
+
63
 
64
+ ### Recommendations
65
+ - Answers should be validated through the use of external sources.
66
+ - Disparities between the data contributors and the direct and indirect users of the technology should inform developers in assessing what constitutes an appropriate use case.
67
+ - Further research is needed to attribute model generations to sources in the training data, especially in cases where the model copies answers from the training data.
68
 
69
+ ## Training Details
70
 
71
+ ### Training Data
72
+ Original datasets are described in [the LLaMA Model Card](https://github.com/facebookresearch/llama/blob/main/MODEL_CARD.md#training-dataset).
73
+ Fine-tuning datasets for this model are based on [Stack Exchange Paired](https://huggingface.co/datasets/lvwerra/stack-exchange-paired), which consists of questions and answers from various domains in Stack Exchange, such as programming, mathematics, physics, and more. Specifically:
74
+
75
+ **Traditional Fine-tuning:** [https://huggingface.co/datasets/lvwerra/stack-exchange-paired/tree/main/data/finetune](https://huggingface.co/datasets/lvwerra/stack-exchange-paired/tree/main/data/finetune)
76
+
77
+ **RL Fine-tuning:** [https://huggingface.co/datasets/lvwerra/stack-exchange-paired/tree/main/data/rl](https://huggingface.co/datasets/lvwerra/stack-exchange-paired/tree/main/data/rl)
78
+
79
+ **Reward Model:** [https://huggingface.co/trl-lib/llama-7b-se-rm-peft](https://huggingface.co/trl-lib/llama-7b-se-rm-peft)
80
+
81
+ ### Training Procedure
82
+ The model was first fine-tuned on the Stack Exchange question and answer pairs and then RL fine-tuned using a Stack Exchange Reward Model.
83
+ It is trained to respond to prompts with the following template:
84
+
85
+ ```
86
+ Question: <Query>
87
+
88
+ Answer: <Response>
89
+ ```
90
+
91
+
92
+ ## Citation
93
+
94
+ **BibTeX:**
95
+ ```
96
  @misc {beeching2023stackllama,
97
  author = { Edward Beeching and
98
  Younes Belkada and
 
108
  doi = { 10.57967/hf/0513 },
109
  publisher = { Hugging Face Blog }
110
  }
111
+ ```
112
+
113
+ ## Model Card Authors
114
+ [Nathan Lambert](https://huggingface.co/natolambert), [Leandro von Werra](https://huggingface.co/lvwerra), [Edward Beeching](https://huggingface.co/edbeeching), [Kashif Rasul](https://huggingface.co/kashif), [Younes Belkada](https://huggingface.co/ybelkada), [Margaret Mitchell](https://huggingface.co/meg)