amy-hyunji-lee
commited on
Commit
•
185dfab
1
Parent(s):
dd31dc0
Update README.md
Browse files
README.md
CHANGED
@@ -1,3 +1,23 @@
|
|
1 |
---
|
2 |
license: cc-by-nc-4.0
|
3 |
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
---
|
2 |
license: cc-by-nc-4.0
|
3 |
---
|
4 |
+
This model is a generation model trained via [semiparametric token-sequence co-supervision](https://github.com/kaistAI/Semiparametric_Token-Sequence_Co-Supervision) on top of Llama2-7B.
|
5 |
+
The embedding model which constructs the nonparametric sequence embedding spaces is in [here](https://huggingface.co/kaist-ai/cosupervision-emb_seq-Llama2_7b).
|
6 |
+
The models are trained on information-seeking datasets provided by [self-rag](https://selfrag.github.io/) with co-supervision from next token prediction (NTP) and next sequence prediction (NSP).
|
7 |
+
In the inference step, the model generates a response by retrieving relevant sequences.
|
8 |
+
See full descriptions in our paper.
|
9 |
+
|
10 |
+
### Usage
|
11 |
+
Here, we show an easy way to quickly download our model from HuggingFace.
|
12 |
+
Make sure to install dependencies listed at requirements.txt.
|
13 |
+
To run our full inference pipeline with embedding model, please use our [code](https://github.com/kaistAI/Semiparametric_Token-Sequence_Co-Supervision).
|
14 |
+
|
15 |
+
```
|
16 |
+
from transformers import AutoTokenizer, LlamaForCausalLM
|
17 |
+
|
18 |
+
model = LlamaForCausalLM.from_pretrained(
|
19 |
+
"kaist-ai/cosupervision-emb_seq-Llama2_7b",
|
20 |
+
load_in_8bit=True if train_config.quantization else None,
|
21 |
+
device_map="auto" if train_config.quantization else None,
|
22 |
+
)
|
23 |
+
```
|