Update README.md (#5)
Browse files- Update README.md (2a1691feecb72701f55f748559342674567f0364)
README.md
CHANGED
@@ -23,19 +23,4 @@ The template used to build a prompt for the Instruct model is defined as follows
|
|
23 |
```
|
24 |
Note that `<s>` and `</s>` are special tokens for beginning of string (BOS) and end of string (EOS) while [INST] and [/INST] are regular strings.
|
25 |
|
26 |
-
As reference, here is the pseudo-code used to tokenize instructions during fine-tuning:
|
27 |
-
```python
|
28 |
-
def tokenize(text):
|
29 |
-
return tok.encode(text, add_special_tokens=False)
|
30 |
-
|
31 |
-
[BOS_ID] +
|
32 |
-
tokenize("[INST]") + tokenize(USER_MESSAGE_1) + tokenize("[/INST]") +
|
33 |
-
tokenize(BOT_MESSAGE_1) + [EOS_ID] +
|
34 |
-
…
|
35 |
-
tokenize("[INST]") + tokenize(USER_MESSAGE_N) + tokenize("[/INST]") +
|
36 |
-
tokenize(BOT_MESSAGE_N) + [EOS_ID]
|
37 |
-
```
|
38 |
-
|
39 |
-
In the pseudo-code above, note that the `tokenize` method should not add a BOS or EOS token automatically, but should add a prefix space.
|
40 |
-
|
41 |
## Run the model
|
|
|
23 |
```
|
24 |
Note that `<s>` and `</s>` are special tokens for beginning of string (BOS) and end of string (EOS) while [INST] and [/INST] are regular strings.
|
25 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
26 |
## Run the model
|