Saibo-creator
commited on
Commit
•
d6e37fb
1
Parent(s):
2fa4473
Update README.md
Browse files
README.md
CHANGED
@@ -132,7 +132,7 @@ In the Masked Language Model showroom, the tokens have a prefix **Ġ**. This see
|
|
132 |
I know in case of BPE tokenizer(ROBERTA's tokenizer), the symbol Ġ means the end of a new token and the majority of tokens in vocabs of pre-trained tokenizers start with Ġ.
|
133 |
|
134 |
For example
|
135 |
-
```
|
136 |
import transformers
|
137 |
tokenizer = transformers.RobertaTokenizer.from_pretrained('roberta-base')
|
138 |
print(tokenizer.tokenize('I love salad'))
|
|
|
132 |
I know in case of BPE tokenizer(ROBERTA's tokenizer), the symbol Ġ means the end of a new token and the majority of tokens in vocabs of pre-trained tokenizers start with Ġ.
|
133 |
|
134 |
For example
|
135 |
+
```python
|
136 |
import transformers
|
137 |
tokenizer = transformers.RobertaTokenizer.from_pretrained('roberta-base')
|
138 |
print(tokenizer.tokenize('I love salad'))
|