Saibo-creator commited on
Commit
d6e37fb
1 Parent(s): 2fa4473

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +1 -1
README.md CHANGED
@@ -132,7 +132,7 @@ In the Masked Language Model showroom, the tokens have a prefix **Ġ**. This see
132
  I know in case of BPE tokenizer(ROBERTA's tokenizer), the symbol Ġ means the end of a new token and the majority of tokens in vocabs of pre-trained tokenizers start with Ġ.
133
 
134
  For example
135
- ```
136
  import transformers
137
  tokenizer = transformers.RobertaTokenizer.from_pretrained('roberta-base')
138
  print(tokenizer.tokenize('I love salad'))
 
132
  I know in case of BPE tokenizer(ROBERTA's tokenizer), the symbol Ġ means the end of a new token and the majority of tokens in vocabs of pre-trained tokenizers start with Ġ.
133
 
134
  For example
135
+ ```python
136
  import transformers
137
  tokenizer = transformers.RobertaTokenizer.from_pretrained('roberta-base')
138
  print(tokenizer.tokenize('I love salad'))