oyxy2019's picture
Upload tokenizer
1996cb7
raw
history blame contribute delete
151 Bytes
{
"cls_token": "<|beginoftext|>",
"mask_token": "[MASK]",
"pad_token": "<|endoftext|>",
"sep_token": "<|endoftext|>",
"unk_token": "[UNK]"
}