clotho-simple-tokenizer / special_tokens_map.json
wsntxxn's picture
Upload tokenizer
1a9a647 verified
{
"additional_special_tokens": [
"<start>",
"<end>"
],
"bos_token": "<start>",
"eos_token": "<end>",
"unk_token": "<unk>"
}