stablelm-2-1_6b / tokenizer_config.json
jon-tow's picture
fix: add default special tokens
9f2def9 verified
raw
history blame
224 Bytes
{
"tokenizer_class": "Arcade100kTokenizer",
"auto_map": {
"AutoTokenizer": [
"tokenization_arcade100k.Arcade100kTokenizer",
null
]
},
"eos_token": "<|endoftext|>",
"pad_token": "<|endoftext|>"
}