dulia-13b-8k-alpha / added_tokens.json
onlywicked's picture
feat: add tokenizer
f46968f
raw
history blame
151 Bytes
{
"<pad>": 32000,
"<|assistant|>": 32001,
"<|prefix_begin|>": 32004,
"<|prefix_end|>": 32002,
"<|prompter|>": 32005,
"<|system|>": 32003
}