sarashina2-13b / tokenizer_config.json

Commit History

use LlamaTokenizer
d5feed5

ryo0634 commited on

Use tool tokens
fd6d47c

kajyuuen commited on

Use FastTokenizer
a295f29

kajyuuen commited on

add tokenizer_config
b283e16

sho-takase commited on