internlm2-chat-1_8b / tokenization_internlm2_fast.py

Commit History

fix no white space when using stream_chat with fast tokenizer
f842a52

x54-729 commited on

update tokenizers
38ae535

ZwwWayne commited on