Hugging Face's logo

internlm
/
internlm2-chat-20b

Text Generation
Transformers
Safetensors
internlm2
conversational
custom_code
Model card Files Community
11

tokenizer adds default eos_token_id while build_inputs function adds eos_token too!

#9
by suzhu001 - opened
Discussion
suzhu001

https://huggingface.co/internlm/internlm2-chat-20b/blob/3f710f76f56f8c40dc5dd800dbe66f3341cb2c87/modeling_internlm2.py#L1146

suzhu001 changed discussion title from tokenizer add default eos_token_id while build_inputs function add eos_token too! to tokenizer adds default eos_token_id while build_inputs function adds eos_token too!

Sign up or log in to comment