Why did you change the eos_token in tokenizer_config.json file?
#7
by
PhilipMay
- opened
When I compare this tokenizer_config.json with the tokenizer_config.json of microsoft/Phi-3-mini-4k-instruct I see that
you changed the eos_token from"eos_token": "<|endoftext|>", to "eos_token": "<|end|>",.
May I ask why you did this change?
For chat. I'm using trtllm, and this eos_token works better for me.