I could be wrong, but I think the <think> tag needs to be removed from the last bit of the jinja template in the tokenizer_config.json
As is, interfaces like open-webui don't display the opening tag, but changing {{- '<|im_start|>assistant\\n<think>\\n' }}\n{%- endif %}\n to {{- '<|im_start|>assistant\\n'}}\n{%- endif %}\n fixes this. I could be missing something important though.
Edit: And I'm getting better results removing the new line, {{- '<|im_start|>assistant\\n' }}{%- endif %}\n
It's part of the add_generation_prompt so I think it default adds during the assistant turn.
Are you saying you get better results if we don add it by default?
With it added, open-webui and other frontends don't display the opening think tag, when I remove it, they do. Figured it's worth mentioning even if it's not worth changing.
Also, thank you, Unsloth is amazing.
With it added, open-webui and other frontends don't display the opening think tag, when I remove it, they do. Figured it's worth mentioning even if it's not worth changing.
Also, thank you, Unsloth is amazing.
You can check qwen model page, if you don't add <think>\n before generation, there's a chance the model will skip reasoning and perform worse, so they add it in chat template.