Chat prompt template
Hey guys, in README it's said that you're using ChatML prompt template.
This prompt template should start with:
<|im_start|>system
But, I see that the prompt template that you have in tokenizer_config, does not start messages with <|im_start|>system.
Instead, it's formatted like this:
SYSTEM PROMPT GOES HERE\n
<|im_start|>user
HI!<|im_end|>
<|im_start|>assistant
So, should I update the chat template to match ChatML or stick with the one you have in the tokenizer config?
We trained with the chatML template in axolotl, which should match the readme, the gen config was taken from upstream instruct to fix some other issues people were facing.
tl;dr: use proper chatML.
If you'd like you could PR the correct jinja template and we'll merge it. (unless someone else gets to it before that)
Sure, I can open a PR. Would you like me to make a change to tokenizer_config or upload jinja template separately in other way?
I can also run the model with this change to see if this breaks anything.
This is the template I'm using btw:
{% for message in messages %}{{'<|im_start|>' + message['role'] + '\n' + message['content']}}{% if (loop.last and add_generation_prompt) or not loop.last %}{{ '<|im_end|>' + '\n'}}{% endif %}{% endfor %}\n{% if add_generation_prompt and messages[-1]['role'] != 'assistant' %}{{ '<|im_start|>assistant\n' }}{% endif %}
I can also run the model with this change to see if this breaks anything.
that'd be great yes
Will you be re-uploading the model with this change?
there's no need to reupload, if the above template works we'll just hotswap it in the repo configs.
Sounds good :-)
I'll run it tomorrow and open the PR.
Will the config file work with the GGUF I have? Sorry, I'm not massively experienced with this.
the gguf would need to be redone or the metadata adjusted at the very least iirc
I'm sure someone will be willing to redo it :-)