[Train] pad_token_id set as "!"
#7
by
eungizoa
- opened
According to the special_tokens_map, the pad_token is set as "!".
I want to fine-tune this model to perform specific task, and rom the trl.SFTTrainer document, it says that pad_token_id should be set differently as eos_token_id.
"Make sure to have a pad_token_id which is different from eos_token_id which can result in the model not properly predicting EOS (End of Sentence) tokens during generation." (https://huggingface.co/docs/trl/sft_trainer)
But the string "!" is a very common string, so I wonder if I can use this pad_token during training.
Can I train this model as below?
"tokenizer.pad_token_id = tokenizer.eos_token_id"
Or should I use this pad_token ("!") without any modification?