How to set the attention mask for inference ?
#13
by
jgsmcmahon
- opened
MSG appears at Inference : The attention mask and the pad token id were not set. As a consequence, you may observe unexpected behavior. Please pass your input's attention_mask to obtain reliable results. Setting pad_token_id to eos_token_id:2 for open-end generation.
Please let me know what I need to do to get past this issue ? Many thanks :-)
Just pass the full inputs from the tokenizer. inputs = tokenizer(["Hey how are you?"], ['Good'], padding = True) will return the attention mask. But if there is no padding token you should specify tokenizer.pad_token = tokenizer.eos_token
Ok thanks will try this :-)
Add this after loading in the tokenizer from pretrained:
if tokenizer.pad_token is None:
tokenizer.pad_token = tokenizer.eos_token
And then change the generate line to this:
generated_ids = model.generate(model_inputs, pad_token_id=tokenizer.pad_token_id, max_new_tokens=2000, do_sample=True)