HAT tokenizer ValueError

#2
by BarryC7 - opened

from transformers import AutoTokenizer, AutoModelForSequenceClassification
tokenizer = AutoTokenizer.from_pretrained("kiddothe2b/hierarchical-transformer-base-4096", trust_remote_code=True)
tokenizer('i want to test something')

show error 'You should supply an encoding or a list of encodings to this method that includes input_ids, but you provided []'

Minimum working example:

from transformers import AutoTokenizer, AutoModelForSequenceClassification
tokenizer = AutoTokenizer.from_pretrained("kiddothe2b/hierarchical-transformer-base-4096", trust_remote_code=True)
tokenizer(['i want to test something'], padding='max_length', max_length=4096)
kiddothe2b changed discussion status to closed

How do you make the type of input_ids tensor

Sign up or log in to comment