ValueError: Tokenizer class GemmaTokenizer does not exist or is not currently imported.

#53
by medmac01 - opened

I'm currently trying to use gemma-7b-it with chat template, however it can't find the tokenizer, throwing this error:

model_id = "google/gemma-7b"

tokenizer = AutoTokenizer.from_pretrained(model_id)

ValueError Traceback (most recent call last)
Input In [10], in <cell line: 3>()
1 model_id = "google/gemma-7b"
----> 3 tokenizer = AutoTokenizer.from_pretrained(model_id, use_auth_token=True)

File ~/opt/anaconda3/lib/python3.9/site-packages/transformers/models/auto/tokenization_auto.py:688, in AutoTokenizer.from_pretrained(cls, pretrained_model_name_or_path, *inputs, **kwargs)
686 tokenizer_class = tokenizer_class_from_name(tokenizer_class_candidate)
687 if tokenizer_class is None:
--> 688 raise ValueError(
689 f"Tokenizer class {tokenizer_class_candidate} does not exist or is not currently imported."
690 )
691 return tokenizer_class.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
693 # Otherwise we have to be creative.
694 # if model is an encoder decoder, the encoder tokenizer class is used by default

ValueError: Tokenizer class GemmaTokenizer does not exist or is not currently imported.

update: you have to import GemmaTokenizer

from transformers import GemmaTokenizer

make sure that tokenizer is up to date

medmac01 changed discussion status to closed
Your need to confirm your account before you can post a new comment.

Sign up or log in to comment