ValueError: Couldn't instantiate the backend tokenizer from one of:
#1
by
doublelift
- opened
ValueError: Couldn't instantiate the backend tokenizer from one of:
(1) a tokenizers library serialization file,
(2) a slow tokenizer instance to convert or
(3) an equivalent slow tokenizer class to instantiate and convert.
You need to have sentencepiece installed to convert a slow tokenizer to a fast one.
已经安装了sentencepiece，但是还是报错了。
尝试使用LlamaTokenizer代替AutoTokenizer则运行成功了。
请问这样的解决方案合理么，问题出在哪里？