INX-TEXT

/

Bailong-instruct-7B

like 44

Follow

INX-TEXT-AI 6

我嘗試使用 AutoTokenzier, 但遇到以下的錯誤

docker-llm-1    |   File "/app/app.py", line 15, in load_model
docker-llm-1    |     tokenizer = AutoTokenizer.from_pretrained("INX-TEXT/Bailong-instruct-7B")
docker-llm-1    |   File "/usr/local/lib/python3.10/dist-packages/transformers/models/auto/tokenization_auto.py", line 814, in from_pretrained
docker-llm-1    |     return tokenizer_class.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
docker-llm-1    |   File "/usr/local/lib/python3.10/dist-packages/transformers/tokenization_utils_base.py", line 2029, in from_pretrained
docker-llm-1    |     return cls._from_pretrained(
docker-llm-1    |   File "/usr/local/lib/python3.10/dist-packages/transformers/tokenization_utils_base.py", line 2261, in _from_pretrained
docker-llm-1    |     tokenizer = cls(*init_inputs, **init_kwargs)
docker-llm-1    |   File "/usr/local/lib/python3.10/dist-packages/transformers/models/llama/tokenization_llama_fast.py", line 124, in __init__
docker-llm-1    |     super().__init__(
docker-llm-1    |   File "/usr/local/lib/python3.10/dist-packages/transformers/tokenization_utils_fast.py", line 120, in __init__
docker-llm-1    |     raise ValueError(
docker-llm-1    | ValueError: Couldn't instantiate the backend tokenizer from one of: 
docker-llm-1    | (1) a `tokenizers` library serialization file, 
docker-llm-1    | (2) a slow tokenizer instance to convert or 
docker-llm-1    | (3) an equivalent slow tokenizer class to instantiate and convert. 
docker-llm-1    | You need to have sentencepiece installed to convert a slow tokenizer to a fast one.

我的 transformer=4.37.2 想請問上面的問題該如何解決 謝謝