Wrong tokenizer given "expected str, bytes or os.PathLike object, not NoneType"
Running both the 4B and 2B versions raises Exceptions for me "expected str, bytes or os.PathLike object, not NoneType" when trying to load the tokenizer.
I think the issue is that the LlamaTokenizer expects a tokenizer.model, but this repo contains a tokenizer.json instead.
Here's how I solved it:
- Run the model once, so it has all the files downloaded
- Copy over tokenizer.model (in snapshots)and the file it points to (in blobs) (the file in blobs first, so the pointer works) from the non-adapted version of InternVL2 (either the 2B or 4B version)
- Delete tokenizer.model in the .no_exist directory (it was cached as missing during the initial download run)
Results seem to be good, but if you could add the "real" tokenizer.model, that would be great. Thanks!
I cannot find the tokenizer.model in snapshots file.
They are the 2B or 4B paths I linked
I use 4B but couldn't find the tokenizer.model in snapshots file, In your solution, the second item "Copy over tokenizer.model (in snapshots)and the file it points to (in blobs) (the file in blobs first, so the pointer works) from the non-adapted version of InternVL2 (either the 2B or 4B version)", I have no idea what to copy