microsoft

/

Phi-3-small-8k-instruct

like 150

This change deals with the scenario where user has restricted internet access and thus fails to download the tokenizer file from https://openaipublic.blob.core.windows.net/encodings/cl100k_base.tiktoken by directly calling base = tiktoken.get_encoding("cl100k_base"). The workaround is to download the cl100k_base.tiktoken file from this repo to user's local. It is assumed that user should be able to access files on huggingface hub.