open_llama_3b / tokenizer.json

Commit History

Enable LlamaTokenizerFast and AutoTokenizer to load in seconds rather than 5 minutes.
379edd8

danielhanchen commited on