how did you guys pretrain the tokenizer using tiktoken ?
#9
by
StephennFernandes
- opened
how did you guys pretrain the tokenizer using tiktoken, i am unable to find on steps to train tiktoken on my own corpus. but it seems you and many other with models like phi-3, llama-3 have trained using tiktoken. please help me out