Question about tokenizer.json size
#8
by
eatbeans2
- opened
There was a discussion in this reddit post about many finetunes having an incorrect and oversized tokenizer.json files. I noticed that the tokenizer for this model is almost 2MB larger than the 2407 version. Is this expected?