Create tokenizer.json
#1
by
Jilt
- opened
tokenizer.json will be required for conversions (for example to CT2 format). It also makes it a consistent file type across different whisper models.
Thanks for this
ylacombe
changed pull request status to
merged