ZinengTang
/

tulip-tokenizer

Model card Files Files and versions Community

tulip-tokenizer / tokenizer.model

Commit History

Update tokenizer by filtering out foreign language tokens

b3b3dbb
verified

ZinengTang commited on 7 days ago