ZinengTang
/

tulip-tokenizer

SHA256: ee9b31b05e6557d414859b8deba332a2e0f1721a6be750a76f6defffaac08af4
Pointer size: 133 Bytes
Size of remote file: 43.6 MB

tulip-tokenizer / tokenizer.json

Update tokenizer by filtering out foreign language tokens

b3b3dbb verified 7 days ago

43.6 MB

This file is stored with Git LFS . It is too big to display, but you can still download it.

Git Large File Storage (LFS) replaces large files with text pointers inside Git, while storing the file contents on a remote server. More info.