added tokenizer from v4 (#1)

- added tokenizer from v4 (af6e819932545e15cfee3b9acf503457df552dc9)

Co-authored-by: neural_worm <neuralworm@users.noreply.huggingface.co>

Files changed (2) hide show

tokenizer.json ADDED Viewed

The diff for this file is too large to render. See raw diff

tokenizer.model ADDED Viewed

+version https://git-lfs.github.com/spec/v1
+oid sha256:f440c53d2cc6f14a7ed7124dea5f5a7402fb4fc95bccb5d8be6d0f7e74d327ed
+size 568229