ArthurConmy commited on
Commit
b389e23
1 Parent(s): c76d915

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +1 -1
README.md CHANGED
@@ -1,4 +1,4 @@
1
- This is Neel's tokenizer but with `"tokenizer_class": "GPTNeoXTokenizer"` in the config.
2
 
3
  # Neel's README
4
 
 
1
+ This is https://huggingface.co/NeelNanda/gpt-neox-tokenizer-digits but with a fix that makes this work on `tokenizers == 0.14` after a breaking change involving `added_tokens`: https://github.com/huggingface/tokenizers/issues/1358
2
 
3
  # Neel's README
4