eliebak
/

the-tokenizer-v2

Inference Endpoints

Model card Files Files and versions Community

eliebak HF staff commited on Jun 17, 2024

Commit

17b1f92

·

verified ·

1 Parent(s): 4da8c10

fix vocab size by hand

Files changed (1) hide show

tokenizer_config.json +1 -1

tokenizer_config.json CHANGED Viewed

@@ -292,5 +292,5 @@
   "model_max_length": 1000000000000000019884624838656,
   "tokenizer_class": "GPT2Tokenizer",
   "unk_token": "<|endoftext|>",
-  "vocab_size": 49152
 }

   "model_max_length": 1000000000000000019884624838656,
   "tokenizer_class": "GPT2Tokenizer",
   "unk_token": "<|endoftext|>",
+  "vocab_size": 49154
 }