aehrc
/

cxrmate-ed

anicolson commited on 27 days ago

Commit

af37377

•

1 Parent(s): 80559a6

Upload tokenizer

Files changed (2) hide show

tokenizer.json CHANGED Viewed

The diff for this file is too large to render. See raw diff

tokenizer_config.json CHANGED Viewed

@@ -53,6 +53,7 @@
   "clean_up_tokenization_spaces": true,
   "cls_token": "[BOS]",
   "eos_token": "[EOS]",
   "mask_token": "[MASK]",
   "model_max_length": 1000000000000000019884624838656,
   "pad_token": "[PAD]",

   "clean_up_tokenization_spaces": true,
   "cls_token": "[BOS]",
   "eos_token": "[EOS]",
+  "extra_special_tokens": {},
   "mask_token": "[MASK]",
   "model_max_length": 1000000000000000019884624838656,
   "pad_token": "[PAD]",