Hugging Face's logo

DazMashaly
/
whisper_tokenizer

Transformers
Inference Endpoints
Model card Files Community
DazMashaly commited on
Commit
6daa99b
•
1 Parent(s): c0ba2ad

Upload tokenizer

Browse files
Files changed (4) hide show
  1. added_tokens.json +1 -0
  2. merges.txt +0 -0
  3. tokenizer.json +0 -0
  4. vocab.json +0 -0
added_tokens.json CHANGED
@@ -1518,6 +1518,7 @@
1518
  "<|da|>": 50285,
1519
  "<|de|>": 50261,
1520
  "<|el|>": 50281,
 
1521
  "<|en|>": 50259,
1522
  "<|es|>": 50262,
1523
  "<|et|>": 50307,
 
1518
  "<|da|>": 50285,
1519
  "<|de|>": 50261,
1520
  "<|el|>": 50281,
1521
+ "<|endoftext|>": 50257,
1522
  "<|en|>": 50259,
1523
  "<|es|>": 50262,
1524
  "<|et|>": 50307,
merges.txt CHANGED
The diff for this file is too large to render. See raw diff
 
tokenizer.json CHANGED
The diff for this file is too large to render. See raw diff
 
vocab.json CHANGED
The diff for this file is too large to render. See raw diff