This file should be removed - it's no longer in the upstream repo
Browse filesIt was broken in the original release, and they've since removed it. They've specified the special_tokens as added_tokens, but they're already listed in the tokenizer and special_tokens_map.json.
This file being here breaks making GGUF quants, and might break some other things too.
In the upstream repo they've moved to the new style Transformers special_tokens config
- added_tokens.json +0 -5
added_tokens.json
DELETED
@@ -1,5 +0,0 @@
|
|
1 |
-
{
|
2 |
-
"</s>": 2,
|
3 |
-
"<s>": 1,
|
4 |
-
"<unk>": 0
|
5 |
-
}
|
|
|
|
|
|
|
|
|
|
|
|