Commit
•
492f305
1
Parent(s):
243c5b8
added english-32000, english-50256
Browse files
README.md
CHANGED
|
@@ -28,9 +28,9 @@ Trained models can be downloaded from here:
|
|
| 28 |
|-----------------|------------|--------|-------------
|
| 29 |
| english-100256 | 100256 | UTF-8 | [download](https://huggingface.co/alasdairforsythe/tokenmonster/resolve/main/english-100256.vocab)
|
| 30 |
| english-65536 | 65536 | UTF-8 | in-progress
|
| 31 |
-
| english-50256 | 50256 | UTF-8 |
|
| 32 |
| english-40000 | 40000 | UTF-8 | in-progress
|
| 33 |
-
| english-32000 | 32000 | UTF-8 |
|
| 34 |
| english-24000 | 24000 | UTF-8 | in-progress
|
| 35 |
| code-100256 | 100256 | UTF-8 | [download](https://huggingface.co/alasdairforsythe/tokenmonster/resolve/main/code-100256.vocab)
|
| 36 |
| code-65536 | 65536 | UTF-8 | in-progress
|
|
|
|
| 28 |
|-----------------|------------|--------|-------------
|
| 29 |
| english-100256 | 100256 | UTF-8 | [download](https://huggingface.co/alasdairforsythe/tokenmonster/resolve/main/english-100256.vocab)
|
| 30 |
| english-65536 | 65536 | UTF-8 | in-progress
|
| 31 |
+
| english-50256 | 50256 | UTF-8 | [download](https://huggingface.co/alasdairforsythe/tokenmonster/resolve/main/english-50256.vocab)
|
| 32 |
| english-40000 | 40000 | UTF-8 | in-progress
|
| 33 |
+
| english-32000 | 32000 | UTF-8 | [download](https://huggingface.co/alasdairforsythe/tokenmonster/resolve/main/english-32000.vocab)
|
| 34 |
| english-24000 | 24000 | UTF-8 | in-progress
|
| 35 |
| code-100256 | 100256 | UTF-8 | [download](https://huggingface.co/alasdairforsythe/tokenmonster/resolve/main/code-100256.vocab)
|
| 36 |
| code-65536 | 65536 | UTF-8 | in-progress
|