Models
Datasets
Spaces
Posts
Docs
Pricing
Log In
Sign Up

Spaces:

yhavinga
/

dutch-tokenizer-arena

Running

App Files Files Community

dutch-tokenizer-arena / vocab /gpt_35_turbo

3 contributors

History: 10 commits

xu-song's picture

update

9820e00 10 months ago

README.md

1.65 kB

fix chatglm; new feature about add_special_tokens; 10 months ago
__init__.py

2.38 kB

update 10 months ago
cl100k_base.tiktoken

1.68 MB

update about 1 year ago
decode_test.py

291 Bytes

fix chatglm; new feature about add_special_tokens; 10 months ago
test_arena.py

199 Bytes

add more tokenizers 12 months ago
test_tiktoken.py

1.09 kB

fix chatglm; new feature about add_special_tokens; 10 months ago
tiktoken_decode_test.py

965 Bytes

add more tokenizers 12 months ago
vocab.jsonl

3.78 MB

add more tokenizers 12 months ago