Models
Datasets
Spaces
Posts
Docs
Enterprise
Pricing
Log In
Sign Up

Spaces:

xu-song
/

tokenizer-arena

Running

App Files Files Community

Fetching metadata from the HF Docker repository...

tokenizer-arena / vocab /gpt_35_turbo

2 contributors

History: 11 commits

xu-song's picture

fix tokenize

e6543ac 11 months ago

README.md

1.65 kB

fix chatglm; new feature about add_special_tokens; 12 months ago
__init__.py

2.42 kB

fix tokenize 11 months ago
cl100k_base.tiktoken

1.68 MB

update over 1 year ago
decode_test.py

291 Bytes

fix chatglm; new feature about add_special_tokens; 12 months ago
test_arena.py

199 Bytes

add more tokenizers about 1 year ago
test_tiktoken.py

1.09 kB

fix chatglm; new feature about add_special_tokens; 12 months ago
tiktoken_decode_test.py

965 Bytes

add more tokenizers about 1 year ago
vocab.jsonl

3.78 MB

add more tokenizers about 1 year ago