Hugging Face's logo

Spaces:
xu-song
/
tokenizer-arena
Running

App Files Community
1
xu-song commited on
Commit
2442c83
·
1 Parent(s): bbefe94

fix olmo

Browse files
Files changed (1) hide show
  1. vocab/olmo_7b/__init__.py +1 -1
vocab/olmo_7b/__init__.py CHANGED
@@ -1,4 +1,4 @@
1
 
2
  from transformers import AutoTokenizer
3
 
4
- tokenizer = AutoTokenizer.from_pretrained("allenai/OLMo-7B")
 
1
 
2
  from transformers import AutoTokenizer
3
 
4
+ tokenizer = AutoTokenizer.from_pretrained("allenai/OLMo-7B", trust_remote_code=True)