Spaces:

yhavinga
/

dutch-tokenizer-arena

Running

add more tokenizer

293bad6 12 months ago

134 Bytes

	from transformers import AutoTokenizer

	tokenizer = AutoTokenizer.from_pretrained("lmsys/fastchat-t5-3b-v1.0", trust_remote_code=True)