tokenizers
Collection
trained and adapted tokenizers - various
•
14 items
•
Updated
This is a ByteLevelBPETokenizer
trained on the JeanKaddour/minipile
dataset with the aim to create a compact English-only tokenizer.
load with AutoTokenizer, i.e.:
from transformers import AutoTokenizer
tk = AutoTokenizer.from_pretrained('BEE-spoke-data/MiniTokenizer-20480')
tk