cognitivess / cognitivess_model /tokenization_cognitivess_fast.py
cognitivess's picture
Update cognitivess_model/tokenization_cognitivess_fast.py
b2f5c84 verified
raw
history blame
204 Bytes
from transformers import LlamaTokenizerFast
from .tokenization_cognitivess import CognitivessTokenizer
class CognitivessTokenizerFast(LlamaTokenizerFast):
slow_tokenizer_class = CognitivessTokenizer