codebert-llvm-ic-v0 / tokenizer_config.json
iyaja's picture
add tokenizer
92cad50
raw history blame
No virus
78 Bytes
{"max_len_single_sentence": 512, "tokenizer_class": "PreTrainedTokenizerFast"}