polyglot-ko-5.8b / tokenizer_config.json
jason9693's picture
Upload with huggingface_hub
602dc2f
raw
history blame
90 Bytes
{"pad_token": "<pad>", "eos_token": "<eos>", "tokenizer_class": "PreTrainedTokenizerFast"}