special_tokens_map.json · thevox/en-nb-7b at main

en-nb-7b / special_tokens_map.json

thevox's picture

Upload tokenizer

97842e0 about 1 year ago

history blame contribute delete

215 Bytes

	{
	"additional_special_tokens": [
	"<unk>",
	"<s>",
	"</s>"
	],
	"bos_token": "<s>",
	"cls_token": "</s>",
	"eos_token": "</s>",
	"pad_token": "<unk>",
	"sep_token": "</s>",
	"unk_token": "<unk>"
	}