StripedHyena-Hessian-7B / tokenizer_config.json

fix fast tokenizer for transformers >= "from_slow": false,

f340f48 verified 10 months ago

142 Bytes

	{
	"unk_token": "</s>",
	"bos_token": "<s>",
	"eos_token": "</s>",
	"from_slow": false,
	"tokenizer_class": "LlamaTokenizer"
	}