InstaDeepAI
/

segment_nt

Feature Extraction

Model card Files Files and versions Community

segment_nt / tokenizer_config.json

hdallatorre's picture

Upload tokenizer

5ef47a0 verified 10 months ago

129 Bytes

	{
	"clean_up_tokenization_spaces": true,
	"eos_token": null,
	"model_max_length": 2048,
	"tokenizer_class": "EsmTokenizer"
	}