zirui3
/

llm-multilingual-tokenizer

Model card Files Files and versions Community

llm-multilingual-tokenizer / README.md

zirui3's picture

Upload README.md

84c361f over 1 year ago

|

123 Bytes



	# summary
	multilingual tokenizer trained on multilingual data by using the SentencePiece library and the BPE algorithm.