parameters: Namespace(corpus_dir='../datasets/online_novel//data.txt', output_dir='../models/baby-chinese-llama2', model_type='bpe', max_sentence_length=4096, vocab_size=32000, max_lines=1000000, shuffle_lines=True, pad_id=3, normalization_rule_name='identity', character_coverage=0.9995, action='export')
 
trainer_interface.cc(428) LOG(INFO) Normalizing sentences...
trainer_interface.cc(537) LOG(INFO) all chars count=796343461
trainer_interface.cc(548) LOG(INFO) Done: 99.95% characters are covered.
trainer_interface.cc(558) LOG(INFO) Alphabet size=5013
trainer_interface.cc(559) LOG(INFO) Final character coverage=0.9995
trainer_interface.cc(591) LOG(INFO) Done! preprocessed 800000 sentences.
trainer_interface.cc(597) LOG(INFO) Tokenizing input sentences with whitespace: 800000
trainer_interface.cc(608) LOG(INFO) Done! 1021909

Raw corpus
Total lines: 2995508
Total tokens: 1827.13MB
Mean: 610, Median: 606.0,
5th percentile: 546.0,
25th percentile: 580.0,
75th percentile: 636.0,
95th percentile: 686.0,
99th percentile: 722.0,
max: 2657