| trainer_interface.cc(537) LOG(INFO) all chars count=796343461 | |
| trainer_interface.cc(548) LOG(INFO) Done: 99.95% characters are covered. | |
| trainer_interface.cc(558) LOG(INFO) Alphabet size=5013 | |
| trainer_interface.cc(559) LOG(INFO) Final character coverage=0.9995 | |
| trainer_interface.cc(591) LOG(INFO) Done! preprocessed 800000 sentences. | |
| trainer_interface.cc(597) LOG(INFO) Tokenizing input sentences with whitespace: 800000 | |
| trainer_interface.cc(608) LOG(INFO) Done! 1021909 | |
| corpus_dir='../datasets/online_novel//data.txt', output_dir='../models/baby-chinese-llama2/64k', model_type='bpe', max_sentence_length=4096, vocab_size=64000, max_lines=1000000, shuffle_lines=True, pad_id=3, normalization_rule_name='identity', character_coverage=0.9995, action='export') | |
| Total tokens: 1686.54MB | |
| Mean: 563.0230508481366, Median: 559.0, | |
| 5th percentile: 504.0, | |
| 25th percentile: 535.0, | |
| 75th percentile: 587.0, | |
| 95th percentile: 634.0, | |
| 99th percentile: 669.0, | |
| max: 2657 |