2023-09-16 11:59:03,468 44k INFO {'train': {'log_interval': 200, 'eval_interval': 800, 'seed': 1234, 'epochs': 10000, 'learning_rate': 5e-05, 'betas': [0.8, 0.99], 'eps': 1e-09, 'batch_size': 80, 'fp16_run': True, 'half_type': 'fp16', 'lr_decay': 0.999875, 'segment_size': 10240, 'init_lr_ratio': 1, 'warmup_epochs': 0, 'c_mel': 45, 'c_kl': 1.0, 'use_sr': True, 'max_speclen': 512, 'port': '8001', 'keep_ckpts': 5, 'all_in_mem': True, 'vol_aug': True}, 'data': {'training_files': 'filelists/train.txt', 'validation_files': 'filelists/val.txt', 'max_wav_value': 32768.0, 'sampling_rate': 44100, 'filter_length': 2048, 'hop_length': 512, 'win_length': 2048, 'n_mel_channels': 80, 'mel_fmin': 0.0, 'mel_fmax': 22050, 'unit_interpolate_mode': 'nearest'}, 'model': {'inter_channels': 192, 'hidden_channels': 192, 'filter_channels': 768, 'n_heads': 2, 'n_layers': 6, 'kernel_size': 3, 'p_dropout': 0.1, 'resblock': '1', 'resblock_kernel_sizes': [3, 7, 11], 'resblock_dilation_sizes': [[1, 3, 5], [1, 3, 5], [1, 3, 5]], 'upsample_rates': [8, 8, 2, 2, 2], 'upsample_initial_channel': 512, 'upsample_kernel_sizes': [16, 16, 4, 4, 4], 'n_layers_q': 3, 'n_layers_trans_flow': 3, 'n_flow_layer': 4, 'use_spectral_norm': False, 'gin_channels': 768, 'ssl_dim': 768, 'n_speakers': 3, 'vocoder_name': 'nsf-hifigan', 'speech_encoder': 'vec768l12', 'speaker_embedding': False, 'vol_embedding': True, 'use_depthwise_conv': False, 'flow_share_parameter': False, 'use_automatic_f0_prediction': True, 'use_transformer_flow': False}, 'spk': {'KATO_SHIHO': 0, 'MATSUDA_KONOKA': 1, 'SAITO_KYOKO': 2}, 'model_dir': './logs/44k'} 2023-09-16 12:00:13,660 44k INFO emb_g.weight is not in the checkpoint 2023-09-16 12:00:13,660 44k INFO emb_vol.weight is not in the checkpoint 2023-09-16 12:00:13,660 44k INFO emb_vol.bias is not in the checkpoint 2023-09-16 12:00:13,713 44k INFO Loaded checkpoint './logs/44k/G_0.pth' (iteration 0) 2023-09-16 12:00:14,008 44k INFO Loaded checkpoint './logs/44k/D_0.pth' (iteration 0) 2023-09-16 12:01:33,612 44k INFO ====> Epoch: 1, cost 150.16 s 2023-09-16 12:02:33,615 44k INFO ====> Epoch: 2, cost 60.00 s 2023-09-16 12:03:33,783 44k INFO ====> Epoch: 3, cost 60.17 s 2023-09-16 12:04:33,691 44k INFO ====> Epoch: 4, cost 59.91 s 2023-09-16 12:05:33,228 44k INFO ====> Epoch: 5, cost 59.54 s 2023-09-16 12:06:32,952 44k INFO ====> Epoch: 6, cost 59.72 s 2023-09-16 12:06:48,240 44k INFO Train Epoch: 7 [22%] 2023-09-16 12:06:48,241 44k INFO Losses: [2.415888547897339, 2.495821714401245, 13.114778518676758, 27.233884811401367, 1.8150787353515625], step: 200, lr: 4.996251171679705e-05, reference_loss: 47.07545471191406 2023-09-16 12:07:33,089 44k INFO ====> Epoch: 7, cost 60.14 s 2023-09-16 12:08:33,304 44k INFO ====> Epoch: 8, cost 60.21 s 2023-09-16 12:09:32,836 44k INFO ====> Epoch: 9, cost 59.53 s 2023-09-16 12:10:32,788 44k INFO ====> Epoch: 10, cost 59.95 s 2023-09-16 12:11:32,795 44k INFO ====> Epoch: 11, cost 60.01 s 2023-09-16 12:12:32,995 44k INFO ====> Epoch: 12, cost 60.20 s 2023-09-16 12:13:03,548 44k INFO Train Epoch: 13 [47%] 2023-09-16 12:13:03,549 44k INFO Losses: [2.4585509300231934, 2.510573387145996, 12.793045043945312, 26.4683895111084, 1.7211201190948486], step: 400, lr: 4.992505154102165e-05, reference_loss: 45.95167922973633 2023-09-16 12:13:33,789 44k INFO ====> Epoch: 13, cost 60.79 s 2023-09-16 12:14:34,083 44k INFO ====> Epoch: 14, cost 60.29 s 2023-09-16 12:15:33,544 44k INFO ====> Epoch: 15, cost 59.46 s 2023-09-16 12:16:33,502 44k INFO ====> Epoch: 16, cost 59.96 s 2023-09-16 12:17:33,837 44k INFO ====> Epoch: 17, cost 60.33 s 2023-09-16 12:18:33,685 44k INFO ====> Epoch: 18, cost 59.85 s 2023-09-16 12:19:19,756 44k INFO Train Epoch: 19 [72%] 2023-09-16 12:19:19,757 44k INFO Losses: [2.393871784210205, 2.474231719970703, 11.761624336242676, 25.432668685913086, 1.4525039196014404], step: 600, lr: 4.9887619451599816e-05, reference_loss: 43.51490020751953 2023-09-16 12:19:34,369 44k INFO ====> Epoch: 19, cost 60.68 s 2023-09-16 12:20:34,415 44k INFO ====> Epoch: 20, cost 60.05 s 2023-09-16 12:21:33,401 44k INFO ====> Epoch: 21, cost 58.99 s 2023-09-16 12:22:33,430 44k INFO ====> Epoch: 22, cost 60.03 s 2023-09-16 12:23:33,791 44k INFO ====> Epoch: 23, cost 60.36 s 2023-09-16 12:24:34,016 44k INFO ====> Epoch: 24, cost 60.23 s 2023-09-16 12:25:33,666 44k INFO Train Epoch: 25 [97%] 2023-09-16 12:25:33,667 44k INFO Losses: [2.252323865890503, 2.5726516246795654, 12.512992858886719, 23.331615447998047, 1.180820107460022], step: 800, lr: 4.985021542747336e-05, reference_loss: 41.850406646728516 2023-09-16 12:25:57,852 44k INFO Saving model and optimizer state at iteration 25 to ./logs/44k/G_800.pth 2023-09-16 12:26:01,289 44k INFO Saving model and optimizer state at iteration 25 to ./logs/44k/D_800.pth 2023-09-16 12:26:02,561 44k INFO ====> Epoch: 25, cost 88.55 s 2023-09-16 12:27:02,623 44k INFO ====> Epoch: 26, cost 60.06 s 2023-09-16 12:28:02,163 44k INFO ====> Epoch: 27, cost 59.54 s 2023-09-16 12:29:01,925 44k INFO ====> Epoch: 28, cost 59.76 s 2023-09-16 12:30:02,319 44k INFO ====> Epoch: 29, cost 60.39 s 2023-09-16 12:31:02,874 44k INFO ====> Epoch: 30, cost 60.55 s 2023-09-16 12:32:02,969 44k INFO ====> Epoch: 31, cost 60.10 s 2023-09-16 12:32:18,030 44k INFO Train Epoch: 32 [22%] 2023-09-16 12:32:18,031 44k INFO Losses: [2.4512696266174316, 2.4131946563720703, 12.147953987121582, 25.461721420288086, 1.4842323064804077], step: 1000, lr: 4.980661284266894e-05, reference_loss: 43.9583740234375 2023-09-16 12:33:03,050 44k INFO ====> Epoch: 32, cost 60.08 s 2023-09-16 12:34:02,770 44k INFO ====> Epoch: 33, cost 59.72 s 2023-09-16 12:35:01,755 44k INFO ====> Epoch: 34, cost 58.99 s 2023-09-16 12:36:02,055 44k INFO ====> Epoch: 35, cost 60.30 s 2023-09-16 12:37:01,770 44k INFO ====> Epoch: 36, cost 59.72 s 2023-09-16 12:38:01,200 44k INFO ====> Epoch: 37, cost 59.43 s 2023-09-16 12:38:31,789 44k INFO Train Epoch: 38 [47%] 2023-09-16 12:38:31,790 44k INFO Losses: [2.323206901550293, 2.540818691253662, 12.201950073242188, 25.417898178100586, 1.3928589820861816], step: 1200, lr: 4.976926955451643e-05, reference_loss: 43.876731872558594 2023-09-16 12:39:01,438 44k INFO ====> Epoch: 38, cost 60.24 s 2023-09-16 12:40:01,394 44k INFO ====> Epoch: 39, cost 59.96 s 2023-09-16 12:41:01,279 44k INFO ====> Epoch: 40, cost 59.88 s 2023-09-16 12:42:01,622 44k INFO ====> Epoch: 41, cost 60.34 s 2023-09-16 12:43:01,247 44k INFO ====> Epoch: 42, cost 59.62 s 2023-09-16 12:44:01,160 44k INFO ====> Epoch: 43, cost 59.91 s 2023-09-16 12:44:46,695 44k INFO Train Epoch: 44 [72%] 2023-09-16 12:44:46,696 44k INFO Losses: [2.3850936889648438, 2.503847122192383, 12.184123992919922, 24.419109344482422, 1.422132134437561], step: 1400, lr: 4.973195426507915e-05, reference_loss: 42.914302825927734 2023-09-16 12:45:01,383 44k INFO ====> Epoch: 44, cost 60.22 s 2023-09-16 12:46:01,377 44k INFO ====> Epoch: 45, cost 59.99 s 2023-09-16 12:47:00,748 44k INFO ====> Epoch: 46, cost 59.37 s 2023-09-16 12:48:00,555 44k INFO ====> Epoch: 47, cost 59.81 s 2023-09-16 12:49:00,534 44k INFO ====> Epoch: 48, cost 59.98 s 2023-09-16 12:49:59,825 44k INFO ====> Epoch: 49, cost 59.29 s 2023-09-16 12:50:59,670 44k INFO Train Epoch: 50 [97%] 2023-09-16 12:50:59,671 44k INFO Losses: [2.292370557785034, 2.7866265773773193, 13.440462112426758, 23.218414306640625, 0.7844018936157227], step: 1600, lr: 4.969466695336463e-05, reference_loss: 42.522274017333984 2023-09-16 12:51:16,760 44k INFO Saving model and optimizer state at iteration 50 to ./logs/44k/G_1600.pth 2023-09-16 12:51:20,345 44k INFO Saving model and optimizer state at iteration 50 to ./logs/44k/D_1600.pth 2023-09-16 12:51:20,894 44k INFO ====> Epoch: 50, cost 81.07 s 2023-09-16 12:52:20,515 44k INFO ====> Epoch: 51, cost 59.62 s 2023-09-16 12:53:20,572 44k INFO ====> Epoch: 52, cost 60.06 s 2023-09-16 12:54:20,449 44k INFO ====> Epoch: 53, cost 59.88 s 2023-09-16 12:55:20,809 44k INFO ====> Epoch: 54, cost 60.36 s 2023-09-16 12:56:21,212 44k INFO ====> Epoch: 55, cost 60.40 s 2023-09-16 12:57:20,668 44k INFO ====> Epoch: 56, cost 59.46 s 2023-09-16 12:57:35,516 44k INFO Train Epoch: 57 [22%] 2023-09-16 12:57:35,517 44k INFO Losses: [2.336550712585449, 2.6066040992736816, 12.860333442687988, 25.091625213623047, 1.4051625728607178], step: 1800, lr: 4.9651200422446335e-05, reference_loss: 44.30027389526367 2023-09-16 12:58:20,838 44k INFO ====> Epoch: 57, cost 60.17 s 2023-09-16 12:59:20,037 44k INFO ====> Epoch: 58, cost 59.20 s 2023-09-16 13:00:19,832 44k INFO ====> Epoch: 59, cost 59.80 s 2023-09-16 13:01:19,635 44k INFO ====> Epoch: 60, cost 59.80 s 2023-09-16 13:02:18,996 44k INFO ====> Epoch: 61, cost 59.36 s 2023-09-16 13:03:18,684 44k INFO ====> Epoch: 62, cost 59.69 s 2023-09-16 13:03:49,098 44k INFO Train Epoch: 63 [47%] 2023-09-16 13:03:49,099 44k INFO Losses: [2.346595287322998, 2.4011011123657227, 12.764424324035645, 24.9774227142334, 1.327884554862976], step: 2000, lr: 4.961397365719026e-05, reference_loss: 43.81742858886719 2023-09-16 13:04:19,377 44k INFO ====> Epoch: 63, cost 60.69 s 2023-09-16 13:05:19,040 44k INFO ====> Epoch: 64, cost 59.66 s 2023-09-16 13:06:18,831 44k INFO ====> Epoch: 65, cost 59.79 s 2023-09-16 13:07:18,996 44k INFO ====> Epoch: 66, cost 60.17 s 2023-09-16 13:08:18,539 44k INFO ====> Epoch: 67, cost 59.54 s 2023-09-16 13:09:17,651 44k INFO ====> Epoch: 68, cost 59.11 s 2023-09-16 13:10:03,817 44k INFO Train Epoch: 69 [72%] 2023-09-16 13:10:03,818 44k INFO Losses: [2.3135056495666504, 2.355234384536743, 12.109992980957031, 23.99466896057129, 1.3531808853149414], step: 2200, lr: 4.9576774803284575e-05, reference_loss: 42.12657928466797 2023-09-16 13:10:18,040 44k INFO ====> Epoch: 69, cost 60.39 s 2023-09-16 13:11:17,657 44k INFO ====> Epoch: 70, cost 59.62 s 2023-09-16 13:12:16,807 44k INFO ====> Epoch: 71, cost 59.15 s 2023-09-16 13:13:16,611 44k INFO ====> Epoch: 72, cost 59.80 s 2023-09-16 13:14:16,361 44k INFO ====> Epoch: 73, cost 59.75 s 2023-09-16 13:15:15,740 44k INFO ====> Epoch: 74, cost 59.38 s 2023-09-16 13:16:15,049 44k INFO Train Epoch: 75 [97%] 2023-09-16 13:16:15,054 44k INFO Losses: [2.051313877105713, 2.710628032684326, 9.76130485534668, 21.36376190185547, 0.6048551201820374], step: 2400, lr: 4.9539603839802285e-05, reference_loss: 36.49186325073242 2023-09-16 13:16:32,304 44k INFO Saving model and optimizer state at iteration 75 to ./logs/44k/G_2400.pth 2023-09-16 13:16:35,778 44k INFO Saving model and optimizer state at iteration 75 to ./logs/44k/D_2400.pth 2023-09-16 13:16:36,753 44k INFO ====> Epoch: 75, cost 81.01 s 2023-09-16 13:17:36,649 44k INFO ====> Epoch: 76, cost 59.90 s 2023-09-16 13:18:35,824 44k INFO ====> Epoch: 77, cost 59.18 s 2023-09-16 13:19:35,289 44k INFO ====> Epoch: 78, cost 59.46 s 2023-09-16 13:20:34,578 44k INFO ====> Epoch: 79, cost 59.29 s 2023-09-16 13:21:34,271 44k INFO ====> Epoch: 80, cost 59.69 s 2023-09-16 13:22:34,257 44k INFO ====> Epoch: 81, cost 59.99 s 2023-09-16 13:22:49,080 44k INFO Train Epoch: 82 [22%] 2023-09-16 13:22:49,081 44k INFO Losses: [2.3384785652160645, 2.461658477783203, 12.518563270568848, 24.61665916442871, 1.334166169166565], step: 2600, lr: 4.949627293823888e-05, reference_loss: 43.269527435302734 2023-09-16 13:23:34,434 44k INFO ====> Epoch: 82, cost 60.18 s 2023-09-16 13:24:33,600 44k INFO ====> Epoch: 83, cost 59.17 s 2023-09-16 13:25:33,098 44k INFO ====> Epoch: 84, cost 59.50 s 2023-09-16 13:26:32,309 44k INFO ====> Epoch: 85, cost 59.21 s 2023-09-16 13:27:31,363 44k INFO ====> Epoch: 86, cost 59.05 s 2023-09-16 13:28:30,067 44k INFO ====> Epoch: 87, cost 58.70 s 2023-09-16 13:29:00,495 44k INFO Train Epoch: 88 [47%] 2023-09-16 13:29:00,497 44k INFO Losses: [2.2890403270721436, 2.4867539405822754, 12.045918464660645, 24.606721878051758, 1.2668800354003906], step: 2800, lr: 4.945916233229089e-05, reference_loss: 42.695316314697266 2023-09-16 13:29:30,284 44k INFO ====> Epoch: 88, cost 60.22 s 2023-09-16 13:30:29,922 44k INFO ====> Epoch: 89, cost 59.64 s 2023-09-16 13:31:29,601 44k INFO ====> Epoch: 90, cost 59.68 s 2023-09-16 13:32:28,805 44k INFO ====> Epoch: 91, cost 59.20 s 2023-09-16 13:33:28,112 44k INFO ====> Epoch: 92, cost 59.31 s 2023-09-16 13:34:27,472 44k INFO ====> Epoch: 93, cost 59.36 s 2023-09-16 13:35:13,319 44k INFO Train Epoch: 94 [72%] 2023-09-16 13:35:13,321 44k INFO Losses: [2.339224338531494, 2.347076654434204, 12.213699340820312, 23.748262405395508, 1.3135976791381836], step: 3000, lr: 4.942207955060102e-05, reference_loss: 41.96186065673828 2023-09-16 13:35:28,075 44k INFO ====> Epoch: 94, cost 60.60 s 2023-09-16 13:36:27,396 44k INFO ====> Epoch: 95, cost 59.32 s 2023-09-16 13:37:25,950 44k INFO ====> Epoch: 96, cost 58.55 s 2023-09-16 13:38:25,737 44k INFO ====> Epoch: 97, cost 59.79 s 2023-09-16 13:39:25,241 44k INFO ====> Epoch: 98, cost 59.50 s 2023-09-16 13:40:24,405 44k INFO ====> Epoch: 99, cost 59.16 s 2023-09-16 13:41:23,433 44k INFO Train Epoch: 100 [97%] 2023-09-16 13:41:23,434 44k INFO Losses: [2.2206830978393555, 2.5247113704681396, 7.965087890625, 20.766080856323242, 0.4962790608406067], step: 3200, lr: 4.9385024572307584e-05, reference_loss: 33.972843170166016 2023-09-16 13:41:39,600 44k INFO Saving model and optimizer state at iteration 100 to ./logs/44k/G_3200.pth 2023-09-16 13:41:43,090 44k INFO Saving model and optimizer state at iteration 100 to ./logs/44k/D_3200.pth 2023-09-16 13:41:43,767 44k INFO ====> Epoch: 100, cost 79.36 s 2023-09-16 13:42:43,218 44k INFO ====> Epoch: 101, cost 59.45 s 2023-09-16 13:43:42,472 44k INFO ====> Epoch: 102, cost 59.25 s 2023-09-16 13:44:42,148 44k INFO ====> Epoch: 103, cost 59.68 s 2023-09-16 13:45:41,611 44k INFO ====> Epoch: 104, cost 59.46 s 2023-09-16 13:46:41,017 44k INFO ====> Epoch: 105, cost 59.41 s 2023-09-16 13:47:41,415 44k INFO ====> Epoch: 106, cost 60.40 s 2023-09-16 13:47:56,419 44k INFO Train Epoch: 107 [22%] 2023-09-16 13:47:56,420 44k INFO Losses: [2.3962960243225098, 2.5544235706329346, 11.372119903564453, 24.395463943481445, 1.268775224685669], step: 3400, lr: 4.934182887689248e-05, reference_loss: 41.98707962036133 2023-09-16 13:48:41,100 44k INFO ====> Epoch: 107, cost 59.69 s 2023-09-16 13:49:39,980 44k INFO ====> Epoch: 108, cost 58.88 s 2023-09-16 13:50:39,532 44k INFO ====> Epoch: 109, cost 59.55 s 2023-09-16 13:51:38,398 44k INFO ====> Epoch: 110, cost 58.87 s 2023-09-16 13:52:37,198 44k INFO ====> Epoch: 111, cost 58.80 s 2023-09-16 13:53:36,345 44k INFO ====> Epoch: 112, cost 59.15 s 2023-09-16 13:54:06,315 44k INFO Train Epoch: 113 [47%] 2023-09-16 13:54:06,316 44k INFO Losses: [2.4065022468566895, 2.4323601722717285, 11.039319038391113, 24.273286819458008, 1.2752240896224976], step: 3600, lr: 4.93048340677987e-05, reference_loss: 41.42668914794922 2023-09-16 13:54:36,333 44k INFO ====> Epoch: 113, cost 59.99 s 2023-09-16 13:55:35,710 44k INFO ====> Epoch: 114, cost 59.38 s 2023-09-16 13:56:35,331 44k INFO ====> Epoch: 115, cost 59.62 s 2023-09-16 13:57:34,162 44k INFO ====> Epoch: 116, cost 58.83 s 2023-09-16 13:58:32,765 44k INFO ====> Epoch: 117, cost 58.60 s 2023-09-16 13:59:32,118 44k INFO ====> Epoch: 118, cost 59.35 s 2023-09-16 14:00:17,183 44k INFO Train Epoch: 119 [72%] 2023-09-16 14:00:17,184 44k INFO Losses: [2.308865785598755, 2.514223575592041, 12.485522270202637, 23.504226684570312, 1.2794419527053833], step: 3800, lr: 4.926786699614252e-05, reference_loss: 42.092281341552734 2023-09-16 14:00:31,741 44k INFO ====> Epoch: 119, cost 59.62 s 2023-09-16 14:01:30,698 44k INFO ====> Epoch: 120, cost 58.96 s 2023-09-16 14:02:28,818 44k INFO ====> Epoch: 121, cost 58.12 s 2023-09-16 14:03:27,945 44k INFO ====> Epoch: 122, cost 59.13 s 2023-09-16 14:04:26,387 44k INFO ====> Epoch: 123, cost 58.44 s 2023-09-16 14:05:25,577 44k INFO ====> Epoch: 124, cost 59.19 s 2023-09-16 14:06:23,942 44k INFO Train Epoch: 125 [97%] 2023-09-16 14:06:23,943 44k INFO Losses: [2.4853389263153076, 2.2905263900756836, 11.668732643127441, 21.952922821044922, 0.5164315104484558], step: 4000, lr: 4.923092764112739e-05, reference_loss: 38.9139518737793 2023-09-16 14:06:41,326 44k INFO Saving model and optimizer state at iteration 125 to ./logs/44k/G_4000.pth 2023-09-16 14:06:44,400 44k INFO Saving model and optimizer state at iteration 125 to ./logs/44k/D_4000.pth 2023-09-16 14:06:44,932 44k INFO ====> Epoch: 125, cost 79.36 s 2023-09-16 14:07:43,300 44k INFO ====> Epoch: 126, cost 58.37 s 2023-09-16 14:08:41,927 44k INFO ====> Epoch: 127, cost 58.63 s 2023-09-16 14:09:39,662 44k INFO ====> Epoch: 128, cost 57.73 s 2023-09-16 14:10:37,652 44k INFO ====> Epoch: 129, cost 57.99 s 2023-09-16 14:11:36,328 44k INFO ====> Epoch: 130, cost 58.68 s 2023-09-16 14:12:35,398 44k INFO ====> Epoch: 131, cost 59.07 s 2023-09-16 14:12:50,024 44k INFO Train Epoch: 132 [22%] 2023-09-16 14:12:50,025 44k INFO Losses: [2.3675007820129395, 2.672095537185669, 13.927635192871094, 24.46503257751465, 1.2853904962539673], step: 4200, lr: 4.918786672997454e-05, reference_loss: 44.717655181884766 2023-09-16 14:13:34,429 44k INFO ====> Epoch: 132, cost 59.03 s 2023-09-16 14:14:33,322 44k INFO ====> Epoch: 133, cost 58.89 s 2023-09-16 14:15:32,258 44k INFO ====> Epoch: 134, cost 58.94 s 2023-09-16 14:16:31,047 44k INFO ====> Epoch: 135, cost 58.79 s 2023-09-16 14:17:30,127 44k INFO ====> Epoch: 136, cost 59.08 s 2023-09-16 14:18:28,712 44k INFO ====> Epoch: 137, cost 58.58 s 2023-09-16 14:18:58,938 44k INFO Train Epoch: 138 [47%] 2023-09-16 14:18:58,939 44k INFO Losses: [2.3306212425231934, 2.4024248123168945, 12.219969749450684, 24.37347984313965, 1.2190802097320557], step: 4400, lr: 4.915098735641209e-05, reference_loss: 42.54557800292969 2023-09-16 14:19:28,400 44k INFO ====> Epoch: 138, cost 59.69 s 2023-09-16 14:20:27,422 44k INFO ====> Epoch: 139, cost 59.02 s 2023-09-16 14:21:26,088 44k INFO ====> Epoch: 140, cost 58.67 s 2023-09-16 14:22:24,800 44k INFO ====> Epoch: 141, cost 58.71 s 2023-09-16 14:23:23,459 44k INFO ====> Epoch: 142, cost 58.66 s 2023-09-16 14:24:21,984 44k INFO ====> Epoch: 143, cost 58.53 s 2023-09-16 14:25:06,888 44k INFO Train Epoch: 144 [72%] 2023-09-16 14:25:06,889 44k INFO Losses: [2.3279178142547607, 2.3704748153686523, 11.742562294006348, 23.34153175354004, 1.2400590181350708], step: 4600, lr: 4.9114135633737644e-05, reference_loss: 41.022544860839844 2023-09-16 14:25:21,010 44k INFO ====> Epoch: 144, cost 59.03 s 2023-09-16 14:26:20,022 44k INFO ====> Epoch: 145, cost 59.01 s 2023-09-16 14:27:18,668 44k INFO ====> Epoch: 146, cost 58.65 s 2023-09-16 14:28:17,020 44k INFO ====> Epoch: 147, cost 58.35 s 2023-09-16 14:29:15,850 44k INFO ====> Epoch: 148, cost 58.83 s 2023-09-16 14:30:15,189 44k INFO ====> Epoch: 149, cost 59.34 s 2023-09-16 14:31:13,776 44k INFO Train Epoch: 150 [97%] 2023-09-16 14:31:13,777 44k INFO Losses: [2.411179780960083, 2.211221933364868, 7.418693542480469, 20.31729507446289, 0.4073795676231384], step: 4800, lr: 4.907731154121953e-05, reference_loss: 32.765769958496094 2023-09-16 14:31:29,145 44k INFO Saving model and optimizer state at iteration 150 to ./logs/44k/G_4800.pth 2023-09-16 14:31:32,338 44k INFO Saving model and optimizer state at iteration 150 to ./logs/44k/D_4800.pth 2023-09-16 14:31:33,367 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_800.pth 2023-09-16 14:31:33,369 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_800.pth 2023-09-16 14:31:33,380 44k INFO ====> Epoch: 150, cost 78.19 s 2023-09-16 14:32:31,984 44k INFO ====> Epoch: 151, cost 58.60 s 2023-09-16 14:33:30,980 44k INFO ====> Epoch: 152, cost 59.00 s 2023-09-16 14:34:29,767 44k INFO ====> Epoch: 153, cost 58.79 s 2023-09-16 14:35:28,214 44k INFO ====> Epoch: 154, cost 58.45 s 2023-09-16 14:36:26,935 44k INFO ====> Epoch: 155, cost 58.72 s 2023-09-16 14:37:25,463 44k INFO ====> Epoch: 156, cost 58.53 s 2023-09-16 14:37:40,134 44k INFO Train Epoch: 157 [22%] 2023-09-16 14:37:40,135 44k INFO Losses: [2.350313186645508, 2.685488224029541, 11.413718223571777, 23.99038314819336, 1.238631248474121], step: 5000, lr: 4.9034384993759324e-05, reference_loss: 41.67853546142578 2023-09-16 14:38:24,451 44k INFO ====> Epoch: 157, cost 58.99 s 2023-09-16 14:39:23,278 44k INFO ====> Epoch: 158, cost 58.83 s 2023-09-16 14:40:20,967 44k INFO ====> Epoch: 159, cost 57.69 s 2023-09-16 14:41:19,407 44k INFO ====> Epoch: 160, cost 58.44 s 2023-09-16 14:42:17,848 44k INFO ====> Epoch: 161, cost 58.44 s 2023-09-16 14:43:15,959 44k INFO ====> Epoch: 162, cost 58.11 s 2023-09-16 14:43:45,913 44k INFO Train Epoch: 163 [47%] 2023-09-16 14:43:45,915 44k INFO Losses: [2.296901226043701, 2.421416997909546, 12.184147834777832, 24.00956153869629, 1.1922332048416138], step: 5200, lr: 4.899762069553275e-05, reference_loss: 42.1042594909668 2023-09-16 14:44:15,001 44k INFO ====> Epoch: 163, cost 59.04 s 2023-09-16 14:45:13,309 44k INFO ====> Epoch: 164, cost 58.31 s 2023-09-16 14:46:11,700 44k INFO ====> Epoch: 165, cost 58.39 s 2023-09-16 14:47:10,050 44k INFO ====> Epoch: 166, cost 58.35 s 2023-09-16 14:48:08,546 44k INFO ====> Epoch: 167, cost 58.50 s 2023-09-16 14:49:07,781 44k INFO ====> Epoch: 168, cost 59.24 s 2023-09-16 14:49:52,494 44k INFO Train Epoch: 169 [72%] 2023-09-16 14:49:52,495 44k INFO Losses: [2.36460280418396, 2.4558842182159424, 10.932869911193848, 23.47553062438965, 1.2257301807403564], step: 5400, lr: 4.896088396191466e-05, reference_loss: 40.454620361328125 2023-09-16 14:50:07,025 44k INFO ====> Epoch: 169, cost 59.24 s 2023-09-16 14:51:05,741 44k INFO ====> Epoch: 170, cost 58.72 s 2023-09-16 14:52:03,762 44k INFO ====> Epoch: 171, cost 58.02 s 2023-09-16 14:53:02,254 44k INFO ====> Epoch: 172, cost 58.49 s 2023-09-16 14:54:01,136 44k INFO ====> Epoch: 173, cost 58.88 s 2023-09-16 14:54:59,768 44k INFO ====> Epoch: 174, cost 58.63 s 2023-09-16 14:55:58,372 44k INFO Train Epoch: 175 [97%] 2023-09-16 14:55:58,373 44k INFO Losses: [2.1895029544830322, 2.5551979541778564, 9.302614212036133, 21.405208587646484, 0.3923245966434479], step: 5600, lr: 4.892417477223804e-05, reference_loss: 35.8448486328125 2023-09-16 14:56:13,515 44k INFO Saving model and optimizer state at iteration 175 to ./logs/44k/G_5600.pth 2023-09-16 14:56:17,226 44k INFO Saving model and optimizer state at iteration 175 to ./logs/44k/D_5600.pth 2023-09-16 14:56:17,901 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_1600.pth 2023-09-16 14:56:17,902 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_1600.pth 2023-09-16 14:56:17,903 44k INFO ====> Epoch: 175, cost 78.14 s 2023-09-16 14:57:16,947 44k INFO ====> Epoch: 176, cost 59.04 s 2023-09-16 14:58:15,477 44k INFO ====> Epoch: 177, cost 58.53 s 2023-09-16 14:59:14,418 44k INFO ====> Epoch: 178, cost 58.94 s 2023-09-16 15:00:12,753 44k INFO ====> Epoch: 179, cost 58.33 s 2023-09-16 15:01:11,586 44k INFO ====> Epoch: 180, cost 58.83 s 2023-09-16 15:02:10,176 44k INFO ====> Epoch: 181, cost 58.59 s 2023-09-16 15:02:25,167 44k INFO Train Epoch: 182 [22%] 2023-09-16 15:02:25,168 44k INFO Losses: [2.3585941791534424, 2.617913007736206, 11.995933532714844, 24.01803970336914, 1.3171135187149048], step: 5800, lr: 4.8881382169213154e-05, reference_loss: 42.30759048461914 2023-09-16 15:03:09,689 44k INFO ====> Epoch: 182, cost 59.51 s 2023-09-16 15:04:08,188 44k INFO ====> Epoch: 183, cost 58.50 s 2023-09-16 15:05:06,981 44k INFO ====> Epoch: 184, cost 58.79 s 2023-09-16 15:06:05,518 44k INFO ====> Epoch: 185, cost 58.54 s 2023-09-16 15:07:04,429 44k INFO ====> Epoch: 186, cost 58.91 s 2023-09-16 15:08:03,211 44k INFO ====> Epoch: 187, cost 58.78 s 2023-09-16 15:08:33,218 44k INFO Train Epoch: 188 [47%] 2023-09-16 15:08:33,219 44k INFO Losses: [2.354800224304199, 2.435034990310669, 11.404458999633789, 24.232505798339844, 1.155534267425537], step: 6000, lr: 4.884473258725093e-05, reference_loss: 41.582332611083984 2023-09-16 15:09:02,879 44k INFO ====> Epoch: 188, cost 59.67 s 2023-09-16 15:10:01,550 44k INFO ====> Epoch: 189, cost 58.67 s 2023-09-16 15:11:00,817 44k INFO ====> Epoch: 190, cost 59.27 s 2023-09-16 15:11:59,801 44k INFO ====> Epoch: 191, cost 58.98 s 2023-09-16 15:12:58,634 44k INFO ====> Epoch: 192, cost 58.83 s 2023-09-16 15:13:56,958 44k INFO ====> Epoch: 193, cost 58.32 s 2023-09-16 15:14:42,132 44k INFO Train Epoch: 194 [72%] 2023-09-16 15:14:42,133 44k INFO Losses: [2.3305485248565674, 2.4458463191986084, 12.188104629516602, 23.189729690551758, 1.2064619064331055], step: 6200, lr: 4.880811048388686e-05, reference_loss: 41.36069107055664 2023-09-16 15:14:56,808 44k INFO ====> Epoch: 194, cost 59.85 s 2023-09-16 15:15:55,378 44k INFO ====> Epoch: 195, cost 58.57 s 2023-09-16 15:16:53,779 44k INFO ====> Epoch: 196, cost 58.40 s 2023-09-16 15:17:52,859 44k INFO ====> Epoch: 197, cost 59.08 s 2023-09-16 15:18:51,384 44k INFO ====> Epoch: 198, cost 58.53 s 2023-09-16 15:19:50,476 44k INFO ====> Epoch: 199, cost 59.09 s 2023-09-16 15:20:48,988 44k INFO Train Epoch: 200 [97%] 2023-09-16 15:20:48,989 44k INFO Losses: [2.3198413848876953, 2.3342342376708984, 10.613605499267578, 20.360027313232422, 0.3546128273010254], step: 6400, lr: 4.877151583851844e-05, reference_loss: 35.982322692871094 2023-09-16 15:21:04,311 44k INFO Saving model and optimizer state at iteration 200 to ./logs/44k/G_6400.pth 2023-09-16 15:21:07,686 44k INFO Saving model and optimizer state at iteration 200 to ./logs/44k/D_6400.pth 2023-09-16 15:21:08,201 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_2400.pth 2023-09-16 15:21:08,202 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_2400.pth 2023-09-16 15:21:08,203 44k INFO ====> Epoch: 200, cost 77.73 s 2023-09-16 15:22:06,389 44k INFO ====> Epoch: 201, cost 58.19 s 2023-09-16 15:23:04,641 44k INFO ====> Epoch: 202, cost 58.25 s 2023-09-16 15:24:03,621 44k INFO ====> Epoch: 203, cost 58.98 s 2023-09-16 15:25:02,057 44k INFO ====> Epoch: 204, cost 58.44 s 2023-09-16 15:26:00,655 44k INFO ====> Epoch: 205, cost 58.60 s 2023-09-16 15:26:59,486 44k INFO ====> Epoch: 206, cost 58.83 s 2023-09-16 15:27:14,377 44k INFO Train Epoch: 207 [22%] 2023-09-16 15:27:14,378 44k INFO Losses: [2.3860666751861572, 2.6496214866638184, 11.761467933654785, 23.65882682800293, 1.1724090576171875], step: 6600, lr: 4.872885676197979e-05, reference_loss: 41.628395080566406 2023-09-16 15:27:58,641 44k INFO ====> Epoch: 207, cost 59.15 s 2023-09-16 15:28:57,346 44k INFO ====> Epoch: 208, cost 58.70 s 2023-09-16 15:29:55,799 44k INFO ====> Epoch: 209, cost 58.45 s 2023-09-16 15:30:54,449 44k INFO ====> Epoch: 210, cost 58.65 s 2023-09-16 15:31:53,117 44k INFO ====> Epoch: 211, cost 58.67 s 2023-09-16 15:32:51,675 44k INFO ====> Epoch: 212, cost 58.56 s 2023-09-16 15:33:21,728 44k INFO Train Epoch: 213 [47%] 2023-09-16 15:33:21,729 44k INFO Losses: [2.3026514053344727, 2.4771149158477783, 11.325963020324707, 23.92774200439453, 1.1380294561386108], step: 6800, lr: 4.86923215383308e-05, reference_loss: 41.17150115966797 2023-09-16 15:33:51,315 44k INFO ====> Epoch: 213, cost 59.64 s 2023-09-16 15:34:50,126 44k INFO ====> Epoch: 214, cost 58.81 s 2023-09-16 15:35:48,452 44k INFO ====> Epoch: 215, cost 58.33 s 2023-09-16 15:36:47,212 44k INFO ====> Epoch: 216, cost 58.76 s 2023-09-16 15:37:45,529 44k INFO ====> Epoch: 217, cost 58.32 s 2023-09-16 15:38:44,113 44k INFO ====> Epoch: 218, cost 58.58 s 2023-09-16 15:39:29,299 44k INFO Train Epoch: 219 [72%] 2023-09-16 15:39:29,300 44k INFO Losses: [2.3307342529296875, 2.507298469543457, 11.879077911376953, 23.33123779296875, 1.1742091178894043], step: 7000, lr: 4.865581370753804e-05, reference_loss: 41.222557067871094 2023-09-16 15:39:43,670 44k INFO ====> Epoch: 219, cost 59.56 s 2023-09-16 15:40:41,936 44k INFO ====> Epoch: 220, cost 58.27 s 2023-09-16 15:41:40,349 44k INFO ====> Epoch: 221, cost 58.41 s 2023-09-16 15:42:38,965 44k INFO ====> Epoch: 222, cost 58.62 s 2023-09-16 15:43:38,089 44k INFO ====> Epoch: 223, cost 59.12 s 2023-09-16 15:44:36,638 44k INFO ====> Epoch: 224, cost 58.55 s 2023-09-16 15:45:35,231 44k INFO Train Epoch: 225 [97%] 2023-09-16 15:45:35,232 44k INFO Losses: [2.1165480613708496, 2.4751036167144775, 10.339865684509277, 20.52815818786621, 0.3209353983402252], step: 7200, lr: 4.8619333249063276e-05, reference_loss: 35.780609130859375 2023-09-16 15:45:50,640 44k INFO Saving model and optimizer state at iteration 225 to ./logs/44k/G_7200.pth 2023-09-16 15:45:54,156 44k INFO Saving model and optimizer state at iteration 225 to ./logs/44k/D_7200.pth 2023-09-16 15:45:54,874 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_3200.pth 2023-09-16 15:45:54,876 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_3200.pth 2023-09-16 15:45:54,876 44k INFO ====> Epoch: 225, cost 78.24 s 2023-09-16 15:46:53,246 44k INFO ====> Epoch: 226, cost 58.37 s 2023-09-16 15:47:51,639 44k INFO ====> Epoch: 227, cost 58.39 s 2023-09-16 15:48:49,937 44k INFO ====> Epoch: 228, cost 58.30 s 2023-09-16 15:49:48,316 44k INFO ====> Epoch: 229, cost 58.38 s 2023-09-16 15:50:46,613 44k INFO ====> Epoch: 230, cost 58.30 s 2023-09-16 15:51:45,647 44k INFO ====> Epoch: 231, cost 59.03 s 2023-09-16 15:52:00,739 44k INFO Train Epoch: 232 [22%] 2023-09-16 15:52:00,740 44k INFO Losses: [2.2993242740631104, 2.7130074501037598, 13.352561950683594, 24.043598175048828, 1.1999115943908691], step: 7400, lr: 4.8576807282365885e-05, reference_loss: 43.60840606689453 2023-09-16 15:52:44,858 44k INFO ====> Epoch: 232, cost 59.21 s 2023-09-16 15:53:43,542 44k INFO ====> Epoch: 233, cost 58.68 s 2023-09-16 15:54:42,085 44k INFO ====> Epoch: 234, cost 58.54 s 2023-09-16 15:55:40,165 44k INFO ====> Epoch: 235, cost 58.08 s 2023-09-16 15:56:38,612 44k INFO ====> Epoch: 236, cost 58.45 s 2023-09-16 15:57:37,742 44k INFO ====> Epoch: 237, cost 59.13 s 2023-09-16 15:58:07,425 44k INFO Train Epoch: 238 [47%] 2023-09-16 15:58:07,426 44k INFO Losses: [2.3233273029327393, 2.427129030227661, 12.343849182128906, 24.076566696166992, 1.1437433958053589], step: 7600, lr: 4.854038606019595e-05, reference_loss: 42.314613342285156 2023-09-16 15:58:36,899 44k INFO ====> Epoch: 238, cost 59.16 s 2023-09-16 15:59:35,636 44k INFO ====> Epoch: 239, cost 58.74 s 2023-09-16 16:00:34,611 44k INFO ====> Epoch: 240, cost 58.98 s 2023-09-16 16:01:33,060 44k INFO ====> Epoch: 241, cost 58.45 s 2023-09-16 16:02:31,656 44k INFO ====> Epoch: 242, cost 58.60 s 2023-09-16 16:03:30,286 44k INFO ====> Epoch: 243, cost 58.63 s 2023-09-16 16:04:15,210 44k INFO Train Epoch: 244 [72%] 2023-09-16 16:04:15,211 44k INFO Losses: [2.345794439315796, 2.426962375640869, 11.364263534545898, 23.000486373901367, 1.1717053651809692], step: 7800, lr: 4.850399214540784e-05, reference_loss: 40.30921173095703 2023-09-16 16:04:29,322 44k INFO ====> Epoch: 244, cost 59.04 s 2023-09-16 16:05:27,720 44k INFO ====> Epoch: 245, cost 58.40 s 2023-09-16 16:06:26,746 44k INFO ====> Epoch: 246, cost 59.03 s 2023-09-16 16:07:25,484 44k INFO ====> Epoch: 247, cost 58.74 s 2023-09-16 16:08:24,060 44k INFO ====> Epoch: 248, cost 58.58 s 2023-09-16 16:09:23,115 44k INFO ====> Epoch: 249, cost 59.06 s 2023-09-16 16:10:21,691 44k INFO Train Epoch: 250 [97%] 2023-09-16 16:10:21,692 44k INFO Losses: [2.3851068019866943, 2.4012701511383057, 10.109452247619629, 20.45076560974121, 0.24854479730129242], step: 8000, lr: 4.846762551752742e-05, reference_loss: 35.59514236450195 2023-09-16 16:10:39,545 44k INFO Saving model and optimizer state at iteration 250 to ./logs/44k/G_8000.pth 2023-09-16 16:10:41,906 44k INFO Saving model and optimizer state at iteration 250 to ./logs/44k/D_8000.pth 2023-09-16 16:10:42,464 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_4000.pth 2023-09-16 16:10:42,465 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_4000.pth 2023-09-16 16:10:42,466 44k INFO ====> Epoch: 250, cost 79.35 s 2023-09-16 16:11:41,036 44k INFO ====> Epoch: 251, cost 58.57 s 2023-09-16 16:12:39,767 44k INFO ====> Epoch: 252, cost 58.73 s 2023-09-16 16:13:38,840 44k INFO ====> Epoch: 253, cost 59.07 s 2023-09-16 16:14:37,061 44k INFO ====> Epoch: 254, cost 58.22 s 2023-09-16 16:15:35,586 44k INFO ====> Epoch: 255, cost 58.53 s 2023-09-16 16:16:34,204 44k INFO ====> Epoch: 256, cost 58.62 s 2023-09-16 16:16:49,016 44k INFO Train Epoch: 257 [22%] 2023-09-16 16:16:49,017 44k INFO Losses: [2.2936227321624756, 2.645134210586548, 13.104780197143555, 23.892406463623047, 1.1339136362075806], step: 8200, lr: 4.842523224532639e-05, reference_loss: 43.06985855102539 2023-09-16 16:17:33,222 44k INFO ====> Epoch: 257, cost 59.02 s 2023-09-16 16:18:31,683 44k INFO ====> Epoch: 258, cost 58.46 s 2023-09-16 16:19:30,684 44k INFO ====> Epoch: 259, cost 59.00 s 2023-09-16 16:20:29,499 44k INFO ====> Epoch: 260, cost 58.81 s 2023-09-16 16:21:28,665 44k INFO ====> Epoch: 261, cost 59.17 s 2023-09-16 16:22:26,878 44k INFO ====> Epoch: 262, cost 58.21 s 2023-09-16 16:22:56,882 44k INFO Train Epoch: 263 [47%] 2023-09-16 16:22:56,883 44k INFO Losses: [2.2718052864074707, 2.427567958831787, 12.038663864135742, 23.86661720275879, 1.0823076963424683], step: 8400, lr: 4.838892466891475e-05, reference_loss: 41.68695831298828 2023-09-16 16:23:26,404 44k INFO ====> Epoch: 263, cost 59.53 s 2023-09-16 16:24:25,100 44k INFO ====> Epoch: 264, cost 58.70 s 2023-09-16 16:25:24,200 44k INFO ====> Epoch: 265, cost 59.10 s 2023-09-16 16:26:22,846 44k INFO ====> Epoch: 266, cost 58.65 s 2023-09-16 16:27:21,346 44k INFO ====> Epoch: 267, cost 58.50 s 2023-09-16 16:28:19,327 44k INFO ====> Epoch: 268, cost 57.98 s 2023-09-16 16:29:04,248 44k INFO Train Epoch: 269 [72%] 2023-09-16 16:29:04,249 44k INFO Losses: [2.314242362976074, 2.4435787200927734, 11.48090934753418, 22.853073120117188, 1.1435083150863647], step: 8600, lr: 4.8352644314677255e-05, reference_loss: 40.235313415527344 2023-09-16 16:29:18,415 44k INFO ====> Epoch: 269, cost 59.09 s 2023-09-16 16:30:16,486 44k INFO ====> Epoch: 270, cost 58.07 s 2023-09-16 16:31:14,723 44k INFO ====> Epoch: 271, cost 58.24 s 2023-09-16 16:32:13,147 44k INFO ====> Epoch: 272, cost 58.42 s 2023-09-16 16:33:12,030 44k INFO ====> Epoch: 273, cost 58.88 s 2023-09-16 16:34:10,784 44k INFO ====> Epoch: 274, cost 58.75 s 2023-09-16 16:35:09,668 44k INFO Train Epoch: 275 [97%] 2023-09-16 16:35:09,670 44k INFO Losses: [2.26580810546875, 2.5182790756225586, 10.672371864318848, 19.71672821044922, 0.24391864240169525], step: 8800, lr: 4.831639116220366e-05, reference_loss: 35.41710662841797 2023-09-16 16:35:28,690 44k INFO Saving model and optimizer state at iteration 275 to ./logs/44k/G_8800.pth 2023-09-16 16:35:31,913 44k INFO Saving model and optimizer state at iteration 275 to ./logs/44k/D_8800.pth 2023-09-16 16:35:33,050 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_4800.pth 2023-09-16 16:35:33,057 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_4800.pth 2023-09-16 16:35:33,078 44k INFO ====> Epoch: 275, cost 82.29 s 2023-09-16 16:36:32,079 44k INFO ====> Epoch: 276, cost 59.00 s 2023-09-16 16:37:30,657 44k INFO ====> Epoch: 277, cost 58.58 s 2023-09-16 16:38:29,195 44k INFO ====> Epoch: 278, cost 58.54 s 2023-09-16 16:39:27,916 44k INFO ====> Epoch: 279, cost 58.72 s 2023-09-16 16:40:26,216 44k INFO ====> Epoch: 280, cost 58.30 s 2023-09-16 16:41:24,989 44k INFO ====> Epoch: 281, cost 58.77 s 2023-09-16 16:41:40,047 44k INFO Train Epoch: 282 [22%] 2023-09-16 16:41:40,048 44k INFO Losses: [2.2841501235961914, 2.6894373893737793, 12.8440523147583, 23.686574935913086, 1.1450332403182983], step: 9000, lr: 4.82741301704501e-05, reference_loss: 42.64925003051758 2023-09-16 16:42:24,083 44k INFO ====> Epoch: 282, cost 59.09 s 2023-09-16 16:43:22,528 44k INFO ====> Epoch: 283, cost 58.44 s 2023-09-16 16:44:21,247 44k INFO ====> Epoch: 284, cost 58.72 s 2023-09-16 16:45:20,162 44k INFO ====> Epoch: 285, cost 58.91 s 2023-09-16 16:46:18,639 44k INFO ====> Epoch: 286, cost 58.48 s 2023-09-16 16:47:17,077 44k INFO ====> Epoch: 287, cost 58.44 s 2023-09-16 16:47:46,802 44k INFO Train Epoch: 288 [47%] 2023-09-16 16:47:46,803 44k INFO Losses: [2.3420870304107666, 2.4923737049102783, 11.226630210876465, 23.80913734436035, 1.0852935314178467], step: 9200, lr: 4.823793588518598e-05, reference_loss: 40.95552062988281 2023-09-16 16:48:16,021 44k INFO ====> Epoch: 288, cost 58.94 s 2023-09-16 16:49:14,312 44k INFO ====> Epoch: 289, cost 58.29 s 2023-09-16 16:50:12,794 44k INFO ====> Epoch: 290, cost 58.48 s 2023-09-16 16:51:11,325 44k INFO ====> Epoch: 291, cost 58.53 s 2023-09-16 16:52:10,567 44k INFO ====> Epoch: 292, cost 59.24 s 2023-09-16 16:53:09,491 44k INFO ====> Epoch: 293, cost 58.92 s 2023-09-16 16:53:54,451 44k INFO Train Epoch: 294 [72%] 2023-09-16 16:53:54,451 44k INFO Losses: [2.280744791030884, 2.4551944732666016, 12.03333568572998, 22.995790481567383, 1.1131656169891357], step: 9400, lr: 4.820176873715419e-05, reference_loss: 40.87823486328125 2023-09-16 16:54:08,498 44k INFO ====> Epoch: 294, cost 59.01 s 2023-09-16 16:55:07,065 44k INFO ====> Epoch: 295, cost 58.57 s 2023-09-16 16:56:05,792 44k INFO ====> Epoch: 296, cost 58.73 s 2023-09-16 16:57:04,755 44k INFO ====> Epoch: 297, cost 58.96 s 2023-09-16 16:58:03,682 44k INFO ====> Epoch: 298, cost 58.93 s 2023-09-16 16:59:02,772 44k INFO ====> Epoch: 299, cost 59.09 s 2023-09-16 17:00:01,579 44k INFO Train Epoch: 300 [97%] 2023-09-16 17:00:01,580 44k INFO Losses: [2.3280608654022217, 2.467665433883667, 5.814445495605469, 19.27834701538086, 0.20333941280841827], step: 9600, lr: 4.816562870600816e-05, reference_loss: 30.09185791015625 2023-09-16 17:00:19,143 44k INFO Saving model and optimizer state at iteration 300 to ./logs/44k/G_9600.pth 2023-09-16 17:00:22,599 44k INFO Saving model and optimizer state at iteration 300 to ./logs/44k/D_9600.pth 2023-09-16 17:00:23,145 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_5600.pth 2023-09-16 17:00:23,146 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_5600.pth 2023-09-16 17:00:23,147 44k INFO ====> Epoch: 300, cost 80.37 s 2023-09-16 17:01:22,152 44k INFO ====> Epoch: 301, cost 59.01 s 2023-09-16 17:02:21,357 44k INFO ====> Epoch: 302, cost 59.21 s 2023-09-16 17:03:20,041 44k INFO ====> Epoch: 303, cost 58.68 s 2023-09-16 17:04:18,572 44k INFO ====> Epoch: 304, cost 58.53 s 2023-09-16 17:05:16,838 44k INFO ====> Epoch: 305, cost 58.27 s 2023-09-16 17:06:15,693 44k INFO ====> Epoch: 306, cost 58.85 s 2023-09-16 17:06:30,628 44k INFO Train Epoch: 307 [22%] 2023-09-16 17:06:30,629 44k INFO Losses: [2.3072702884674072, 2.5170576572418213, 12.869349479675293, 23.582056045532227, 1.1076796054840088], step: 9800, lr: 4.812349958194515e-05, reference_loss: 42.38341522216797 2023-09-16 17:07:15,457 44k INFO ====> Epoch: 307, cost 59.76 s 2023-09-16 17:08:13,862 44k INFO ====> Epoch: 308, cost 58.40 s 2023-09-16 17:09:12,271 44k INFO ====> Epoch: 309, cost 58.41 s 2023-09-16 17:10:10,615 44k INFO ====> Epoch: 310, cost 58.34 s 2023-09-16 17:11:09,142 44k INFO ====> Epoch: 311, cost 58.53 s 2023-09-16 17:12:07,707 44k INFO ====> Epoch: 312, cost 58.57 s 2023-09-16 17:12:37,874 44k INFO Train Epoch: 313 [47%] 2023-09-16 17:12:37,875 44k INFO Losses: [2.2739038467407227, 2.3874309062957764, 11.755609512329102, 23.8424129486084, 1.0870471000671387], step: 10000, lr: 4.8087418234324244e-05, reference_loss: 41.346405029296875 2023-09-16 17:13:07,123 44k INFO ====> Epoch: 313, cost 59.42 s 2023-09-16 17:14:05,591 44k INFO ====> Epoch: 314, cost 58.47 s 2023-09-16 17:15:04,284 44k INFO ====> Epoch: 315, cost 58.69 s 2023-09-16 17:16:03,015 44k INFO ====> Epoch: 316, cost 58.73 s 2023-09-16 17:17:01,057 44k INFO ====> Epoch: 317, cost 58.04 s 2023-09-16 17:17:59,448 44k INFO ====> Epoch: 318, cost 58.39 s 2023-09-16 17:18:44,656 44k INFO Train Epoch: 319 [72%] 2023-09-16 17:18:44,657 44k INFO Losses: [2.340914249420166, 2.4280502796173096, 11.75757122039795, 22.883378982543945, 1.1001849174499512], step: 10200, lr: 4.80513639392589e-05, reference_loss: 40.51009750366211 2023-09-16 17:18:58,898 44k INFO ====> Epoch: 319, cost 59.45 s 2023-09-16 17:19:57,258 44k INFO ====> Epoch: 320, cost 58.36 s 2023-09-16 17:20:56,092 44k INFO ====> Epoch: 321, cost 58.83 s 2023-09-16 17:21:54,741 44k INFO ====> Epoch: 322, cost 58.65 s 2023-09-16 17:22:53,289 44k INFO ====> Epoch: 323, cost 58.55 s 2023-09-16 17:23:51,794 44k INFO ====> Epoch: 324, cost 58.51 s 2023-09-16 17:24:49,669 44k INFO Train Epoch: 325 [97%] 2023-09-16 17:24:49,670 44k INFO Losses: [2.117244005203247, 2.5343236923217773, 8.269760131835938, 19.406015396118164, 0.24247339367866516], step: 10400, lr: 4.8015336676466044e-05, reference_loss: 32.56981658935547 2023-09-16 17:25:05,900 44k INFO Saving model and optimizer state at iteration 325 to ./logs/44k/G_10400.pth 2023-09-16 17:25:08,886 44k INFO Saving model and optimizer state at iteration 325 to ./logs/44k/D_10400.pth 2023-09-16 17:25:09,404 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_6400.pth 2023-09-16 17:25:09,405 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_6400.pth 2023-09-16 17:25:09,406 44k INFO ====> Epoch: 325, cost 77.61 s 2023-09-16 17:26:08,024 44k INFO ====> Epoch: 326, cost 58.62 s 2023-09-16 17:27:06,870 44k INFO ====> Epoch: 327, cost 58.85 s 2023-09-16 17:28:06,095 44k INFO ====> Epoch: 328, cost 59.23 s 2023-09-16 17:29:04,436 44k INFO ====> Epoch: 329, cost 58.34 s 2023-09-16 17:30:03,102 44k INFO ====> Epoch: 330, cost 58.67 s 2023-09-16 17:31:01,334 44k INFO ====> Epoch: 331, cost 58.23 s 2023-09-16 17:31:16,246 44k INFO Train Epoch: 332 [22%] 2023-09-16 17:31:16,247 44k INFO Losses: [2.3363513946533203, 2.6025750637054443, 13.023463249206543, 23.586917877197266, 1.0915031433105469], step: 10600, lr: 4.797333900862458e-05, reference_loss: 42.640811920166016 2023-09-16 17:32:01,042 44k INFO ====> Epoch: 332, cost 59.71 s 2023-09-16 17:32:59,465 44k INFO ====> Epoch: 333, cost 58.42 s 2023-09-16 17:33:57,607 44k INFO ====> Epoch: 334, cost 58.14 s 2023-09-16 17:34:56,053 44k INFO ====> Epoch: 335, cost 58.45 s 2023-09-16 17:35:55,038 44k INFO ====> Epoch: 336, cost 58.99 s 2023-09-16 17:36:54,380 44k INFO ====> Epoch: 337, cost 59.34 s 2023-09-16 17:37:24,355 44k INFO Train Epoch: 338 [47%] 2023-09-16 17:37:24,356 44k INFO Losses: [2.2901158332824707, 2.377485990524292, 12.612651824951172, 23.82915496826172, 1.0494086742401123], step: 10800, lr: 4.793737024624565e-05, reference_loss: 42.158817291259766 2023-09-16 17:37:53,860 44k INFO ====> Epoch: 338, cost 59.48 s 2023-09-16 17:38:52,781 44k INFO ====> Epoch: 339, cost 58.92 s 2023-09-16 17:39:51,599 44k INFO ====> Epoch: 340, cost 58.82 s 2023-09-16 17:40:50,803 44k INFO ====> Epoch: 341, cost 59.20 s 2023-09-16 17:41:49,394 44k INFO ====> Epoch: 342, cost 58.59 s 2023-09-16 17:42:47,958 44k INFO ====> Epoch: 343, cost 58.56 s 2023-09-16 17:43:33,396 44k INFO Train Epoch: 344 [72%] 2023-09-16 17:43:33,398 44k INFO Losses: [2.3094277381896973, 2.395076274871826, 12.045500755310059, 22.6531925201416, 1.0855735540390015], step: 11000, lr: 4.790142845200973e-05, reference_loss: 40.488773345947266 2023-09-16 17:43:47,963 44k INFO ====> Epoch: 344, cost 60.00 s 2023-09-16 17:44:46,693 44k INFO ====> Epoch: 345, cost 58.73 s 2023-09-16 17:45:45,619 44k INFO ====> Epoch: 346, cost 58.93 s 2023-09-16 17:46:44,016 44k INFO ====> Epoch: 347, cost 58.40 s 2023-09-16 17:47:42,336 44k INFO ====> Epoch: 348, cost 58.32 s 2023-09-16 17:48:40,870 44k INFO ====> Epoch: 349, cost 58.53 s 2023-09-16 17:49:39,199 44k INFO Train Epoch: 350 [97%] 2023-09-16 17:49:39,200 44k INFO Losses: [2.1475062370300293, 2.568636417388916, 9.825151443481445, 19.0902042388916, 0.13980357348918915], step: 11200, lr: 4.786551360569703e-05, reference_loss: 33.771305084228516 2023-09-16 17:49:55,781 44k INFO Saving model and optimizer state at iteration 350 to ./logs/44k/G_11200.pth 2023-09-16 17:49:59,109 44k INFO Saving model and optimizer state at iteration 350 to ./logs/44k/D_11200.pth 2023-09-16 17:50:00,192 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_7200.pth 2023-09-16 17:50:00,194 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_7200.pth 2023-09-16 17:50:00,194 44k INFO ====> Epoch: 350, cost 79.32 s 2023-09-16 17:50:58,921 44k INFO ====> Epoch: 351, cost 58.73 s 2023-09-16 17:51:57,366 44k INFO ====> Epoch: 352, cost 58.45 s 2023-09-16 17:52:55,491 44k INFO ====> Epoch: 353, cost 58.12 s 2023-09-16 17:53:53,889 44k INFO ====> Epoch: 354, cost 58.40 s 2023-09-16 17:54:52,668 44k INFO ====> Epoch: 355, cost 58.78 s 2023-09-16 17:55:51,927 44k INFO ====> Epoch: 356, cost 59.26 s 2023-09-16 17:56:06,857 44k INFO Train Epoch: 357 [22%] 2023-09-16 17:56:06,858 44k INFO Losses: [2.306858539581299, 2.6192243099212646, 12.497739791870117, 23.623796463012695, 1.07131826877594], step: 11400, lr: 4.782364698389204e-05, reference_loss: 42.11893844604492 2023-09-16 17:56:51,313 44k INFO ====> Epoch: 357, cost 59.39 s 2023-09-16 17:57:50,130 44k INFO ====> Epoch: 358, cost 58.82 s 2023-09-16 17:58:48,998 44k INFO ====> Epoch: 359, cost 58.87 s 2023-09-16 17:59:47,467 44k INFO ====> Epoch: 360, cost 58.47 s 2023-09-16 18:00:46,373 44k INFO ====> Epoch: 361, cost 58.91 s 2023-09-16 18:01:45,040 44k INFO ====> Epoch: 362, cost 58.67 s 2023-09-16 18:02:14,518 44k INFO Train Epoch: 363 [47%] 2023-09-16 18:02:14,519 44k INFO Losses: [2.242711067199707, 2.5575661659240723, 12.48481273651123, 23.584312438964844, 1.0403454303741455], step: 11600, lr: 4.778779045545343e-05, reference_loss: 41.90974426269531 2023-09-16 18:02:44,547 44k INFO ====> Epoch: 363, cost 59.51 s 2023-09-16 18:03:43,245 44k INFO ====> Epoch: 364, cost 58.70 s 2023-09-16 18:04:41,770 44k INFO ====> Epoch: 365, cost 58.53 s 2023-09-16 18:05:40,629 44k INFO ====> Epoch: 366, cost 58.86 s 2023-09-16 18:06:39,706 44k INFO ====> Epoch: 367, cost 59.08 s 2023-09-16 18:07:38,056 44k INFO ====> Epoch: 368, cost 58.35 s 2023-09-16 18:08:23,026 44k INFO Train Epoch: 369 [72%] 2023-09-16 18:08:23,027 44k INFO Losses: [2.3077714443206787, 2.483860492706299, 12.855877876281738, 22.95302963256836, 1.0582799911499023], step: 11800, lr: 4.775196081100868e-05, reference_loss: 41.65882110595703 2023-09-16 18:08:37,207 44k INFO ====> Epoch: 369, cost 59.15 s 2023-09-16 18:09:35,427 44k INFO ====> Epoch: 370, cost 58.22 s 2023-09-16 18:10:34,281 44k INFO ====> Epoch: 371, cost 58.85 s 2023-09-16 18:11:32,965 44k INFO ====> Epoch: 372, cost 58.68 s 2023-09-16 18:12:32,008 44k INFO ====> Epoch: 373, cost 59.04 s 2023-09-16 18:13:30,818 44k INFO ====> Epoch: 374, cost 58.81 s 2023-09-16 18:14:29,104 44k INFO Train Epoch: 375 [97%] 2023-09-16 18:14:29,105 44k INFO Losses: [2.143369674682617, 2.451078414916992, 8.408323287963867, 18.972890853881836, 0.12158255279064178], step: 12000, lr: 4.771615803040109e-05, reference_loss: 32.09724426269531 2023-09-16 18:14:46,537 44k INFO Saving model and optimizer state at iteration 375 to ./logs/44k/G_12000.pth 2023-09-16 18:14:48,753 44k INFO Saving model and optimizer state at iteration 375 to ./logs/44k/D_12000.pth 2023-09-16 18:14:49,325 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_8000.pth 2023-09-16 18:14:49,327 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_8000.pth 2023-09-16 18:14:49,327 44k INFO ====> Epoch: 375, cost 78.51 s 2023-09-16 18:15:49,369 44k INFO ====> Epoch: 376, cost 60.04 s 2023-09-16 18:16:49,468 44k INFO ====> Epoch: 377, cost 60.10 s 2023-09-16 18:17:48,753 44k INFO ====> Epoch: 378, cost 59.28 s 2023-09-16 18:18:48,070 44k INFO ====> Epoch: 379, cost 59.32 s 2023-09-16 18:19:47,985 44k INFO ====> Epoch: 380, cost 59.91 s 2023-09-16 18:20:47,835 44k INFO ====> Epoch: 381, cost 59.85 s 2023-09-16 18:21:03,260 44k INFO Train Epoch: 382 [22%] 2023-09-16 18:21:03,261 44k INFO Losses: [2.3252856731414795, 2.4889543056488037, 12.968439102172852, 23.6538143157959, 1.073981761932373], step: 12200, lr: 4.7674422045727384e-05, reference_loss: 42.510475158691406 2023-09-16 18:21:48,757 44k INFO ====> Epoch: 382, cost 60.92 s 2023-09-16 18:22:48,390 44k INFO ====> Epoch: 383, cost 59.63 s 2023-09-16 18:23:47,761 44k INFO ====> Epoch: 384, cost 59.37 s 2023-09-16 18:24:47,038 44k INFO ====> Epoch: 385, cost 59.28 s 2023-09-16 18:25:46,490 44k INFO ====> Epoch: 386, cost 59.45 s 2023-09-16 18:26:46,325 44k INFO ====> Epoch: 387, cost 59.84 s 2023-09-16 18:27:16,773 44k INFO Train Epoch: 388 [47%] 2023-09-16 18:27:16,774 44k INFO Losses: [2.264629364013672, 2.5261735916137695, 12.505435943603516, 23.58946418762207, 1.0394909381866455], step: 12400, lr: 4.763867740102364e-05, reference_loss: 41.925193786621094 2023-09-16 18:27:46,722 44k INFO ====> Epoch: 388, cost 60.40 s 2023-09-16 18:28:45,770 44k INFO ====> Epoch: 389, cost 59.05 s 2023-09-16 18:29:45,303 44k INFO ====> Epoch: 390, cost 59.53 s 2023-09-16 18:30:44,448 44k INFO ====> Epoch: 391, cost 59.15 s 2023-09-16 18:31:44,105 44k INFO ====> Epoch: 392, cost 59.66 s 2023-09-16 18:32:43,476 44k INFO ====> Epoch: 393, cost 59.37 s 2023-09-16 18:33:29,266 44k INFO Train Epoch: 394 [72%] 2023-09-16 18:33:29,267 44k INFO Losses: [2.251061201095581, 2.5661237239837646, 11.88205337524414, 22.540912628173828, 1.0661029815673828], step: 12600, lr: 4.7602959556427164e-05, reference_loss: 40.306251525878906 2023-09-16 18:33:43,922 44k INFO ====> Epoch: 394, cost 60.45 s 2023-09-16 18:34:43,826 44k INFO ====> Epoch: 395, cost 59.90 s 2023-09-16 18:35:43,281 44k INFO ====> Epoch: 396, cost 59.46 s 2023-09-16 18:36:43,242 44k INFO ====> Epoch: 397, cost 59.96 s 2023-09-16 18:37:42,675 44k INFO ====> Epoch: 398, cost 59.43 s 2023-09-16 18:38:42,200 44k INFO ====> Epoch: 399, cost 59.52 s 2023-09-16 18:39:41,642 44k INFO Train Epoch: 400 [97%] 2023-09-16 18:39:41,643 44k INFO Losses: [1.809727430343628, 2.7921559810638428, 12.47323989868164, 18.8375244140625, 0.12455820292234421], step: 12800, lr: 4.756726849184417e-05, reference_loss: 36.03720474243164 2023-09-16 18:39:58,340 44k INFO Saving model and optimizer state at iteration 400 to ./logs/44k/G_12800.pth 2023-09-16 18:40:01,462 44k INFO Saving model and optimizer state at iteration 400 to ./logs/44k/D_12800.pth 2023-09-16 18:40:02,606 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_8800.pth 2023-09-16 18:40:02,608 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_8800.pth 2023-09-16 18:40:02,629 44k INFO ====> Epoch: 400, cost 80.43 s 2023-09-16 18:41:03,160 44k INFO ====> Epoch: 401, cost 60.53 s 2023-09-16 18:42:02,513 44k INFO ====> Epoch: 402, cost 59.35 s 2023-09-16 18:43:01,983 44k INFO ====> Epoch: 403, cost 59.47 s 2023-09-16 18:44:01,153 44k INFO ====> Epoch: 404, cost 59.17 s 2023-09-16 18:45:00,524 44k INFO ====> Epoch: 405, cost 59.37 s 2023-09-16 18:46:00,149 44k INFO ====> Epoch: 406, cost 59.62 s 2023-09-16 18:46:15,555 44k INFO Train Epoch: 407 [22%] 2023-09-16 18:46:15,556 44k INFO Losses: [2.3184242248535156, 2.529550075531006, 12.746094703674316, 23.5833797454834, 1.1019816398620605], step: 13000, lr: 4.752566273667251e-05, reference_loss: 42.2794303894043 2023-09-16 18:47:01,051 44k INFO ====> Epoch: 407, cost 60.90 s 2023-09-16 18:48:01,116 44k INFO ====> Epoch: 408, cost 60.07 s 2023-09-16 18:49:01,226 44k INFO ====> Epoch: 409, cost 60.11 s 2023-09-16 18:50:00,885 44k INFO ====> Epoch: 410, cost 59.66 s 2023-09-16 18:51:00,936 44k INFO ====> Epoch: 411, cost 60.05 s 2023-09-16 18:52:00,343 44k INFO ====> Epoch: 412, cost 59.41 s 2023-09-16 18:52:31,155 44k INFO Train Epoch: 413 [47%] 2023-09-16 18:52:31,156 44k INFO Losses: [2.285881757736206, 2.4832353591918945, 11.654311180114746, 23.19939613342285, 1.0028486251831055], step: 13200, lr: 4.7490029626590895e-05, reference_loss: 40.625675201416016 2023-09-16 18:53:00,895 44k INFO ====> Epoch: 413, cost 60.55 s 2023-09-16 18:54:01,008 44k INFO ====> Epoch: 414, cost 60.11 s 2023-09-16 18:55:01,236 44k INFO ====> Epoch: 415, cost 60.23 s 2023-09-16 18:56:01,147 44k INFO ====> Epoch: 416, cost 59.91 s 2023-09-16 18:57:00,801 44k INFO ====> Epoch: 417, cost 59.65 s 2023-09-16 18:58:00,772 44k INFO ====> Epoch: 418, cost 59.97 s 2023-09-16 18:58:46,783 44k INFO Train Epoch: 419 [72%] 2023-09-16 18:58:46,784 44k INFO Losses: [2.3032612800598145, 2.4761452674865723, 11.921201705932617, 22.580970764160156, 1.0665831565856934], step: 13400, lr: 4.7454423232991734e-05, reference_loss: 40.34815979003906 2023-09-16 18:59:01,590 44k INFO ====> Epoch: 419, cost 60.82 s 2023-09-16 19:00:01,380 44k INFO ====> Epoch: 420, cost 59.79 s 2023-09-16 19:01:01,330 44k INFO ====> Epoch: 421, cost 59.95 s 2023-09-16 19:02:00,826 44k INFO ====> Epoch: 422, cost 59.50 s 2023-09-16 19:03:00,694 44k INFO ====> Epoch: 423, cost 59.87 s 2023-09-16 19:04:00,537 44k INFO ====> Epoch: 424, cost 59.84 s 2023-09-16 19:05:00,364 44k INFO Train Epoch: 425 [97%] 2023-09-16 19:05:00,365 44k INFO Losses: [2.277089834213257, 2.529310941696167, 10.782842636108398, 19.87291717529297, 0.16931936144828796], step: 13600, lr: 4.741884353584391e-05, reference_loss: 35.63147735595703 2023-09-16 19:05:17,397 44k INFO Saving model and optimizer state at iteration 425 to ./logs/44k/G_13600.pth 2023-09-16 19:05:21,042 44k INFO Saving model and optimizer state at iteration 425 to ./logs/44k/D_13600.pth 2023-09-16 19:05:21,668 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_9600.pth 2023-09-16 19:05:21,670 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_9600.pth 2023-09-16 19:05:21,670 44k INFO ====> Epoch: 425, cost 81.13 s 2023-09-16 19:06:21,464 44k INFO ====> Epoch: 426, cost 59.79 s 2023-09-16 19:07:21,264 44k INFO ====> Epoch: 427, cost 59.80 s 2023-09-16 19:08:21,113 44k INFO ====> Epoch: 428, cost 59.85 s 2023-09-16 19:09:20,851 44k INFO ====> Epoch: 429, cost 59.74 s 2023-09-16 19:10:20,977 44k INFO ====> Epoch: 430, cost 60.13 s 2023-09-16 19:11:20,663 44k INFO ====> Epoch: 431, cost 59.69 s 2023-09-16 19:11:35,710 44k INFO Train Epoch: 432 [22%] 2023-09-16 19:11:35,711 44k INFO Losses: [2.318906545639038, 2.5783965587615967, 13.056275367736816, 23.463642120361328, 1.049059510231018], step: 13800, lr: 4.737736760381696e-05, reference_loss: 42.46628189086914 2023-09-16 19:12:21,294 44k INFO ====> Epoch: 432, cost 60.63 s 2023-09-16 19:13:21,618 44k INFO ====> Epoch: 433, cost 60.32 s 2023-09-16 19:14:21,803 44k INFO ====> Epoch: 434, cost 60.18 s 2023-09-16 19:15:20,952 44k INFO ====> Epoch: 435, cost 59.15 s 2023-09-16 19:16:20,859 44k INFO ====> Epoch: 436, cost 59.91 s 2023-09-16 19:17:19,990 44k INFO ====> Epoch: 437, cost 59.13 s 2023-09-16 19:17:50,422 44k INFO Train Epoch: 438 [47%] 2023-09-16 19:17:50,423 44k INFO Losses: [2.2192094326019287, 2.52052640914917, 13.454492568969727, 23.550182342529297, 1.0086121559143066], step: 14000, lr: 4.7341845680334117e-05, reference_loss: 42.75302505493164 2023-09-16 19:18:20,460 44k INFO ====> Epoch: 438, cost 60.47 s 2023-09-16 19:19:20,355 44k INFO ====> Epoch: 439, cost 59.90 s 2023-09-16 19:20:19,801 44k INFO ====> Epoch: 440, cost 59.45 s 2023-09-16 19:21:19,689 44k INFO ====> Epoch: 441, cost 59.89 s 2023-09-16 19:22:19,757 44k INFO ====> Epoch: 442, cost 60.07 s 2023-09-16 19:23:19,693 44k INFO ====> Epoch: 443, cost 59.94 s 2023-09-16 19:24:05,569 44k INFO Train Epoch: 444 [72%] 2023-09-16 19:24:05,570 44k INFO Losses: [2.3082690238952637, 2.4295170307159424, 11.905156135559082, 22.6258487701416, 1.0415350198745728], step: 14200, lr: 4.730635038996982e-05, reference_loss: 40.310325622558594 2023-09-16 19:24:20,038 44k INFO ====> Epoch: 444, cost 60.34 s 2023-09-16 19:25:19,469 44k INFO ====> Epoch: 445, cost 59.43 s 2023-09-16 19:26:19,453 44k INFO ====> Epoch: 446, cost 59.98 s 2023-09-16 19:27:19,608 44k INFO ====> Epoch: 447, cost 60.15 s 2023-09-16 19:28:19,827 44k INFO ====> Epoch: 448, cost 60.22 s 2023-09-16 19:29:19,837 44k INFO ====> Epoch: 449, cost 60.01 s 2023-09-16 19:30:19,402 44k INFO Train Epoch: 450 [97%] 2023-09-16 19:30:19,404 44k INFO Losses: [2.104605197906494, 2.7016584873199463, 8.369061470031738, 19.054794311523438, 0.052650030702352524], step: 14400, lr: 4.7270881712755474e-05, reference_loss: 32.28276824951172 2023-09-16 19:30:36,565 44k INFO Saving model and optimizer state at iteration 450 to ./logs/44k/G_14400.pth 2023-09-16 19:30:39,451 44k INFO Saving model and optimizer state at iteration 450 to ./logs/44k/D_14400.pth 2023-09-16 19:30:40,030 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_10400.pth 2023-09-16 19:30:40,032 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_10400.pth 2023-09-16 19:30:40,032 44k INFO ====> Epoch: 450, cost 80.20 s 2023-09-16 19:31:40,014 44k INFO ====> Epoch: 451, cost 59.98 s 2023-09-16 19:32:39,413 44k INFO ====> Epoch: 452, cost 59.40 s 2023-09-16 19:33:39,584 44k INFO ====> Epoch: 453, cost 60.17 s 2023-09-16 19:34:39,695 44k INFO ====> Epoch: 454, cost 60.11 s 2023-09-16 19:35:39,272 44k INFO ====> Epoch: 455, cost 59.58 s 2023-09-16 19:36:38,357 44k INFO ====> Epoch: 456, cost 59.08 s 2023-09-16 19:36:53,321 44k INFO Train Epoch: 457 [22%] 2023-09-16 19:36:53,322 44k INFO Losses: [2.365976333618164, 2.5024116039276123, 12.863001823425293, 23.381567001342773, 1.0253925323486328], step: 14600, lr: 4.7229535198783855e-05, reference_loss: 42.13835144042969 2023-09-16 19:37:37,895 44k INFO ====> Epoch: 457, cost 59.54 s 2023-09-16 19:38:36,716 44k INFO ====> Epoch: 458, cost 58.82 s 2023-09-16 19:39:35,819 44k INFO ====> Epoch: 459, cost 59.10 s 2023-09-16 19:40:34,969 44k INFO ====> Epoch: 460, cost 59.15 s 2023-09-16 19:41:33,347 44k INFO ====> Epoch: 461, cost 58.38 s 2023-09-16 19:42:32,030 44k INFO ====> Epoch: 462, cost 58.68 s 2023-09-16 19:43:01,802 44k INFO Train Epoch: 463 [47%] 2023-09-16 19:43:01,803 44k INFO Losses: [2.3118598461151123, 2.534996747970581, 12.109854698181152, 23.4189510345459, 0.9948534369468689], step: 14800, lr: 4.7194124114962335e-05, reference_loss: 41.37051773071289 2023-09-16 19:43:31,009 44k INFO ====> Epoch: 463, cost 58.98 s 2023-09-16 19:44:29,458 44k INFO ====> Epoch: 464, cost 58.45 s 2023-09-16 19:45:28,499 44k INFO ====> Epoch: 465, cost 59.04 s 2023-09-16 19:46:27,284 44k INFO ====> Epoch: 466, cost 58.79 s 2023-09-16 19:47:25,439 44k INFO ====> Epoch: 467, cost 58.15 s 2023-09-16 19:48:23,504 44k INFO ====> Epoch: 468, cost 58.06 s 2023-09-16 19:49:08,155 44k INFO Train Epoch: 469 [72%] 2023-09-16 19:49:08,156 44k INFO Losses: [2.2332561016082764, 2.5682907104492188, 12.872559547424316, 22.35428810119629, 1.0195051431655884], step: 15000, lr: 4.715873958115559e-05, reference_loss: 41.04790115356445 2023-09-16 19:49:22,573 44k INFO ====> Epoch: 469, cost 59.07 s 2023-09-16 19:50:21,987 44k INFO ====> Epoch: 470, cost 59.41 s 2023-09-16 19:51:20,506 44k INFO ====> Epoch: 471, cost 58.52 s 2023-09-16 19:52:18,862 44k INFO ====> Epoch: 472, cost 58.36 s 2023-09-16 19:53:17,342 44k INFO ====> Epoch: 473, cost 58.48 s 2023-09-16 19:54:15,643 44k INFO ====> Epoch: 474, cost 58.30 s 2023-09-16 19:55:14,050 44k INFO Train Epoch: 475 [97%] 2023-09-16 19:55:14,052 44k INFO Losses: [1.9509992599487305, 2.6935105323791504, 7.880533695220947, 18.6888427734375, 0.03252580761909485], step: 15200, lr: 4.712338157745734e-05, reference_loss: 31.24641227722168 2023-09-16 19:55:30,850 44k INFO Saving model and optimizer state at iteration 475 to ./logs/44k/G_15200.pth 2023-09-16 19:55:34,225 44k INFO Saving model and optimizer state at iteration 475 to ./logs/44k/D_15200.pth 2023-09-16 19:55:35,293 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_11200.pth 2023-09-16 19:55:35,295 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_11200.pth 2023-09-16 19:55:35,296 44k INFO ====> Epoch: 475, cost 79.65 s 2023-09-16 19:56:33,619 44k INFO ====> Epoch: 476, cost 58.32 s 2023-09-16 19:57:32,456 44k INFO ====> Epoch: 477, cost 58.84 s 2023-09-16 19:58:31,109 44k INFO ====> Epoch: 478, cost 58.65 s 2023-09-16 19:59:29,908 44k INFO ====> Epoch: 479, cost 58.80 s 2023-09-16 20:00:28,683 44k INFO ====> Epoch: 480, cost 58.78 s 2023-09-16 20:01:27,425 44k INFO ====> Epoch: 481, cost 58.74 s 2023-09-16 20:01:42,278 44k INFO Train Epoch: 482 [22%] 2023-09-16 20:01:42,279 44k INFO Losses: [2.316898822784424, 2.5798535346984863, 12.736610412597656, 23.237014770507812, 1.0049892663955688], step: 15400, lr: 4.7082164077715716e-05, reference_loss: 41.8753662109375 2023-09-16 20:02:26,543 44k INFO ====> Epoch: 482, cost 59.12 s 2023-09-16 20:03:25,356 44k INFO ====> Epoch: 483, cost 58.81 s 2023-09-16 20:04:24,128 44k INFO ====> Epoch: 484, cost 58.77 s 2023-09-16 20:05:22,773 44k INFO ====> Epoch: 485, cost 58.64 s 2023-09-16 20:06:21,084 44k INFO ====> Epoch: 486, cost 58.31 s 2023-09-16 20:07:20,075 44k INFO ====> Epoch: 487, cost 58.99 s 2023-09-16 20:07:49,924 44k INFO Train Epoch: 488 [47%] 2023-09-16 20:07:49,925 44k INFO Losses: [2.2829458713531494, 2.5709478855133057, 12.695841789245605, 23.35027313232422, 0.9583988785743713], step: 15600, lr: 4.704686348770066e-05, reference_loss: 41.8584098815918 2023-09-16 20:08:19,169 44k INFO ====> Epoch: 488, cost 59.09 s 2023-09-16 20:09:17,917 44k INFO ====> Epoch: 489, cost 58.75 s 2023-09-16 20:10:16,771 44k INFO ====> Epoch: 490, cost 58.85 s 2023-09-16 20:11:15,268 44k INFO ====> Epoch: 491, cost 58.50 s 2023-09-16 20:12:13,926 44k INFO ====> Epoch: 492, cost 58.66 s 2023-09-16 20:13:13,044 44k INFO ====> Epoch: 493, cost 59.12 s 2023-09-16 20:13:58,170 44k INFO Train Epoch: 494 [72%] 2023-09-16 20:13:58,171 44k INFO Losses: [2.2300868034362793, 2.5533671379089355, 13.131590843200684, 22.640575408935547, 1.045279860496521], step: 15800, lr: 4.7011589364855904e-05, reference_loss: 41.60089874267578 2023-09-16 20:14:12,559 44k INFO ====> Epoch: 494, cost 59.52 s 2023-09-16 20:15:11,685 44k INFO ====> Epoch: 495, cost 59.13 s 2023-09-16 20:16:10,543 44k INFO ====> Epoch: 496, cost 58.86 s 2023-09-16 20:17:08,641 44k INFO ====> Epoch: 497, cost 58.10 s 2023-09-16 20:18:07,213 44k INFO ====> Epoch: 498, cost 58.57 s 2023-09-16 20:19:05,829 44k INFO ====> Epoch: 499, cost 58.62 s 2023-09-16 20:20:04,783 44k INFO Train Epoch: 500 [97%] 2023-09-16 20:20:04,785 44k INFO Losses: [2.2327396869659424, 2.466789722442627, 7.687862873077393, 18.67744255065918, 0.13621366024017334], step: 16000, lr: 4.697634168933729e-05, reference_loss: 31.201047897338867 2023-09-16 20:20:21,699 44k INFO Saving model and optimizer state at iteration 500 to ./logs/44k/G_16000.pth 2023-09-16 20:20:24,450 44k INFO Saving model and optimizer state at iteration 500 to ./logs/44k/D_16000.pth 2023-09-16 20:20:24,999 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_12000.pth 2023-09-16 20:20:25,000 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_12000.pth 2023-09-16 20:20:25,001 44k INFO ====> Epoch: 500, cost 79.17 s 2023-09-16 20:21:23,306 44k INFO ====> Epoch: 501, cost 58.31 s 2023-09-16 20:22:21,882 44k INFO ====> Epoch: 502, cost 58.58 s 2023-09-16 20:23:20,284 44k INFO ====> Epoch: 503, cost 58.40 s 2023-09-16 20:24:18,983 44k INFO ====> Epoch: 504, cost 58.70 s 2023-09-16 20:25:17,309 44k INFO ====> Epoch: 505, cost 58.33 s 2023-09-16 20:26:16,093 44k INFO ====> Epoch: 506, cost 58.78 s 2023-09-16 20:26:30,848 44k INFO Train Epoch: 507 [22%] 2023-09-16 20:26:30,849 44k INFO Losses: [2.321916341781616, 2.713341474533081, 13.398487091064453, 23.39422607421875, 1.0232032537460327], step: 16200, lr: 4.693525280126035e-05, reference_loss: 42.85117721557617 2023-09-16 20:27:14,872 44k INFO ====> Epoch: 507, cost 58.78 s 2023-09-16 20:28:12,991 44k INFO ====> Epoch: 508, cost 58.12 s 2023-09-16 20:29:11,855 44k INFO ====> Epoch: 509, cost 58.86 s 2023-09-16 20:30:10,919 44k INFO ====> Epoch: 510, cost 59.06 s 2023-09-16 20:31:09,200 44k INFO ====> Epoch: 511, cost 58.28 s 2023-09-16 20:32:08,398 44k INFO ====> Epoch: 512, cost 59.20 s 2023-09-16 20:32:37,930 44k INFO Train Epoch: 513 [47%] 2023-09-16 20:32:37,931 44k INFO Losses: [2.257777452468872, 2.4959940910339355, 12.89642333984375, 23.438934326171875, 0.9826414585113525], step: 16400, lr: 4.6900062360276036e-05, reference_loss: 42.071773529052734 2023-09-16 20:33:07,120 44k INFO ====> Epoch: 513, cost 58.72 s 2023-09-16 20:34:05,888 44k INFO ====> Epoch: 514, cost 58.77 s 2023-09-16 20:35:04,266 44k INFO ====> Epoch: 515, cost 58.38 s 2023-09-16 20:36:03,396 44k INFO ====> Epoch: 516, cost 59.13 s 2023-09-16 20:37:02,534 44k INFO ====> Epoch: 517, cost 59.14 s 2023-09-16 20:38:01,164 44k INFO ====> Epoch: 518, cost 58.63 s 2023-09-16 20:38:46,227 44k INFO Train Epoch: 519 [72%] 2023-09-16 20:38:46,228 44k INFO Losses: [2.2881667613983154, 2.366387128829956, 12.607439041137695, 22.52545738220215, 1.0061533451080322], step: 16600, lr: 4.686489830387608e-05, reference_loss: 40.79360580444336 2023-09-16 20:39:00,737 44k INFO ====> Epoch: 519, cost 59.57 s 2023-09-16 20:39:59,126 44k INFO ====> Epoch: 520, cost 58.39 s 2023-09-16 20:40:57,607 44k INFO ====> Epoch: 521, cost 58.48 s 2023-09-16 20:41:56,538 44k INFO ====> Epoch: 522, cost 58.93 s 2023-09-16 20:42:54,748 44k INFO ====> Epoch: 523, cost 58.21 s 2023-09-16 20:43:53,201 44k INFO ====> Epoch: 524, cost 58.45 s 2023-09-16 20:44:51,886 44k INFO Train Epoch: 525 [97%] 2023-09-16 20:44:51,887 44k INFO Losses: [1.9956929683685303, 2.6649396419525146, 9.327006340026855, 18.393260955810547, 0.021422285586595535], step: 16800, lr: 4.6829760612278214e-05, reference_loss: 32.402320861816406 2023-09-16 20:45:08,248 44k INFO Saving model and optimizer state at iteration 525 to ./logs/44k/G_16800.pth 2023-09-16 20:45:11,187 44k INFO Saving model and optimizer state at iteration 525 to ./logs/44k/D_16800.pth 2023-09-16 20:45:12,169 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_12800.pth 2023-09-16 20:45:12,170 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_12800.pth 2023-09-16 20:45:12,182 44k INFO ====> Epoch: 525, cost 78.98 s 2023-09-16 20:46:11,272 44k INFO ====> Epoch: 526, cost 59.09 s 2023-09-16 20:47:09,583 44k INFO ====> Epoch: 527, cost 58.31 s 2023-09-16 20:48:08,722 44k INFO ====> Epoch: 528, cost 59.14 s 2023-09-16 20:49:07,750 44k INFO ====> Epoch: 529, cost 59.03 s 2023-09-16 20:50:06,381 44k INFO ====> Epoch: 530, cost 58.63 s 2023-09-16 20:51:04,918 44k INFO ====> Epoch: 531, cost 58.54 s 2023-09-16 20:51:20,065 44k INFO Train Epoch: 532 [22%] 2023-09-16 20:51:20,066 44k INFO Losses: [2.4091620445251465, 2.562915086746216, 11.476977348327637, 22.97216796875, 0.988628089427948], step: 17000, lr: 4.6788799934556806e-05, reference_loss: 40.40985107421875 2023-09-16 20:52:04,657 44k INFO ====> Epoch: 532, cost 59.74 s 2023-09-16 20:53:03,386 44k INFO ====> Epoch: 533, cost 58.73 s 2023-09-16 20:54:01,587 44k INFO ====> Epoch: 534, cost 58.20 s 2023-09-16 20:55:00,541 44k INFO ====> Epoch: 535, cost 58.95 s 2023-09-16 20:55:58,752 44k INFO ====> Epoch: 536, cost 58.21 s 2023-09-16 20:56:56,988 44k INFO ====> Epoch: 537, cost 58.24 s 2023-09-16 20:57:26,820 44k INFO Train Epoch: 538 [47%] 2023-09-16 20:57:26,821 44k INFO Losses: [2.2473154067993164, 2.5370190143585205, 12.066885948181152, 23.108198165893555, 0.9465435743331909], step: 17200, lr: 4.6753719298903336e-05, reference_loss: 40.90596389770508 2023-09-16 20:57:56,333 44k INFO ====> Epoch: 538, cost 59.34 s 2023-09-16 20:58:54,827 44k INFO ====> Epoch: 539, cost 58.49 s 2023-09-16 20:59:53,279 44k INFO ====> Epoch: 540, cost 58.45 s 2023-09-16 21:00:52,528 44k INFO ====> Epoch: 541, cost 59.25 s 2023-09-16 21:01:51,358 44k INFO ====> Epoch: 542, cost 58.83 s 2023-09-16 21:02:49,827 44k INFO ====> Epoch: 543, cost 58.47 s 2023-09-16 21:03:34,445 44k INFO Train Epoch: 544 [72%] 2023-09-16 21:03:34,446 44k INFO Losses: [2.319063663482666, 2.375405788421631, 12.723000526428223, 22.634496688842773, 1.0329595804214478], step: 17400, lr: 4.671866496550597e-05, reference_loss: 41.084922790527344 2023-09-16 21:03:48,856 44k INFO ====> Epoch: 544, cost 59.03 s 2023-09-16 21:04:47,697 44k INFO ====> Epoch: 545, cost 58.84 s 2023-09-16 21:05:46,304 44k INFO ====> Epoch: 546, cost 58.61 s 2023-09-16 21:06:44,664 44k INFO ====> Epoch: 547, cost 58.36 s 2023-09-16 21:07:43,196 44k INFO ====> Epoch: 548, cost 58.53 s 2023-09-16 21:08:41,237 44k INFO ====> Epoch: 549, cost 58.04 s 2023-09-16 21:09:39,277 44k INFO Train Epoch: 550 [97%] 2023-09-16 21:09:39,279 44k INFO Losses: [2.133967876434326, 2.6392970085144043, 8.647675514221191, 18.632863998413086, -0.016205433756113052], step: 17600, lr: 4.6683636914644154e-05, reference_loss: 32.03759765625 2023-09-16 21:09:55,649 44k INFO Saving model and optimizer state at iteration 550 to ./logs/44k/G_17600.pth 2023-09-16 21:09:59,198 44k INFO Saving model and optimizer state at iteration 550 to ./logs/44k/D_17600.pth 2023-09-16 21:09:59,883 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_13600.pth 2023-09-16 21:09:59,884 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_13600.pth 2023-09-16 21:09:59,885 44k INFO ====> Epoch: 550, cost 78.65 s 2023-09-16 21:10:58,340 44k INFO ====> Epoch: 551, cost 58.45 s 2023-09-16 21:11:57,692 44k INFO ====> Epoch: 552, cost 59.35 s 2023-09-16 21:12:56,297 44k INFO ====> Epoch: 553, cost 58.61 s 2023-09-16 21:13:54,994 44k INFO ====> Epoch: 554, cost 58.70 s 2023-09-16 21:14:53,750 44k INFO ====> Epoch: 555, cost 58.76 s 2023-09-16 21:15:51,807 44k INFO ====> Epoch: 556, cost 58.06 s 2023-09-16 21:16:06,640 44k INFO Train Epoch: 557 [22%] 2023-09-16 21:16:06,641 44k INFO Losses: [2.326284170150757, 2.49539852142334, 12.861845016479492, 23.3042049407959, 0.9967472553253174], step: 17800, lr: 4.664280404722132e-05, reference_loss: 41.98447799682617 2023-09-16 21:16:51,341 44k INFO ====> Epoch: 557, cost 59.53 s 2023-09-16 21:17:50,355 44k INFO ====> Epoch: 558, cost 59.01 s 2023-09-16 21:18:49,039 44k INFO ====> Epoch: 559, cost 58.68 s 2023-09-16 21:19:47,796 44k INFO ====> Epoch: 560, cost 58.76 s 2023-09-16 21:20:46,476 44k INFO ====> Epoch: 561, cost 58.68 s 2023-09-16 21:21:45,038 44k INFO ====> Epoch: 562, cost 58.56 s 2023-09-16 21:22:15,051 44k INFO Train Epoch: 563 [47%] 2023-09-16 21:22:15,052 44k INFO Losses: [2.0480105876922607, 3.0134074687957764, 13.329144477844238, 23.214723587036133, 0.948180079460144], step: 18000, lr: 4.660783287427127e-05, reference_loss: 42.553466796875 2023-09-16 21:22:44,308 44k INFO ====> Epoch: 563, cost 59.27 s 2023-09-16 21:23:43,240 44k INFO ====> Epoch: 564, cost 58.93 s 2023-09-16 21:24:41,923 44k INFO ====> Epoch: 565, cost 58.68 s 2023-09-16 21:25:40,490 44k INFO ====> Epoch: 566, cost 58.57 s 2023-09-16 21:26:39,455 44k INFO ====> Epoch: 567, cost 58.96 s 2023-09-16 21:27:38,352 44k INFO ====> Epoch: 568, cost 58.90 s 2023-09-16 21:28:23,034 44k INFO Train Epoch: 569 [72%] 2023-09-16 21:28:23,035 44k INFO Losses: [2.2859573364257812, 2.5168981552124023, 12.599772453308105, 22.561613082885742, 1.035539150238037], step: 18200, lr: 4.6572887921505934e-05, reference_loss: 40.999778747558594 2023-09-16 21:28:37,310 44k INFO ====> Epoch: 569, cost 58.96 s 2023-09-16 21:29:35,609 44k INFO ====> Epoch: 570, cost 58.30 s 2023-09-16 21:30:34,813 44k INFO ====> Epoch: 571, cost 59.20 s 2023-09-16 21:31:33,942 44k INFO ====> Epoch: 572, cost 59.13 s 2023-09-16 21:32:32,779 44k INFO ====> Epoch: 573, cost 58.84 s 2023-09-16 21:33:31,703 44k INFO ====> Epoch: 574, cost 58.92 s 2023-09-16 21:34:30,134 44k INFO Train Epoch: 575 [97%] 2023-09-16 21:34:30,135 44k INFO Losses: [2.045819044113159, 2.601414918899536, 10.069657325744629, 18.891963958740234, -0.0007341513992287219], step: 18400, lr: 4.6537969169266317e-05, reference_loss: 33.608123779296875 2023-09-16 21:34:45,599 44k INFO Saving model and optimizer state at iteration 575 to ./logs/44k/G_18400.pth 2023-09-16 21:34:47,800 44k INFO Saving model and optimizer state at iteration 575 to ./logs/44k/D_18400.pth 2023-09-16 21:34:48,323 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_14400.pth 2023-09-16 21:34:48,324 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_14400.pth 2023-09-16 21:34:48,325 44k INFO ====> Epoch: 575, cost 76.62 s 2023-09-16 21:35:46,847 44k INFO ====> Epoch: 576, cost 58.52 s 2023-09-16 21:36:45,230 44k INFO ====> Epoch: 577, cost 58.38 s 2023-09-16 21:37:44,167 44k INFO ====> Epoch: 578, cost 58.94 s 2023-09-16 21:38:42,680 44k INFO ====> Epoch: 579, cost 58.51 s 2023-09-16 21:39:41,677 44k INFO ====> Epoch: 580, cost 59.00 s 2023-09-16 21:40:40,098 44k INFO ====> Epoch: 581, cost 58.42 s 2023-09-16 21:40:55,023 44k INFO Train Epoch: 582 [22%] 2023-09-16 21:40:55,024 44k INFO Losses: [2.2351889610290527, 2.7000434398651123, 14.224446296691895, 23.141801834106445, 1.0269855260849], step: 18600, lr: 4.6497263713333416e-05, reference_loss: 43.328468322753906 2023-09-16 21:41:39,546 44k INFO ====> Epoch: 582, cost 59.45 s 2023-09-16 21:42:38,394 44k INFO ====> Epoch: 583, cost 58.85 s 2023-09-16 21:43:37,043 44k INFO ====> Epoch: 584, cost 58.65 s 2023-09-16 21:44:35,710 44k INFO ====> Epoch: 585, cost 58.67 s 2023-09-16 21:45:34,210 44k INFO ====> Epoch: 586, cost 58.50 s 2023-09-16 21:46:32,959 44k INFO ====> Epoch: 587, cost 58.75 s 2023-09-16 21:47:02,887 44k INFO Train Epoch: 588 [47%] 2023-09-16 21:47:02,888 44k INFO Losses: [2.2353477478027344, 2.5656960010528564, 12.490243911743164, 23.053133010864258, 0.9326230883598328], step: 18800, lr: 4.6462401661528456e-05, reference_loss: 41.27704620361328 2023-09-16 21:47:32,636 44k INFO ====> Epoch: 588, cost 59.68 s 2023-09-16 21:48:31,615 44k INFO ====> Epoch: 589, cost 58.98 s 2023-09-16 21:49:30,286 44k INFO ====> Epoch: 590, cost 58.67 s 2023-09-16 21:50:28,910 44k INFO ====> Epoch: 591, cost 58.62 s 2023-09-16 21:51:27,509 44k INFO ====> Epoch: 592, cost 58.60 s 2023-09-16 21:52:26,195 44k INFO ====> Epoch: 593, cost 58.69 s 2023-09-16 21:53:11,172 44k INFO Train Epoch: 594 [72%] 2023-09-16 21:53:11,173 44k INFO Losses: [2.27258038520813, 2.4788668155670166, 12.254258155822754, 22.340789794921875, 0.9994801878929138], step: 19000, lr: 4.6427565748092926e-05, reference_loss: 40.345977783203125 2023-09-16 21:53:25,229 44k INFO ====> Epoch: 594, cost 59.03 s 2023-09-16 21:54:23,736 44k INFO ====> Epoch: 595, cost 58.51 s 2023-09-16 21:55:22,436 44k INFO ====> Epoch: 596, cost 58.70 s 2023-09-16 21:56:21,122 44k INFO ====> Epoch: 597, cost 58.69 s 2023-09-16 21:57:20,090 44k INFO ====> Epoch: 598, cost 58.97 s 2023-09-16 21:58:18,612 44k INFO ====> Epoch: 599, cost 58.52 s 2023-09-16 21:59:17,674 44k INFO Train Epoch: 600 [97%] 2023-09-16 21:59:17,675 44k INFO Losses: [2.0762054920196533, 2.704420804977417, 12.033459663391113, 19.229494094848633, 0.15018364787101746], step: 19200, lr: 4.639275595342915e-05, reference_loss: 36.193763732910156 2023-09-16 21:59:34,194 44k INFO Saving model and optimizer state at iteration 600 to ./logs/44k/G_19200.pth 2023-09-16 21:59:37,771 44k INFO Saving model and optimizer state at iteration 600 to ./logs/44k/D_19200.pth 2023-09-16 21:59:38,762 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_15200.pth 2023-09-16 21:59:38,764 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_15200.pth 2023-09-16 21:59:38,764 44k INFO ====> Epoch: 600, cost 80.15 s 2023-09-16 22:00:37,529 44k INFO ====> Epoch: 601, cost 58.76 s 2023-09-16 22:01:36,141 44k INFO ====> Epoch: 602, cost 58.61 s 2023-09-16 22:02:34,367 44k INFO ====> Epoch: 603, cost 58.23 s 2023-09-16 22:03:33,108 44k INFO ====> Epoch: 604, cost 58.74 s 2023-09-16 22:04:31,944 44k INFO ====> Epoch: 605, cost 58.84 s 2023-09-16 22:05:30,226 44k INFO ====> Epoch: 606, cost 58.28 s 2023-09-16 22:05:45,193 44k INFO Train Epoch: 607 [22%] 2023-09-16 22:05:45,194 44k INFO Losses: [2.312204360961914, 2.66912579536438, 13.193926811218262, 23.006824493408203, 0.9763622879981995], step: 19400, lr: 4.6352177511421956e-05, reference_loss: 42.158447265625 2023-09-16 22:06:29,659 44k INFO ====> Epoch: 607, cost 59.43 s 2023-09-16 22:07:28,099 44k INFO ====> Epoch: 608, cost 58.44 s 2023-09-16 22:08:27,074 44k INFO ====> Epoch: 609, cost 58.98 s 2023-09-16 22:09:26,102 44k INFO ====> Epoch: 610, cost 59.03 s 2023-09-16 22:10:24,419 44k INFO ====> Epoch: 611, cost 58.32 s 2023-09-16 22:11:22,638 44k INFO ====> Epoch: 612, cost 58.22 s 2023-09-16 22:11:52,252 44k INFO Train Epoch: 613 [47%] 2023-09-16 22:11:52,253 44k INFO Losses: [2.266063690185547, 2.609699010848999, 12.740920066833496, 23.086620330810547, 0.9176176190376282], step: 19600, lr: 4.631742424026951e-05, reference_loss: 41.62092208862305 2023-09-16 22:12:21,374 44k INFO ====> Epoch: 613, cost 58.74 s 2023-09-16 22:13:19,872 44k INFO ====> Epoch: 614, cost 58.50 s 2023-09-16 22:14:18,369 44k INFO ====> Epoch: 615, cost 58.50 s 2023-09-16 22:15:16,803 44k INFO ====> Epoch: 616, cost 58.43 s 2023-09-16 22:16:14,897 44k INFO ====> Epoch: 617, cost 58.09 s 2023-09-16 22:17:13,500 44k INFO ====> Epoch: 618, cost 58.60 s 2023-09-16 22:17:58,845 44k INFO Train Epoch: 619 [72%] 2023-09-16 22:17:58,846 44k INFO Losses: [2.2811996936798096, 2.570067882537842, 12.128145217895508, 22.352920532226562, 0.9793955683708191], step: 19800, lr: 4.62826970259265e-05, reference_loss: 40.311729431152344 2023-09-16 22:18:12,975 44k INFO ====> Epoch: 619, cost 59.47 s 2023-09-16 22:19:11,509 44k INFO ====> Epoch: 620, cost 58.53 s 2023-09-16 22:20:10,030 44k INFO ====> Epoch: 621, cost 58.52 s 2023-09-16 22:21:08,301 44k INFO ====> Epoch: 622, cost 58.27 s 2023-09-16 22:22:06,893 44k INFO ====> Epoch: 623, cost 58.59 s 2023-09-16 22:23:05,323 44k INFO ====> Epoch: 624, cost 58.43 s 2023-09-16 22:24:03,935 44k INFO Train Epoch: 625 [97%] 2023-09-16 22:24:03,936 44k INFO Losses: [1.904895305633545, 2.8078737258911133, 11.378426551818848, 19.120887756347656, -0.010197311639785767], step: 20000, lr: 4.624799584885641e-05, reference_loss: 35.20188522338867 2023-09-16 22:24:21,317 44k INFO Saving model and optimizer state at iteration 625 to ./logs/44k/G_20000.pth 2023-09-16 22:24:24,001 44k INFO Saving model and optimizer state at iteration 625 to ./logs/44k/D_20000.pth 2023-09-16 22:24:24,530 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_16000.pth 2023-09-16 22:24:24,532 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_16000.pth 2023-09-16 22:24:24,532 44k INFO ====> Epoch: 625, cost 79.21 s 2023-09-16 22:25:23,634 44k INFO ====> Epoch: 626, cost 59.10 s 2023-09-16 22:26:22,424 44k INFO ====> Epoch: 627, cost 58.79 s 2023-09-16 22:27:21,475 44k INFO ====> Epoch: 628, cost 59.05 s 2023-09-16 22:28:20,368 44k INFO ====> Epoch: 629, cost 58.89 s 2023-09-16 22:29:19,903 44k INFO ====> Epoch: 630, cost 59.53 s 2023-09-16 22:30:18,809 44k INFO ====> Epoch: 631, cost 58.91 s 2023-09-16 22:30:33,535 44k INFO Train Epoch: 632 [22%] 2023-09-16 22:30:33,536 44k INFO Losses: [2.3025131225585938, 2.4788694381713867, 12.966777801513672, 23.16344451904297, 0.9667849540710449], step: 20200, lr: 4.62075440244512e-05, reference_loss: 41.87839126586914 2023-09-16 22:31:17,998 44k INFO ====> Epoch: 632, cost 59.19 s 2023-09-16 22:32:16,759 44k INFO ====> Epoch: 633, cost 58.76 s 2023-09-16 22:33:15,344 44k INFO ====> Epoch: 634, cost 58.59 s 2023-09-16 22:34:14,654 44k INFO ====> Epoch: 635, cost 59.31 s 2023-09-16 22:35:13,072 44k INFO ====> Epoch: 636, cost 58.42 s 2023-09-16 22:36:11,990 44k INFO ====> Epoch: 637, cost 58.92 s 2023-09-16 22:36:42,375 44k INFO Train Epoch: 638 [47%] 2023-09-16 22:36:42,376 44k INFO Losses: [2.232002019882202, 2.583597421646118, 13.052815437316895, 22.903406143188477, 0.9260207414627075], step: 20400, lr: 4.617289919452116e-05, reference_loss: 41.69784164428711 2023-09-16 22:37:12,030 44k INFO ====> Epoch: 638, cost 60.04 s 2023-09-16 22:38:10,895 44k INFO ====> Epoch: 639, cost 58.86 s 2023-09-16 22:39:09,573 44k INFO ====> Epoch: 640, cost 58.68 s 2023-09-16 22:40:08,062 44k INFO ====> Epoch: 641, cost 58.49 s 2023-09-16 22:41:06,655 44k INFO ====> Epoch: 642, cost 58.59 s 2023-09-16 22:42:04,689 44k INFO ====> Epoch: 643, cost 58.03 s 2023-09-16 22:42:49,519 44k INFO Train Epoch: 644 [72%] 2023-09-16 22:42:49,520 44k INFO Losses: [2.2906389236450195, 2.5133326053619385, 12.399672508239746, 22.30893325805664, 0.9703485369682312], step: 20600, lr: 4.613828034009505e-05, reference_loss: 40.48292541503906 2023-09-16 22:43:03,947 44k INFO ====> Epoch: 644, cost 59.26 s 2023-09-16 22:44:03,131 44k INFO ====> Epoch: 645, cost 59.18 s 2023-09-16 22:45:02,063 44k INFO ====> Epoch: 646, cost 58.93 s 2023-09-16 22:46:00,927 44k INFO ====> Epoch: 647, cost 58.86 s 2023-09-16 22:46:59,600 44k INFO ====> Epoch: 648, cost 58.67 s 2023-09-16 22:47:58,338 44k INFO ====> Epoch: 649, cost 58.74 s 2023-09-16 22:48:56,799 44k INFO Train Epoch: 650 [97%] 2023-09-16 22:48:56,800 44k INFO Losses: [2.1669130325317383, 2.4785962104797363, 10.93682861328125, 18.377866744995117, -0.01521299034357071], step: 20800, lr: 4.610368744169732e-05, reference_loss: 33.94499206542969 2023-09-16 22:49:13,118 44k INFO Saving model and optimizer state at iteration 650 to ./logs/44k/G_20800.pth 2023-09-16 22:49:16,310 44k INFO Saving model and optimizer state at iteration 650 to ./logs/44k/D_20800.pth 2023-09-16 22:49:17,403 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_16800.pth 2023-09-16 22:49:17,405 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_16800.pth 2023-09-16 22:49:17,406 44k INFO ====> Epoch: 650, cost 79.07 s 2023-09-16 22:50:15,862 44k INFO ====> Epoch: 651, cost 58.46 s 2023-09-16 22:51:14,316 44k INFO ====> Epoch: 652, cost 58.45 s 2023-09-16 22:52:13,137 44k INFO ====> Epoch: 653, cost 58.82 s 2023-09-16 22:53:11,991 44k INFO ====> Epoch: 654, cost 58.85 s 2023-09-16 22:54:10,518 44k INFO ====> Epoch: 655, cost 58.53 s 2023-09-16 22:55:09,454 44k INFO ====> Epoch: 656, cost 58.94 s 2023-09-16 22:55:24,316 44k INFO Train Epoch: 657 [22%] 2023-09-16 22:55:24,317 44k INFO Losses: [2.2700138092041016, 2.666489601135254, 13.197489738464355, 23.111888885498047, 0.9786016941070557], step: 21000, lr: 4.606336183980704e-05, reference_loss: 42.224483489990234 2023-09-16 22:56:08,713 44k INFO ====> Epoch: 657, cost 59.26 s 2023-09-16 22:57:07,221 44k INFO ====> Epoch: 658, cost 58.51 s 2023-09-16 22:58:05,953 44k INFO ====> Epoch: 659, cost 58.73 s 2023-09-16 22:59:04,287 44k INFO ====> Epoch: 660, cost 58.33 s 2023-09-16 23:00:03,333 44k INFO ====> Epoch: 661, cost 59.05 s 2023-09-16 23:01:02,137 44k INFO ====> Epoch: 662, cost 58.80 s 2023-09-16 23:01:31,562 44k INFO Train Epoch: 663 [47%] 2023-09-16 23:01:31,563 44k INFO Losses: [2.2028658390045166, 2.5656683444976807, 13.8521146774292, 23.208850860595703, 0.9141803979873657], step: 21200, lr: 4.602882511272843e-05, reference_loss: 42.74367904663086 2023-09-16 23:02:01,284 44k INFO ====> Epoch: 663, cost 59.15 s 2023-09-16 23:03:00,129 44k INFO ====> Epoch: 664, cost 58.84 s 2023-09-16 23:03:58,062 44k INFO ====> Epoch: 665, cost 57.93 s 2023-09-16 23:04:57,094 44k INFO ====> Epoch: 666, cost 59.03 s 2023-09-16 23:05:56,242 44k INFO ====> Epoch: 667, cost 59.15 s 2023-09-16 23:06:54,883 44k INFO ====> Epoch: 668, cost 58.64 s 2023-09-16 23:07:39,643 44k INFO Train Epoch: 669 [72%] 2023-09-16 23:07:39,644 44k INFO Losses: [2.305558681488037, 2.5389652252197266, 12.03804874420166, 22.245532989501953, 0.975671648979187], step: 21400, lr: 4.5994314280101916e-05, reference_loss: 40.10377502441406 2023-09-16 23:07:53,875 44k INFO ====> Epoch: 669, cost 58.99 s 2023-09-16 23:08:52,761 44k INFO ====> Epoch: 670, cost 58.89 s 2023-09-16 23:09:51,642 44k INFO ====> Epoch: 671, cost 58.88 s 2023-09-16 23:10:50,571 44k INFO ====> Epoch: 672, cost 58.93 s 2023-09-16 23:11:48,731 44k INFO ====> Epoch: 673, cost 58.16 s 2023-09-16 23:12:47,326 44k INFO ====> Epoch: 674, cost 58.60 s 2023-09-16 23:13:46,313 44k INFO Train Epoch: 675 [97%] 2023-09-16 23:13:46,314 44k INFO Losses: [2.0164592266082764, 2.729809522628784, 11.74462604522705, 18.919065475463867, -0.09301292896270752], step: 21600, lr: 4.5959829322512754e-05, reference_loss: 35.31694412231445 2023-09-16 23:14:01,999 44k INFO Saving model and optimizer state at iteration 675 to ./logs/44k/G_21600.pth 2023-09-16 23:14:05,541 44k INFO Saving model and optimizer state at iteration 675 to ./logs/44k/D_21600.pth 2023-09-16 23:14:06,217 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_17600.pth 2023-09-16 23:14:06,219 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_17600.pth 2023-09-16 23:14:06,219 44k INFO ====> Epoch: 675, cost 78.89 s 2023-09-16 23:15:04,653 44k INFO ====> Epoch: 676, cost 58.43 s 2023-09-16 23:16:03,691 44k INFO ====> Epoch: 677, cost 59.04 s 2023-09-16 23:17:02,368 44k INFO ====> Epoch: 678, cost 58.68 s 2023-09-16 23:18:00,746 44k INFO ====> Epoch: 679, cost 58.38 s 2023-09-16 23:18:59,194 44k INFO ====> Epoch: 680, cost 58.45 s 2023-09-16 23:19:57,583 44k INFO ====> Epoch: 681, cost 58.39 s 2023-09-16 23:20:12,543 44k INFO Train Epoch: 682 [22%] 2023-09-16 23:20:12,544 44k INFO Losses: [2.316908597946167, 2.60872483253479, 12.814970970153809, 23.061630249023438, 0.9499024748802185], step: 21800, lr: 4.5919629549283146e-05, reference_loss: 41.75213623046875 2023-09-16 23:20:56,796 44k INFO ====> Epoch: 682, cost 59.21 s 2023-09-16 23:21:56,171 44k INFO ====> Epoch: 683, cost 59.37 s 2023-09-16 23:22:55,200 44k INFO ====> Epoch: 684, cost 59.03 s 2023-09-16 23:23:53,946 44k INFO ====> Epoch: 685, cost 58.75 s 2023-09-16 23:24:52,479 44k INFO ====> Epoch: 686, cost 58.53 s 2023-09-16 23:25:50,729 44k INFO ====> Epoch: 687, cost 58.25 s 2023-09-16 23:26:20,420 44k INFO Train Epoch: 688 [47%] 2023-09-16 23:26:20,421 44k INFO Losses: [2.2045726776123047, 2.5230836868286133, 13.27503490447998, 23.01807975769043, 0.9179019331932068], step: 22000, lr: 4.5885200587740786e-05, reference_loss: 41.93867111206055 2023-09-16 23:26:49,540 44k INFO ====> Epoch: 688, cost 58.81 s 2023-09-16 23:27:47,863 44k INFO ====> Epoch: 689, cost 58.32 s 2023-09-16 23:28:46,708 44k INFO ====> Epoch: 690, cost 58.84 s 2023-09-16 23:29:45,407 44k INFO ====> Epoch: 691, cost 58.70 s 2023-09-16 23:30:43,376 44k INFO ====> Epoch: 692, cost 57.97 s 2023-09-16 23:31:41,436 44k INFO ====> Epoch: 693, cost 58.06 s 2023-09-16 23:32:26,268 44k INFO Train Epoch: 694 [72%] 2023-09-16 23:32:26,269 44k INFO Losses: [2.263392925262451, 2.4652650356292725, 12.183664321899414, 22.428363800048828, 1.039743185043335], step: 22200, lr: 4.585079743985163e-05, reference_loss: 40.380428314208984 2023-09-16 23:32:40,386 44k INFO ====> Epoch: 694, cost 58.95 s 2023-09-16 23:33:39,190 44k INFO ====> Epoch: 695, cost 58.80 s 2023-09-16 23:34:37,482 44k INFO ====> Epoch: 696, cost 58.29 s 2023-09-16 23:35:35,605 44k INFO ====> Epoch: 697, cost 58.12 s 2023-09-16 23:36:34,450 44k INFO ====> Epoch: 698, cost 58.84 s 2023-09-16 23:37:32,759 44k INFO ====> Epoch: 699, cost 58.31 s 2023-09-16 23:38:31,593 44k INFO Train Epoch: 700 [97%] 2023-09-16 23:38:31,594 44k INFO Losses: [1.9338105916976929, 2.5791189670562744, 9.876747131347656, 18.466983795166016, 0.044963616877794266], step: 22400, lr: 4.5816420086261495e-05, reference_loss: 32.9016227722168 2023-09-16 23:38:48,657 44k INFO Saving model and optimizer state at iteration 700 to ./logs/44k/G_22400.pth 2023-09-16 23:38:50,804 44k INFO Saving model and optimizer state at iteration 700 to ./logs/44k/D_22400.pth 2023-09-16 23:38:51,319 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_18400.pth 2023-09-16 23:38:51,320 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_18400.pth 2023-09-16 23:38:51,321 44k INFO ====> Epoch: 700, cost 78.56 s 2023-09-16 23:39:49,649 44k INFO ====> Epoch: 701, cost 58.33 s 2023-09-16 23:40:48,116 44k INFO ====> Epoch: 702, cost 58.47 s 2023-09-16 23:41:47,149 44k INFO ====> Epoch: 703, cost 59.03 s 2023-09-16 23:42:45,918 44k INFO ====> Epoch: 704, cost 58.77 s 2023-09-16 23:43:44,778 44k INFO ====> Epoch: 705, cost 58.86 s 2023-09-16 23:44:43,376 44k INFO ====> Epoch: 706, cost 58.60 s 2023-09-16 23:44:58,277 44k INFO Train Epoch: 707 [22%] 2023-09-16 23:44:58,278 44k INFO Losses: [2.295949697494507, 2.5899813175201416, 13.67569351196289, 23.065961837768555, 0.9477071166038513], step: 22600, lr: 4.5776345749067245e-05, reference_loss: 42.575294494628906 2023-09-16 23:45:42,416 44k INFO ====> Epoch: 707, cost 59.04 s 2023-09-16 23:46:41,334 44k INFO ====> Epoch: 708, cost 58.92 s 2023-09-16 23:47:40,091 44k INFO ====> Epoch: 709, cost 58.76 s 2023-09-16 23:48:37,892 44k INFO ====> Epoch: 710, cost 57.80 s 2023-09-16 23:49:36,781 44k INFO ====> Epoch: 711, cost 58.89 s 2023-09-16 23:50:35,419 44k INFO ====> Epoch: 712, cost 58.64 s 2023-09-16 23:51:05,293 44k INFO Train Epoch: 713 [47%] 2023-09-16 23:51:05,294 44k INFO Losses: [2.1919045448303223, 2.5906991958618164, 12.848188400268555, 22.766525268554688, 0.8971822261810303], step: 22800, lr: 4.57420242167985e-05, reference_loss: 41.294498443603516 2023-09-16 23:51:34,260 44k INFO ====> Epoch: 713, cost 58.84 s 2023-09-16 23:52:32,979 44k INFO ====> Epoch: 714, cost 58.72 s 2023-09-16 23:53:31,590 44k INFO ====> Epoch: 715, cost 58.61 s 2023-09-16 23:54:30,664 44k INFO ====> Epoch: 716, cost 59.07 s 2023-09-16 23:55:29,626 44k INFO ====> Epoch: 717, cost 58.96 s 2023-09-16 23:56:28,437 44k INFO ====> Epoch: 718, cost 58.81 s 2023-09-16 23:57:13,522 44k INFO Train Epoch: 719 [72%] 2023-09-16 23:57:13,523 44k INFO Losses: [2.2625041007995605, 2.5366930961608887, 12.20931625366211, 22.242576599121094, 0.9543431401252747], step: 23000, lr: 4.570772841763618e-05, reference_loss: 40.2054328918457 2023-09-16 23:57:27,505 44k INFO ====> Epoch: 719, cost 59.07 s 2023-09-16 23:58:26,480 44k INFO ====> Epoch: 720, cost 58.97 s 2023-09-16 23:59:24,886 44k INFO ====> Epoch: 721, cost 58.41 s 2023-09-17 00:00:23,287 44k INFO ====> Epoch: 722, cost 58.40 s 2023-09-17 00:01:22,055 44k INFO ====> Epoch: 723, cost 58.77 s 2023-09-17 00:02:20,375 44k INFO ====> Epoch: 724, cost 58.32 s 2023-09-17 00:03:19,127 44k INFO Train Epoch: 725 [97%] 2023-09-17 00:03:19,128 44k INFO Losses: [1.793247103691101, 2.9266839027404785, 12.794977188110352, 18.740297317504883, -0.08972270786762238], step: 23200, lr: 4.5673458332286506e-05, reference_loss: 36.16548156738281 2023-09-17 00:03:35,386 44k INFO Saving model and optimizer state at iteration 725 to ./logs/44k/G_23200.pth 2023-09-17 00:03:38,712 44k INFO Saving model and optimizer state at iteration 725 to ./logs/44k/D_23200.pth 2023-09-17 00:03:39,663 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_19200.pth 2023-09-17 00:03:39,666 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_19200.pth 2023-09-17 00:03:39,667 44k INFO ====> Epoch: 725, cost 79.29 s 2023-09-17 00:04:38,470 44k INFO ====> Epoch: 726, cost 58.80 s 2023-09-17 00:05:36,959 44k INFO ====> Epoch: 727, cost 58.49 s 2023-09-17 00:06:35,458 44k INFO ====> Epoch: 728, cost 58.50 s 2023-09-17 00:07:34,224 44k INFO ====> Epoch: 729, cost 58.77 s 2023-09-17 00:08:32,348 44k INFO ====> Epoch: 730, cost 58.12 s 2023-09-17 00:09:30,755 44k INFO ====> Epoch: 731, cost 58.41 s 2023-09-17 00:09:45,725 44k INFO Train Epoch: 732 [22%] 2023-09-17 00:09:45,726 44k INFO Losses: [2.349515199661255, 2.5777242183685303, 12.56545352935791, 22.986906051635742, 0.923302948474884], step: 23400, lr: 4.563350903972744e-05, reference_loss: 41.40290069580078 2023-09-17 00:10:30,050 44k INFO ====> Epoch: 732, cost 59.29 s 2023-09-17 00:11:28,844 44k INFO ====> Epoch: 733, cost 58.79 s 2023-09-17 00:12:27,394 44k INFO ====> Epoch: 734, cost 58.55 s 2023-09-17 00:13:26,311 44k INFO ====> Epoch: 735, cost 58.92 s 2023-09-17 00:14:24,723 44k INFO ====> Epoch: 736, cost 58.41 s 2023-09-17 00:15:23,314 44k INFO ====> Epoch: 737, cost 58.59 s 2023-09-17 00:15:52,672 44k INFO Train Epoch: 738 [47%] 2023-09-17 00:15:52,673 44k INFO Losses: [2.1835737228393555, 2.693561553955078, 13.261792182922363, 22.83609390258789, 0.8973855376243591], step: 23600, lr: 4.559929460151892e-05, reference_loss: 41.872406005859375 2023-09-17 00:16:22,235 44k INFO ====> Epoch: 738, cost 58.92 s 2023-09-17 00:17:20,823 44k INFO ====> Epoch: 739, cost 58.59 s 2023-09-17 00:18:19,545 44k INFO ====> Epoch: 740, cost 58.72 s 2023-09-17 00:19:17,838 44k INFO ====> Epoch: 741, cost 58.29 s 2023-09-17 00:20:15,783 44k INFO ====> Epoch: 742, cost 57.94 s 2023-09-17 00:21:14,357 44k INFO ====> Epoch: 743, cost 58.57 s 2023-09-17 00:21:59,069 44k INFO Train Epoch: 744 [72%] 2023-09-17 00:21:59,070 44k INFO Losses: [2.23189115524292, 2.5218403339385986, 12.776825904846191, 22.225276947021484, 0.9494567513465881], step: 23800, lr: 4.556510581612139e-05, reference_loss: 40.705291748046875 2023-09-17 00:22:13,190 44k INFO ====> Epoch: 744, cost 58.83 s 2023-09-17 00:23:11,751 44k INFO ====> Epoch: 745, cost 58.56 s 2023-09-17 00:24:10,342 44k INFO ====> Epoch: 746, cost 58.59 s 2023-09-17 00:25:08,825 44k INFO ====> Epoch: 747, cost 58.48 s 2023-09-17 00:26:07,504 44k INFO ====> Epoch: 748, cost 58.68 s 2023-09-17 00:27:06,319 44k INFO ====> Epoch: 749, cost 58.81 s 2023-09-17 00:28:04,649 44k INFO Train Epoch: 750 [97%] 2023-09-17 00:28:04,650 44k INFO Losses: [1.7320187091827393, 2.9112560749053955, 12.596366882324219, 18.40655517578125, -0.1365424543619156], step: 24000, lr: 4.553094266430124e-05, reference_loss: 35.50965118408203 2023-09-17 00:28:21,041 44k INFO Saving model and optimizer state at iteration 750 to ./logs/44k/G_24000.pth 2023-09-17 00:28:24,503 44k INFO Saving model and optimizer state at iteration 750 to ./logs/44k/D_24000.pth 2023-09-17 00:28:25,045 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_20000.pth 2023-09-17 00:28:25,047 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_20000.pth 2023-09-17 00:28:25,047 44k INFO ====> Epoch: 750, cost 78.73 s 2023-09-17 00:29:24,748 44k INFO ====> Epoch: 751, cost 59.70 s 2023-09-17 00:30:24,571 44k INFO ====> Epoch: 752, cost 59.82 s 2023-09-17 00:31:23,834 44k INFO ====> Epoch: 753, cost 59.26 s 2023-09-17 00:32:23,458 44k INFO ====> Epoch: 754, cost 59.62 s 2023-09-17 00:33:23,007 44k INFO ====> Epoch: 755, cost 59.55 s 2023-09-17 00:34:22,239 44k INFO ====> Epoch: 756, cost 59.23 s 2023-09-17 00:34:37,245 44k INFO Train Epoch: 757 [22%] 2023-09-17 00:34:37,247 44k INFO Losses: [2.336958408355713, 2.5724642276763916, 13.31881332397461, 23.153139114379883, 0.9555040597915649], step: 24200, lr: 4.549111802619844e-05, reference_loss: 42.336883544921875 2023-09-17 00:35:22,626 44k INFO ====> Epoch: 757, cost 60.39 s 2023-09-17 00:36:21,950 44k INFO ====> Epoch: 758, cost 59.32 s 2023-09-17 00:37:21,351 44k INFO ====> Epoch: 759, cost 59.40 s 2023-09-17 00:38:20,837 44k INFO ====> Epoch: 760, cost 59.49 s 2023-09-17 00:39:20,124 44k INFO ====> Epoch: 761, cost 59.29 s 2023-09-17 00:40:19,450 44k INFO ====> Epoch: 762, cost 59.33 s 2023-09-17 00:40:49,491 44k INFO Train Epoch: 763 [47%] 2023-09-17 00:40:49,492 44k INFO Losses: [2.153183937072754, 2.6174299716949463, 13.572967529296875, 22.899309158325195, 0.8901001214981079], step: 24400, lr: 4.545701034788274e-05, reference_loss: 42.13298797607422 2023-09-17 00:41:19,445 44k INFO ====> Epoch: 763, cost 60.00 s 2023-09-17 00:42:19,455 44k INFO ====> Epoch: 764, cost 60.01 s 2023-09-17 00:43:18,813 44k INFO ====> Epoch: 765, cost 59.36 s 2023-09-17 00:44:18,205 44k INFO ====> Epoch: 766, cost 59.39 s 2023-09-17 00:45:17,712 44k INFO ====> Epoch: 767, cost 59.51 s 2023-09-17 00:46:16,862 44k INFO ====> Epoch: 768, cost 59.15 s 2023-09-17 00:47:02,650 44k INFO Train Epoch: 769 [72%] 2023-09-17 00:47:02,651 44k INFO Losses: [2.2197554111480713, 2.5217721462249756, 13.441725730895996, 22.42739486694336, 0.9704839587211609], step: 24600, lr: 4.542292824233311e-05, reference_loss: 41.58113479614258 2023-09-17 00:47:16,950 44k INFO ====> Epoch: 769, cost 60.09 s 2023-09-17 00:48:15,734 44k INFO ====> Epoch: 770, cost 58.78 s 2023-09-17 00:49:15,303 44k INFO ====> Epoch: 771, cost 59.57 s 2023-09-17 00:50:14,919 44k INFO ====> Epoch: 772, cost 59.62 s 2023-09-17 00:51:14,062 44k INFO ====> Epoch: 773, cost 59.14 s 2023-09-17 00:52:14,050 44k INFO ====> Epoch: 774, cost 59.99 s 2023-09-17 00:53:13,360 44k INFO Train Epoch: 775 [97%] 2023-09-17 00:53:13,361 44k INFO Losses: [1.8591504096984863, 2.7009079456329346, 8.19754695892334, 17.56157875061035, -0.08859074860811234], step: 24800, lr: 4.538887169037598e-05, reference_loss: 30.230592727661133 2023-09-17 00:53:30,293 44k INFO Saving model and optimizer state at iteration 775 to ./logs/44k/G_24800.pth 2023-09-17 00:53:33,427 44k INFO Saving model and optimizer state at iteration 775 to ./logs/44k/D_24800.pth 2023-09-17 00:53:34,408 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_20800.pth 2023-09-17 00:53:34,420 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_20800.pth 2023-09-17 00:53:34,431 44k INFO ====> Epoch: 775, cost 80.38 s 2023-09-17 00:54:34,030 44k INFO ====> Epoch: 776, cost 59.60 s 2023-09-17 00:55:33,471 44k INFO ====> Epoch: 777, cost 59.44 s 2023-09-17 00:56:33,271 44k INFO ====> Epoch: 778, cost 59.80 s 2023-09-17 00:57:33,003 44k INFO ====> Epoch: 779, cost 59.73 s 2023-09-17 00:58:32,604 44k INFO ====> Epoch: 780, cost 59.60 s 2023-09-17 00:59:31,672 44k INFO ====> Epoch: 781, cost 59.07 s 2023-09-17 00:59:46,980 44k INFO Train Epoch: 782 [22%] 2023-09-17 00:59:46,981 44k INFO Losses: [2.257446765899658, 2.6359028816223145, 13.630996704101562, 23.16592788696289, 0.9448270797729492], step: 25000, lr: 4.5349171317768046e-05, reference_loss: 42.635101318359375 2023-09-17 01:00:32,694 44k INFO ====> Epoch: 782, cost 61.02 s 2023-09-17 01:01:32,795 44k INFO ====> Epoch: 783, cost 60.10 s 2023-09-17 01:02:32,141 44k INFO ====> Epoch: 784, cost 59.35 s 2023-09-17 01:03:31,970 44k INFO ====> Epoch: 785, cost 59.83 s 2023-09-17 01:04:32,086 44k INFO ====> Epoch: 786, cost 60.12 s 2023-09-17 01:05:31,352 44k INFO ====> Epoch: 787, cost 59.27 s 2023-09-17 01:06:02,023 44k INFO Train Epoch: 788 [47%] 2023-09-17 01:06:02,024 44k INFO Losses: [2.2011208534240723, 2.6180522441864014, 13.113093376159668, 22.61589241027832, 0.8838010430335999], step: 25200, lr: 4.5315170066220454e-05, reference_loss: 41.43195724487305 2023-09-17 01:06:31,997 44k INFO ====> Epoch: 788, cost 60.65 s 2023-09-17 01:07:32,174 44k INFO ====> Epoch: 789, cost 60.18 s 2023-09-17 01:08:32,607 44k INFO ====> Epoch: 790, cost 60.43 s 2023-09-17 01:09:32,727 44k INFO ====> Epoch: 791, cost 60.12 s 2023-09-17 01:10:32,883 44k INFO ====> Epoch: 792, cost 60.16 s 2023-09-17 01:11:32,989 44k INFO ====> Epoch: 793, cost 60.11 s 2023-09-17 01:12:18,794 44k INFO Train Epoch: 794 [72%] 2023-09-17 01:12:18,795 44k INFO Losses: [2.17690110206604, 2.485109806060791, 13.15184211730957, 22.18655776977539, 0.9240463972091675], step: 25400, lr: 4.528119430764381e-05, reference_loss: 40.92445755004883 2023-09-17 01:12:33,172 44k INFO ====> Epoch: 794, cost 60.18 s 2023-09-17 01:13:33,157 44k INFO ====> Epoch: 795, cost 59.99 s 2023-09-17 01:14:33,196 44k INFO ====> Epoch: 796, cost 60.04 s 2023-09-17 01:15:33,102 44k INFO ====> Epoch: 797, cost 59.91 s 2023-09-17 01:16:33,048 44k INFO ====> Epoch: 798, cost 59.95 s 2023-09-17 01:17:32,702 44k INFO ====> Epoch: 799, cost 59.65 s 2023-09-17 01:18:32,222 44k INFO Train Epoch: 800 [97%] 2023-09-17 01:18:32,223 44k INFO Losses: [1.8717453479766846, 2.694432497024536, 10.510148048400879, 18.182153701782227, -0.13088037073612213], step: 25600, lr: 4.5247244022924354e-05, reference_loss: 33.12759780883789 2023-09-17 01:18:48,133 44k INFO Saving model and optimizer state at iteration 800 to ./logs/44k/G_25600.pth 2023-09-17 01:18:51,571 44k INFO Saving model and optimizer state at iteration 800 to ./logs/44k/D_25600.pth 2023-09-17 01:18:52,231 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_21600.pth 2023-09-17 01:18:52,233 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_21600.pth 2023-09-17 01:18:52,233 44k INFO ====> Epoch: 800, cost 79.53 s 2023-09-17 01:19:52,002 44k INFO ====> Epoch: 801, cost 59.77 s 2023-09-17 01:20:51,327 44k INFO ====> Epoch: 802, cost 59.33 s 2023-09-17 01:21:50,581 44k INFO ====> Epoch: 803, cost 59.25 s 2023-09-17 01:22:49,490 44k INFO ====> Epoch: 804, cost 58.91 s 2023-09-17 01:23:48,032 44k INFO ====> Epoch: 805, cost 58.54 s 2023-09-17 01:24:46,895 44k INFO ====> Epoch: 806, cost 58.86 s 2023-09-17 01:25:01,695 44k INFO Train Epoch: 807 [22%] 2023-09-17 01:25:01,696 44k INFO Losses: [2.2970693111419678, 2.674434185028076, 13.643845558166504, 23.0612850189209, 0.9475618600845337], step: 25800, lr: 4.5207667528063535e-05, reference_loss: 42.62419891357422 2023-09-17 01:25:46,004 44k INFO ====> Epoch: 807, cost 59.11 s 2023-09-17 01:26:44,469 44k INFO ====> Epoch: 808, cost 58.47 s 2023-09-17 01:27:42,896 44k INFO ====> Epoch: 809, cost 58.43 s 2023-09-17 01:28:41,728 44k INFO ====> Epoch: 810, cost 58.83 s 2023-09-17 01:29:40,137 44k INFO ====> Epoch: 811, cost 58.41 s 2023-09-17 01:30:38,975 44k INFO ====> Epoch: 812, cost 58.84 s 2023-09-17 01:31:08,720 44k INFO Train Epoch: 813 [47%] 2023-09-17 01:31:08,721 44k INFO Losses: [2.2183613777160645, 2.5757665634155273, 11.559014320373535, 22.561620712280273, 0.8629869222640991], step: 26000, lr: 4.5173772371198794e-05, reference_loss: 39.77775192260742 2023-09-17 01:31:38,459 44k INFO ====> Epoch: 813, cost 59.48 s 2023-09-17 01:32:36,828 44k INFO ====> Epoch: 814, cost 58.37 s 2023-09-17 01:33:35,316 44k INFO ====> Epoch: 815, cost 58.49 s 2023-09-17 01:34:34,259 44k INFO ====> Epoch: 816, cost 58.94 s 2023-09-17 01:35:32,701 44k INFO ====> Epoch: 817, cost 58.44 s 2023-09-17 01:36:31,265 44k INFO ====> Epoch: 818, cost 58.56 s 2023-09-17 01:37:16,579 44k INFO Train Epoch: 819 [72%] 2023-09-17 01:37:16,580 44k INFO Losses: [2.2384192943573, 2.410367250442505, 12.681600570678711, 22.225833892822266, 0.9224737286567688], step: 26200, lr: 4.513990262775884e-05, reference_loss: 40.478694915771484 2023-09-17 01:37:31,029 44k INFO ====> Epoch: 819, cost 59.76 s 2023-09-17 01:38:29,564 44k INFO ====> Epoch: 820, cost 58.54 s 2023-09-17 01:39:27,549 44k INFO ====> Epoch: 821, cost 57.98 s 2023-09-17 01:40:26,064 44k INFO ====> Epoch: 822, cost 58.52 s 2023-09-17 01:41:25,158 44k INFO ====> Epoch: 823, cost 59.09 s 2023-09-17 01:42:23,991 44k INFO ====> Epoch: 824, cost 58.83 s 2023-09-17 01:43:22,757 44k INFO Train Epoch: 825 [97%] 2023-09-17 01:43:22,758 44k INFO Losses: [1.9712873697280884, 2.6549086570739746, 7.899457931518555, 17.583913803100586, -0.09727788716554642], step: 26400, lr: 4.510605827868957e-05, reference_loss: 30.01228904724121 2023-09-17 01:43:39,621 44k INFO Saving model and optimizer state at iteration 825 to ./logs/44k/G_26400.pth 2023-09-17 01:43:41,314 44k INFO Saving model and optimizer state at iteration 825 to ./logs/44k/D_26400.pth 2023-09-17 01:43:41,896 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_22400.pth 2023-09-17 01:43:41,897 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_22400.pth 2023-09-17 01:43:41,898 44k INFO ====> Epoch: 825, cost 77.91 s 2023-09-17 01:44:40,334 44k INFO ====> Epoch: 826, cost 58.44 s 2023-09-17 01:45:38,615 44k INFO ====> Epoch: 827, cost 58.28 s 2023-09-17 01:46:37,489 44k INFO ====> Epoch: 828, cost 58.87 s 2023-09-17 01:47:35,847 44k INFO ====> Epoch: 829, cost 58.36 s 2023-09-17 01:48:34,339 44k INFO ====> Epoch: 830, cost 58.49 s 2023-09-17 01:49:32,734 44k INFO ====> Epoch: 831, cost 58.39 s 2023-09-17 01:49:47,694 44k INFO Train Epoch: 832 [22%] 2023-09-17 01:49:47,695 44k INFO Losses: [2.298842668533325, 2.5851056575775146, 13.15904426574707, 22.834814071655273, 0.9302595853805542], step: 26600, lr: 4.5066605275038036e-05, reference_loss: 41.808067321777344 2023-09-17 01:50:32,197 44k INFO ====> Epoch: 832, cost 59.46 s 2023-09-17 01:51:30,872 44k INFO ====> Epoch: 833, cost 58.68 s 2023-09-17 01:52:29,635 44k INFO ====> Epoch: 834, cost 58.76 s 2023-09-17 01:53:28,265 44k INFO ====> Epoch: 835, cost 58.63 s 2023-09-17 01:54:27,280 44k INFO ====> Epoch: 836, cost 59.01 s 2023-09-17 01:55:25,718 44k INFO ====> Epoch: 837, cost 58.44 s 2023-09-17 01:55:55,616 44k INFO Train Epoch: 838 [47%] 2023-09-17 01:55:55,617 44k INFO Losses: [2.2063980102539062, 2.548725128173828, 12.031522750854492, 22.61162567138672, 0.8750007152557373], step: 26800, lr: 4.50328158818071e-05, reference_loss: 40.27326965332031 2023-09-17 01:56:25,111 44k INFO ====> Epoch: 838, cost 59.39 s 2023-09-17 01:57:23,765 44k INFO ====> Epoch: 839, cost 58.65 s 2023-09-17 01:58:22,195 44k INFO ====> Epoch: 840, cost 58.43 s 2023-09-17 01:59:20,826 44k INFO ====> Epoch: 841, cost 58.63 s 2023-09-17 02:00:19,808 44k INFO ====> Epoch: 842, cost 58.98 s 2023-09-17 02:01:18,297 44k INFO ====> Epoch: 843, cost 58.49 s 2023-09-17 02:02:03,139 44k INFO Train Epoch: 844 [72%] 2023-09-17 02:02:03,140 44k INFO Losses: [2.2046098709106445, 2.7109830379486084, 13.351774215698242, 22.180652618408203, 0.9450808167457581], step: 27000, lr: 4.499905182270303e-05, reference_loss: 41.39310073852539 2023-09-17 02:02:17,122 44k INFO ====> Epoch: 844, cost 58.83 s 2023-09-17 02:03:15,830 44k INFO ====> Epoch: 845, cost 58.71 s 2023-09-17 02:04:14,079 44k INFO ====> Epoch: 846, cost 58.25 s 2023-09-17 02:05:12,519 44k INFO ====> Epoch: 847, cost 58.44 s 2023-09-17 02:06:10,857 44k INFO ====> Epoch: 848, cost 58.34 s 2023-09-17 02:07:09,402 44k INFO ====> Epoch: 849, cost 58.54 s 2023-09-17 02:08:07,604 44k INFO Train Epoch: 850 [97%] 2023-09-17 02:08:07,605 44k INFO Losses: [1.6573290824890137, 2.943274736404419, 12.701081275939941, 18.000228881835938, -0.14536084234714508], step: 27200, lr: 4.4965313078731154e-05, reference_loss: 35.15655517578125 2023-09-17 02:08:22,663 44k INFO Saving model and optimizer state at iteration 850 to ./logs/44k/G_27200.pth 2023-09-17 02:08:26,057 44k INFO Saving model and optimizer state at iteration 850 to ./logs/44k/D_27200.pth 2023-09-17 02:08:27,054 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_23200.pth 2023-09-17 02:08:27,061 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_23200.pth 2023-09-17 02:08:27,067 44k INFO ====> Epoch: 850, cost 77.67 s 2023-09-17 02:09:25,979 44k INFO ====> Epoch: 851, cost 58.91 s 2023-09-17 02:10:24,373 44k INFO ====> Epoch: 852, cost 58.39 s 2023-09-17 02:11:22,727 44k INFO ====> Epoch: 853, cost 58.35 s 2023-09-17 02:12:21,970 44k INFO ====> Epoch: 854, cost 59.24 s 2023-09-17 02:13:20,822 44k INFO ====> Epoch: 855, cost 58.85 s 2023-09-17 02:14:19,105 44k INFO ====> Epoch: 856, cost 58.28 s 2023-09-17 02:14:33,984 44k INFO Train Epoch: 857 [22%] 2023-09-17 02:14:33,985 44k INFO Losses: [2.269883871078491, 2.6083033084869385, 14.831443786621094, 23.142986297607422, 0.9258599281311035], step: 27400, lr: 4.492598318095719e-05, reference_loss: 43.77847671508789 2023-09-17 02:15:18,492 44k INFO ====> Epoch: 857, cost 59.39 s 2023-09-17 02:16:17,316 44k INFO ====> Epoch: 858, cost 58.82 s 2023-09-17 02:17:16,277 44k INFO ====> Epoch: 859, cost 58.96 s 2023-09-17 02:18:14,508 44k INFO ====> Epoch: 860, cost 58.23 s 2023-09-17 02:19:13,314 44k INFO ====> Epoch: 861, cost 58.81 s 2023-09-17 02:20:12,681 44k INFO ====> Epoch: 862, cost 59.37 s 2023-09-17 02:20:42,377 44k INFO Train Epoch: 863 [47%] 2023-09-17 02:20:42,378 44k INFO Losses: [2.1977710723876953, 2.6459901332855225, 13.607035636901855, 22.86985969543457, 0.8915404081344604], step: 27600, lr: 4.489229922134401e-05, reference_loss: 42.212196350097656 2023-09-17 02:21:11,759 44k INFO ====> Epoch: 863, cost 59.08 s 2023-09-17 02:22:10,054 44k INFO ====> Epoch: 864, cost 58.29 s 2023-09-17 02:23:08,266 44k INFO ====> Epoch: 865, cost 58.21 s 2023-09-17 02:24:06,928 44k INFO ====> Epoch: 866, cost 58.66 s 2023-09-17 02:25:05,088 44k INFO ====> Epoch: 867, cost 58.16 s 2023-09-17 02:26:03,798 44k INFO ====> Epoch: 868, cost 58.71 s 2023-09-17 02:26:48,669 44k INFO Train Epoch: 869 [72%] 2023-09-17 02:26:48,670 44k INFO Losses: [2.2515134811401367, 2.4955413341522217, 13.292341232299805, 22.272058486938477, 0.9371263980865479], step: 27800, lr: 4.4858640516807185e-05, reference_loss: 41.24858093261719 2023-09-17 02:27:03,077 44k INFO ====> Epoch: 869, cost 59.28 s 2023-09-17 02:28:01,808 44k INFO ====> Epoch: 870, cost 58.73 s 2023-09-17 02:29:00,270 44k INFO ====> Epoch: 871, cost 58.46 s 2023-09-17 02:29:58,605 44k INFO ====> Epoch: 872, cost 58.33 s 2023-09-17 02:30:57,379 44k INFO ====> Epoch: 873, cost 58.77 s 2023-09-17 02:31:56,497 44k INFO ====> Epoch: 874, cost 59.12 s 2023-09-17 02:32:55,189 44k INFO Train Epoch: 875 [97%] 2023-09-17 02:32:55,190 44k INFO Losses: [1.7258665561676025, 2.895817518234253, 11.198967933654785, 17.405107498168945, -0.18805302679538727], step: 28000, lr: 4.482500704841131e-05, reference_loss: 33.0377082824707 2023-09-17 02:33:10,536 44k INFO Saving model and optimizer state at iteration 875 to ./logs/44k/G_28000.pth 2023-09-17 02:33:14,094 44k INFO Saving model and optimizer state at iteration 875 to ./logs/44k/D_28000.pth 2023-09-17 02:33:14,664 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_24000.pth 2023-09-17 02:33:14,665 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_24000.pth 2023-09-17 02:33:14,665 44k INFO ====> Epoch: 875, cost 78.17 s 2023-09-17 02:34:12,740 44k INFO ====> Epoch: 876, cost 58.07 s 2023-09-17 02:35:11,244 44k INFO ====> Epoch: 877, cost 58.50 s 2023-09-17 02:36:10,153 44k INFO ====> Epoch: 878, cost 58.91 s 2023-09-17 02:37:09,037 44k INFO ====> Epoch: 879, cost 58.88 s 2023-09-17 02:38:07,481 44k INFO ====> Epoch: 880, cost 58.44 s 2023-09-17 02:39:06,200 44k INFO ====> Epoch: 881, cost 58.72 s 2023-09-17 02:39:21,177 44k INFO Train Epoch: 882 [22%] 2023-09-17 02:39:21,178 44k INFO Losses: [2.286689043045044, 2.6277318000793457, 13.90092945098877, 23.050750732421875, 0.9014525413513184], step: 28200, lr: 4.4785799872385556e-05, reference_loss: 42.76755142211914 2023-09-17 02:40:05,427 44k INFO ====> Epoch: 882, cost 59.23 s 2023-09-17 02:41:03,514 44k INFO ====> Epoch: 883, cost 58.09 s 2023-09-17 02:42:02,235 44k INFO ====> Epoch: 884, cost 58.72 s 2023-09-17 02:43:01,311 44k INFO ====> Epoch: 885, cost 59.08 s 2023-09-17 02:43:59,970 44k INFO ====> Epoch: 886, cost 58.66 s 2023-09-17 02:44:58,799 44k INFO ====> Epoch: 887, cost 58.83 s 2023-09-17 02:45:28,513 44k INFO Train Epoch: 888 [47%] 2023-09-17 02:45:28,514 44k INFO Losses: [2.2228987216949463, 2.604438543319702, 12.08128833770752, 22.735803604125977, 0.8433781862258911], step: 28400, lr: 4.4752221017403816e-05, reference_loss: 40.48781204223633 2023-09-17 02:45:57,921 44k INFO ====> Epoch: 888, cost 59.12 s 2023-09-17 02:46:55,939 44k INFO ====> Epoch: 889, cost 58.02 s 2023-09-17 02:47:54,502 44k INFO ====> Epoch: 890, cost 58.56 s 2023-09-17 02:48:53,139 44k INFO ====> Epoch: 891, cost 58.64 s 2023-09-17 02:49:51,890 44k INFO ====> Epoch: 892, cost 58.75 s 2023-09-17 02:50:50,154 44k INFO ====> Epoch: 893, cost 58.26 s 2023-09-17 02:51:35,692 44k INFO Train Epoch: 894 [72%] 2023-09-17 02:51:35,695 44k INFO Losses: [2.2496368885040283, 2.5005760192871094, 12.840408325195312, 22.279842376708984, 0.9212618470191956], step: 28600, lr: 4.4718667338694587e-05, reference_loss: 40.791725158691406 2023-09-17 02:51:49,699 44k INFO ====> Epoch: 894, cost 59.55 s 2023-09-17 02:52:48,245 44k INFO ====> Epoch: 895, cost 58.55 s 2023-09-17 02:53:46,688 44k INFO ====> Epoch: 896, cost 58.44 s 2023-09-17 02:54:45,211 44k INFO ====> Epoch: 897, cost 58.52 s 2023-09-17 02:55:43,460 44k INFO ====> Epoch: 898, cost 58.25 s 2023-09-17 02:56:41,726 44k INFO ====> Epoch: 899, cost 58.27 s 2023-09-17 02:57:40,864 44k INFO Train Epoch: 900 [97%] 2023-09-17 02:57:40,865 44k INFO Losses: [1.8490550518035889, 2.634406089782715, 7.7183308601379395, 16.993322372436523, -0.17952902615070343], step: 28800, lr: 4.468513881738155e-05, reference_loss: 29.01558494567871 2023-09-17 02:57:58,222 44k INFO Saving model and optimizer state at iteration 900 to ./logs/44k/G_28800.pth 2023-09-17 02:58:01,701 44k INFO Saving model and optimizer state at iteration 900 to ./logs/44k/D_28800.pth 2023-09-17 02:58:02,231 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_24800.pth 2023-09-17 02:58:02,234 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_24800.pth 2023-09-17 02:58:02,234 44k INFO ====> Epoch: 900, cost 80.51 s 2023-09-17 02:59:00,719 44k INFO ====> Epoch: 901, cost 58.48 s 2023-09-17 02:59:59,422 44k INFO ====> Epoch: 902, cost 58.70 s 2023-09-17 03:00:58,012 44k INFO ====> Epoch: 903, cost 58.59 s 2023-09-17 03:01:56,425 44k INFO ====> Epoch: 904, cost 58.41 s 2023-09-17 03:02:54,865 44k INFO ====> Epoch: 905, cost 58.44 s 2023-09-17 03:03:52,756 44k INFO ====> Epoch: 906, cost 57.89 s 2023-09-17 03:04:07,711 44k INFO Train Epoch: 907 [22%] 2023-09-17 03:04:07,712 44k INFO Losses: [2.33132266998291, 2.4665074348449707, 12.3078031539917, 22.659616470336914, 0.8811843991279602], step: 29000, lr: 4.4646053980173236e-05, reference_loss: 40.64643478393555 2023-09-17 03:04:51,988 44k INFO ====> Epoch: 907, cost 59.23 s 2023-09-17 03:05:50,524 44k INFO ====> Epoch: 908, cost 58.54 s 2023-09-17 03:06:49,214 44k INFO ====> Epoch: 909, cost 58.69 s 2023-09-17 03:07:47,505 44k INFO ====> Epoch: 910, cost 58.29 s 2023-09-17 03:08:45,693 44k INFO ====> Epoch: 911, cost 58.19 s 2023-09-17 03:09:44,484 44k INFO ====> Epoch: 912, cost 58.79 s 2023-09-17 03:10:14,435 44k INFO Train Epoch: 913 [47%] 2023-09-17 03:10:14,436 44k INFO Losses: [2.1758687496185303, 2.497500419616699, 13.535901069641113, 22.753141403198242, 0.8530375361442566], step: 29200, lr: 4.461257990186317e-05, reference_loss: 41.81544876098633 2023-09-17 03:10:43,698 44k INFO ====> Epoch: 913, cost 59.21 s 2023-09-17 03:11:42,455 44k INFO ====> Epoch: 914, cost 58.76 s 2023-09-17 03:12:41,264 44k INFO ====> Epoch: 915, cost 58.81 s 2023-09-17 03:13:39,775 44k INFO ====> Epoch: 916, cost 58.51 s 2023-09-17 03:14:38,146 44k INFO ====> Epoch: 917, cost 58.37 s 2023-09-17 03:15:36,824 44k INFO ====> Epoch: 918, cost 58.68 s 2023-09-17 03:16:21,704 44k INFO Train Epoch: 919 [72%] 2023-09-17 03:16:21,705 44k INFO Losses: [2.2461793422698975, 2.5130615234375, 12.8385648727417, 22.213396072387695, 0.881579577922821], step: 29400, lr: 4.457913092126766e-05, reference_loss: 40.69278335571289 2023-09-17 03:16:36,236 44k INFO ====> Epoch: 919, cost 59.41 s 2023-09-17 03:17:34,989 44k INFO ====> Epoch: 920, cost 58.75 s 2023-09-17 03:18:33,590 44k INFO ====> Epoch: 921, cost 58.60 s 2023-09-17 03:19:32,782 44k INFO ====> Epoch: 922, cost 59.19 s 2023-09-17 03:20:31,515 44k INFO ====> Epoch: 923, cost 58.73 s 2023-09-17 03:21:30,174 44k INFO ====> Epoch: 924, cost 58.66 s 2023-09-17 03:22:28,377 44k INFO Train Epoch: 925 [97%] 2023-09-17 03:22:28,378 44k INFO Losses: [1.8086473941802979, 3.0142672061920166, 13.527506828308105, 18.155517578125, -0.1510510891675949], step: 29600, lr: 4.454570701956929e-05, reference_loss: 36.354888916015625 2023-09-17 03:22:45,096 44k INFO Saving model and optimizer state at iteration 925 to ./logs/44k/G_29600.pth 2023-09-17 03:22:48,507 44k INFO Saving model and optimizer state at iteration 925 to ./logs/44k/D_29600.pth 2023-09-17 03:22:49,208 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_25600.pth 2023-09-17 03:22:49,210 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_25600.pth 2023-09-17 03:22:49,210 44k INFO ====> Epoch: 925, cost 79.04 s 2023-09-17 03:23:47,754 44k INFO ====> Epoch: 926, cost 58.54 s 2023-09-17 03:24:46,246 44k INFO ====> Epoch: 927, cost 58.49 s 2023-09-17 03:25:45,402 44k INFO ====> Epoch: 928, cost 59.16 s 2023-09-17 03:26:43,863 44k INFO ====> Epoch: 929, cost 58.46 s 2023-09-17 03:27:42,942 44k INFO ====> Epoch: 930, cost 59.08 s 2023-09-17 03:28:41,984 44k INFO ====> Epoch: 931, cost 59.04 s 2023-09-17 03:28:56,698 44k INFO Train Epoch: 932 [22%] 2023-09-17 03:28:56,699 44k INFO Losses: [2.322528600692749, 2.714061737060547, 12.457982063293457, 22.747812271118164, 0.9031170010566711], step: 29800, lr: 4.4506744139442534e-05, reference_loss: 41.145503997802734 2023-09-17 03:29:40,788 44k INFO ====> Epoch: 932, cost 58.80 s 2023-09-17 03:30:39,407 44k INFO ====> Epoch: 933, cost 58.62 s 2023-09-17 03:31:38,252 44k INFO ====> Epoch: 934, cost 58.85 s 2023-09-17 03:32:36,939 44k INFO ====> Epoch: 935, cost 58.69 s 2023-09-17 03:33:36,090 44k INFO ====> Epoch: 936, cost 59.15 s 2023-09-17 03:34:35,077 44k INFO ====> Epoch: 937, cost 58.99 s 2023-09-17 03:35:05,099 44k INFO Train Epoch: 938 [47%] 2023-09-17 03:35:05,100 44k INFO Losses: [2.1962945461273193, 2.608454942703247, 13.263845443725586, 22.75813102722168, 0.8539586067199707], step: 30000, lr: 4.447337451086772e-05, reference_loss: 41.680686950683594 2023-09-17 03:35:34,773 44k INFO ====> Epoch: 938, cost 59.70 s 2023-09-17 03:36:33,382 44k INFO ====> Epoch: 939, cost 58.61 s 2023-09-17 03:37:32,200 44k INFO ====> Epoch: 940, cost 58.82 s 2023-09-17 03:38:30,509 44k INFO ====> Epoch: 941, cost 58.31 s 2023-09-17 03:39:29,483 44k INFO ====> Epoch: 942, cost 58.97 s 2023-09-17 03:40:28,319 44k INFO ====> Epoch: 943, cost 58.84 s 2023-09-17 03:41:13,395 44k INFO Train Epoch: 944 [72%] 2023-09-17 03:41:13,396 44k INFO Losses: [2.2279529571533203, 2.4473438262939453, 13.753270149230957, 22.030681610107422, 0.9064825773239136], step: 30200, lr: 4.444002990169463e-05, reference_loss: 41.36573028564453 2023-09-17 03:41:27,748 44k INFO ====> Epoch: 944, cost 59.43 s 2023-09-17 03:42:26,184 44k INFO ====> Epoch: 945, cost 58.44 s 2023-09-17 03:43:25,260 44k INFO ====> Epoch: 946, cost 59.08 s 2023-09-17 03:44:23,433 44k INFO ====> Epoch: 947, cost 58.17 s 2023-09-17 03:45:22,238 44k INFO ====> Epoch: 948, cost 58.81 s 2023-09-17 03:46:20,904 44k INFO ====> Epoch: 949, cost 58.67 s 2023-09-17 03:47:19,777 44k INFO Train Epoch: 950 [97%] 2023-09-17 03:47:19,778 44k INFO Losses: [1.9343032836914062, 2.8160457611083984, 12.08349895477295, 17.381772994995117, -0.19466929137706757], step: 30400, lr: 4.440671029316458e-05, reference_loss: 34.02095413208008 2023-09-17 03:47:36,740 44k INFO Saving model and optimizer state at iteration 950 to ./logs/44k/G_30400.pth 2023-09-17 03:47:40,392 44k INFO Saving model and optimizer state at iteration 950 to ./logs/44k/D_30400.pth 2023-09-17 03:47:40,932 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_26400.pth 2023-09-17 03:47:40,933 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_26400.pth 2023-09-17 03:47:40,934 44k INFO ====> Epoch: 950, cost 80.03 s 2023-09-17 03:48:39,616 44k INFO ====> Epoch: 951, cost 58.68 s 2023-09-17 03:49:38,075 44k INFO ====> Epoch: 952, cost 58.46 s 2023-09-17 03:50:36,761 44k INFO ====> Epoch: 953, cost 58.69 s 2023-09-17 03:51:35,393 44k INFO ====> Epoch: 954, cost 58.63 s 2023-09-17 03:52:34,068 44k INFO ====> Epoch: 955, cost 58.68 s 2023-09-17 03:53:32,669 44k INFO ====> Epoch: 956, cost 58.60 s 2023-09-17 03:53:47,514 44k INFO Train Epoch: 957 [22%] 2023-09-17 03:53:47,515 44k INFO Losses: [2.251624345779419, 2.6805520057678223, 14.665009498596191, 22.92404556274414, 0.9492825269699097], step: 30600, lr: 4.4367868989574625e-05, reference_loss: 43.470516204833984 2023-09-17 03:54:32,206 44k INFO ====> Epoch: 957, cost 59.54 s 2023-09-17 03:55:30,798 44k INFO ====> Epoch: 958, cost 58.59 s 2023-09-17 03:56:29,572 44k INFO ====> Epoch: 959, cost 58.77 s 2023-09-17 03:57:28,441 44k INFO ====> Epoch: 960, cost 58.87 s 2023-09-17 03:58:27,126 44k INFO ====> Epoch: 961, cost 58.68 s 2023-09-17 03:59:26,446 44k INFO ====> Epoch: 962, cost 59.32 s 2023-09-17 03:59:56,496 44k INFO Train Epoch: 963 [47%] 2023-09-17 03:59:56,497 44k INFO Losses: [2.2130775451660156, 2.4970052242279053, 12.459375381469727, 22.574777603149414, 0.8549054861068726], step: 30800, lr: 4.433460348481877e-05, reference_loss: 40.59914016723633 2023-09-17 04:00:26,124 44k INFO ====> Epoch: 963, cost 59.68 s 2023-09-17 04:01:24,628 44k INFO ====> Epoch: 964, cost 58.50 s 2023-09-17 04:02:23,498 44k INFO ====> Epoch: 965, cost 58.87 s 2023-09-17 04:03:22,145 44k INFO ====> Epoch: 966, cost 58.65 s 2023-09-17 04:04:21,220 44k INFO ====> Epoch: 967, cost 59.08 s 2023-09-17 04:05:19,403 44k INFO ====> Epoch: 968, cost 58.18 s 2023-09-17 04:06:04,750 44k INFO Train Epoch: 969 [72%] 2023-09-17 04:06:04,751 44k INFO Losses: [2.17602801322937, 2.548927068710327, 13.699070930480957, 22.1005916595459, 0.8974735736846924], step: 31000, lr: 4.430136292139618e-05, reference_loss: 41.42209243774414 2023-09-17 04:06:19,267 44k INFO ====> Epoch: 969, cost 59.86 s 2023-09-17 04:07:18,338 44k INFO ====> Epoch: 970, cost 59.07 s 2023-09-17 04:08:17,149 44k INFO ====> Epoch: 971, cost 58.81 s 2023-09-17 04:09:16,769 44k INFO ====> Epoch: 972, cost 59.62 s 2023-09-17 04:10:15,684 44k INFO ====> Epoch: 973, cost 58.91 s 2023-09-17 04:11:14,822 44k INFO ====> Epoch: 974, cost 59.14 s 2023-09-17 04:12:13,709 44k INFO Train Epoch: 975 [97%] 2023-09-17 04:12:13,710 44k INFO Losses: [1.683647871017456, 2.824279546737671, 10.249406814575195, 17.44159507751465, -0.12002642452716827], step: 31200, lr: 4.4268147280606693e-05, reference_loss: 32.07890319824219 2023-09-17 04:12:30,536 44k INFO Saving model and optimizer state at iteration 975 to ./logs/44k/G_31200.pth 2023-09-17 04:12:33,681 44k INFO Saving model and optimizer state at iteration 975 to ./logs/44k/D_31200.pth 2023-09-17 04:12:34,858 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_27200.pth 2023-09-17 04:12:34,875 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_27200.pth 2023-09-17 04:12:34,897 44k INFO ====> Epoch: 975, cost 80.07 s 2023-09-17 04:13:33,645 44k INFO ====> Epoch: 976, cost 58.75 s 2023-09-17 04:14:31,978 44k INFO ====> Epoch: 977, cost 58.33 s 2023-09-17 04:15:30,487 44k INFO ====> Epoch: 978, cost 58.51 s 2023-09-17 04:16:29,388 44k INFO ====> Epoch: 979, cost 58.90 s 2023-09-17 04:17:27,811 44k INFO ====> Epoch: 980, cost 58.42 s 2023-09-17 04:18:25,712 44k INFO ====> Epoch: 981, cost 57.90 s 2023-09-17 04:18:40,573 44k INFO Train Epoch: 982 [22%] 2023-09-17 04:18:40,574 44k INFO Losses: [2.2731664180755615, 2.628171443939209, 12.984395027160645, 22.754196166992188, 0.900371253490448], step: 31400, lr: 4.422942717419621e-05, reference_loss: 41.54030227661133 2023-09-17 04:19:24,920 44k INFO ====> Epoch: 982, cost 59.21 s 2023-09-17 04:20:23,541 44k INFO ====> Epoch: 983, cost 58.62 s 2023-09-17 04:21:22,119 44k INFO ====> Epoch: 984, cost 58.58 s 2023-09-17 04:22:20,385 44k INFO ====> Epoch: 985, cost 58.27 s 2023-09-17 04:23:19,238 44k INFO ====> Epoch: 986, cost 58.85 s 2023-09-17 04:24:17,811 44k INFO ====> Epoch: 987, cost 58.57 s 2023-09-17 04:24:47,910 44k INFO Train Epoch: 988 [47%] 2023-09-17 04:24:47,911 44k INFO Losses: [2.2010316848754883, 2.6897778511047363, 13.657269477844238, 22.806730270385742, 0.8376214504241943], step: 31600, lr: 4.419626546836e-05, reference_loss: 42.19242858886719 2023-09-17 04:25:17,280 44k INFO ====> Epoch: 988, cost 59.47 s 2023-09-17 04:26:15,802 44k INFO ====> Epoch: 989, cost 58.52 s 2023-09-17 04:27:14,456 44k INFO ====> Epoch: 990, cost 58.65 s 2023-09-17 04:28:13,380 44k INFO ====> Epoch: 991, cost 58.92 s 2023-09-17 04:29:11,893 44k INFO ====> Epoch: 992, cost 58.51 s 2023-09-17 04:30:10,141 44k INFO ====> Epoch: 993, cost 58.25 s 2023-09-17 04:30:54,912 44k INFO Train Epoch: 994 [72%] 2023-09-17 04:30:54,913 44k INFO Losses: [2.2019295692443848, 2.6545305252075195, 13.988818168640137, 22.281328201293945, 0.909777045249939], step: 31800, lr: 4.416312862603218e-05, reference_loss: 42.03638458251953 2023-09-17 04:31:09,137 44k INFO ====> Epoch: 994, cost 59.00 s 2023-09-17 04:32:07,959 44k INFO ====> Epoch: 995, cost 58.82 s 2023-09-17 04:33:06,359 44k INFO ====> Epoch: 996, cost 58.40 s 2023-09-17 04:34:05,231 44k INFO ====> Epoch: 997, cost 58.87 s 2023-09-17 04:35:03,457 44k INFO ====> Epoch: 998, cost 58.23 s 2023-09-17 04:36:02,089 44k INFO ====> Epoch: 999, cost 58.63 s 2023-09-17 04:37:00,455 44k INFO Train Epoch: 1000 [97%] 2023-09-17 04:37:00,456 44k INFO Losses: [1.7904367446899414, 2.935420513153076, 12.501069068908691, 16.945024490356445, -0.2147098332643509], step: 32000, lr: 4.413001662857095e-05, reference_loss: 33.95724105834961 2023-09-17 04:37:17,645 44k INFO Saving model and optimizer state at iteration 1000 to ./logs/44k/G_32000.pth 2023-09-17 04:37:21,206 44k INFO Saving model and optimizer state at iteration 1000 to ./logs/44k/D_32000.pth 2023-09-17 04:37:21,874 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_28000.pth 2023-09-17 04:37:21,876 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_28000.pth 2023-09-17 04:37:21,876 44k INFO ====> Epoch: 1000, cost 79.79 s 2023-09-17 04:38:20,415 44k INFO ====> Epoch: 1001, cost 58.54 s 2023-09-17 04:39:19,194 44k INFO ====> Epoch: 1002, cost 58.78 s 2023-09-17 04:40:17,653 44k INFO ====> Epoch: 1003, cost 58.46 s 2023-09-17 04:41:16,253 44k INFO ====> Epoch: 1004, cost 58.60 s 2023-09-17 04:42:14,875 44k INFO ====> Epoch: 1005, cost 58.62 s 2023-09-17 04:43:13,992 44k INFO ====> Epoch: 1006, cost 59.12 s 2023-09-17 04:43:28,739 44k INFO Train Epoch: 1007 [22%] 2023-09-17 04:43:28,740 44k INFO Losses: [2.3288052082061768, 2.598698377609253, 12.68195915222168, 22.702617645263672, 0.9420521259307861], step: 32200, lr: 4.409141734116632e-05, reference_loss: 41.25413131713867 2023-09-17 04:44:13,279 44k INFO ====> Epoch: 1007, cost 59.29 s 2023-09-17 04:45:12,047 44k INFO ====> Epoch: 1008, cost 58.77 s 2023-09-17 04:46:10,392 44k INFO ====> Epoch: 1009, cost 58.34 s 2023-09-17 04:47:09,239 44k INFO ====> Epoch: 1010, cost 58.85 s 2023-09-17 04:48:07,893 44k INFO ====> Epoch: 1011, cost 58.65 s 2023-09-17 04:49:06,693 44k INFO ====> Epoch: 1012, cost 58.80 s 2023-09-17 04:49:36,522 44k INFO Train Epoch: 1013 [47%] 2023-09-17 04:49:36,523 44k INFO Losses: [2.1364450454711914, 2.714231491088867, 12.883227348327637, 22.50527000427246, 0.860404372215271], step: 32400, lr: 4.405835911036422e-05, reference_loss: 41.099578857421875 2023-09-17 04:50:05,965 44k INFO ====> Epoch: 1013, cost 59.27 s 2023-09-17 04:51:04,487 44k INFO ====> Epoch: 1014, cost 58.52 s 2023-09-17 04:52:03,069 44k INFO ====> Epoch: 1015, cost 58.58 s 2023-09-17 04:53:02,096 44k INFO ====> Epoch: 1016, cost 59.03 s 2023-09-17 04:54:00,641 44k INFO ====> Epoch: 1017, cost 58.55 s 2023-09-17 04:54:58,863 44k INFO ====> Epoch: 1018, cost 58.22 s 2023-09-17 04:55:43,665 44k INFO Train Epoch: 1019 [72%] 2023-09-17 04:55:43,666 44k INFO Losses: [2.241421699523926, 2.480940103530884, 12.629622459411621, 22.018545150756836, 0.9288999438285828], step: 32600, lr: 4.402532566548848e-05, reference_loss: 40.29943084716797 2023-09-17 04:55:58,103 44k INFO ====> Epoch: 1019, cost 59.24 s 2023-09-17 04:56:56,813 44k INFO ====> Epoch: 1020, cost 58.71 s 2023-09-17 04:57:55,079 44k INFO ====> Epoch: 1021, cost 58.27 s 2023-09-17 04:58:53,670 44k INFO ====> Epoch: 1022, cost 58.59 s 2023-09-17 04:59:52,184 44k INFO ====> Epoch: 1023, cost 58.51 s 2023-09-17 05:00:50,745 44k INFO ====> Epoch: 1024, cost 58.56 s 2023-09-17 05:01:49,838 44k INFO Train Epoch: 1025 [97%] 2023-09-17 05:01:49,839 44k INFO Losses: [1.7514399290084839, 2.883885383605957, 10.968294143676758, 17.07666015625, -0.22269606590270996], step: 32800, lr: 4.399231698795547e-05, reference_loss: 32.457584381103516 2023-09-17 05:02:06,986 44k INFO Saving model and optimizer state at iteration 1025 to ./logs/44k/G_32800.pth 2023-09-17 05:02:10,386 44k INFO Saving model and optimizer state at iteration 1025 to ./logs/44k/D_32800.pth 2023-09-17 05:02:10,916 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_28800.pth 2023-09-17 05:02:10,919 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_28800.pth 2023-09-17 05:02:10,919 44k INFO ====> Epoch: 1025, cost 80.17 s 2023-09-17 05:03:09,104 44k INFO ====> Epoch: 1026, cost 58.19 s 2023-09-17 05:04:07,817 44k INFO ====> Epoch: 1027, cost 58.71 s 2023-09-17 05:05:06,375 44k INFO ====> Epoch: 1028, cost 58.56 s 2023-09-17 05:06:05,249 44k INFO ====> Epoch: 1029, cost 58.87 s 2023-09-17 05:07:04,036 44k INFO ====> Epoch: 1030, cost 58.79 s 2023-09-17 05:08:02,041 44k INFO ====> Epoch: 1031, cost 58.00 s 2023-09-17 05:08:16,686 44k INFO Train Epoch: 1032 [22%] 2023-09-17 05:08:16,687 44k INFO Losses: [2.248016357421875, 2.6002557277679443, 12.447113037109375, 22.715335845947266, 0.8722014427185059], step: 33000, lr: 4.39538381425631e-05, reference_loss: 40.88291931152344 2023-09-17 05:09:00,801 44k INFO ====> Epoch: 1032, cost 58.76 s 2023-09-17 05:09:59,563 44k INFO ====> Epoch: 1033, cost 58.76 s 2023-09-17 05:10:58,149 44k INFO ====> Epoch: 1034, cost 58.59 s 2023-09-17 05:11:56,545 44k INFO ====> Epoch: 1035, cost 58.40 s 2023-09-17 05:12:54,987 44k INFO ====> Epoch: 1036, cost 58.44 s 2023-09-17 05:13:53,887 44k INFO ====> Epoch: 1037, cost 58.90 s 2023-09-17 05:14:23,624 44k INFO Train Epoch: 1038 [47%] 2023-09-17 05:14:23,625 44k INFO Losses: [2.1859054565429688, 2.590054512023926, 12.703890800476074, 22.66695213317871, 0.8181431889533997], step: 33200, lr: 4.3920883063920205e-05, reference_loss: 40.96494674682617 2023-09-17 05:14:52,810 44k INFO ====> Epoch: 1038, cost 58.92 s 2023-09-17 05:15:50,985 44k INFO ====> Epoch: 1039, cost 58.17 s 2023-09-17 05:16:49,582 44k INFO ====> Epoch: 1040, cost 58.60 s 2023-09-17 05:17:47,912 44k INFO ====> Epoch: 1041, cost 58.33 s 2023-09-17 05:18:46,384 44k INFO ====> Epoch: 1042, cost 58.47 s 2023-09-17 05:19:45,427 44k INFO ====> Epoch: 1043, cost 59.04 s 2023-09-17 05:20:30,284 44k INFO Train Epoch: 1044 [72%] 2023-09-17 05:20:30,285 44k INFO Losses: [2.226125955581665, 2.480888843536377, 13.110255241394043, 21.939870834350586, 0.872590959072113], step: 33400, lr: 4.3887952693863714e-05, reference_loss: 40.629730224609375 2023-09-17 05:20:44,302 44k INFO ====> Epoch: 1044, cost 58.87 s 2023-09-17 05:21:43,094 44k INFO ====> Epoch: 1045, cost 58.79 s 2023-09-17 05:22:41,418 44k INFO ====> Epoch: 1046, cost 58.32 s 2023-09-17 05:23:39,731 44k INFO ====> Epoch: 1047, cost 58.31 s 2023-09-17 05:24:38,255 44k INFO ====> Epoch: 1048, cost 58.52 s 2023-09-17 05:25:36,357 44k INFO ====> Epoch: 1049, cost 58.10 s 2023-09-17 05:26:34,940 44k INFO Train Epoch: 1050 [97%] 2023-09-17 05:26:34,941 44k INFO Losses: [1.524716854095459, 3.09145450592041, 11.870292663574219, 17.576692581176758, -0.15662910044193268], step: 33600, lr: 4.385504701386801e-05, reference_loss: 33.90652847290039 2023-09-17 05:26:51,432 44k INFO Saving model and optimizer state at iteration 1050 to ./logs/44k/G_33600.pth 2023-09-17 05:26:54,846 44k INFO Saving model and optimizer state at iteration 1050 to ./logs/44k/D_33600.pth 2023-09-17 05:26:55,948 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_29600.pth 2023-09-17 05:26:55,950 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_29600.pth 2023-09-17 05:26:55,951 44k INFO ====> Epoch: 1050, cost 79.59 s 2023-09-17 05:27:54,387 44k INFO ====> Epoch: 1051, cost 58.44 s 2023-09-17 05:28:53,500 44k INFO ====> Epoch: 1052, cost 59.11 s 2023-09-17 05:29:51,915 44k INFO ====> Epoch: 1053, cost 58.42 s 2023-09-17 05:30:50,611 44k INFO ====> Epoch: 1054, cost 58.70 s 2023-09-17 05:31:49,350 44k INFO ====> Epoch: 1055, cost 58.74 s 2023-09-17 05:32:48,094 44k INFO ====> Epoch: 1056, cost 58.74 s 2023-09-17 05:33:03,061 44k INFO Train Epoch: 1057 [22%] 2023-09-17 05:33:03,062 44k INFO Losses: [2.2056427001953125, 2.7034988403320312, 13.52332592010498, 22.64900016784668, 0.8650357723236084], step: 33800, lr: 4.381668823467064e-05, reference_loss: 41.946502685546875 2023-09-17 05:33:47,443 44k INFO ====> Epoch: 1057, cost 59.35 s 2023-09-17 05:34:46,577 44k INFO ====> Epoch: 1058, cost 59.13 s 2023-09-17 05:35:46,116 44k INFO ====> Epoch: 1059, cost 59.54 s 2023-09-17 05:36:44,535 44k INFO ====> Epoch: 1060, cost 58.42 s 2023-09-17 05:37:43,534 44k INFO ====> Epoch: 1061, cost 59.00 s 2023-09-17 05:38:42,336 44k INFO ====> Epoch: 1062, cost 58.80 s 2023-09-17 05:39:12,243 44k INFO Train Epoch: 1063 [47%] 2023-09-17 05:39:12,244 44k INFO Losses: [2.229809522628784, 2.7551188468933105, 13.694644927978516, 22.790687561035156, 0.8282544612884521], step: 34000, lr: 4.3783835986319494e-05, reference_loss: 42.29851531982422 2023-09-17 05:39:41,374 44k INFO ====> Epoch: 1063, cost 59.04 s 2023-09-17 05:40:40,183 44k INFO ====> Epoch: 1064, cost 58.81 s 2023-09-17 05:41:38,899 44k INFO ====> Epoch: 1065, cost 58.72 s 2023-09-17 05:42:37,818 44k INFO ====> Epoch: 1066, cost 58.92 s 2023-09-17 05:43:36,598 44k INFO ====> Epoch: 1067, cost 58.78 s 2023-09-17 05:44:35,088 44k INFO ====> Epoch: 1068, cost 58.49 s 2023-09-17 05:45:20,362 44k INFO Train Epoch: 1069 [72%] 2023-09-17 05:45:20,363 44k INFO Losses: [2.232753276824951, 2.48469877243042, 12.51541519165039, 21.750158309936523, 0.885416567325592], step: 34200, lr: 4.375100836945616e-05, reference_loss: 39.86844253540039 2023-09-17 05:45:34,577 44k INFO ====> Epoch: 1069, cost 59.49 s 2023-09-17 05:46:33,707 44k INFO ====> Epoch: 1070, cost 59.13 s 2023-09-17 05:47:31,922 44k INFO ====> Epoch: 1071, cost 58.21 s 2023-09-17 05:48:30,853 44k INFO ====> Epoch: 1072, cost 58.93 s 2023-09-17 05:49:29,516 44k INFO ====> Epoch: 1073, cost 58.66 s 2023-09-17 05:50:28,224 44k INFO ====> Epoch: 1074, cost 58.71 s 2023-09-17 05:51:26,983 44k INFO Train Epoch: 1075 [97%] 2023-09-17 05:51:26,984 44k INFO Losses: [1.8974690437316895, 2.8047585487365723, 12.817920684814453, 17.450077056884766, -0.13112598657608032], step: 34400, lr: 4.3718205365612784e-05, reference_loss: 34.8390998840332 2023-09-17 05:51:44,045 44k INFO Saving model and optimizer state at iteration 1075 to ./logs/44k/G_34400.pth 2023-09-17 05:51:47,601 44k INFO Saving model and optimizer state at iteration 1075 to ./logs/44k/D_34400.pth 2023-09-17 05:51:48,145 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_30400.pth 2023-09-17 05:51:48,147 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_30400.pth 2023-09-17 05:51:48,147 44k INFO ====> Epoch: 1075, cost 79.92 s 2023-09-17 05:52:46,843 44k INFO ====> Epoch: 1076, cost 58.70 s 2023-09-17 05:53:46,061 44k INFO ====> Epoch: 1077, cost 59.22 s 2023-09-17 05:54:44,948 44k INFO ====> Epoch: 1078, cost 58.89 s 2023-09-17 05:55:44,324 44k INFO ====> Epoch: 1079, cost 59.38 s 2023-09-17 05:56:43,517 44k INFO ====> Epoch: 1080, cost 59.19 s 2023-09-17 05:57:42,384 44k INFO ====> Epoch: 1081, cost 58.87 s 2023-09-17 05:57:57,297 44k INFO Train Epoch: 1082 [22%] 2023-09-17 05:57:57,299 44k INFO Losses: [2.1707468032836914, 2.676020622253418, 14.201153755187988, 22.686904907226562, 0.9322684407234192], step: 34600, lr: 4.367996627796582e-05, reference_loss: 42.66709518432617 2023-09-17 05:58:42,089 44k INFO ====> Epoch: 1082, cost 59.70 s 2023-09-17 05:59:40,589 44k INFO ====> Epoch: 1083, cost 58.50 s 2023-09-17 06:00:39,798 44k INFO ====> Epoch: 1084, cost 59.21 s 2023-09-17 06:01:38,466 44k INFO ====> Epoch: 1085, cost 58.67 s 2023-09-17 06:02:36,831 44k INFO ====> Epoch: 1086, cost 58.37 s 2023-09-17 06:03:35,806 44k INFO ====> Epoch: 1087, cost 58.97 s 2023-09-17 06:04:05,278 44k INFO Train Epoch: 1088 [47%] 2023-09-17 06:04:05,279 44k INFO Losses: [2.1765294075012207, 2.5041134357452393, 13.200522422790527, 22.333995819091797, 0.8177309632301331], step: 34800, lr: 4.364721653904334e-05, reference_loss: 41.03289031982422 2023-09-17 06:04:34,971 44k INFO ====> Epoch: 1088, cost 59.17 s 2023-09-17 06:05:33,456 44k INFO ====> Epoch: 1089, cost 58.48 s 2023-09-17 06:06:31,738 44k INFO ====> Epoch: 1090, cost 58.28 s 2023-09-17 06:07:30,170 44k INFO ====> Epoch: 1091, cost 58.43 s 2023-09-17 06:08:28,358 44k INFO ====> Epoch: 1092, cost 58.19 s 2023-09-17 06:09:26,456 44k INFO ====> Epoch: 1093, cost 58.10 s 2023-09-17 06:10:11,642 44k INFO Train Epoch: 1094 [72%] 2023-09-17 06:10:11,643 44k INFO Losses: [2.211902618408203, 2.498786211013794, 13.752366065979004, 22.230031967163086, 0.9116554856300354], step: 35000, lr: 4.361449135475061e-05, reference_loss: 41.604740142822266 2023-09-17 06:10:25,683 44k INFO ====> Epoch: 1094, cost 59.23 s 2023-09-17 06:11:24,762 44k INFO ====> Epoch: 1095, cost 59.08 s 2023-09-17 06:12:23,426 44k INFO ====> Epoch: 1096, cost 58.66 s 2023-09-17 06:13:21,906 44k INFO ====> Epoch: 1097, cost 58.48 s 2023-09-17 06:14:20,488 44k INFO ====> Epoch: 1098, cost 58.58 s 2023-09-17 06:15:19,214 44k INFO ====> Epoch: 1099, cost 58.73 s 2023-09-17 06:16:17,483 44k INFO Train Epoch: 1100 [97%] 2023-09-17 06:16:17,484 44k INFO Losses: [1.6045360565185547, 2.9301562309265137, 11.060593605041504, 17.392019271850586, -0.2577070891857147], step: 35200, lr: 4.358179070667742e-05, reference_loss: 32.72959899902344 2023-09-17 06:16:34,782 44k INFO Saving model and optimizer state at iteration 1100 to ./logs/44k/G_35200.pth 2023-09-17 06:16:38,040 44k INFO Saving model and optimizer state at iteration 1100 to ./logs/44k/D_35200.pth 2023-09-17 06:16:39,308 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_31200.pth 2023-09-17 06:16:39,310 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_31200.pth 2023-09-17 06:16:39,310 44k INFO ====> Epoch: 1100, cost 80.10 s 2023-09-17 06:17:38,364 44k INFO ====> Epoch: 1101, cost 59.05 s 2023-09-17 06:18:37,001 44k INFO ====> Epoch: 1102, cost 58.64 s 2023-09-17 06:19:35,289 44k INFO ====> Epoch: 1103, cost 58.29 s 2023-09-17 06:20:33,840 44k INFO ====> Epoch: 1104, cost 58.55 s 2023-09-17 06:21:32,680 44k INFO ====> Epoch: 1105, cost 58.84 s 2023-09-17 06:22:31,415 44k INFO ====> Epoch: 1106, cost 58.74 s 2023-09-17 06:22:46,330 44k INFO Train Epoch: 1107 [22%] 2023-09-17 06:22:46,331 44k INFO Losses: [2.2615106105804443, 2.648381471633911, 13.054556846618652, 22.376399993896484, 0.8721454739570618], step: 35400, lr: 4.35436709371053e-05, reference_loss: 41.21299743652344 2023-09-17 06:23:30,907 44k INFO ====> Epoch: 1107, cost 59.49 s 2023-09-17 06:24:29,397 44k INFO ====> Epoch: 1108, cost 58.49 s 2023-09-17 06:25:27,821 44k INFO ====> Epoch: 1109, cost 58.42 s 2023-09-17 06:26:26,349 44k INFO ====> Epoch: 1110, cost 58.53 s 2023-09-17 06:27:25,159 44k INFO ====> Epoch: 1111, cost 58.81 s 2023-09-17 06:28:23,791 44k INFO ====> Epoch: 1112, cost 58.63 s 2023-09-17 06:28:53,686 44k INFO Train Epoch: 1113 [47%] 2023-09-17 06:28:53,687 44k INFO Losses: [2.2061264514923096, 2.558946371078491, 13.204610824584961, 22.622690200805664, 0.8093796372413635], step: 35600, lr: 4.351102338774957e-05, reference_loss: 41.401756286621094 2023-09-17 06:29:23,313 44k INFO ====> Epoch: 1113, cost 59.52 s 2023-09-17 06:30:22,005 44k INFO ====> Epoch: 1114, cost 58.69 s 2023-09-17 06:31:20,200 44k INFO ====> Epoch: 1115, cost 58.20 s 2023-09-17 06:32:18,997 44k INFO ====> Epoch: 1116, cost 58.80 s 2023-09-17 06:33:17,180 44k INFO ====> Epoch: 1117, cost 58.18 s 2023-09-17 06:34:15,886 44k INFO ====> Epoch: 1118, cost 58.71 s 2023-09-17 06:35:01,315 44k INFO Train Epoch: 1119 [72%] 2023-09-17 06:35:01,316 44k INFO Losses: [2.2567968368530273, 2.5584635734558105, 11.97916316986084, 21.876436233520508, 0.8667891621589661], step: 35800, lr: 4.347840031640537e-05, reference_loss: 39.53765106201172 2023-09-17 06:35:15,874 44k INFO ====> Epoch: 1119, cost 59.99 s 2023-09-17 06:36:14,435 44k INFO ====> Epoch: 1120, cost 58.56 s 2023-09-17 06:37:13,287 44k INFO ====> Epoch: 1121, cost 58.85 s 2023-09-17 06:38:11,573 44k INFO ====> Epoch: 1122, cost 58.29 s 2023-09-17 06:39:09,962 44k INFO ====> Epoch: 1123, cost 58.39 s 2023-09-17 06:40:08,611 44k INFO ====> Epoch: 1124, cost 58.65 s 2023-09-17 06:41:07,139 44k INFO Train Epoch: 1125 [97%] 2023-09-17 06:41:07,141 44k INFO Losses: [1.8058459758758545, 2.99287748336792, 12.794397354125977, 17.16851043701172, -0.2802201509475708], step: 36000, lr: 4.344580170471992e-05, reference_loss: 34.48141098022461 2023-09-17 06:41:23,355 44k INFO Saving model and optimizer state at iteration 1125 to ./logs/44k/G_36000.pth 2023-09-17 06:41:26,733 44k INFO Saving model and optimizer state at iteration 1125 to ./logs/44k/D_36000.pth 2023-09-17 06:41:27,415 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_32000.pth 2023-09-17 06:41:27,417 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_32000.pth 2023-09-17 06:41:27,417 44k INFO ====> Epoch: 1125, cost 78.81 s 2023-09-17 06:42:26,530 44k INFO ====> Epoch: 1126, cost 59.11 s 2023-09-17 06:43:25,838 44k INFO ====> Epoch: 1127, cost 59.31 s 2023-09-17 06:44:25,251 44k INFO ====> Epoch: 1128, cost 59.41 s 2023-09-17 06:45:24,823 44k INFO ====> Epoch: 1129, cost 59.57 s 2023-09-17 06:46:24,328 44k INFO ====> Epoch: 1130, cost 59.51 s 2023-09-17 06:47:23,642 44k INFO ====> Epoch: 1131, cost 59.31 s 2023-09-17 06:47:38,661 44k INFO Train Epoch: 1132 [22%] 2023-09-17 06:47:38,662 44k INFO Losses: [2.251323938369751, 2.7131850719451904, 13.679019927978516, 22.58336067199707, 0.8729291558265686], step: 36200, lr: 4.340780088091241e-05, reference_loss: 42.09981918334961 2023-09-17 06:48:23,293 44k INFO ====> Epoch: 1132, cost 59.65 s 2023-09-17 06:49:22,800 44k INFO ====> Epoch: 1133, cost 59.51 s 2023-09-17 06:50:22,073 44k INFO ====> Epoch: 1134, cost 59.27 s 2023-09-17 06:51:21,834 44k INFO ====> Epoch: 1135, cost 59.76 s 2023-09-17 06:52:20,700 44k INFO ====> Epoch: 1136, cost 58.87 s 2023-09-17 06:53:20,480 44k INFO ====> Epoch: 1137, cost 59.78 s 2023-09-17 06:53:50,530 44k INFO Train Epoch: 1138 [47%] 2023-09-17 06:53:50,531 44k INFO Losses: [2.219158172607422, 2.685431718826294, 13.80644416809082, 22.58989143371582, 0.8166085481643677], step: 36400, lr: 4.3375255202259584e-05, reference_loss: 42.11753463745117 2023-09-17 06:54:20,248 44k INFO ====> Epoch: 1138, cost 59.77 s 2023-09-17 06:55:19,697 44k INFO ====> Epoch: 1139, cost 59.45 s 2023-09-17 06:56:19,561 44k INFO ====> Epoch: 1140, cost 59.86 s 2023-09-17 06:57:18,402 44k INFO ====> Epoch: 1141, cost 58.84 s 2023-09-17 06:58:18,012 44k INFO ====> Epoch: 1142, cost 59.61 s 2023-09-17 06:59:17,205 44k INFO ====> Epoch: 1143, cost 59.19 s 2023-09-17 07:00:03,472 44k INFO Train Epoch: 1144 [72%] 2023-09-17 07:00:03,473 44k INFO Losses: [2.220240354537964, 2.5504910945892334, 12.593310356140137, 21.853466033935547, 0.8830547332763672], step: 36600, lr: 4.334273392523914e-05, reference_loss: 40.100563049316406 2023-09-17 07:00:17,823 44k INFO ====> Epoch: 1144, cost 60.62 s 2023-09-17 07:01:17,503 44k INFO ====> Epoch: 1145, cost 59.68 s 2023-09-17 07:02:16,567 44k INFO ====> Epoch: 1146, cost 59.06 s 2023-09-17 07:03:15,716 44k INFO ====> Epoch: 1147, cost 59.15 s 2023-09-17 07:04:14,316 44k INFO ====> Epoch: 1148, cost 58.60 s 2023-09-17 07:05:13,740 44k INFO ====> Epoch: 1149, cost 59.42 s 2023-09-17 07:06:12,466 44k INFO Train Epoch: 1150 [97%] 2023-09-17 07:06:12,467 44k INFO Losses: [1.671386957168579, 2.933117389678955, 10.322138786315918, 17.582027435302734, -0.2838082015514374], step: 36800, lr: 4.331023703155555e-05, reference_loss: 32.22486114501953 2023-09-17 07:06:29,685 44k INFO Saving model and optimizer state at iteration 1150 to ./logs/44k/G_36800.pth 2023-09-17 07:06:32,403 44k INFO Saving model and optimizer state at iteration 1150 to ./logs/44k/D_36800.pth 2023-09-17 07:06:32,938 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_32800.pth 2023-09-17 07:06:32,940 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_32800.pth 2023-09-17 07:06:32,940 44k INFO ====> Epoch: 1150, cost 79.20 s 2023-09-17 07:07:30,888 44k INFO ====> Epoch: 1151, cost 57.95 s 2023-09-17 07:08:29,187 44k INFO ====> Epoch: 1152, cost 58.30 s 2023-09-17 07:09:27,720 44k INFO ====> Epoch: 1153, cost 58.53 s 2023-09-17 07:10:26,376 44k INFO ====> Epoch: 1154, cost 58.66 s 2023-09-17 07:11:24,953 44k INFO ====> Epoch: 1155, cost 58.58 s 2023-09-17 07:12:23,779 44k INFO ====> Epoch: 1156, cost 58.83 s 2023-09-17 07:12:38,853 44k INFO Train Epoch: 1157 [22%] 2023-09-17 07:12:38,854 44k INFO Losses: [2.2610526084899902, 2.685152769088745, 13.737847328186035, 22.94330406188965, 0.8831143975257874], step: 37000, lr: 4.3272354782364154e-05, reference_loss: 42.51047134399414 2023-09-17 07:13:23,524 44k INFO ====> Epoch: 1157, cost 59.75 s 2023-09-17 07:14:22,391 44k INFO ====> Epoch: 1158, cost 58.87 s 2023-09-17 07:15:20,897 44k INFO ====> Epoch: 1159, cost 58.51 s 2023-09-17 07:16:19,120 44k INFO ====> Epoch: 1160, cost 58.22 s 2023-09-17 07:17:17,513 44k INFO ====> Epoch: 1161, cost 58.39 s 2023-09-17 07:18:15,775 44k INFO ====> Epoch: 1162, cost 58.26 s 2023-09-17 07:18:45,745 44k INFO Train Epoch: 1163 [47%] 2023-09-17 07:18:45,746 44k INFO Losses: [2.1793928146362305, 2.5970261096954346, 13.373133659362793, 22.56240463256836, 0.8234155774116516], step: 37200, lr: 4.3239910656545354e-05, reference_loss: 41.535369873046875 2023-09-17 07:19:15,204 44k INFO ====> Epoch: 1163, cost 59.43 s 2023-09-17 07:20:14,272 44k INFO ====> Epoch: 1164, cost 59.07 s 2023-09-17 07:21:12,745 44k INFO ====> Epoch: 1165, cost 58.47 s 2023-09-17 07:22:11,308 44k INFO ====> Epoch: 1166, cost 58.56 s 2023-09-17 07:23:09,720 44k INFO ====> Epoch: 1167, cost 58.41 s 2023-09-17 07:24:07,925 44k INFO ====> Epoch: 1168, cost 58.20 s 2023-09-17 07:24:53,102 44k INFO Train Epoch: 1169 [72%] 2023-09-17 07:24:53,103 44k INFO Losses: [2.1874208450317383, 2.5323147773742676, 13.312544822692871, 22.07503318786621, 0.8650914430618286], step: 37400, lr: 4.320749085621809e-05, reference_loss: 40.972408294677734 2023-09-17 07:25:07,581 44k INFO ====> Epoch: 1169, cost 59.66 s 2023-09-17 07:26:06,622 44k INFO ====> Epoch: 1170, cost 59.04 s 2023-09-17 07:27:05,463 44k INFO ====> Epoch: 1171, cost 58.84 s 2023-09-17 07:28:03,758 44k INFO ====> Epoch: 1172, cost 58.29 s 2023-09-17 07:29:02,706 44k INFO ====> Epoch: 1173, cost 58.95 s 2023-09-17 07:30:02,139 44k INFO ====> Epoch: 1174, cost 59.43 s 2023-09-17 07:31:00,412 44k INFO Train Epoch: 1175 [97%] 2023-09-17 07:31:00,413 44k INFO Losses: [1.5940070152282715, 2.9533796310424805, 9.01489543914795, 17.120223999023438, -0.2641695737838745], step: 37600, lr: 4.317509536314396e-05, reference_loss: 30.418336868286133 2023-09-17 07:31:17,333 44k INFO Saving model and optimizer state at iteration 1175 to ./logs/44k/G_37600.pth 2023-09-17 07:31:20,653 44k INFO Saving model and optimizer state at iteration 1175 to ./logs/44k/D_37600.pth 2023-09-17 07:31:21,763 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_33600.pth 2023-09-17 07:31:21,764 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_33600.pth 2023-09-17 07:31:21,765 44k INFO ====> Epoch: 1175, cost 79.63 s 2023-09-17 07:32:20,330 44k INFO ====> Epoch: 1176, cost 58.57 s 2023-09-17 07:33:18,932 44k INFO ====> Epoch: 1177, cost 58.60 s 2023-09-17 07:34:17,831 44k INFO ====> Epoch: 1178, cost 58.90 s 2023-09-17 07:35:16,492 44k INFO ====> Epoch: 1179, cost 58.66 s 2023-09-17 07:36:14,376 44k INFO ====> Epoch: 1180, cost 57.88 s 2023-09-17 07:37:12,300 44k INFO ====> Epoch: 1181, cost 57.92 s 2023-09-17 07:37:27,148 44k INFO Train Epoch: 1182 [22%] 2023-09-17 07:37:27,149 44k INFO Losses: [2.2895753383636475, 2.687044858932495, 14.536599159240723, 22.79338836669922, 0.8575903177261353], step: 37800, lr: 4.313733131857832e-05, reference_loss: 43.16419982910156 2023-09-17 07:38:11,191 44k INFO ====> Epoch: 1182, cost 58.89 s 2023-09-17 07:39:09,558 44k INFO ====> Epoch: 1183, cost 58.37 s 2023-09-17 07:40:08,324 44k INFO ====> Epoch: 1184, cost 58.77 s 2023-09-17 07:41:07,487 44k INFO ====> Epoch: 1185, cost 59.16 s 2023-09-17 07:42:05,813 44k INFO ====> Epoch: 1186, cost 58.33 s 2023-09-17 07:43:04,160 44k INFO ====> Epoch: 1187, cost 58.35 s 2023-09-17 07:43:33,801 44k INFO Train Epoch: 1188 [47%] 2023-09-17 07:43:33,802 44k INFO Losses: [2.1722002029418945, 2.670271158218384, 13.700238227844238, 22.742517471313477, 0.8202165961265564], step: 38000, lr: 4.3104988428716506e-05, reference_loss: 42.10544204711914 2023-09-17 07:44:03,285 44k INFO ====> Epoch: 1188, cost 59.13 s 2023-09-17 07:45:01,851 44k INFO ====> Epoch: 1189, cost 58.57 s 2023-09-17 07:46:00,653 44k INFO ====> Epoch: 1190, cost 58.80 s 2023-09-17 07:46:59,024 44k INFO ====> Epoch: 1191, cost 58.37 s 2023-09-17 07:47:57,721 44k INFO ====> Epoch: 1192, cost 58.70 s 2023-09-17 07:48:56,330 44k INFO ====> Epoch: 1193, cost 58.61 s 2023-09-17 07:49:41,463 44k INFO Train Epoch: 1194 [72%] 2023-09-17 07:49:41,464 44k INFO Losses: [2.2820498943328857, 2.48564076423645, 13.13198184967041, 21.933786392211914, 0.8829327821731567], step: 38200, lr: 4.307266978844299e-05, reference_loss: 40.716392517089844 2023-09-17 07:49:55,817 44k INFO ====> Epoch: 1194, cost 59.49 s 2023-09-17 07:50:54,415 44k INFO ====> Epoch: 1195, cost 58.60 s 2023-09-17 07:51:52,265 44k INFO ====> Epoch: 1196, cost 57.85 s 2023-09-17 07:52:51,033 44k INFO ====> Epoch: 1197, cost 58.77 s 2023-09-17 07:53:49,695 44k INFO ====> Epoch: 1198, cost 58.66 s 2023-09-17 07:54:48,440 44k INFO ====> Epoch: 1199, cost 58.74 s 2023-09-17 07:55:47,328 44k INFO Train Epoch: 1200 [97%] 2023-09-17 07:55:47,329 44k INFO Losses: [1.6518254280090332, 3.0420918464660645, 14.158689498901367, 17.600919723510742, -0.3206920921802521], step: 38400, lr: 4.3040375379576254e-05, reference_loss: 36.132835388183594 2023-09-17 07:56:04,294 44k INFO Saving model and optimizer state at iteration 1200 to ./logs/44k/G_38400.pth 2023-09-17 07:56:07,777 44k INFO Saving model and optimizer state at iteration 1200 to ./logs/44k/D_38400.pth 2023-09-17 07:56:08,456 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_34400.pth 2023-09-17 07:56:08,457 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_34400.pth 2023-09-17 07:56:08,457 44k INFO ====> Epoch: 1200, cost 80.02 s 2023-09-17 07:57:06,508 44k INFO ====> Epoch: 1201, cost 58.05 s 2023-09-17 07:58:04,920 44k INFO ====> Epoch: 1202, cost 58.41 s 2023-09-17 07:59:03,236 44k INFO ====> Epoch: 1203, cost 58.32 s 2023-09-17 08:00:01,853 44k INFO ====> Epoch: 1204, cost 58.62 s 2023-09-17 08:01:00,645 44k INFO ====> Epoch: 1205, cost 58.79 s 2023-09-17 08:01:58,929 44k INFO ====> Epoch: 1206, cost 58.28 s 2023-09-17 08:02:13,844 44k INFO Train Epoch: 1207 [22%] 2023-09-17 08:02:13,845 44k INFO Losses: [2.285097122192383, 2.819688320159912, 13.86864185333252, 22.748077392578125, 0.873090386390686], step: 38600, lr: 4.300272917080043e-05, reference_loss: 42.5945930480957 2023-09-17 08:02:58,484 44k INFO ====> Epoch: 1207, cost 59.56 s 2023-09-17 08:03:56,838 44k INFO ====> Epoch: 1208, cost 58.35 s 2023-09-17 08:04:55,212 44k INFO ====> Epoch: 1209, cost 58.37 s 2023-09-17 08:05:53,721 44k INFO ====> Epoch: 1210, cost 58.51 s 2023-09-17 08:06:52,054 44k INFO ====> Epoch: 1211, cost 58.33 s 2023-09-17 08:07:50,334 44k INFO ====> Epoch: 1212, cost 58.28 s 2023-09-17 08:08:19,824 44k INFO Train Epoch: 1213 [47%] 2023-09-17 08:08:19,825 44k INFO Losses: [2.1676571369171143, 2.68210768699646, 14.137528419494629, 22.893495559692383, 0.8436429500579834], step: 38800, lr: 4.297048720100734e-05, reference_loss: 42.72443389892578 2023-09-17 08:08:48,740 44k INFO ====> Epoch: 1213, cost 58.41 s 2023-09-17 08:09:47,243 44k INFO ====> Epoch: 1214, cost 58.50 s 2023-09-17 08:10:46,267 44k INFO ====> Epoch: 1215, cost 59.02 s 2023-09-17 08:11:44,858 44k INFO ====> Epoch: 1216, cost 58.59 s 2023-09-17 08:12:43,243 44k INFO ====> Epoch: 1217, cost 58.39 s 2023-09-17 08:13:41,743 44k INFO ====> Epoch: 1218, cost 58.50 s 2023-09-17 08:14:27,147 44k INFO Train Epoch: 1219 [72%] 2023-09-17 08:14:27,148 44k INFO Losses: [2.19512939453125, 2.5321974754333496, 13.198273658752441, 21.975486755371094, 0.8791207075119019], step: 39000, lr: 4.2938269405136134e-05, reference_loss: 40.78020477294922 2023-09-17 08:14:41,320 44k INFO ====> Epoch: 1219, cost 59.58 s 2023-09-17 08:15:40,302 44k INFO ====> Epoch: 1220, cost 58.98 s 2023-09-17 08:16:39,373 44k INFO ====> Epoch: 1221, cost 59.07 s 2023-09-17 08:17:37,575 44k INFO ====> Epoch: 1222, cost 58.20 s 2023-09-17 08:18:36,213 44k INFO ====> Epoch: 1223, cost 58.64 s 2023-09-17 08:19:35,238 44k INFO ====> Epoch: 1224, cost 59.03 s 2023-09-17 08:20:34,038 44k INFO Train Epoch: 1225 [97%] 2023-09-17 08:20:34,039 44k INFO Losses: [2.0671615600585938, 2.570662021636963, 10.418466567993164, 16.66939353942871, -0.18292909860610962], step: 39200, lr: 4.290607576506204e-05, reference_loss: 31.542753219604492 2023-09-17 08:20:49,324 44k INFO Saving model and optimizer state at iteration 1225 to ./logs/44k/G_39200.pth 2023-09-17 08:20:52,344 44k INFO Saving model and optimizer state at iteration 1225 to ./logs/44k/D_39200.pth 2023-09-17 08:20:53,347 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_35200.pth 2023-09-17 08:20:53,349 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_35200.pth 2023-09-17 08:20:53,370 44k INFO ====> Epoch: 1225, cost 78.13 s 2023-09-17 08:21:52,088 44k INFO ====> Epoch: 1226, cost 58.72 s 2023-09-17 08:22:50,713 44k INFO ====> Epoch: 1227, cost 58.63 s 2023-09-17 08:23:49,216 44k INFO ====> Epoch: 1228, cost 58.50 s 2023-09-17 08:24:47,587 44k INFO ====> Epoch: 1229, cost 58.37 s 2023-09-17 08:25:46,586 44k INFO ====> Epoch: 1230, cost 59.00 s 2023-09-17 08:26:45,102 44k INFO ====> Epoch: 1231, cost 58.52 s 2023-09-17 08:27:00,063 44k INFO Train Epoch: 1232 [22%] 2023-09-17 08:27:00,064 44k INFO Losses: [2.2159435749053955, 2.6127545833587646, 13.79740047454834, 22.597742080688477, 0.8673611283302307], step: 39400, lr: 4.286854702439104e-05, reference_loss: 42.09120178222656 2023-09-17 08:27:44,214 44k INFO ====> Epoch: 1232, cost 59.11 s 2023-09-17 08:28:42,700 44k INFO ====> Epoch: 1233, cost 58.49 s 2023-09-17 08:29:40,963 44k INFO ====> Epoch: 1234, cost 58.26 s 2023-09-17 08:30:39,604 44k INFO ====> Epoch: 1235, cost 58.64 s 2023-09-17 08:31:38,542 44k INFO ====> Epoch: 1236, cost 58.94 s 2023-09-17 08:32:37,452 44k INFO ====> Epoch: 1237, cost 58.91 s 2023-09-17 08:33:07,389 44k INFO Train Epoch: 1238 [47%] 2023-09-17 08:33:07,390 44k INFO Losses: [2.1151766777038574, 2.7219083309173584, 14.540569305419922, 22.64134407043457, 0.8326835036277771], step: 39600, lr: 4.283640565976405e-05, reference_loss: 42.851680755615234 2023-09-17 08:33:36,390 44k INFO ====> Epoch: 1238, cost 58.94 s 2023-09-17 08:34:34,771 44k INFO ====> Epoch: 1239, cost 58.38 s 2023-09-17 08:35:33,488 44k INFO ====> Epoch: 1240, cost 58.72 s 2023-09-17 08:36:32,172 44k INFO ====> Epoch: 1241, cost 58.68 s 2023-09-17 08:37:30,786 44k INFO ====> Epoch: 1242, cost 58.61 s 2023-09-17 08:38:29,785 44k INFO ====> Epoch: 1243, cost 59.00 s 2023-09-17 08:39:14,562 44k INFO Train Epoch: 1244 [72%] 2023-09-17 08:39:14,563 44k INFO Losses: [2.1992247104644775, 2.592285633087158, 13.411892890930176, 21.86207389831543, 0.8652067184448242], step: 39800, lr: 4.2804288393628645e-05, reference_loss: 40.93068313598633 2023-09-17 08:39:28,699 44k INFO ====> Epoch: 1244, cost 58.91 s 2023-09-17 08:40:27,153 44k INFO ====> Epoch: 1245, cost 58.45 s 2023-09-17 08:41:25,789 44k INFO ====> Epoch: 1246, cost 58.64 s 2023-09-17 08:42:24,437 44k INFO ====> Epoch: 1247, cost 58.65 s 2023-09-17 08:43:22,772 44k INFO ====> Epoch: 1248, cost 58.33 s 2023-09-17 08:44:21,407 44k INFO ====> Epoch: 1249, cost 58.64 s 2023-09-17 08:45:19,782 44k INFO Train Epoch: 1250 [97%] 2023-09-17 08:45:19,783 44k INFO Losses: [1.4408928155899048, 3.146688222885132, 11.798787117004395, 17.24960708618164, -0.2955852150917053], step: 40000, lr: 4.2772195207916614e-05, reference_loss: 33.340389251708984 2023-09-17 08:45:36,360 44k INFO Saving model and optimizer state at iteration 1250 to ./logs/44k/G_40000.pth 2023-09-17 08:45:39,789 44k INFO Saving model and optimizer state at iteration 1250 to ./logs/44k/D_40000.pth 2023-09-17 08:45:40,464 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_36000.pth 2023-09-17 08:45:40,465 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_36000.pth 2023-09-17 08:45:40,465 44k INFO ====> Epoch: 1250, cost 79.06 s 2023-09-17 08:46:38,732 44k INFO ====> Epoch: 1251, cost 58.27 s 2023-09-17 08:47:37,341 44k INFO ====> Epoch: 1252, cost 58.61 s 2023-09-17 08:48:35,491 44k INFO ====> Epoch: 1253, cost 58.15 s 2023-09-17 08:49:34,245 44k INFO ====> Epoch: 1254, cost 58.75 s 2023-09-17 08:50:32,387 44k INFO ====> Epoch: 1255, cost 58.14 s 2023-09-17 08:51:30,692 44k INFO ====> Epoch: 1256, cost 58.30 s 2023-09-17 08:51:45,539 44k INFO Train Epoch: 1257 [22%] 2023-09-17 08:51:45,540 44k INFO Losses: [2.2176105976104736, 2.642571449279785, 13.593343734741211, 22.49705696105957, 0.9262765049934387], step: 40200, lr: 4.2734783568812714e-05, reference_loss: 41.876861572265625 2023-09-17 08:52:30,048 44k INFO ====> Epoch: 1257, cost 59.36 s 2023-09-17 08:53:28,522 44k INFO ====> Epoch: 1258, cost 58.47 s 2023-09-17 08:54:27,170 44k INFO ====> Epoch: 1259, cost 58.65 s 2023-09-17 08:55:25,644 44k INFO ====> Epoch: 1260, cost 58.47 s 2023-09-17 08:56:24,531 44k INFO ====> Epoch: 1261, cost 58.89 s 2023-09-17 08:57:23,085 44k INFO ====> Epoch: 1262, cost 58.55 s 2023-09-17 08:57:52,985 44k INFO Train Epoch: 1263 [47%] 2023-09-17 08:57:52,986 44k INFO Losses: [2.234438419342041, 2.7131948471069336, 13.035626411437988, 22.57465171813965, 0.7986294627189636], step: 40400, lr: 4.2702742495431816e-05, reference_loss: 41.35654067993164 2023-09-17 08:58:22,411 44k INFO ====> Epoch: 1263, cost 59.33 s 2023-09-17 08:59:21,327 44k INFO ====> Epoch: 1264, cost 58.92 s 2023-09-17 09:00:20,047 44k INFO ====> Epoch: 1265, cost 58.72 s 2023-09-17 09:01:18,626 44k INFO ====> Epoch: 1266, cost 58.58 s 2023-09-17 09:02:17,108 44k INFO ====> Epoch: 1267, cost 58.48 s 2023-09-17 09:03:15,770 44k INFO ====> Epoch: 1268, cost 58.66 s 2023-09-17 09:04:00,964 44k INFO Train Epoch: 1269 [72%] 2023-09-17 09:04:00,965 44k INFO Losses: [2.273592233657837, 2.519523859024048, 12.722355842590332, 21.649011611938477, 0.8562586903572083], step: 40600, lr: 4.267072544534758e-05, reference_loss: 40.0207405090332 2023-09-17 09:04:15,282 44k INFO ====> Epoch: 1269, cost 59.51 s 2023-09-17 09:05:13,939 44k INFO ====> Epoch: 1270, cost 58.66 s 2023-09-17 09:06:12,340 44k INFO ====> Epoch: 1271, cost 58.40 s 2023-09-17 09:07:10,359 44k INFO ====> Epoch: 1272, cost 58.02 s 2023-09-17 09:08:09,091 44k INFO ====> Epoch: 1273, cost 58.73 s 2023-09-17 09:09:07,862 44k INFO ====> Epoch: 1274, cost 58.77 s 2023-09-17 09:10:06,920 44k INFO Train Epoch: 1275 [97%] 2023-09-17 09:10:06,921 44k INFO Losses: [1.6980161666870117, 2.7821238040924072, 9.206704139709473, 16.277137756347656, -0.22030219435691833], step: 40800, lr: 4.263873240054817e-05, reference_loss: 29.74367904663086 2023-09-17 09:10:24,352 44k INFO Saving model and optimizer state at iteration 1275 to ./logs/44k/G_40800.pth 2023-09-17 09:10:27,694 44k INFO Saving model and optimizer state at iteration 1275 to ./logs/44k/D_40800.pth 2023-09-17 09:10:28,191 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_36800.pth 2023-09-17 09:10:28,193 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_36800.pth 2023-09-17 09:10:28,193 44k INFO ====> Epoch: 1275, cost 80.33 s 2023-09-17 09:11:26,582 44k INFO ====> Epoch: 1276, cost 58.39 s 2023-09-17 09:12:24,677 44k INFO ====> Epoch: 1277, cost 58.10 s 2023-09-17 09:13:23,198 44k INFO ====> Epoch: 1278, cost 58.52 s 2023-09-17 09:14:21,977 44k INFO ====> Epoch: 1279, cost 58.78 s 2023-09-17 09:15:20,778 44k INFO ====> Epoch: 1280, cost 58.80 s 2023-09-17 09:16:19,356 44k INFO ====> Epoch: 1281, cost 58.58 s 2023-09-17 09:16:34,038 44k INFO Train Epoch: 1282 [22%] 2023-09-17 09:16:34,038 44k INFO Losses: [2.2505764961242676, 2.72989559173584, 14.014870643615723, 22.515949249267578, 0.885944664478302], step: 41000, lr: 4.260143749761735e-05, reference_loss: 42.39723587036133 2023-09-17 09:17:18,121 44k INFO ====> Epoch: 1282, cost 58.77 s 2023-09-17 09:18:16,336 44k INFO ====> Epoch: 1283, cost 58.21 s 2023-09-17 09:19:14,671 44k INFO ====> Epoch: 1284, cost 58.33 s 2023-09-17 09:20:13,597 44k INFO ====> Epoch: 1285, cost 58.93 s 2023-09-17 09:21:12,039 44k INFO ====> Epoch: 1286, cost 58.44 s 2023-09-17 09:22:10,731 44k INFO ====> Epoch: 1287, cost 58.69 s 2023-09-17 09:22:40,815 44k INFO Train Epoch: 1288 [47%] 2023-09-17 09:22:40,816 44k INFO Losses: [2.153798818588257, 2.7106540203094482, 12.740349769592285, 22.14080047607422, 0.7920676469802856], step: 41200, lr: 4.2569496402542076e-05, reference_loss: 40.53767395019531 2023-09-17 09:23:10,521 44k INFO ====> Epoch: 1288, cost 59.79 s 2023-09-17 09:24:09,291 44k INFO ====> Epoch: 1289, cost 58.77 s 2023-09-17 09:25:07,481 44k INFO ====> Epoch: 1290, cost 58.19 s 2023-09-17 09:26:06,178 44k INFO ====> Epoch: 1291, cost 58.70 s 2023-09-17 09:27:04,927 44k INFO ====> Epoch: 1292, cost 58.75 s 2023-09-17 09:28:03,348 44k INFO ====> Epoch: 1293, cost 58.42 s 2023-09-17 09:28:48,358 44k INFO Train Epoch: 1294 [72%] 2023-09-17 09:28:48,359 44k INFO Losses: [2.226715326309204, 2.5498735904693604, 13.953295707702637, 22.03077507019043, 0.8755699992179871], step: 41400, lr: 4.253757925580316e-05, reference_loss: 41.636226654052734 2023-09-17 09:29:02,489 44k INFO ====> Epoch: 1294, cost 59.14 s 2023-09-17 09:30:00,775 44k INFO ====> Epoch: 1295, cost 58.29 s 2023-09-17 09:30:59,323 44k INFO ====> Epoch: 1296, cost 58.55 s 2023-09-17 09:31:57,105 44k INFO ====> Epoch: 1297, cost 57.78 s 2023-09-17 09:32:55,045 44k INFO ====> Epoch: 1298, cost 57.94 s 2023-09-17 09:33:53,808 44k INFO ====> Epoch: 1299, cost 58.76 s 2023-09-17 09:34:52,072 44k INFO Train Epoch: 1300 [97%] 2023-09-17 09:34:52,073 44k INFO Losses: [1.7362120151519775, 2.8403873443603516, 11.198220252990723, 16.033653259277344, -0.30267471075057983], step: 41600, lr: 4.250568603944497e-05, reference_loss: 31.505796432495117 2023-09-17 09:35:08,227 44k INFO Saving model and optimizer state at iteration 1300 to ./logs/44k/G_41600.pth 2023-09-17 09:35:11,529 44k INFO Saving model and optimizer state at iteration 1300 to ./logs/44k/D_41600.pth 2023-09-17 09:35:12,620 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_37600.pth 2023-09-17 09:35:12,621 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_37600.pth 2023-09-17 09:35:12,622 44k INFO ====> Epoch: 1300, cost 78.81 s 2023-09-17 09:36:11,156 44k INFO ====> Epoch: 1301, cost 58.53 s 2023-09-17 09:37:09,226 44k INFO ====> Epoch: 1302, cost 58.07 s 2023-09-17 09:38:08,243 44k INFO ====> Epoch: 1303, cost 59.02 s 2023-09-17 09:39:07,028 44k INFO ====> Epoch: 1304, cost 58.78 s 2023-09-17 09:40:05,515 44k INFO ====> Epoch: 1305, cost 58.49 s 2023-09-17 09:41:03,916 44k INFO ====> Epoch: 1306, cost 58.40 s 2023-09-17 09:41:18,860 44k INFO Train Epoch: 1307 [22%] 2023-09-17 09:41:18,861 44k INFO Losses: [2.256754159927368, 2.6370458602905273, 12.958683013916016, 22.482999801635742, 0.8694770336151123], step: 41800, lr: 4.2468507508433384e-05, reference_loss: 41.2049560546875 2023-09-17 09:42:02,874 44k INFO ====> Epoch: 1307, cost 58.96 s 2023-09-17 09:43:01,693 44k INFO ====> Epoch: 1308, cost 58.82 s 2023-09-17 09:44:00,482 44k INFO ====> Epoch: 1309, cost 58.79 s 2023-09-17 09:44:59,297 44k INFO ====> Epoch: 1310, cost 58.82 s 2023-09-17 09:45:57,869 44k INFO ====> Epoch: 1311, cost 58.57 s 2023-09-17 09:46:55,967 44k INFO ====> Epoch: 1312, cost 58.10 s 2023-09-17 09:47:25,740 44k INFO Train Epoch: 1313 [47%] 2023-09-17 09:47:25,741 44k INFO Losses: [2.1225485801696777, 2.5908823013305664, 13.086212158203125, 22.303098678588867, 0.8039345145225525], step: 42000, lr: 4.243666607969972e-05, reference_loss: 40.90667724609375 2023-09-17 09:47:55,134 44k INFO ====> Epoch: 1313, cost 59.17 s 2023-09-17 09:48:53,453 44k INFO ====> Epoch: 1314, cost 58.32 s 2023-09-17 09:49:51,683 44k INFO ====> Epoch: 1315, cost 58.23 s 2023-09-17 09:50:50,168 44k INFO ====> Epoch: 1316, cost 58.49 s 2023-09-17 09:51:48,699 44k INFO ====> Epoch: 1317, cost 58.53 s 2023-09-17 09:52:47,395 44k INFO ====> Epoch: 1318, cost 58.70 s 2023-09-17 09:53:32,731 44k INFO Train Epoch: 1319 [72%] 2023-09-17 09:53:32,732 44k INFO Losses: [2.1819446086883545, 2.6002492904663086, 12.662819862365723, 21.59967613220215, 0.8656669855117798], step: 42200, lr: 4.240484852457602e-05, reference_loss: 39.91035842895508 2023-09-17 09:53:46,863 44k INFO ====> Epoch: 1319, cost 59.47 s 2023-09-17 09:54:45,374 44k INFO ====> Epoch: 1320, cost 58.51 s 2023-09-17 09:55:44,064 44k INFO ====> Epoch: 1321, cost 58.69 s 2023-09-17 09:56:42,547 44k INFO ====> Epoch: 1322, cost 58.48 s 2023-09-17 09:57:41,364 44k INFO ====> Epoch: 1323, cost 58.82 s 2023-09-17 09:58:39,979 44k INFO ====> Epoch: 1324, cost 58.61 s 2023-09-17 09:59:38,263 44k INFO Train Epoch: 1325 [97%] 2023-09-17 09:59:38,264 44k INFO Losses: [1.6028310060501099, 3.0287375450134277, 8.673271179199219, 16.665639877319336, -0.3191334903240204], step: 42400, lr: 4.2373054825162655e-05, reference_loss: 29.65134620666504 2023-09-17 09:59:55,033 44k INFO Saving model and optimizer state at iteration 1325 to ./logs/44k/G_42400.pth 2023-09-17 09:59:58,480 44k INFO Saving model and optimizer state at iteration 1325 to ./logs/44k/D_42400.pth 2023-09-17 09:59:58,988 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_38400.pth 2023-09-17 09:59:58,989 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_38400.pth 2023-09-17 09:59:58,989 44k INFO ====> Epoch: 1325, cost 79.01 s 2023-09-17 10:00:57,849 44k INFO ====> Epoch: 1326, cost 58.86 s 2023-09-17 10:01:56,677 44k INFO ====> Epoch: 1327, cost 58.83 s 2023-09-17 10:02:55,428 44k INFO ====> Epoch: 1328, cost 58.75 s 2023-09-17 10:03:53,743 44k INFO ====> Epoch: 1329, cost 58.31 s 2023-09-17 10:04:52,709 44k INFO ====> Epoch: 1330, cost 58.97 s 2023-09-17 10:05:51,429 44k INFO ====> Epoch: 1331, cost 58.72 s 2023-09-17 10:06:06,350 44k INFO Train Epoch: 1332 [22%] 2023-09-17 10:06:06,351 44k INFO Losses: [2.2925941944122314, 2.6851694583892822, 13.47168254852295, 22.662199020385742, 0.855513870716095], step: 42600, lr: 4.233599230295301e-05, reference_loss: 41.967159271240234 2023-09-17 10:06:50,991 44k INFO ====> Epoch: 1332, cost 59.56 s 2023-09-17 10:07:50,099 44k INFO ====> Epoch: 1333, cost 59.11 s 2023-09-17 10:08:48,504 44k INFO ====> Epoch: 1334, cost 58.40 s 2023-09-17 10:09:47,134 44k INFO ====> Epoch: 1335, cost 58.63 s 2023-09-17 10:10:45,627 44k INFO ====> Epoch: 1336, cost 58.49 s 2023-09-17 10:11:43,944 44k INFO ====> Epoch: 1337, cost 58.32 s 2023-09-17 10:12:13,816 44k INFO Train Epoch: 1338 [47%] 2023-09-17 10:12:13,817 44k INFO Losses: [2.154574394226074, 2.603640079498291, 13.922074317932129, 22.604583740234375, 0.8129111528396606], step: 42800, lr: 4.230425022957039e-05, reference_loss: 42.09778594970703 2023-09-17 10:12:43,075 44k INFO ====> Epoch: 1338, cost 59.13 s 2023-09-17 10:13:41,877 44k INFO ====> Epoch: 1339, cost 58.80 s 2023-09-17 10:14:40,867 44k INFO ====> Epoch: 1340, cost 58.99 s 2023-09-17 10:15:39,442 44k INFO ====> Epoch: 1341, cost 58.58 s 2023-09-17 10:16:38,019 44k INFO ====> Epoch: 1342, cost 58.58 s 2023-09-17 10:17:36,644 44k INFO ====> Epoch: 1343, cost 58.62 s 2023-09-17 10:18:21,989 44k INFO Train Epoch: 1344 [72%] 2023-09-17 10:18:21,991 44k INFO Losses: [2.2338836193084717, 2.5254292488098145, 13.893916130065918, 21.840473175048828, 0.8704183101654053], step: 43000, lr: 4.227253195530449e-05, reference_loss: 41.36412048339844 2023-09-17 10:18:36,063 44k INFO ====> Epoch: 1344, cost 59.42 s 2023-09-17 10:19:34,953 44k INFO ====> Epoch: 1345, cost 58.89 s 2023-09-17 10:20:33,721 44k INFO ====> Epoch: 1346, cost 58.77 s 2023-09-17 10:21:33,153 44k INFO ====> Epoch: 1347, cost 59.43 s 2023-09-17 10:22:31,615 44k INFO ====> Epoch: 1348, cost 58.46 s 2023-09-17 10:23:30,182 44k INFO ====> Epoch: 1349, cost 58.57 s 2023-09-17 10:24:28,623 44k INFO Train Epoch: 1350 [97%] 2023-09-17 10:24:28,624 44k INFO Losses: [1.4597262144088745, 3.2393879890441895, 14.393470764160156, 17.094877243041992, -0.3696559965610504], step: 43200, lr: 4.2240837462311564e-05, reference_loss: 35.817806243896484 2023-09-17 10:24:45,602 44k INFO Saving model and optimizer state at iteration 1350 to ./logs/44k/G_43200.pth 2023-09-17 10:24:48,802 44k INFO Saving model and optimizer state at iteration 1350 to ./logs/44k/D_43200.pth 2023-09-17 10:24:49,773 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_39200.pth 2023-09-17 10:24:49,785 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_39200.pth 2023-09-17 10:24:49,796 44k INFO ====> Epoch: 1350, cost 79.61 s 2023-09-17 10:25:48,324 44k INFO ====> Epoch: 1351, cost 58.53 s 2023-09-17 10:26:46,867 44k INFO ====> Epoch: 1352, cost 58.54 s 2023-09-17 10:27:45,643 44k INFO ====> Epoch: 1353, cost 58.78 s 2023-09-17 10:28:44,092 44k INFO ====> Epoch: 1354, cost 58.45 s 2023-09-17 10:29:42,747 44k INFO ====> Epoch: 1355, cost 58.65 s 2023-09-17 10:30:41,292 44k INFO ====> Epoch: 1356, cost 58.55 s 2023-09-17 10:30:56,068 44k INFO Train Epoch: 1357 [22%] 2023-09-17 10:30:56,069 44k INFO Losses: [2.2127137184143066, 2.762192726135254, 14.736741065979004, 22.633527755737305, 0.8857632875442505], step: 43400, lr: 4.220389058691962e-05, reference_loss: 43.23094177246094 2023-09-17 10:31:40,531 44k INFO ====> Epoch: 1357, cost 59.24 s 2023-09-17 10:32:39,242 44k INFO ====> Epoch: 1358, cost 58.71 s 2023-09-17 10:33:37,402 44k INFO ====> Epoch: 1359, cost 58.16 s 2023-09-17 10:34:35,824 44k INFO ====> Epoch: 1360, cost 58.42 s 2023-09-17 10:35:34,647 44k INFO ====> Epoch: 1361, cost 58.82 s 2023-09-17 10:36:33,490 44k INFO ====> Epoch: 1362, cost 58.84 s 2023-09-17 10:37:03,462 44k INFO Train Epoch: 1363 [47%] 2023-09-17 10:37:03,463 44k INFO Losses: [2.169995069503784, 2.7457096576690674, 13.932250022888184, 22.27685546875, 0.8153136372566223], step: 43600, lr: 4.2172247558867844e-05, reference_loss: 41.94012451171875 2023-09-17 10:37:33,029 44k INFO ====> Epoch: 1363, cost 59.54 s 2023-09-17 10:38:32,340 44k INFO ====> Epoch: 1364, cost 59.31 s 2023-09-17 10:39:30,500 44k INFO ====> Epoch: 1365, cost 58.16 s 2023-09-17 10:40:28,784 44k INFO ====> Epoch: 1366, cost 58.28 s 2023-09-17 10:41:27,333 44k INFO ====> Epoch: 1367, cost 58.55 s 2023-09-17 10:42:26,078 44k INFO ====> Epoch: 1368, cost 58.75 s 2023-09-17 10:43:10,933 44k INFO Train Epoch: 1369 [72%] 2023-09-17 10:43:10,934 44k INFO Losses: [2.186283826828003, 2.6739583015441895, 13.987104415893555, 21.880725860595703, 0.8676244616508484], step: 43800, lr: 4.2140628255671993e-05, reference_loss: 41.595699310302734 2023-09-17 10:43:25,282 44k INFO ====> Epoch: 1369, cost 59.20 s 2023-09-17 10:44:24,210 44k INFO ====> Epoch: 1370, cost 58.93 s 2023-09-17 10:45:22,394 44k INFO ====> Epoch: 1371, cost 58.18 s 2023-09-17 10:46:20,693 44k INFO ====> Epoch: 1372, cost 58.30 s 2023-09-17 10:47:19,187 44k INFO ====> Epoch: 1373, cost 58.49 s 2023-09-17 10:48:17,394 44k INFO ====> Epoch: 1374, cost 58.21 s 2023-09-17 10:49:16,254 44k INFO Train Epoch: 1375 [97%] 2023-09-17 10:49:16,255 44k INFO Losses: [1.8092546463012695, 2.744197368621826, 7.741121768951416, 16.16757583618164, -0.3363402783870697], step: 44000, lr: 4.210903265954401e-05, reference_loss: 28.125810623168945 2023-09-17 10:49:32,985 44k INFO Saving model and optimizer state at iteration 1375 to ./logs/44k/G_44000.pth 2023-09-17 10:49:36,384 44k INFO Saving model and optimizer state at iteration 1375 to ./logs/44k/D_44000.pth 2023-09-17 10:49:37,481 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_40000.pth 2023-09-17 10:49:37,484 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_40000.pth 2023-09-17 10:49:37,485 44k INFO ====> Epoch: 1375, cost 80.09 s 2023-09-17 10:50:36,154 44k INFO ====> Epoch: 1376, cost 58.67 s 2023-09-17 10:51:35,280 44k INFO ====> Epoch: 1377, cost 59.13 s 2023-09-17 10:52:33,491 44k INFO ====> Epoch: 1378, cost 58.21 s 2023-09-17 10:53:32,272 44k INFO ====> Epoch: 1379, cost 58.78 s 2023-09-17 10:54:30,324 44k INFO ====> Epoch: 1380, cost 58.05 s 2023-09-17 10:55:29,066 44k INFO ====> Epoch: 1381, cost 58.74 s 2023-09-17 10:55:44,246 44k INFO Train Epoch: 1382 [22%] 2023-09-17 10:55:44,247 44k INFO Losses: [2.257650852203369, 2.743943929672241, 14.030277252197266, 22.629024505615234, 0.8696538209915161], step: 44200, lr: 4.207220107011505e-05, reference_loss: 42.53055191040039 2023-09-17 10:56:28,289 44k INFO ====> Epoch: 1382, cost 59.22 s 2023-09-17 10:57:27,203 44k INFO ====> Epoch: 1383, cost 58.91 s 2023-09-17 10:58:25,943 44k INFO ====> Epoch: 1384, cost 58.74 s 2023-09-17 10:59:24,904 44k INFO ====> Epoch: 1385, cost 58.96 s 2023-09-17 11:00:23,544 44k INFO ====> Epoch: 1386, cost 58.64 s 2023-09-17 11:01:21,814 44k INFO ====> Epoch: 1387, cost 58.27 s 2023-09-17 11:01:51,599 44k INFO Train Epoch: 1388 [47%] 2023-09-17 11:01:51,601 44k INFO Losses: [2.1068639755249023, 2.737964391708374, 14.700569152832031, 22.590822219848633, 0.8185808658599854], step: 44400, lr: 4.20406567783413e-05, reference_loss: 42.954803466796875 2023-09-17 11:02:21,009 44k INFO ====> Epoch: 1388, cost 59.19 s 2023-09-17 11:03:20,307 44k INFO ====> Epoch: 1389, cost 59.30 s 2023-09-17 11:04:19,035 44k INFO ====> Epoch: 1390, cost 58.73 s 2023-09-17 11:05:17,724 44k INFO ====> Epoch: 1391, cost 58.69 s 2023-09-17 11:06:16,438 44k INFO ====> Epoch: 1392, cost 58.71 s 2023-09-17 11:07:15,641 44k INFO ====> Epoch: 1393, cost 59.20 s 2023-09-17 11:08:00,598 44k INFO Train Epoch: 1394 [72%] 2023-09-17 11:08:00,599 44k INFO Losses: [2.2246756553649902, 2.5469353199005127, 12.45228099822998, 21.841510772705078, 0.8704016208648682], step: 44600, lr: 4.200913613739441e-05, reference_loss: 39.93580627441406 2023-09-17 11:08:14,590 44k INFO ====> Epoch: 1394, cost 58.95 s 2023-09-17 11:09:13,475 44k INFO ====> Epoch: 1395, cost 58.89 s 2023-09-17 11:10:12,334 44k INFO ====> Epoch: 1396, cost 58.86 s 2023-09-17 11:11:11,119 44k INFO ====> Epoch: 1397, cost 58.79 s 2023-09-17 11:12:09,649 44k INFO ====> Epoch: 1398, cost 58.53 s 2023-09-17 11:13:08,625 44k INFO ====> Epoch: 1399, cost 58.98 s 2023-09-17 11:14:07,712 44k INFO Train Epoch: 1400 [97%] 2023-09-17 11:14:07,713 44k INFO Losses: [1.6830875873565674, 2.8545639514923096, 9.247572898864746, 16.371353149414062, -0.41429004073143005], step: 44800, lr: 4.1977639129541807e-05, reference_loss: 29.74228858947754 2023-09-17 11:14:24,343 44k INFO Saving model and optimizer state at iteration 1400 to ./logs/44k/G_44800.pth 2023-09-17 11:14:27,074 44k INFO Saving model and optimizer state at iteration 1400 to ./logs/44k/D_44800.pth 2023-09-17 11:14:27,595 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_40800.pth 2023-09-17 11:14:27,596 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_40800.pth 2023-09-17 11:14:27,596 44k INFO ====> Epoch: 1400, cost 78.97 s 2023-09-17 11:15:26,169 44k INFO ====> Epoch: 1401, cost 58.57 s 2023-09-17 11:16:24,906 44k INFO ====> Epoch: 1402, cost 58.74 s 2023-09-17 11:17:23,096 44k INFO ====> Epoch: 1403, cost 58.19 s 2023-09-17 11:18:21,452 44k INFO ====> Epoch: 1404, cost 58.36 s 2023-09-17 11:19:19,727 44k INFO ====> Epoch: 1405, cost 58.27 s 2023-09-17 11:20:18,398 44k INFO ====> Epoch: 1406, cost 58.67 s 2023-09-17 11:20:33,240 44k INFO Train Epoch: 1407 [22%] 2023-09-17 11:20:33,241 44k INFO Losses: [2.2006964683532715, 2.8911187648773193, 14.252410888671875, 22.54142189025879, 0.8655394315719604], step: 45000, lr: 4.1940922466347085e-05, reference_loss: 42.751190185546875 2023-09-17 11:21:17,216 44k INFO ====> Epoch: 1407, cost 58.82 s 2023-09-17 11:22:16,166 44k INFO ====> Epoch: 1408, cost 58.95 s 2023-09-17 11:23:15,025 44k INFO ====> Epoch: 1409, cost 58.86 s 2023-09-17 11:24:13,406 44k INFO ====> Epoch: 1410, cost 58.38 s 2023-09-17 11:25:12,068 44k INFO ====> Epoch: 1411, cost 58.66 s 2023-09-17 11:26:10,583 44k INFO ====> Epoch: 1412, cost 58.51 s 2023-09-17 11:26:40,481 44k INFO Train Epoch: 1413 [47%] 2023-09-17 11:26:40,482 44k INFO Losses: [2.1972899436950684, 2.5974738597869873, 12.63243293762207, 22.074344635009766, 0.7982392907142639], step: 45200, lr: 4.190947660276286e-05, reference_loss: 40.299781799316406 2023-09-17 11:27:09,775 44k INFO ====> Epoch: 1413, cost 59.19 s 2023-09-17 11:28:08,095 44k INFO ====> Epoch: 1414, cost 58.32 s 2023-09-17 11:29:06,974 44k INFO ====> Epoch: 1415, cost 58.88 s 2023-09-17 11:30:05,385 44k INFO ====> Epoch: 1416, cost 58.41 s 2023-09-17 11:31:03,353 44k INFO ====> Epoch: 1417, cost 57.97 s 2023-09-17 11:32:02,247 44k INFO ====> Epoch: 1418, cost 58.89 s 2023-09-17 11:32:47,514 44k INFO Train Epoch: 1419 [72%] 2023-09-17 11:32:47,515 44k INFO Losses: [2.209775924682617, 2.547107696533203, 12.779711723327637, 21.64844512939453, 0.8501259088516235], step: 45400, lr: 4.187805431620742e-05, reference_loss: 40.03516387939453 2023-09-17 11:33:01,864 44k INFO ====> Epoch: 1419, cost 59.62 s 2023-09-17 11:33:59,856 44k INFO ====> Epoch: 1420, cost 57.99 s 2023-09-17 11:34:58,079 44k INFO ====> Epoch: 1421, cost 58.22 s 2023-09-17 11:35:56,425 44k INFO ====> Epoch: 1422, cost 58.35 s 2023-09-17 11:36:55,144 44k INFO ====> Epoch: 1423, cost 58.72 s 2023-09-17 11:37:53,474 44k INFO ====> Epoch: 1424, cost 58.33 s 2023-09-17 11:38:52,593 44k INFO Train Epoch: 1425 [97%] 2023-09-17 11:38:52,594 44k INFO Losses: [1.8763923645019531, 2.7127182483673096, 9.238999366760254, 16.442094802856445, -0.3476368486881256], step: 45600, lr: 4.184665558900353e-05, reference_loss: 29.92256736755371 2023-09-17 11:39:09,660 44k INFO Saving model and optimizer state at iteration 1425 to ./logs/44k/G_45600.pth 2023-09-17 11:39:12,954 44k INFO Saving model and optimizer state at iteration 1425 to ./logs/44k/D_45600.pth 2023-09-17 11:39:14,046 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_41600.pth 2023-09-17 11:39:14,048 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_41600.pth 2023-09-17 11:39:14,049 44k INFO ====> Epoch: 1425, cost 80.57 s 2023-09-17 11:40:13,160 44k INFO ====> Epoch: 1426, cost 59.11 s 2023-09-17 11:41:11,977 44k INFO ====> Epoch: 1427, cost 58.82 s 2023-09-17 11:42:10,930 44k INFO ====> Epoch: 1428, cost 58.95 s 2023-09-17 11:43:09,429 44k INFO ====> Epoch: 1429, cost 58.50 s 2023-09-17 11:44:08,931 44k INFO ====> Epoch: 1430, cost 59.50 s 2023-09-17 11:45:08,070 44k INFO ====> Epoch: 1431, cost 59.14 s 2023-09-17 11:45:23,110 44k INFO Train Epoch: 1432 [22%] 2023-09-17 11:45:23,111 44k INFO Losses: [2.2410993576049805, 2.6725170612335205, 13.381953239440918, 22.198993682861328, 0.8643754720687866], step: 45800, lr: 4.181005349343675e-05, reference_loss: 41.35894012451172 2023-09-17 11:46:07,827 44k INFO ====> Epoch: 1432, cost 59.76 s 2023-09-17 11:47:06,848 44k INFO ====> Epoch: 1433, cost 59.02 s 2023-09-17 11:48:05,593 44k INFO ====> Epoch: 1434, cost 58.75 s 2023-09-17 11:49:03,935 44k INFO ====> Epoch: 1435, cost 58.34 s 2023-09-17 11:50:02,257 44k INFO ====> Epoch: 1436, cost 58.32 s 2023-09-17 11:51:00,800 44k INFO ====> Epoch: 1437, cost 58.54 s 2023-09-17 11:51:30,562 44k INFO Train Epoch: 1438 [47%] 2023-09-17 11:51:30,563 44k INFO Losses: [2.106593132019043, 2.878804922103882, 15.912808418273926, 22.763084411621094, 0.8204991817474365], step: 46000, lr: 4.17787057509149e-05, reference_loss: 44.48179244995117 2023-09-17 11:51:59,545 44k INFO ====> Epoch: 1438, cost 58.74 s 2023-09-17 11:52:58,399 44k INFO ====> Epoch: 1439, cost 58.85 s 2023-09-17 11:53:57,083 44k INFO ====> Epoch: 1440, cost 58.68 s 2023-09-17 11:54:56,109 44k INFO ====> Epoch: 1441, cost 59.03 s 2023-09-17 11:55:54,498 44k INFO ====> Epoch: 1442, cost 58.39 s 2023-09-17 11:56:53,399 44k INFO ====> Epoch: 1443, cost 58.90 s 2023-09-17 11:57:38,063 44k INFO Train Epoch: 1444 [72%] 2023-09-17 11:57:38,064 44k INFO Losses: [2.1948800086975098, 2.6617610454559326, 13.907034873962402, 21.914831161499023, 0.8740463256835938], step: 46200, lr: 4.1747381511854035e-05, reference_loss: 41.55255126953125 2023-09-17 11:57:52,175 44k INFO ====> Epoch: 1444, cost 58.78 s 2023-09-17 11:58:50,288 44k INFO ====> Epoch: 1445, cost 58.11 s 2023-09-17 11:59:48,681 44k INFO ====> Epoch: 1446, cost 58.39 s 2023-09-17 12:00:47,479 44k INFO ====> Epoch: 1447, cost 58.80 s 2023-09-17 12:01:45,829 44k INFO ====> Epoch: 1448, cost 58.35 s 2023-09-17 12:02:44,496 44k INFO ====> Epoch: 1449, cost 58.67 s 2023-09-17 12:03:43,191 44k INFO Train Epoch: 1450 [97%] 2023-09-17 12:03:43,192 44k INFO Losses: [1.5741413831710815, 3.200981855392456, 13.55008316040039, 16.845043182373047, -0.3855477273464203], step: 46400, lr: 4.171608075863207e-05, reference_loss: 34.78470230102539 2023-09-17 12:03:59,826 44k INFO Saving model and optimizer state at iteration 1450 to ./logs/44k/G_46400.pth 2023-09-17 12:04:03,475 44k INFO Saving model and optimizer state at iteration 1450 to ./logs/44k/D_46400.pth 2023-09-17 12:04:04,009 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_42400.pth 2023-09-17 12:04:04,011 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_42400.pth 2023-09-17 12:04:04,011 44k INFO ====> Epoch: 1450, cost 79.52 s 2023-09-17 12:05:02,698 44k INFO ====> Epoch: 1451, cost 58.69 s 2023-09-17 12:06:01,140 44k INFO ====> Epoch: 1452, cost 58.44 s 2023-09-17 12:06:59,979 44k INFO ====> Epoch: 1453, cost 58.84 s 2023-09-17 12:07:58,996 44k INFO ====> Epoch: 1454, cost 59.02 s 2023-09-17 12:08:57,910 44k INFO ====> Epoch: 1455, cost 58.91 s 2023-09-17 12:09:56,578 44k INFO ====> Epoch: 1456, cost 58.67 s 2023-09-17 12:10:11,394 44k INFO Train Epoch: 1457 [22%] 2023-09-17 12:10:11,395 44k INFO Losses: [2.2632651329040527, 2.66969895362854, 13.390568733215332, 22.37081527709961, 0.8308454751968384], step: 46600, lr: 4.167959287320594e-05, reference_loss: 41.52519226074219 2023-09-17 12:10:55,693 44k INFO ====> Epoch: 1457, cost 59.11 s 2023-09-17 12:11:54,465 44k INFO ====> Epoch: 1458, cost 58.77 s 2023-09-17 12:12:52,773 44k INFO ====> Epoch: 1459, cost 58.31 s 2023-09-17 12:13:50,853 44k INFO ====> Epoch: 1460, cost 58.08 s 2023-09-17 12:14:49,885 44k INFO ====> Epoch: 1461, cost 59.03 s 2023-09-17 12:15:48,269 44k INFO ====> Epoch: 1462, cost 58.38 s 2023-09-17 12:16:17,785 44k INFO Train Epoch: 1463 [47%] 2023-09-17 12:16:17,786 44k INFO Losses: [2.1781110763549805, 2.6700656414031982, 14.070731163024902, 22.30245590209961, 0.7990319728851318], step: 46800, lr: 4.164834294557764e-05, reference_loss: 42.02039337158203 2023-09-17 12:16:47,314 44k INFO ====> Epoch: 1463, cost 59.04 s 2023-09-17 12:17:46,084 44k INFO ====> Epoch: 1464, cost 58.77 s 2023-09-17 12:18:44,574 44k INFO ====> Epoch: 1465, cost 58.49 s 2023-09-17 12:19:42,884 44k INFO ====> Epoch: 1466, cost 58.31 s 2023-09-17 12:20:41,672 44k INFO ====> Epoch: 1467, cost 58.79 s 2023-09-17 12:21:40,656 44k INFO ====> Epoch: 1468, cost 58.98 s 2023-09-17 12:22:25,807 44k INFO Train Epoch: 1469 [72%] 2023-09-17 12:22:25,808 44k INFO Losses: [2.2239902019500732, 2.4683597087860107, 13.118192672729492, 21.656179428100586, 0.8410643339157104], step: 47000, lr: 4.161711644807209e-05, reference_loss: 40.30778503417969 2023-09-17 12:22:40,046 44k INFO ====> Epoch: 1469, cost 59.39 s 2023-09-17 12:23:38,961 44k INFO ====> Epoch: 1470, cost 58.92 s 2023-09-17 12:24:38,058 44k INFO ====> Epoch: 1471, cost 59.10 s 2023-09-17 12:25:36,066 44k INFO ====> Epoch: 1472, cost 58.01 s 2023-09-17 12:26:35,098 44k INFO ====> Epoch: 1473, cost 59.03 s 2023-09-17 12:27:33,855 44k INFO ====> Epoch: 1474, cost 58.76 s 2023-09-17 12:28:32,722 44k INFO Train Epoch: 1475 [97%] 2023-09-17 12:28:32,723 44k INFO Losses: [1.9061756134033203, 2.780177354812622, 9.097102165222168, 16.64024543762207, -0.37720954418182373], step: 47200, lr: 4.158591336312217e-05, reference_loss: 30.046491622924805 2023-09-17 12:28:50,316 44k INFO Saving model and optimizer state at iteration 1475 to ./logs/44k/G_47200.pth 2023-09-17 12:28:53,644 44k INFO Saving model and optimizer state at iteration 1475 to ./logs/44k/D_47200.pth 2023-09-17 12:28:54,770 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_43200.pth 2023-09-17 12:28:54,774 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_43200.pth 2023-09-17 12:28:54,775 44k INFO ====> Epoch: 1475, cost 80.92 s 2023-09-17 12:29:53,807 44k INFO ====> Epoch: 1476, cost 59.03 s 2023-09-17 12:30:52,829 44k INFO ====> Epoch: 1477, cost 59.02 s 2023-09-17 12:31:51,875 44k INFO ====> Epoch: 1478, cost 59.05 s 2023-09-17 12:32:50,140 44k INFO ====> Epoch: 1479, cost 58.26 s 2023-09-17 12:33:48,682 44k INFO ====> Epoch: 1480, cost 58.54 s 2023-09-17 12:34:47,295 44k INFO ====> Epoch: 1481, cost 58.61 s 2023-09-17 12:35:02,055 44k INFO Train Epoch: 1482 [22%] 2023-09-17 12:35:02,056 44k INFO Losses: [2.23311710357666, 2.8028857707977295, 13.419537544250488, 22.349084854125977, 0.9102330207824707], step: 47400, lr: 4.154953933146482e-05, reference_loss: 41.71485900878906 2023-09-17 12:35:46,393 44k INFO ====> Epoch: 1482, cost 59.10 s 2023-09-17 12:36:44,518 44k INFO ====> Epoch: 1483, cost 58.13 s 2023-09-17 12:37:43,073 44k INFO ====> Epoch: 1484, cost 58.55 s 2023-09-17 12:38:42,236 44k INFO ====> Epoch: 1485, cost 59.16 s 2023-09-17 12:39:40,805 44k INFO ====> Epoch: 1486, cost 58.57 s 2023-09-17 12:40:39,676 44k INFO ====> Epoch: 1487, cost 58.87 s 2023-09-17 12:41:09,603 44k INFO Train Epoch: 1488 [47%] 2023-09-17 12:41:09,604 44k INFO Losses: [2.15462589263916, 2.6793057918548584, 13.842597007751465, 22.38109588623047, 0.795589029788971], step: 47600, lr: 4.151838691351661e-05, reference_loss: 41.853214263916016 2023-09-17 12:41:38,400 44k INFO ====> Epoch: 1488, cost 58.72 s 2023-09-17 12:42:36,853 44k INFO ====> Epoch: 1489, cost 58.45 s 2023-09-17 12:43:35,604 44k INFO ====> Epoch: 1490, cost 58.75 s 2023-09-17 12:44:33,890 44k INFO ====> Epoch: 1491, cost 58.29 s 2023-09-17 12:45:32,240 44k INFO ====> Epoch: 1492, cost 58.35 s 2023-09-17 12:46:31,110 44k INFO ====> Epoch: 1493, cost 58.87 s 2023-09-17 12:47:16,586 44k INFO Train Epoch: 1494 [72%] 2023-09-17 12:47:16,587 44k INFO Losses: [2.2012691497802734, 2.5989227294921875, 13.723349571228027, 21.782470703125, 0.8870102167129517], step: 47800, lr: 4.148725785258173e-05, reference_loss: 41.19301986694336 2023-09-17 12:47:31,108 44k INFO ====> Epoch: 1494, cost 60.00 s 2023-09-17 12:48:29,675 44k INFO ====> Epoch: 1495, cost 58.57 s 2023-09-17 12:49:28,486 44k INFO ====> Epoch: 1496, cost 58.81 s 2023-09-17 12:50:26,903 44k INFO ====> Epoch: 1497, cost 58.42 s 2023-09-17 12:51:25,410 44k INFO ====> Epoch: 1498, cost 58.51 s 2023-09-17 12:52:24,194 44k INFO ====> Epoch: 1499, cost 58.78 s 2023-09-17 12:53:22,417 44k INFO Train Epoch: 1500 [97%] 2023-09-17 12:53:22,419 44k INFO Losses: [1.4597249031066895, 3.2801353931427, 14.163935661315918, 17.179479598999023, -0.4489765763282776], step: 48000, lr: 4.14561521311479e-05, reference_loss: 35.63429641723633 2023-09-17 12:53:37,922 44k INFO Saving model and optimizer state at iteration 1500 to ./logs/44k/G_48000.pth 2023-09-17 12:53:41,307 44k INFO Saving model and optimizer state at iteration 1500 to ./logs/44k/D_48000.pth 2023-09-17 12:53:41,834 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_44000.pth 2023-09-17 12:53:41,835 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_44000.pth 2023-09-17 12:53:41,836 44k INFO ====> Epoch: 1500, cost 77.64 s 2023-09-17 12:54:41,426 44k INFO ====> Epoch: 1501, cost 59.59 s 2023-09-17 12:55:41,103 44k INFO ====> Epoch: 1502, cost 59.68 s 2023-09-17 12:56:40,379 44k INFO ====> Epoch: 1503, cost 59.28 s 2023-09-17 12:57:39,769 44k INFO ====> Epoch: 1504, cost 59.39 s 2023-09-17 12:58:39,251 44k INFO ====> Epoch: 1505, cost 59.48 s 2023-09-17 12:59:38,814 44k INFO ====> Epoch: 1506, cost 59.56 s 2023-09-17 12:59:54,036 44k INFO Train Epoch: 1507 [22%] 2023-09-17 12:59:54,037 44k INFO Losses: [2.183328866958618, 2.8119442462921143, 13.377996444702148, 22.237165451049805, 0.9237671494483948], step: 48200, lr: 4.141989159799948e-05, reference_loss: 41.534202575683594 2023-09-17 13:00:38,790 44k INFO ====> Epoch: 1507, cost 59.98 s 2023-09-17 13:01:37,687 44k INFO ====> Epoch: 1508, cost 58.90 s 2023-09-17 13:02:37,288 44k INFO ====> Epoch: 1509, cost 59.60 s 2023-09-17 13:03:36,620 44k INFO ====> Epoch: 1510, cost 59.33 s 2023-09-17 13:04:36,210 44k INFO ====> Epoch: 1511, cost 59.59 s 2023-09-17 13:05:34,964 44k INFO ====> Epoch: 1512, cost 58.75 s 2023-09-17 13:06:04,737 44k INFO Train Epoch: 1513 [47%] 2023-09-17 13:06:04,738 44k INFO Losses: [2.1652166843414307, 2.764146327972412, 12.105202674865723, 22.264814376831055, 0.7799170613288879], step: 48400, lr: 4.138883638547025e-05, reference_loss: 40.07929611206055 2023-09-17 13:06:34,200 44k INFO ====> Epoch: 1513, cost 59.24 s 2023-09-17 13:07:33,417 44k INFO ====> Epoch: 1514, cost 59.22 s 2023-09-17 13:08:32,772 44k INFO ====> Epoch: 1515, cost 59.36 s 2023-09-17 13:09:32,575 44k INFO ====> Epoch: 1516, cost 59.80 s 2023-09-17 13:10:32,232 44k INFO ====> Epoch: 1517, cost 59.66 s 2023-09-17 13:11:31,075 44k INFO ====> Epoch: 1518, cost 58.84 s 2023-09-17 13:12:16,581 44k INFO Train Epoch: 1519 [72%] 2023-09-17 13:12:16,583 44k INFO Losses: [2.2108426094055176, 2.654758930206299, 13.280004501342773, 21.941261291503906, 0.9017452597618103], step: 48600, lr: 4.1357804457073056e-05, reference_loss: 40.98861312866211 2023-09-17 13:12:30,914 44k INFO ====> Epoch: 1519, cost 59.84 s 2023-09-17 13:13:30,216 44k INFO ====> Epoch: 1520, cost 59.30 s 2023-09-17 13:14:29,240 44k INFO ====> Epoch: 1521, cost 59.02 s 2023-09-17 13:15:28,801 44k INFO ====> Epoch: 1522, cost 59.56 s 2023-09-17 13:16:28,086 44k INFO ====> Epoch: 1523, cost 59.28 s 2023-09-17 13:17:27,438 44k INFO ====> Epoch: 1524, cost 59.35 s 2023-09-17 13:18:27,108 44k INFO Train Epoch: 1525 [97%] 2023-09-17 13:18:27,110 44k INFO Losses: [1.6938539743423462, 3.217116594314575, 15.296875953674316, 17.03851890563965, -0.45015406608581543], step: 48800, lr: 4.1326795795350275e-05, reference_loss: 36.79621505737305 2023-09-17 13:18:43,845 44k INFO Saving model and optimizer state at iteration 1525 to ./logs/44k/G_48800.pth 2023-09-17 13:18:46,539 44k INFO Saving model and optimizer state at iteration 1525 to ./logs/44k/D_48800.pth 2023-09-17 13:18:47,059 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_44800.pth 2023-09-17 13:18:47,061 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_44800.pth 2023-09-17 13:18:47,061 44k INFO ====> Epoch: 1525, cost 79.62 s 2023-09-17 13:19:46,729 44k INFO ====> Epoch: 1526, cost 59.67 s 2023-09-17 13:20:46,335 44k INFO ====> Epoch: 1527, cost 59.61 s 2023-09-17 13:21:45,730 44k INFO ====> Epoch: 1528, cost 59.39 s 2023-09-17 13:22:44,796 44k INFO ====> Epoch: 1529, cost 59.07 s 2023-09-17 13:23:43,995 44k INFO ====> Epoch: 1530, cost 59.20 s 2023-09-17 13:24:42,915 44k INFO ====> Epoch: 1531, cost 58.92 s 2023-09-17 13:24:58,175 44k INFO Train Epoch: 1532 [22%] 2023-09-17 13:24:58,176 44k INFO Losses: [2.2108662128448486, 2.7778422832489014, 13.127202987670898, 22.09403419494629, 0.837202787399292], step: 49000, lr: 4.129064840655948e-05, reference_loss: 41.047149658203125 2023-09-17 13:25:43,341 44k INFO ====> Epoch: 1532, cost 60.43 s 2023-09-17 13:26:42,411 44k INFO ====> Epoch: 1533, cost 59.07 s 2023-09-17 13:27:41,196 44k INFO ====> Epoch: 1534, cost 58.79 s 2023-09-17 13:28:40,316 44k INFO ====> Epoch: 1535, cost 59.12 s 2023-09-17 13:29:39,527 44k INFO ====> Epoch: 1536, cost 59.21 s 2023-09-17 13:30:38,854 44k INFO ====> Epoch: 1537, cost 59.33 s 2023-09-17 13:31:08,889 44k INFO Train Epoch: 1538 [47%] 2023-09-17 13:31:08,890 44k INFO Losses: [2.1415348052978516, 2.736246109008789, 14.714900016784668, 22.570247650146484, 0.8061184883117676], step: 49200, lr: 4.12596900961375e-05, reference_loss: 42.96904754638672 2023-09-17 13:31:38,812 44k INFO ====> Epoch: 1538, cost 59.96 s 2023-09-17 13:32:38,074 44k INFO ====> Epoch: 1539, cost 59.26 s 2023-09-17 13:33:37,440 44k INFO ====> Epoch: 1540, cost 59.37 s 2023-09-17 13:34:36,708 44k INFO ====> Epoch: 1541, cost 59.27 s 2023-09-17 13:35:35,893 44k INFO ====> Epoch: 1542, cost 59.19 s 2023-09-17 13:36:34,911 44k INFO ====> Epoch: 1543, cost 59.02 s 2023-09-17 13:37:20,311 44k INFO Train Epoch: 1544 [72%] 2023-09-17 13:37:20,313 44k INFO Losses: [2.191436290740967, 2.489797353744507, 13.275944709777832, 21.476547241210938, 0.8704952001571655], step: 49400, lr: 4.1228754997193696e-05, reference_loss: 40.30421829223633 2023-09-17 13:37:34,729 44k INFO ====> Epoch: 1544, cost 59.82 s 2023-09-17 13:38:34,109 44k INFO ====> Epoch: 1545, cost 59.38 s 2023-09-17 13:39:33,281 44k INFO ====> Epoch: 1546, cost 59.17 s 2023-09-17 13:40:32,870 44k INFO ====> Epoch: 1547, cost 59.59 s 2023-09-17 13:41:31,764 44k INFO ====> Epoch: 1548, cost 58.89 s 2023-09-17 13:42:30,988 44k INFO ====> Epoch: 1549, cost 59.22 s 2023-09-17 13:43:30,503 44k INFO Train Epoch: 1550 [97%] 2023-09-17 13:43:30,504 44k INFO Losses: [1.6948790550231934, 2.9192705154418945, 8.780980110168457, 16.83723258972168, -0.4229075312614441], step: 49600, lr: 4.119784309232489e-05, reference_loss: 29.80945587158203 2023-09-17 13:43:48,095 44k INFO Saving model and optimizer state at iteration 1550 to ./logs/44k/G_49600.pth 2023-09-17 13:43:51,423 44k INFO Saving model and optimizer state at iteration 1550 to ./logs/44k/D_49600.pth 2023-09-17 13:43:52,514 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_45600.pth 2023-09-17 13:43:52,515 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_45600.pth 2023-09-17 13:43:52,516 44k INFO ====> Epoch: 1550, cost 81.53 s 2023-09-17 13:44:52,088 44k INFO ====> Epoch: 1551, cost 59.57 s 2023-09-17 13:45:51,388 44k INFO ====> Epoch: 1552, cost 59.30 s 2023-09-17 13:46:50,487 44k INFO ====> Epoch: 1553, cost 59.10 s 2023-09-17 13:47:49,918 44k INFO ====> Epoch: 1554, cost 59.43 s 2023-09-17 13:48:48,670 44k INFO ====> Epoch: 1555, cost 58.75 s 2023-09-17 13:49:48,241 44k INFO ====> Epoch: 1556, cost 59.57 s 2023-09-17 13:50:02,970 44k INFO Train Epoch: 1557 [22%] 2023-09-17 13:50:02,971 44k INFO Losses: [2.193455219268799, 2.7561910152435303, 14.09470272064209, 22.318161010742188, 0.876568078994751], step: 49800, lr: 4.116180849484544e-05, reference_loss: 42.239078521728516 2023-09-17 13:50:48,211 44k INFO ====> Epoch: 1557, cost 59.97 s 2023-09-17 13:51:47,796 44k INFO ====> Epoch: 1558, cost 59.59 s 2023-09-17 13:52:47,020 44k INFO ====> Epoch: 1559, cost 59.22 s 2023-09-17 13:53:46,480 44k INFO ====> Epoch: 1560, cost 59.46 s 2023-09-17 13:54:45,980 44k INFO ====> Epoch: 1561, cost 59.50 s 2023-09-17 13:55:45,665 44k INFO ====> Epoch: 1562, cost 59.68 s 2023-09-17 13:56:15,973 44k INFO Train Epoch: 1563 [47%] 2023-09-17 13:56:15,974 44k INFO Losses: [2.152392864227295, 2.601996898651123, 13.718199729919434, 22.283344268798828, 0.789932906627655], step: 50000, lr: 4.113094678416543e-05, reference_loss: 41.545867919921875 2023-09-17 13:56:45,652 44k INFO ====> Epoch: 1563, cost 59.99 s 2023-09-17 13:57:44,598 44k INFO ====> Epoch: 1564, cost 58.95 s 2023-09-17 13:58:43,733 44k INFO ====> Epoch: 1565, cost 59.13 s 2023-09-17 13:59:43,191 44k INFO ====> Epoch: 1566, cost 59.46 s 2023-09-17 14:00:42,953 44k INFO ====> Epoch: 1567, cost 59.76 s 2023-09-17 14:01:42,377 44k INFO ====> Epoch: 1568, cost 59.42 s 2023-09-17 14:02:28,011 44k INFO Train Epoch: 1569 [72%] 2023-09-17 14:02:28,012 44k INFO Losses: [2.1156466007232666, 2.741691827774048, 14.139958381652832, 21.8016300201416, 0.870171070098877], step: 50200, lr: 4.110010821253642e-05, reference_loss: 41.669097900390625 2023-09-17 14:02:42,037 44k INFO ====> Epoch: 1569, cost 59.66 s 2023-09-17 14:03:41,517 44k INFO ====> Epoch: 1570, cost 59.48 s 2023-09-17 14:04:40,890 44k INFO ====> Epoch: 1571, cost 59.37 s 2023-09-17 14:05:39,909 44k INFO ====> Epoch: 1572, cost 59.02 s 2023-09-17 14:06:38,882 44k INFO ====> Epoch: 1573, cost 58.97 s 2023-09-17 14:07:38,732 44k INFO ====> Epoch: 1574, cost 59.85 s 2023-09-17 14:08:37,999 44k INFO Train Epoch: 1575 [97%] 2023-09-17 14:08:38,000 44k INFO Losses: [1.3700848817825317, 3.1595821380615234, 13.65395450592041, 17.026275634765625, -0.46373534202575684], step: 50400, lr: 4.106929276260955e-05, reference_loss: 34.74616241455078 2023-09-17 14:08:53,649 44k INFO Saving model and optimizer state at iteration 1575 to ./logs/44k/G_50400.pth 2023-09-17 14:08:57,227 44k INFO Saving model and optimizer state at iteration 1575 to ./logs/44k/D_50400.pth 2023-09-17 14:08:57,909 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_46400.pth 2023-09-17 14:08:57,911 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_46400.pth 2023-09-17 14:08:57,911 44k INFO ====> Epoch: 1575, cost 79.18 s 2023-09-17 14:09:56,707 44k INFO ====> Epoch: 1576, cost 58.80 s 2023-09-17 14:10:56,244 44k INFO ====> Epoch: 1577, cost 59.54 s 2023-09-17 14:11:55,513 44k INFO ====> Epoch: 1578, cost 59.27 s 2023-09-17 14:12:54,475 44k INFO ====> Epoch: 1579, cost 58.96 s 2023-09-17 14:13:53,736 44k INFO ====> Epoch: 1580, cost 59.26 s 2023-09-17 14:14:53,263 44k INFO ====> Epoch: 1581, cost 59.53 s 2023-09-17 14:15:08,375 44k INFO Train Epoch: 1582 [22%] 2023-09-17 14:15:08,376 44k INFO Losses: [2.2299509048461914, 2.7616517543792725, 13.207463264465332, 22.086605072021484, 0.8961745500564575], step: 50600, lr: 4.103337060449682e-05, reference_loss: 41.181846618652344 2023-09-17 14:15:52,989 44k INFO ====> Epoch: 1582, cost 59.73 s 2023-09-17 14:16:52,323 44k INFO ====> Epoch: 1583, cost 59.33 s 2023-09-17 14:17:51,535 44k INFO ====> Epoch: 1584, cost 59.21 s 2023-09-17 14:18:50,557 44k INFO ====> Epoch: 1585, cost 59.02 s 2023-09-17 14:19:50,096 44k INFO ====> Epoch: 1586, cost 59.54 s 2023-09-17 14:20:49,493 44k INFO ====> Epoch: 1587, cost 59.40 s 2023-09-17 14:21:20,111 44k INFO Train Epoch: 1588 [47%] 2023-09-17 14:21:20,112 44k INFO Losses: [2.1410908699035645, 2.6834309101104736, 13.349028587341309, 22.135746002197266, 0.7807456254959106], step: 50800, lr: 4.100260519213695e-05, reference_loss: 41.09004211425781 2023-09-17 14:21:49,465 44k INFO ====> Epoch: 1588, cost 59.97 s 2023-09-17 14:22:48,840 44k INFO ====> Epoch: 1589, cost 59.38 s 2023-09-17 14:23:47,908 44k INFO ====> Epoch: 1590, cost 59.07 s 2023-09-17 14:24:47,487 44k INFO ====> Epoch: 1591, cost 59.58 s 2023-09-17 14:25:46,271 44k INFO ====> Epoch: 1592, cost 58.78 s 2023-09-17 14:26:45,487 44k INFO ====> Epoch: 1593, cost 59.22 s 2023-09-17 14:27:30,942 44k INFO Train Epoch: 1594 [72%] 2023-09-17 14:27:30,943 44k INFO Losses: [2.153994083404541, 2.626734972000122, 14.215911865234375, 21.67413902282715, 0.8574249148368835], step: 51000, lr: 4.097186284662692e-05, reference_loss: 41.52820587158203 2023-09-17 14:27:45,211 44k INFO ====> Epoch: 1594, cost 59.72 s 2023-09-17 14:28:44,227 44k INFO ====> Epoch: 1595, cost 59.02 s 2023-09-17 14:29:43,517 44k INFO ====> Epoch: 1596, cost 59.29 s 2023-09-17 14:30:42,963 44k INFO ====> Epoch: 1597, cost 59.45 s 2023-09-17 14:31:42,672 44k INFO ====> Epoch: 1598, cost 59.71 s 2023-09-17 14:32:41,939 44k INFO ====> Epoch: 1599, cost 59.27 s 2023-09-17 14:33:40,162 44k INFO Train Epoch: 1600 [97%] 2023-09-17 14:33:40,163 44k INFO Losses: [1.838470458984375, 2.7550148963928223, 7.951935768127441, 15.86609935760498, -0.4336448013782501], step: 51200, lr: 4.094114355067199e-05, reference_loss: 27.977876663208008 2023-09-17 14:33:57,246 44k INFO Saving model and optimizer state at iteration 1600 to ./logs/44k/G_51200.pth 2023-09-17 14:34:00,538 44k INFO Saving model and optimizer state at iteration 1600 to ./logs/44k/D_51200.pth 2023-09-17 14:34:01,507 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_47200.pth 2023-09-17 14:34:01,509 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_47200.pth 2023-09-17 14:34:01,510 44k INFO ====> Epoch: 1600, cost 79.57 s 2023-09-17 14:35:00,829 44k INFO ====> Epoch: 1601, cost 59.32 s 2023-09-17 14:36:00,221 44k INFO ====> Epoch: 1602, cost 59.39 s 2023-09-17 14:36:59,229 44k INFO ====> Epoch: 1603, cost 59.01 s 2023-09-17 14:37:58,087 44k INFO ====> Epoch: 1604, cost 58.86 s 2023-09-17 14:38:57,330 44k INFO ====> Epoch: 1605, cost 59.24 s 2023-09-17 14:39:56,929 44k INFO ====> Epoch: 1606, cost 59.60 s 2023-09-17 14:40:11,860 44k INFO Train Epoch: 1607 [22%] 2023-09-17 14:40:11,861 44k INFO Losses: [2.1407954692840576, 2.7909903526306152, 15.229604721069336, 22.266780853271484, 0.854574978351593], step: 51400, lr: 4.09053334810795e-05, reference_loss: 43.282745361328125 2023-09-17 14:40:56,864 44k INFO ====> Epoch: 1607, cost 59.93 s 2023-09-17 14:41:56,606 44k INFO ====> Epoch: 1608, cost 59.74 s 2023-09-17 14:42:55,543 44k INFO ====> Epoch: 1609, cost 58.94 s 2023-09-17 14:43:54,873 44k INFO ====> Epoch: 1610, cost 59.33 s 2023-09-17 14:44:54,035 44k INFO ====> Epoch: 1611, cost 59.16 s 2023-09-17 14:45:53,484 44k INFO ====> Epoch: 1612, cost 59.45 s 2023-09-17 14:46:23,679 44k INFO Train Epoch: 1613 [47%] 2023-09-17 14:46:23,680 44k INFO Losses: [2.138786792755127, 2.7304177284240723, 14.481159210205078, 22.498279571533203, 0.7869365811347961], step: 51600, lr: 4.08746640665585e-05, reference_loss: 42.635581970214844 2023-09-17 14:46:53,529 44k INFO ====> Epoch: 1613, cost 60.05 s 2023-09-17 14:47:52,763 44k INFO ====> Epoch: 1614, cost 59.23 s 2023-09-17 14:48:51,724 44k INFO ====> Epoch: 1615, cost 58.96 s 2023-09-17 14:49:51,276 44k INFO ====> Epoch: 1616, cost 59.55 s 2023-09-17 14:50:50,910 44k INFO ====> Epoch: 1617, cost 59.63 s 2023-09-17 14:51:50,520 44k INFO ====> Epoch: 1618, cost 59.61 s 2023-09-17 14:52:36,315 44k INFO Train Epoch: 1619 [72%] 2023-09-17 14:52:36,316 44k INFO Losses: [2.1816964149475098, 2.5864152908325195, 13.627592086791992, 21.787214279174805, 0.8393174409866333], step: 51800, lr: 4.084401764691145e-05, reference_loss: 41.02223587036133 2023-09-17 14:52:50,632 44k INFO ====> Epoch: 1619, cost 60.11 s 2023-09-17 14:53:49,841 44k INFO ====> Epoch: 1620, cost 59.21 s 2023-09-17 14:54:49,165 44k INFO ====> Epoch: 1621, cost 59.32 s 2023-09-17 14:55:48,649 44k INFO ====> Epoch: 1622, cost 59.48 s 2023-09-17 14:56:47,511 44k INFO ====> Epoch: 1623, cost 58.86 s 2023-09-17 14:57:46,524 44k INFO ====> Epoch: 1624, cost 59.01 s 2023-09-17 14:58:45,858 44k INFO Train Epoch: 1625 [97%] 2023-09-17 14:58:45,859 44k INFO Losses: [1.8047206401824951, 2.7856380939483643, 9.35676097869873, 16.18894386291504, -0.4002842605113983], step: 52000, lr: 4.0813394204897576e-05, reference_loss: 29.73577880859375 2023-09-17 14:59:03,237 44k INFO Saving model and optimizer state at iteration 1625 to ./logs/44k/G_52000.pth 2023-09-17 14:59:06,655 44k INFO Saving model and optimizer state at iteration 1625 to ./logs/44k/D_52000.pth 2023-09-17 14:59:07,715 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_48000.pth 2023-09-17 14:59:07,717 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_48000.pth 2023-09-17 14:59:07,717 44k INFO ====> Epoch: 1625, cost 81.19 s 2023-09-17 15:00:07,063 44k INFO ====> Epoch: 1626, cost 59.35 s 2023-09-17 15:01:06,530 44k INFO ====> Epoch: 1627, cost 59.47 s 2023-09-17 15:02:05,430 44k INFO ====> Epoch: 1628, cost 58.90 s 2023-09-17 15:03:04,788 44k INFO ====> Epoch: 1629, cost 59.36 s 2023-09-17 15:04:04,052 44k INFO ====> Epoch: 1630, cost 59.26 s 2023-09-17 15:05:02,650 44k INFO ====> Epoch: 1631, cost 58.60 s 2023-09-17 15:05:17,623 44k INFO Train Epoch: 1632 [22%] 2023-09-17 15:05:17,624 44k INFO Losses: [2.211981773376465, 2.73002028465271, 14.852234840393066, 22.272491455078125, 0.8684206604957581], step: 52200, lr: 4.0777695874073614e-05, reference_loss: 42.93514633178711 2023-09-17 15:06:02,430 44k INFO ====> Epoch: 1632, cost 59.78 s 2023-09-17 15:07:01,867 44k INFO ====> Epoch: 1633, cost 59.44 s 2023-09-17 15:08:00,954 44k INFO ====> Epoch: 1634, cost 59.09 s 2023-09-17 15:08:59,991 44k INFO ====> Epoch: 1635, cost 59.04 s 2023-09-17 15:09:58,945 44k INFO ====> Epoch: 1636, cost 58.95 s 2023-09-17 15:10:58,314 44k INFO ====> Epoch: 1637, cost 59.37 s 2023-09-17 15:11:28,458 44k INFO Train Epoch: 1638 [47%] 2023-09-17 15:11:28,459 44k INFO Losses: [2.1193063259124756, 2.7221872806549072, 13.648557662963867, 22.1920108795166, 0.792159914970398], step: 52400, lr: 4.074712215784779e-05, reference_loss: 41.474220275878906 2023-09-17 15:11:58,374 44k INFO ====> Epoch: 1638, cost 60.06 s 2023-09-17 15:12:57,380 44k INFO ====> Epoch: 1639, cost 59.01 s 2023-09-17 15:13:56,883 44k INFO ====> Epoch: 1640, cost 59.50 s 2023-09-17 15:14:56,081 44k INFO ====> Epoch: 1641, cost 59.20 s 2023-09-17 15:15:55,342 44k INFO ====> Epoch: 1642, cost 59.26 s 2023-09-17 15:16:55,193 44k INFO ====> Epoch: 1643, cost 59.85 s 2023-09-17 15:17:40,808 44k INFO Train Epoch: 1644 [72%] 2023-09-17 15:17:40,809 44k INFO Losses: [2.1837339401245117, 2.5446054935455322, 12.99724006652832, 21.600406646728516, 0.8520742654800415], step: 52600, lr: 4.071657136474462e-05, reference_loss: 40.17805862426758 2023-09-17 15:17:55,349 44k INFO ====> Epoch: 1644, cost 60.16 s 2023-09-17 15:18:54,447 44k INFO ====> Epoch: 1645, cost 59.10 s 2023-09-17 15:19:54,021 44k INFO ====> Epoch: 1646, cost 59.57 s 2023-09-17 15:20:52,974 44k INFO ====> Epoch: 1647, cost 58.95 s 2023-09-17 15:21:52,526 44k INFO ====> Epoch: 1648, cost 59.55 s 2023-09-17 15:22:51,627 44k INFO ====> Epoch: 1649, cost 59.10 s 2023-09-17 15:23:50,908 44k INFO Train Epoch: 1650 [97%] 2023-09-17 15:23:50,909 44k INFO Losses: [1.559007167816162, 3.020951271057129, 10.880867004394531, 16.185449600219727, -0.5138125419616699], step: 52800, lr: 4.0686043477577126e-05, reference_loss: 31.132461547851562 2023-09-17 15:24:08,292 44k INFO Saving model and optimizer state at iteration 1650 to ./logs/44k/G_52800.pth 2023-09-17 15:24:10,972 44k INFO Saving model and optimizer state at iteration 1650 to ./logs/44k/D_52800.pth 2023-09-17 15:24:11,491 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_48800.pth 2023-09-17 15:24:11,492 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_48800.pth 2023-09-17 15:24:11,493 44k INFO ====> Epoch: 1650, cost 79.87 s 2023-09-17 15:25:10,359 44k INFO ====> Epoch: 1651, cost 58.87 s 2023-09-17 15:26:09,544 44k INFO ====> Epoch: 1652, cost 59.19 s 2023-09-17 15:27:09,142 44k INFO ====> Epoch: 1653, cost 59.60 s 2023-09-17 15:28:08,256 44k INFO ====> Epoch: 1654, cost 59.11 s 2023-09-17 15:29:07,809 44k INFO ====> Epoch: 1655, cost 59.55 s 2023-09-17 15:30:07,424 44k INFO ====> Epoch: 1656, cost 59.62 s 2023-09-17 15:30:22,448 44k INFO Train Epoch: 1657 [22%] 2023-09-17 15:30:22,449 44k INFO Losses: [2.2221953868865967, 2.783796787261963, 14.582415580749512, 22.265399932861328, 0.8448291420936584], step: 53000, lr: 4.0650456536861326e-05, reference_loss: 42.69863510131836 2023-09-17 15:31:07,238 44k INFO ====> Epoch: 1657, cost 59.81 s 2023-09-17 15:32:06,984 44k INFO ====> Epoch: 1658, cost 59.75 s 2023-09-17 15:33:06,350 44k INFO ====> Epoch: 1659, cost 59.37 s 2023-09-17 15:34:06,088 44k INFO ====> Epoch: 1660, cost 59.74 s 2023-09-17 15:35:05,934 44k INFO ====> Epoch: 1661, cost 59.85 s 2023-09-17 15:36:05,255 44k INFO ====> Epoch: 1662, cost 59.32 s 2023-09-17 15:36:35,809 44k INFO Train Epoch: 1663 [47%] 2023-09-17 15:36:35,810 44k INFO Losses: [2.0957565307617188, 2.7464449405670166, 14.97374439239502, 22.418012619018555, 0.8229804635047913], step: 53200, lr: 4.0619978220321666e-05, reference_loss: 43.05693435668945 2023-09-17 15:37:05,618 44k INFO ====> Epoch: 1663, cost 60.36 s 2023-09-17 15:38:05,222 44k INFO ====> Epoch: 1664, cost 59.60 s 2023-09-17 15:39:04,483 44k INFO ====> Epoch: 1665, cost 59.26 s 2023-09-17 15:40:03,728 44k INFO ====> Epoch: 1666, cost 59.24 s 2023-09-17 15:41:03,138 44k INFO ====> Epoch: 1667, cost 59.41 s 2023-09-17 15:42:02,528 44k INFO ====> Epoch: 1668, cost 59.39 s 2023-09-17 15:42:47,698 44k INFO Train Epoch: 1669 [72%] 2023-09-17 15:42:47,699 44k INFO Losses: [2.146665096282959, 2.5727782249450684, 14.3754301071167, 21.77250862121582, 0.8439691066741943], step: 53400, lr: 4.058952275537724e-05, reference_loss: 41.71134948730469 2023-09-17 15:43:01,849 44k INFO ====> Epoch: 1669, cost 59.32 s 2023-09-17 15:44:00,716 44k INFO ====> Epoch: 1670, cost 58.87 s 2023-09-17 15:44:59,062 44k INFO ====> Epoch: 1671, cost 58.35 s 2023-09-17 15:45:57,712 44k INFO ====> Epoch: 1672, cost 58.65 s 2023-09-17 15:46:55,715 44k INFO ====> Epoch: 1673, cost 58.00 s 2023-09-17 15:47:54,570 44k INFO ====> Epoch: 1674, cost 58.85 s 2023-09-17 15:48:52,988 44k INFO Train Epoch: 1675 [97%] 2023-09-17 15:48:52,989 44k INFO Losses: [1.514462947845459, 2.9820103645324707, 10.170028686523438, 16.65373992919922, -0.4774441719055176], step: 53600, lr: 4.055909012489472e-05, reference_loss: 30.842798233032227 2023-09-17 15:49:10,354 44k INFO Saving model and optimizer state at iteration 1675 to ./logs/44k/G_53600.pth 2023-09-17 15:49:13,326 44k INFO Saving model and optimizer state at iteration 1675 to ./logs/44k/D_53600.pth 2023-09-17 15:49:14,465 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_49600.pth 2023-09-17 15:49:14,467 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_49600.pth 2023-09-17 15:49:14,467 44k INFO ====> Epoch: 1675, cost 79.90 s 2023-09-17 15:50:13,573 44k INFO ====> Epoch: 1676, cost 59.11 s 2023-09-17 15:51:12,638 44k INFO ====> Epoch: 1677, cost 59.07 s 2023-09-17 15:52:10,789 44k INFO ====> Epoch: 1678, cost 58.15 s 2023-09-17 15:53:09,296 44k INFO ====> Epoch: 1679, cost 58.51 s 2023-09-17 15:54:08,332 44k INFO ====> Epoch: 1680, cost 59.04 s 2023-09-17 15:55:06,639 44k INFO ====> Epoch: 1681, cost 58.31 s 2023-09-17 15:55:21,527 44k INFO Train Epoch: 1682 [22%] 2023-09-17 15:55:21,529 44k INFO Losses: [2.146169900894165, 2.834348201751709, 14.107751846313477, 22.250547409057617, 0.8351293206214905], step: 53800, lr: 4.052361422671463e-05, reference_loss: 42.173946380615234 2023-09-17 15:56:05,629 44k INFO ====> Epoch: 1682, cost 58.99 s 2023-09-17 15:57:04,875 44k INFO ====> Epoch: 1683, cost 59.25 s 2023-09-17 15:58:03,219 44k INFO ====> Epoch: 1684, cost 58.34 s 2023-09-17 15:59:02,461 44k INFO ====> Epoch: 1685, cost 59.24 s 2023-09-17 16:00:01,143 44k INFO ====> Epoch: 1686, cost 58.68 s 2023-09-17 16:01:00,223 44k INFO ====> Epoch: 1687, cost 59.08 s 2023-09-17 16:01:30,293 44k INFO Train Epoch: 1688 [47%] 2023-09-17 16:01:30,294 44k INFO Losses: [2.1312174797058105, 2.732492446899414, 13.872379302978516, 22.33211326599121, 0.7916145920753479], step: 54000, lr: 4.049323101218386e-05, reference_loss: 41.85981750488281 2023-09-17 16:01:59,657 44k INFO ====> Epoch: 1688, cost 59.43 s 2023-09-17 16:02:58,178 44k INFO ====> Epoch: 1689, cost 58.52 s 2023-09-17 16:03:56,396 44k INFO ====> Epoch: 1690, cost 58.22 s 2023-09-17 16:04:55,013 44k INFO ====> Epoch: 1691, cost 58.62 s 2023-09-17 16:05:53,387 44k INFO ====> Epoch: 1692, cost 58.37 s 2023-09-17 16:06:51,663 44k INFO ====> Epoch: 1693, cost 58.28 s 2023-09-17 16:07:36,376 44k INFO Train Epoch: 1694 [72%] 2023-09-17 16:07:36,377 44k INFO Losses: [2.1534626483917236, 2.6539626121520996, 13.967208862304688, 21.71929359436035, 0.8526560068130493], step: 54200, lr: 4.046287057794411e-05, reference_loss: 41.34658432006836 2023-09-17 16:07:50,581 44k INFO ====> Epoch: 1694, cost 58.92 s 2023-09-17 16:08:49,262 44k INFO ====> Epoch: 1695, cost 58.68 s 2023-09-17 16:09:47,875 44k INFO ====> Epoch: 1696, cost 58.61 s 2023-09-17 16:10:46,876 44k INFO ====> Epoch: 1697, cost 59.00 s 2023-09-17 16:11:45,251 44k INFO ====> Epoch: 1698, cost 58.38 s 2023-09-17 16:12:43,506 44k INFO ====> Epoch: 1699, cost 58.25 s 2023-09-17 16:13:41,586 44k INFO Train Epoch: 1700 [97%] 2023-09-17 16:13:41,587 44k INFO Losses: [1.582852840423584, 2.961913824081421, 11.295804977416992, 16.135181427001953, -0.49884530901908875], step: 54400, lr: 4.0432532906915504e-05, reference_loss: 31.47690773010254 2023-09-17 16:13:58,930 44k INFO Saving model and optimizer state at iteration 1700 to ./logs/44k/G_54400.pth 2023-09-17 16:14:02,494 44k INFO Saving model and optimizer state at iteration 1700 to ./logs/44k/D_54400.pth 2023-09-17 16:14:03,190 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_50400.pth 2023-09-17 16:14:03,192 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_50400.pth 2023-09-17 16:14:03,192 44k INFO ====> Epoch: 1700, cost 79.69 s 2023-09-17 16:15:01,491 44k INFO ====> Epoch: 1701, cost 58.30 s 2023-09-17 16:16:00,333 44k INFO ====> Epoch: 1702, cost 58.84 s 2023-09-17 16:16:58,713 44k INFO ====> Epoch: 1703, cost 58.38 s 2023-09-17 16:17:57,212 44k INFO ====> Epoch: 1704, cost 58.50 s 2023-09-17 16:18:55,797 44k INFO ====> Epoch: 1705, cost 58.59 s 2023-09-17 16:19:54,193 44k INFO ====> Epoch: 1706, cost 58.40 s 2023-09-17 16:20:09,162 44k INFO Train Epoch: 1707 [22%] 2023-09-17 16:20:09,163 44k INFO Losses: [2.191889762878418, 2.836854934692383, 14.403891563415527, 22.332853317260742, 0.8461661338806152], step: 54600, lr: 4.0397167704783206e-05, reference_loss: 42.61165237426758 2023-09-17 16:20:53,452 44k INFO ====> Epoch: 1707, cost 59.26 s 2023-09-17 16:21:51,943 44k INFO ====> Epoch: 1708, cost 58.49 s 2023-09-17 16:22:50,973 44k INFO ====> Epoch: 1709, cost 59.03 s 2023-09-17 16:23:50,047 44k INFO ====> Epoch: 1710, cost 59.07 s 2023-09-17 16:24:48,653 44k INFO ====> Epoch: 1711, cost 58.61 s 2023-09-17 16:25:47,424 44k INFO ====> Epoch: 1712, cost 58.77 s 2023-09-17 16:26:17,415 44k INFO Train Epoch: 1713 [47%] 2023-09-17 16:26:17,416 44k INFO Losses: [2.0948758125305176, 2.6959636211395264, 15.117575645446777, 22.373933792114258, 0.7900431752204895], step: 54800, lr: 4.036687929551293e-05, reference_loss: 43.072391510009766 2023-09-17 16:26:47,175 44k INFO ====> Epoch: 1713, cost 59.75 s 2023-09-17 16:27:45,320 44k INFO ====> Epoch: 1714, cost 58.14 s 2023-09-17 16:28:43,873 44k INFO ====> Epoch: 1715, cost 58.55 s 2023-09-17 16:29:43,145 44k INFO ====> Epoch: 1716, cost 59.27 s 2023-09-17 16:30:42,260 44k INFO ====> Epoch: 1717, cost 59.11 s 2023-09-17 16:31:40,817 44k INFO ====> Epoch: 1718, cost 58.56 s 2023-09-17 16:32:26,217 44k INFO Train Epoch: 1719 [72%] 2023-09-17 16:32:26,218 44k INFO Losses: [2.1061553955078125, 2.681844472885132, 15.16109561920166, 21.77120590209961, 0.8380715250968933], step: 55000, lr: 4.033661359545194e-05, reference_loss: 42.558372497558594 2023-09-17 16:32:40,318 44k INFO ====> Epoch: 1719, cost 59.50 s 2023-09-17 16:33:38,657 44k INFO ====> Epoch: 1720, cost 58.34 s 2023-09-17 16:34:37,520 44k INFO ====> Epoch: 1721, cost 58.86 s 2023-09-17 16:35:36,248 44k INFO ====> Epoch: 1722, cost 58.73 s 2023-09-17 16:36:34,480 44k INFO ====> Epoch: 1723, cost 58.23 s 2023-09-17 16:37:32,789 44k INFO ====> Epoch: 1724, cost 58.31 s 2023-09-17 16:38:31,242 44k INFO Train Epoch: 1725 [97%] 2023-09-17 16:38:31,243 44k INFO Losses: [1.979893684387207, 2.708832263946533, 9.442656517028809, 16.2440185546875, -0.4945613741874695], step: 55200, lr: 4.030637058757366e-05, reference_loss: 29.88083839416504 2023-09-17 16:38:49,730 44k INFO Saving model and optimizer state at iteration 1725 to ./logs/44k/G_55200.pth 2023-09-17 16:38:53,289 44k INFO Saving model and optimizer state at iteration 1725 to ./logs/44k/D_55200.pth 2023-09-17 16:38:53,828 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_51200.pth 2023-09-17 16:38:53,829 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_51200.pth 2023-09-17 16:38:53,829 44k INFO ====> Epoch: 1725, cost 81.04 s 2023-09-17 16:39:52,190 44k INFO ====> Epoch: 1726, cost 58.36 s 2023-09-17 16:40:50,694 44k INFO ====> Epoch: 1727, cost 58.50 s 2023-09-17 16:41:49,518 44k INFO ====> Epoch: 1728, cost 58.82 s 2023-09-17 16:42:47,749 44k INFO ====> Epoch: 1729, cost 58.23 s 2023-09-17 16:43:46,232 44k INFO ====> Epoch: 1730, cost 58.48 s 2023-09-17 16:44:45,050 44k INFO ====> Epoch: 1731, cost 58.82 s 2023-09-17 16:44:59,760 44k INFO Train Epoch: 1732 [22%] 2023-09-17 16:44:59,761 44k INFO Losses: [2.222867012023926, 2.7938034534454346, 14.169510841369629, 22.230649948120117, 0.8385384678840637], step: 55400, lr: 4.02711157360824e-05, reference_loss: 42.25537109375 2023-09-17 16:45:43,996 44k INFO ====> Epoch: 1732, cost 58.95 s 2023-09-17 16:46:42,332 44k INFO ====> Epoch: 1733, cost 58.34 s 2023-09-17 16:47:40,573 44k INFO ====> Epoch: 1734, cost 58.24 s 2023-09-17 16:48:39,114 44k INFO ====> Epoch: 1735, cost 58.54 s 2023-09-17 16:49:37,451 44k INFO ====> Epoch: 1736, cost 58.34 s 2023-09-17 16:50:36,610 44k INFO ====> Epoch: 1737, cost 59.16 s 2023-09-17 16:51:06,559 44k INFO Train Epoch: 1738 [47%] 2023-09-17 16:51:06,560 44k INFO Losses: [2.1323399543762207, 2.694030284881592, 13.57571029663086, 22.20551109313965, 0.7673092484474182], step: 55600, lr: 4.0240921836250144e-05, reference_loss: 41.374900817871094 2023-09-17 16:51:36,040 44k INFO ====> Epoch: 1738, cost 59.43 s 2023-09-17 16:52:34,382 44k INFO ====> Epoch: 1739, cost 58.34 s 2023-09-17 16:53:32,779 44k INFO ====> Epoch: 1740, cost 58.40 s 2023-09-17 16:54:31,456 44k INFO ====> Epoch: 1741, cost 58.68 s 2023-09-17 16:55:29,904 44k INFO ====> Epoch: 1742, cost 58.45 s 2023-09-17 16:56:29,135 44k INFO ====> Epoch: 1743, cost 59.23 s 2023-09-17 16:57:14,502 44k INFO Train Epoch: 1744 [72%] 2023-09-17 16:57:14,502 44k INFO Losses: [2.18320631980896, 2.5391690731048584, 13.50413990020752, 21.651582717895508, 0.8544759750366211], step: 55800, lr: 4.021075057476724e-05, reference_loss: 40.732574462890625 2023-09-17 16:57:28,542 44k INFO ====> Epoch: 1744, cost 59.41 s 2023-09-17 16:58:27,048 44k INFO ====> Epoch: 1745, cost 58.51 s 2023-09-17 16:59:26,060 44k INFO ====> Epoch: 1746, cost 59.01 s 2023-09-17 17:00:24,372 44k INFO ====> Epoch: 1747, cost 58.31 s 2023-09-17 17:01:22,827 44k INFO ====> Epoch: 1748, cost 58.46 s 2023-09-17 17:02:21,242 44k INFO ====> Epoch: 1749, cost 58.41 s 2023-09-17 17:03:20,052 44k INFO Train Epoch: 1750 [97%] 2023-09-17 17:03:20,054 44k INFO Losses: [1.3083832263946533, 3.4652161598205566, 15.062191009521484, 16.555866241455078, -0.5444183349609375], step: 56000, lr: 4.0180601934660236e-05, reference_loss: 35.84723663330078 2023-09-17 17:03:36,572 44k INFO Saving model and optimizer state at iteration 1750 to ./logs/44k/G_56000.pth 2023-09-17 17:03:39,915 44k INFO Saving model and optimizer state at iteration 1750 to ./logs/44k/D_56000.pth 2023-09-17 17:03:40,979 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_52000.pth 2023-09-17 17:03:40,980 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_52000.pth 2023-09-17 17:03:40,981 44k INFO ====> Epoch: 1750, cost 79.74 s 2023-09-17 17:04:39,987 44k INFO ====> Epoch: 1751, cost 59.01 s 2023-09-17 17:05:38,393 44k INFO ====> Epoch: 1752, cost 58.41 s 2023-09-17 17:06:37,000 44k INFO ====> Epoch: 1753, cost 58.61 s 2023-09-17 17:07:35,349 44k INFO ====> Epoch: 1754, cost 58.35 s 2023-09-17 17:08:33,464 44k INFO ====> Epoch: 1755, cost 58.12 s 2023-09-17 17:09:31,957 44k INFO ====> Epoch: 1756, cost 58.49 s 2023-09-17 17:09:46,770 44k INFO Train Epoch: 1757 [22%] 2023-09-17 17:09:46,771 44k INFO Losses: [2.197350263595581, 2.7474074363708496, 14.240107536315918, 22.14127540588379, 0.8939344882965088], step: 56200, lr: 4.014545708948103e-05, reference_loss: 42.22007751464844 2023-09-17 17:10:30,759 44k INFO ====> Epoch: 1757, cost 58.80 s 2023-09-17 17:11:29,285 44k INFO ====> Epoch: 1758, cost 58.53 s 2023-09-17 17:12:27,466 44k INFO ====> Epoch: 1759, cost 58.18 s 2023-09-17 17:13:25,918 44k INFO ====> Epoch: 1760, cost 58.45 s 2023-09-17 17:14:23,937 44k INFO ====> Epoch: 1761, cost 58.02 s 2023-09-17 17:15:22,647 44k INFO ====> Epoch: 1762, cost 58.71 s 2023-09-17 17:15:52,330 44k INFO Train Epoch: 1763 [47%] 2023-09-17 17:15:52,331 44k INFO Losses: [2.096961259841919, 2.8687620162963867, 15.172320365905762, 22.49445915222168, 0.794115424156189], step: 56400, lr: 4.011535740418738e-05, reference_loss: 43.42661666870117 2023-09-17 17:16:21,418 44k INFO ====> Epoch: 1763, cost 58.77 s 2023-09-17 17:17:19,469 44k INFO ====> Epoch: 1764, cost 58.05 s 2023-09-17 17:18:18,588 44k INFO ====> Epoch: 1765, cost 59.12 s 2023-09-17 17:19:17,240 44k INFO ====> Epoch: 1766, cost 58.65 s 2023-09-17 17:20:16,468 44k INFO ====> Epoch: 1767, cost 59.23 s 2023-09-17 17:21:15,159 44k INFO ====> Epoch: 1768, cost 58.69 s 2023-09-17 17:22:00,006 44k INFO Train Epoch: 1769 [72%] 2023-09-17 17:22:00,007 44k INFO Losses: [2.099825859069824, 2.616189479827881, 14.322772026062012, 21.62627410888672, 0.8370086550712585], step: 56600, lr: 4.008528028660426e-05, reference_loss: 41.502071380615234 2023-09-17 17:22:14,370 44k INFO ====> Epoch: 1769, cost 59.21 s 2023-09-17 17:23:13,073 44k INFO ====> Epoch: 1770, cost 58.70 s 2023-09-17 17:24:11,653 44k INFO ====> Epoch: 1771, cost 58.58 s 2023-09-17 17:25:10,100 44k INFO ====> Epoch: 1772, cost 58.45 s 2023-09-17 17:26:08,180 44k INFO ====> Epoch: 1773, cost 58.08 s 2023-09-17 17:27:07,150 44k INFO ====> Epoch: 1774, cost 58.97 s 2023-09-17 17:28:06,099 44k INFO Train Epoch: 1775 [97%] 2023-09-17 17:28:06,100 44k INFO Losses: [1.725959062576294, 2.920668125152588, 11.347833633422852, 16.495521545410156, -0.5226644277572632], step: 56800, lr: 4.0055225719811184e-05, reference_loss: 31.967317581176758 2023-09-17 17:28:22,482 44k INFO Saving model and optimizer state at iteration 1775 to ./logs/44k/G_56800.pth 2023-09-17 17:28:26,101 44k INFO Saving model and optimizer state at iteration 1775 to ./logs/44k/D_56800.pth 2023-09-17 17:28:26,652 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_52800.pth 2023-09-17 17:28:26,655 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_52800.pth 2023-09-17 17:28:26,655 44k INFO ====> Epoch: 1775, cost 79.50 s 2023-09-17 17:29:24,668 44k INFO ====> Epoch: 1776, cost 58.01 s 2023-09-17 17:30:23,066 44k INFO ====> Epoch: 1777, cost 58.40 s 2023-09-17 17:31:21,872 44k INFO ====> Epoch: 1778, cost 58.81 s 2023-09-17 17:32:20,419 44k INFO ====> Epoch: 1779, cost 58.55 s 2023-09-17 17:33:18,826 44k INFO ====> Epoch: 1780, cost 58.41 s 2023-09-17 17:34:17,199 44k INFO ====> Epoch: 1781, cost 58.37 s 2023-09-17 17:34:31,829 44k INFO Train Epoch: 1782 [22%] 2023-09-17 17:34:31,830 44k INFO Losses: [2.1560821533203125, 2.8468432426452637, 15.097687721252441, 22.2268123626709, 0.8506879806518555], step: 57000, lr: 4.0020190537689474e-05, reference_loss: 43.1781120300293 2023-09-17 17:35:16,542 44k INFO ====> Epoch: 1782, cost 59.34 s 2023-09-17 17:36:15,011 44k INFO ====> Epoch: 1783, cost 58.47 s 2023-09-17 17:37:13,474 44k INFO ====> Epoch: 1784, cost 58.46 s 2023-09-17 17:38:12,121 44k INFO ====> Epoch: 1785, cost 58.65 s 2023-09-17 17:39:10,731 44k INFO ====> Epoch: 1786, cost 58.61 s 2023-09-17 17:40:09,378 44k INFO ====> Epoch: 1787, cost 58.65 s 2023-09-17 17:40:39,396 44k INFO Train Epoch: 1788 [47%] 2023-09-17 17:40:39,397 44k INFO Losses: [2.0699782371520996, 2.729567527770996, 14.626127243041992, 22.405723571777344, 0.777146577835083], step: 57200, lr: 3.999018477295521e-05, reference_loss: 42.608543395996094 2023-09-17 17:41:08,544 44k INFO ====> Epoch: 1788, cost 59.17 s 2023-09-17 17:42:07,661 44k INFO ====> Epoch: 1789, cost 59.12 s 2023-09-17 17:43:06,367 44k INFO ====> Epoch: 1790, cost 58.71 s 2023-09-17 17:44:05,140 44k INFO ====> Epoch: 1791, cost 58.77 s 2023-09-17 17:45:03,518 44k INFO ====> Epoch: 1792, cost 58.38 s 2023-09-17 17:46:02,310 44k INFO ====> Epoch: 1793, cost 58.79 s 2023-09-17 17:46:47,560 44k INFO Train Epoch: 1794 [72%] 2023-09-17 17:46:47,561 44k INFO Losses: [2.223008155822754, 2.6763148307800293, 13.236116409301758, 21.66032600402832, 0.8302597403526306], step: 57400, lr: 3.996020150551307e-05, reference_loss: 40.62602615356445 2023-09-17 17:47:01,851 44k INFO ====> Epoch: 1794, cost 59.54 s 2023-09-17 17:48:00,523 44k INFO ====> Epoch: 1795, cost 58.67 s 2023-09-17 17:48:59,092 44k INFO ====> Epoch: 1796, cost 58.57 s 2023-09-17 17:49:57,881 44k INFO ====> Epoch: 1797, cost 58.79 s 2023-09-17 17:50:56,540 44k INFO ====> Epoch: 1798, cost 58.66 s 2023-09-17 17:51:55,404 44k INFO ====> Epoch: 1799, cost 58.86 s 2023-09-17 17:52:53,432 44k INFO Train Epoch: 1800 [97%] 2023-09-17 17:52:53,433 44k INFO Losses: [1.7410132884979248, 2.89255428314209, 10.91090202331543, 16.71312141418457, -0.5043822526931763], step: 57600, lr: 3.993024071849536e-05, reference_loss: 31.75320816040039 2023-09-17 17:53:10,031 44k INFO Saving model and optimizer state at iteration 1800 to ./logs/44k/G_57600.pth 2023-09-17 17:53:13,155 44k INFO Saving model and optimizer state at iteration 1800 to ./logs/44k/D_57600.pth 2023-09-17 17:53:14,276 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_53600.pth 2023-09-17 17:53:14,293 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_53600.pth 2023-09-17 17:53:14,309 44k INFO ====> Epoch: 1800, cost 78.90 s 2023-09-17 17:54:13,215 44k INFO ====> Epoch: 1801, cost 58.91 s 2023-09-17 17:55:12,064 44k INFO ====> Epoch: 1802, cost 58.85 s 2023-09-17 17:56:10,558 44k INFO ====> Epoch: 1803, cost 58.49 s 2023-09-17 17:57:09,366 44k INFO ====> Epoch: 1804, cost 58.81 s 2023-09-17 17:58:07,571 44k INFO ====> Epoch: 1805, cost 58.20 s 2023-09-17 17:59:05,770 44k INFO ====> Epoch: 1806, cost 58.20 s 2023-09-17 17:59:20,552 44k INFO Train Epoch: 1807 [22%] 2023-09-17 17:59:20,553 44k INFO Losses: [2.1878607273101807, 2.9473321437835693, 14.58606243133545, 22.306209564208984, 0.8366835713386536], step: 57800, lr: 3.989531485724763e-05, reference_loss: 42.86415100097656 2023-09-17 18:00:04,839 44k INFO ====> Epoch: 1807, cost 59.07 s 2023-09-17 18:01:03,306 44k INFO ====> Epoch: 1808, cost 58.47 s 2023-09-17 18:02:01,986 44k INFO ====> Epoch: 1809, cost 58.68 s 2023-09-17 18:03:00,208 44k INFO ====> Epoch: 1810, cost 58.22 s 2023-09-17 18:03:58,823 44k INFO ====> Epoch: 1811, cost 58.61 s 2023-09-17 18:04:57,232 44k INFO ====> Epoch: 1812, cost 58.41 s 2023-09-17 18:05:27,331 44k INFO Train Epoch: 1813 [47%] 2023-09-17 18:05:27,332 44k INFO Losses: [2.1337687969207764, 2.75177001953125, 14.970064163208008, 22.280786514282227, 0.7773587703704834], step: 58000, lr: 3.986540272001084e-05, reference_loss: 42.91374969482422 2023-09-17 18:05:56,715 44k INFO ====> Epoch: 1813, cost 59.48 s 2023-09-17 18:06:55,479 44k INFO ====> Epoch: 1814, cost 58.76 s 2023-09-17 18:07:54,250 44k INFO ====> Epoch: 1815, cost 58.77 s 2023-09-17 18:08:53,121 44k INFO ====> Epoch: 1816, cost 58.87 s 2023-09-17 18:09:51,663 44k INFO ====> Epoch: 1817, cost 58.54 s 2023-09-17 18:10:50,943 44k INFO ====> Epoch: 1818, cost 59.28 s 2023-09-17 18:11:36,008 44k INFO Train Epoch: 1819 [72%] 2023-09-17 18:11:36,009 44k INFO Losses: [2.1498255729675293, 2.801909923553467, 14.877575874328613, 21.530363082885742, 0.8286932110786438], step: 58200, lr: 3.9835513009867484e-05, reference_loss: 42.1883659362793 2023-09-17 18:11:50,062 44k INFO ====> Epoch: 1819, cost 59.12 s 2023-09-17 18:12:48,656 44k INFO ====> Epoch: 1820, cost 58.59 s 2023-09-17 18:13:47,037 44k INFO ====> Epoch: 1821, cost 58.38 s 2023-09-17 18:14:46,110 44k INFO ====> Epoch: 1822, cost 59.07 s 2023-09-17 18:15:44,683 44k INFO ====> Epoch: 1823, cost 58.57 s 2023-09-17 18:16:43,236 44k INFO ====> Epoch: 1824, cost 58.55 s 2023-09-17 18:17:42,363 44k INFO Train Epoch: 1825 [97%] 2023-09-17 18:17:42,364 44k INFO Losses: [1.296929955482483, 3.620659828186035, 15.917837142944336, 17.19765281677246, -0.5607662200927734], step: 58400, lr: 3.98056457100025e-05, reference_loss: 37.472312927246094 2023-09-17 18:17:59,854 44k INFO Saving model and optimizer state at iteration 1825 to ./logs/44k/G_58400.pth 2023-09-17 18:18:03,242 44k INFO Saving model and optimizer state at iteration 1825 to ./logs/44k/D_58400.pth 2023-09-17 18:18:03,945 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_54400.pth 2023-09-17 18:18:03,945 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_54400.pth 2023-09-17 18:18:03,945 44k INFO ====> Epoch: 1825, cost 80.71 s 2023-09-17 18:19:03,168 44k INFO ====> Epoch: 1826, cost 59.22 s 2023-09-17 18:20:02,167 44k INFO ====> Epoch: 1827, cost 59.00 s 2023-09-17 18:21:00,217 44k INFO ====> Epoch: 1828, cost 58.05 s 2023-09-17 18:21:59,231 44k INFO ====> Epoch: 1829, cost 59.01 s 2023-09-17 18:22:58,195 44k INFO ====> Epoch: 1830, cost 58.96 s 2023-09-17 18:23:56,636 44k INFO ====> Epoch: 1831, cost 58.44 s 2023-09-17 18:24:11,370 44k INFO Train Epoch: 1832 [22%] 2023-09-17 18:24:11,371 44k INFO Losses: [2.2107293605804443, 2.7442030906677246, 13.462907791137695, 22.12620735168457, 0.8658131957054138], step: 58600, lr: 3.9770828828512984e-05, reference_loss: 41.40985870361328 2023-09-17 18:24:56,191 44k INFO ====> Epoch: 1832, cost 59.56 s 2023-09-17 18:25:54,800 44k INFO ====> Epoch: 1833, cost 58.61 s 2023-09-17 18:26:53,270 44k INFO ====> Epoch: 1834, cost 58.47 s 2023-09-17 18:27:51,664 44k INFO ====> Epoch: 1835, cost 58.39 s 2023-09-17 18:28:50,490 44k INFO ====> Epoch: 1836, cost 58.83 s 2023-09-17 18:29:49,103 44k INFO ====> Epoch: 1837, cost 58.61 s 2023-09-17 18:30:19,116 44k INFO Train Epoch: 1838 [47%] 2023-09-17 18:30:19,117 44k INFO Losses: [2.14497709274292, 2.742457151412964, 14.042911529541016, 22.08392333984375, 0.8229214549064636], step: 58800, lr: 3.9741010026626195e-05, reference_loss: 41.837188720703125 2023-09-17 18:30:48,402 44k INFO ====> Epoch: 1838, cost 59.30 s 2023-09-17 18:31:47,543 44k INFO ====> Epoch: 1839, cost 59.14 s 2023-09-17 18:32:46,096 44k INFO ====> Epoch: 1840, cost 58.55 s 2023-09-17 18:33:45,038 44k INFO ====> Epoch: 1841, cost 58.94 s 2023-09-17 18:34:43,826 44k INFO ====> Epoch: 1842, cost 58.79 s 2023-09-17 18:35:42,913 44k INFO ====> Epoch: 1843, cost 59.09 s 2023-09-17 18:36:28,037 44k INFO Train Epoch: 1844 [72%] 2023-09-17 18:36:28,038 44k INFO Losses: [2.1572813987731934, 2.772103786468506, 13.770480155944824, 21.546531677246094, 0.8294051885604858], step: 59000, lr: 3.9711213581853206e-05, reference_loss: 41.0758056640625 2023-09-17 18:36:42,412 44k INFO ====> Epoch: 1844, cost 59.50 s 2023-09-17 18:37:41,140 44k INFO ====> Epoch: 1845, cost 58.73 s 2023-09-17 18:38:39,031 44k INFO ====> Epoch: 1846, cost 57.89 s 2023-09-17 18:39:37,756 44k INFO ====> Epoch: 1847, cost 58.73 s 2023-09-17 18:40:36,322 44k INFO ====> Epoch: 1848, cost 58.57 s 2023-09-17 18:41:35,048 44k INFO ====> Epoch: 1849, cost 58.73 s 2023-09-17 18:42:34,069 44k INFO Train Epoch: 1850 [97%] 2023-09-17 18:42:34,070 44k INFO Losses: [1.713017463684082, 2.816697597503662, 7.060421943664551, 15.363822937011719, -0.41001564264297485], step: 59200, lr: 3.968143947743142e-05, reference_loss: 26.5439453125 2023-09-17 18:42:51,326 44k INFO Saving model and optimizer state at iteration 1850 to ./logs/44k/G_59200.pth 2023-09-17 18:42:54,107 44k INFO Saving model and optimizer state at iteration 1850 to ./logs/44k/D_59200.pth 2023-09-17 18:42:54,606 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_55200.pth 2023-09-17 18:42:54,607 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_55200.pth 2023-09-17 18:42:54,608 44k INFO ====> Epoch: 1850, cost 79.56 s 2023-09-17 18:43:53,301 44k INFO ====> Epoch: 1851, cost 58.69 s 2023-09-17 18:44:52,172 44k INFO ====> Epoch: 1852, cost 58.87 s 2023-09-17 18:45:50,946 44k INFO ====> Epoch: 1853, cost 58.77 s 2023-09-17 18:46:49,353 44k INFO ====> Epoch: 1854, cost 58.41 s 2023-09-17 18:47:47,842 44k INFO ====> Epoch: 1855, cost 58.49 s 2023-09-17 18:48:45,961 44k INFO ====> Epoch: 1856, cost 58.12 s 2023-09-17 18:49:00,666 44k INFO Train Epoch: 1857 [22%] 2023-09-17 18:49:00,667 44k INFO Losses: [2.179001808166504, 2.760432720184326, 14.981268882751465, 22.29775619506836, 0.8354520797729492], step: 59400, lr: 3.9646731235648726e-05, reference_loss: 43.05391311645508 2023-09-17 18:49:45,094 44k INFO ====> Epoch: 1857, cost 59.13 s 2023-09-17 18:50:43,278 44k INFO ====> Epoch: 1858, cost 58.18 s 2023-09-17 18:51:42,233 44k INFO ====> Epoch: 1859, cost 58.96 s 2023-09-17 18:52:40,788 44k INFO ====> Epoch: 1860, cost 58.55 s 2023-09-17 18:53:39,213 44k INFO ====> Epoch: 1861, cost 58.43 s 2023-09-17 18:54:37,498 44k INFO ====> Epoch: 1862, cost 58.29 s 2023-09-17 18:55:07,484 44k INFO Train Epoch: 1863 [47%] 2023-09-17 18:55:07,485 44k INFO Losses: [2.0894877910614014, 2.879209041595459, 15.101669311523438, 22.312929153442383, 0.7741340398788452], step: 59600, lr: 3.961700547787606e-05, reference_loss: 43.15742874145508 2023-09-17 18:55:36,975 44k INFO ====> Epoch: 1863, cost 59.48 s 2023-09-17 18:56:35,411 44k INFO ====> Epoch: 1864, cost 58.44 s 2023-09-17 18:57:33,674 44k INFO ====> Epoch: 1865, cost 58.26 s 2023-09-17 18:58:31,989 44k INFO ====> Epoch: 1866, cost 58.31 s 2023-09-17 18:59:30,585 44k INFO ====> Epoch: 1867, cost 58.60 s 2023-09-17 19:00:28,877 44k INFO ====> Epoch: 1868, cost 58.29 s 2023-09-17 19:01:13,729 44k INFO Train Epoch: 1869 [72%] 2023-09-17 19:01:13,730 44k INFO Losses: [2.1736032962799072, 2.665444850921631, 14.510859489440918, 21.668895721435547, 0.8287925720214844], step: 59800, lr: 3.958730200745591e-05, reference_loss: 41.84759521484375 2023-09-17 19:01:28,159 44k INFO ====> Epoch: 1869, cost 59.28 s 2023-09-17 19:02:26,694 44k INFO ====> Epoch: 1870, cost 58.53 s 2023-09-17 19:03:25,055 44k INFO ====> Epoch: 1871, cost 58.36 s 2023-09-17 19:04:23,456 44k INFO ====> Epoch: 1872, cost 58.40 s 2023-09-17 19:05:22,067 44k INFO ====> Epoch: 1873, cost 58.61 s 2023-09-17 19:06:20,257 44k INFO ====> Epoch: 1874, cost 58.19 s 2023-09-17 19:07:18,386 44k INFO Train Epoch: 1875 [97%] 2023-09-17 19:07:18,388 44k INFO Losses: [1.5977972745895386, 3.284574508666992, 13.054204940795898, 16.528079986572266, -0.5341488718986511], step: 60000, lr: 3.955762080767798e-05, reference_loss: 33.93050765991211 2023-09-17 19:07:35,736 44k INFO Saving model and optimizer state at iteration 1875 to ./logs/44k/G_60000.pth 2023-09-17 19:07:39,148 44k INFO Saving model and optimizer state at iteration 1875 to ./logs/44k/D_60000.pth 2023-09-17 19:07:40,277 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_56000.pth 2023-09-17 19:07:40,279 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_56000.pth 2023-09-17 19:07:40,280 44k INFO ====> Epoch: 1875, cost 80.02 s 2023-09-17 19:08:39,816 44k INFO ====> Epoch: 1876, cost 59.54 s 2023-09-17 19:09:38,720 44k INFO ====> Epoch: 1877, cost 58.90 s 2023-09-17 19:10:38,330 44k INFO ====> Epoch: 1878, cost 59.61 s 2023-09-17 19:11:37,315 44k INFO ====> Epoch: 1879, cost 58.98 s 2023-09-17 19:12:36,978 44k INFO ====> Epoch: 1880, cost 59.66 s 2023-09-17 19:13:36,351 44k INFO ====> Epoch: 1881, cost 59.37 s 2023-09-17 19:13:51,668 44k INFO Train Epoch: 1882 [22%] 2023-09-17 19:13:51,669 44k INFO Losses: [2.159332513809204, 2.8046748638153076, 14.163095474243164, 22.037813186645508, 0.8246472477912903], step: 60200, lr: 3.9523020866611784e-05, reference_loss: 41.98956298828125 2023-09-17 19:14:36,809 44k INFO ====> Epoch: 1882, cost 60.46 s 2023-09-17 19:15:36,102 44k INFO ====> Epoch: 1883, cost 59.29 s 2023-09-17 19:16:35,552 44k INFO ====> Epoch: 1884, cost 59.45 s 2023-09-17 19:17:34,695 44k INFO ====> Epoch: 1885, cost 59.14 s 2023-09-17 19:18:27,562 44k INFO ====> Epoch: 1886, cost 52.87 s 2023-09-17 19:19:14,971 44k INFO ====> Epoch: 1887, cost 47.41 s 2023-09-17 19:19:39,254 44k INFO Train Epoch: 1888 [47%] 2023-09-17 19:19:39,254 44k INFO Losses: [2.083855628967285, 2.703961133956909, 14.944903373718262, 22.195770263671875, 0.7935028672218323], step: 60400, lr: 3.94933878626261e-05, reference_loss: 42.72199249267578 2023-09-17 19:20:02,731 44k INFO ====> Epoch: 1888, cost 47.76 s 2023-09-17 19:20:49,999 44k INFO ====> Epoch: 1889, cost 47.27 s 2023-09-17 19:21:37,346 44k INFO ====> Epoch: 1890, cost 47.35 s 2023-09-17 19:22:24,351 44k INFO ====> Epoch: 1891, cost 47.00 s 2023-09-17 19:23:11,422 44k INFO ====> Epoch: 1892, cost 47.07 s 2023-09-17 19:23:58,330 44k INFO ====> Epoch: 1893, cost 46.91 s 2023-09-17 19:24:34,942 44k INFO Train Epoch: 1894 [72%] 2023-09-17 19:24:34,943 44k INFO Losses: [2.1418423652648926, 2.685828447341919, 14.428901672363281, 21.554401397705078, 0.8228665590286255], step: 60600, lr: 3.9463777076449335e-05, reference_loss: 41.63384246826172 2023-09-17 19:24:46,325 44k INFO ====> Epoch: 1894, cost 48.00 s 2023-09-17 19:25:33,379 44k INFO ====> Epoch: 1895, cost 47.05 s 2023-09-17 19:26:20,623 44k INFO ====> Epoch: 1896, cost 47.24 s 2023-09-17 19:27:08,510 44k INFO ====> Epoch: 1897, cost 47.89 s 2023-09-17 19:27:56,067 44k INFO ====> Epoch: 1898, cost 47.56 s 2023-09-17 19:28:43,295 44k INFO ====> Epoch: 1899, cost 47.23 s 2023-09-17 19:29:30,717 44k INFO Train Epoch: 1900 [97%] 2023-09-17 19:29:30,718 44k INFO Losses: [1.4329674243927002, 3.173757553100586, 12.81321907043457, 16.050561904907227, -0.44373446702957153], step: 60800, lr: 3.943418849142333e-05, reference_loss: 33.026771545410156 2023-09-17 19:29:47,908 44k INFO Saving model and optimizer state at iteration 1900 to ./logs/44k/G_60800.pth 2023-09-17 19:29:51,384 44k INFO Saving model and optimizer state at iteration 1900 to ./logs/44k/D_60800.pth 2023-09-17 19:29:51,900 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_56800.pth 2023-09-17 19:29:51,902 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_56800.pth 2023-09-17 19:29:51,902 44k INFO ====> Epoch: 1900, cost 68.61 s 2023-09-17 19:30:51,348 44k INFO ====> Epoch: 1901, cost 59.45 s 2023-09-17 19:31:50,969 44k INFO ====> Epoch: 1902, cost 59.62 s 2023-09-17 19:32:51,107 44k INFO ====> Epoch: 1903, cost 60.14 s 2023-09-17 19:33:50,823 44k INFO ====> Epoch: 1904, cost 59.72 s 2023-09-17 19:34:50,082 44k INFO ====> Epoch: 1905, cost 59.26 s 2023-09-17 19:35:50,237 44k INFO ====> Epoch: 1906, cost 60.15 s 2023-09-17 19:36:05,514 44k INFO Train Epoch: 1907 [22%] 2023-09-17 19:36:05,515 44k INFO Losses: [2.1097252368927, 2.8386144638061523, 14.76953411102295, 22.191606521606445, 0.8372768759727478], step: 61000, lr: 3.939969651314107e-05, reference_loss: 42.74675369262695 2023-09-17 19:36:50,787 44k INFO ====> Epoch: 1907, cost 60.55 s 2023-09-17 19:37:50,372 44k INFO ====> Epoch: 1908, cost 59.59 s 2023-09-17 19:38:49,809 44k INFO ====> Epoch: 1909, cost 59.44 s 2023-09-17 19:39:48,766 44k INFO ====> Epoch: 1910, cost 58.96 s 2023-09-17 19:40:48,626 44k INFO ====> Epoch: 1911, cost 59.86 s 2023-09-17 19:41:48,642 44k INFO ====> Epoch: 1912, cost 60.02 s 2023-09-17 19:42:19,421 44k INFO Train Epoch: 1913 [47%] 2023-09-17 19:42:19,423 44k INFO Losses: [2.131653070449829, 2.612541675567627, 13.186885833740234, 21.97570037841797, 0.7655084729194641], step: 61200, lr: 3.9370155973521165e-05, reference_loss: 40.67228698730469 2023-09-17 19:42:48,880 44k INFO ====> Epoch: 1913, cost 60.24 s 2023-09-17 19:43:48,445 44k INFO ====> Epoch: 1914, cost 59.57 s 2023-09-17 19:44:48,364 44k INFO ====> Epoch: 1915, cost 59.92 s 2023-09-17 19:45:48,148 44k INFO ====> Epoch: 1916, cost 59.78 s 2023-09-17 19:46:47,960 44k INFO ====> Epoch: 1917, cost 59.81 s 2023-09-17 19:47:47,492 44k INFO ====> Epoch: 1918, cost 59.53 s 2023-09-17 19:48:33,416 44k INFO Train Epoch: 1919 [72%] 2023-09-17 19:48:33,417 44k INFO Losses: [2.1495649814605713, 2.7005343437194824, 13.699087142944336, 21.51883316040039, 0.8323999047279358], step: 61400, lr: 3.934063758238356e-05, reference_loss: 40.900421142578125 2023-09-17 19:48:47,847 44k INFO ====> Epoch: 1919, cost 60.36 s 2023-09-17 19:49:47,136 44k INFO ====> Epoch: 1920, cost 59.29 s 2023-09-17 19:50:46,357 44k INFO ====> Epoch: 1921, cost 59.22 s 2023-09-17 19:51:45,962 44k INFO ====> Epoch: 1922, cost 59.60 s 2023-09-17 19:52:45,126 44k INFO ====> Epoch: 1923, cost 59.16 s 2023-09-17 19:53:44,687 44k INFO ====> Epoch: 1924, cost 59.56 s 2023-09-17 19:54:44,275 44k INFO Train Epoch: 1925 [97%] 2023-09-17 19:54:44,277 44k INFO Losses: [1.3346059322357178, 3.2142155170440674, 12.43571949005127, 16.15402603149414, -0.49033600091934204], step: 61600, lr: 3.931114132312209e-05, reference_loss: 32.64822769165039 2023-09-17 19:55:01,959 44k INFO Saving model and optimizer state at iteration 1925 to ./logs/44k/G_61600.pth 2023-09-17 19:55:05,078 44k INFO Saving model and optimizer state at iteration 1925 to ./logs/44k/D_61600.pth 2023-09-17 19:55:06,329 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_57600.pth 2023-09-17 19:55:06,332 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_57600.pth 2023-09-17 19:55:06,333 44k INFO ====> Epoch: 1925, cost 81.65 s 2023-09-17 19:56:05,925 44k INFO ====> Epoch: 1926, cost 59.59 s 2023-09-17 19:57:05,455 44k INFO ====> Epoch: 1927, cost 59.53 s 2023-09-17 19:58:05,255 44k INFO ====> Epoch: 1928, cost 59.80 s 2023-09-17 19:59:04,651 44k INFO ====> Epoch: 1929, cost 59.40 s 2023-09-17 20:00:03,994 44k INFO ====> Epoch: 1930, cost 59.34 s 2023-09-17 20:01:03,613 44k INFO ====> Epoch: 1931, cost 59.62 s 2023-09-17 20:01:18,693 44k INFO Train Epoch: 1932 [22%] 2023-09-17 20:01:18,694 44k INFO Losses: [2.1771461963653564, 2.802293062210083, 14.999408721923828, 22.07234001159668, 0.8378849029541016], step: 61800, lr: 3.927675697074564e-05, reference_loss: 42.889068603515625 2023-09-17 20:02:03,875 44k INFO ====> Epoch: 1932, cost 60.26 s 2023-09-17 20:03:02,677 44k INFO ====> Epoch: 1933, cost 58.80 s 2023-09-17 20:04:02,179 44k INFO ====> Epoch: 1934, cost 59.50 s 2023-09-17 20:05:02,433 44k INFO ====> Epoch: 1935, cost 60.25 s 2023-09-17 20:06:01,991 44k INFO ====> Epoch: 1936, cost 59.56 s 2023-09-17 20:07:01,389 44k INFO ====> Epoch: 1937, cost 59.40 s 2023-09-17 20:07:31,811 44k INFO Train Epoch: 1938 [47%] 2023-09-17 20:07:31,812 44k INFO Losses: [2.1431500911712646, 2.7893168926239014, 14.693530082702637, 22.09064483642578, 0.7743191719055176], step: 62000, lr: 3.924730860697338e-05, reference_loss: 42.490962982177734 2023-09-17 20:08:01,749 44k INFO ====> Epoch: 1938, cost 60.36 s 2023-09-17 20:09:00,864 44k INFO ====> Epoch: 1939, cost 59.12 s 2023-09-17 20:10:00,589 44k INFO ====> Epoch: 1940, cost 59.72 s 2023-09-17 20:10:59,524 44k INFO ====> Epoch: 1941, cost 58.94 s 2023-09-17 20:11:58,785 44k INFO ====> Epoch: 1942, cost 59.26 s 2023-09-17 20:12:57,891 44k INFO ====> Epoch: 1943, cost 59.11 s 2023-09-17 20:13:43,710 44k INFO Train Epoch: 1944 [72%] 2023-09-17 20:13:43,711 44k INFO Losses: [2.135378360748291, 2.773226499557495, 14.387746810913086, 21.386751174926758, 0.8222253918647766], step: 62200, lr: 3.921788232257314e-05, reference_loss: 41.50532913208008 2023-09-17 20:13:58,336 44k INFO ====> Epoch: 1944, cost 60.45 s 2023-09-17 20:14:57,593 44k INFO ====> Epoch: 1945, cost 59.26 s 2023-09-17 20:15:57,119 44k INFO ====> Epoch: 1946, cost 59.53 s 2023-09-17 20:16:56,202 44k INFO ====> Epoch: 1947, cost 59.08 s 2023-09-17 20:17:55,489 44k INFO ====> Epoch: 1948, cost 59.29 s 2023-09-17 20:18:54,520 44k INFO ====> Epoch: 1949, cost 59.03 s 2023-09-17 20:19:54,116 44k INFO Train Epoch: 1950 [97%] 2023-09-17 20:19:54,117 44k INFO Losses: [1.566652536392212, 2.985023260116577, 11.207581520080566, 15.424947738647461, -0.5884524583816528], step: 62400, lr: 3.918847810099056e-05, reference_loss: 30.59575080871582 2023-09-17 20:20:11,679 44k INFO Saving model and optimizer state at iteration 1950 to ./logs/44k/G_62400.pth 2023-09-17 20:20:15,216 44k INFO Saving model and optimizer state at iteration 1950 to ./logs/44k/D_62400.pth 2023-09-17 20:20:15,929 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_58400.pth 2023-09-17 20:20:15,930 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_58400.pth 2023-09-17 20:20:15,930 44k INFO ====> Epoch: 1950, cost 81.41 s 2023-09-17 20:21:15,837 44k INFO ====> Epoch: 1951, cost 59.91 s 2023-09-17 20:22:15,679 44k INFO ====> Epoch: 1952, cost 59.84 s 2023-09-17 20:23:15,525 44k INFO ====> Epoch: 1953, cost 59.85 s 2023-09-17 20:24:15,028 44k INFO ====> Epoch: 1954, cost 59.50 s 2023-09-17 20:25:14,524 44k INFO ====> Epoch: 1955, cost 59.50 s 2023-09-17 20:26:13,418 44k INFO ====> Epoch: 1956, cost 58.89 s 2023-09-17 20:26:28,469 44k INFO Train Epoch: 1957 [22%] 2023-09-17 20:26:28,469 44k INFO Losses: [2.110640525817871, 2.971642255783081, 14.5835599899292, 21.910476684570312, 0.8186944723129272], step: 62600, lr: 3.9154201038692995e-05, reference_loss: 42.395015716552734 2023-09-17 20:27:13,555 44k INFO ====> Epoch: 1957, cost 60.14 s 2023-09-17 20:28:12,559 44k INFO ====> Epoch: 1958, cost 59.00 s 2023-09-17 20:29:11,979 44k INFO ====> Epoch: 1959, cost 59.42 s 2023-09-17 20:30:12,121 44k INFO ====> Epoch: 1960, cost 60.14 s 2023-09-17 20:31:11,861 44k INFO ====> Epoch: 1961, cost 59.74 s 2023-09-17 20:32:11,594 44k INFO ====> Epoch: 1962, cost 59.73 s 2023-09-17 20:32:41,781 44k INFO Train Epoch: 1963 [47%] 2023-09-17 20:32:41,782 44k INFO Losses: [2.1026148796081543, 2.723895311355591, 13.593849182128906, 22.01645851135254, 0.7777949571609497], step: 62800, lr: 3.912484456315051e-05, reference_loss: 41.21461486816406 2023-09-17 20:33:11,348 44k INFO ====> Epoch: 1963, cost 59.75 s 2023-09-17 20:34:10,851 44k INFO ====> Epoch: 1964, cost 59.50 s 2023-09-17 20:35:10,051 44k INFO ====> Epoch: 1965, cost 59.20 s 2023-09-17 20:36:09,425 44k INFO ====> Epoch: 1966, cost 59.37 s 2023-09-17 20:37:08,642 44k INFO ====> Epoch: 1967, cost 59.22 s 2023-09-17 20:38:07,951 44k INFO ====> Epoch: 1968, cost 59.31 s 2023-09-17 20:38:53,846 44k INFO Train Epoch: 1969 [72%] 2023-09-17 20:38:53,847 44k INFO Losses: [2.1239519119262695, 2.7621426582336426, 14.237062454223633, 21.461536407470703, 0.8236834406852722], step: 63000, lr: 3.909551009808541e-05, reference_loss: 41.40837860107422 2023-09-17 20:39:08,185 44k INFO ====> Epoch: 1969, cost 60.23 s 2023-09-17 20:40:07,419 44k INFO ====> Epoch: 1970, cost 59.23 s 2023-09-17 20:41:06,772 44k INFO ====> Epoch: 1971, cost 59.35 s 2023-09-17 20:42:05,984 44k INFO ====> Epoch: 1972, cost 59.21 s 2023-09-17 20:43:06,000 44k INFO ====> Epoch: 1973, cost 60.02 s 2023-09-17 20:44:05,266 44k INFO ====> Epoch: 1974, cost 59.27 s 2023-09-17 20:45:04,477 44k INFO Train Epoch: 1975 [97%] 2023-09-17 20:45:04,478 44k INFO Losses: [1.5588774681091309, 3.1056108474731445, 9.696378707885742, 16.231046676635742, -0.5633265972137451], step: 63200, lr: 3.9066197626994984e-05, reference_loss: 30.028587341308594 2023-09-17 20:45:21,372 44k INFO Saving model and optimizer state at iteration 1975 to ./logs/44k/G_63200.pth 2023-09-17 20:45:24,139 44k INFO Saving model and optimizer state at iteration 1975 to ./logs/44k/D_63200.pth 2023-09-17 20:45:24,657 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_59200.pth 2023-09-17 20:45:24,659 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_59200.pth 2023-09-17 20:45:24,659 44k INFO ====> Epoch: 1975, cost 79.39 s 2023-09-17 20:46:23,929 44k INFO ====> Epoch: 1976, cost 59.27 s 2023-09-17 20:47:23,291 44k INFO ====> Epoch: 1977, cost 59.36 s 2023-09-17 20:48:23,116 44k INFO ====> Epoch: 1978, cost 59.82 s 2023-09-17 20:49:22,599 44k INFO ====> Epoch: 1979, cost 59.48 s 2023-09-17 20:50:22,760 44k INFO ====> Epoch: 1980, cost 60.16 s 2023-09-17 20:51:22,265 44k INFO ====> Epoch: 1981, cost 59.51 s 2023-09-17 20:51:37,488 44k INFO Train Epoch: 1982 [22%] 2023-09-17 20:51:37,489 44k INFO Losses: [2.0999398231506348, 2.905608892440796, 14.60103988647461, 22.116308212280273, 0.8346441984176636], step: 63400, lr: 3.903202751999725e-05, reference_loss: 42.55754089355469 2023-09-17 20:52:22,544 44k INFO ====> Epoch: 1982, cost 60.28 s 2023-09-17 20:53:21,519 44k INFO ====> Epoch: 1983, cost 58.97 s 2023-09-17 20:54:21,119 44k INFO ====> Epoch: 1984, cost 59.60 s 2023-09-17 20:55:20,354 44k INFO ====> Epoch: 1985, cost 59.24 s 2023-09-17 20:56:19,980 44k INFO ====> Epoch: 1986, cost 59.63 s 2023-09-17 20:57:19,332 44k INFO ====> Epoch: 1987, cost 59.35 s 2023-09-17 20:57:49,751 44k INFO Train Epoch: 1988 [47%] 2023-09-17 20:57:49,752 44k INFO Losses: [2.0669474601745605, 2.805835008621216, 15.234709739685059, 22.081260681152344, 0.7744899392127991], step: 63600, lr: 3.9002762645964146e-05, reference_loss: 42.96324157714844 2023-09-17 20:58:19,103 44k INFO ====> Epoch: 1988, cost 59.77 s 2023-09-17 20:59:18,063 44k INFO ====> Epoch: 1989, cost 58.96 s 2023-09-17 21:00:17,417 44k INFO ====> Epoch: 1990, cost 59.35 s 2023-09-17 21:01:17,333 44k INFO ====> Epoch: 1991, cost 59.92 s 2023-09-17 21:02:16,660 44k INFO ====> Epoch: 1992, cost 59.33 s 2023-09-17 21:03:16,540 44k INFO ====> Epoch: 1993, cost 59.88 s 2023-09-17 21:04:02,202 44k INFO Train Epoch: 1994 [72%] 2023-09-17 21:04:02,203 44k INFO Losses: [2.071631908416748, 2.7586872577667236, 15.563602447509766, 21.758819580078125, 0.8373624086380005], step: 63800, lr: 3.897351971372875e-05, reference_loss: 42.99010467529297 2023-09-17 21:04:16,788 44k INFO ====> Epoch: 1994, cost 60.25 s 2023-09-17 21:05:15,929 44k INFO ====> Epoch: 1995, cost 59.14 s 2023-09-17 21:06:15,681 44k INFO ====> Epoch: 1996, cost 59.75 s 2023-09-17 21:07:14,917 44k INFO ====> Epoch: 1997, cost 59.24 s 2023-09-17 21:08:14,628 44k INFO ====> Epoch: 1998, cost 59.71 s 2023-09-17 21:09:13,889 44k INFO ====> Epoch: 1999, cost 59.26 s 2023-09-17 21:10:13,134 44k INFO Train Epoch: 2000 [97%] 2023-09-17 21:10:13,135 44k INFO Losses: [1.5388123989105225, 3.1503074169158936, 10.944232940673828, 16.113767623901367, -0.5837714672088623], step: 64000, lr: 3.894429870683987e-05, reference_loss: 31.163349151611328 2023-09-17 21:10:28,835 44k INFO Saving model and optimizer state at iteration 2000 to ./logs/44k/G_64000.pth 2023-09-17 21:10:32,442 44k INFO Saving model and optimizer state at iteration 2000 to ./logs/44k/D_64000.pth 2023-09-17 21:10:33,102 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_60000.pth 2023-09-17 21:10:33,104 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_60000.pth 2023-09-17 21:10:33,104 44k INFO ====> Epoch: 2000, cost 79.21 s 2023-09-17 21:11:32,604 44k INFO ====> Epoch: 2001, cost 59.50 s 2023-09-17 21:12:32,158 44k INFO ====> Epoch: 2002, cost 59.55 s 2023-09-17 21:13:31,337 44k INFO ====> Epoch: 2003, cost 59.18 s 2023-09-17 21:14:31,088 44k INFO ====> Epoch: 2004, cost 59.75 s 2023-09-17 21:15:30,945 44k INFO ====> Epoch: 2005, cost 59.86 s 2023-09-17 21:16:30,188 44k INFO ====> Epoch: 2006, cost 59.24 s 2023-09-17 21:16:45,074 44k INFO Train Epoch: 2007 [22%] 2023-09-17 21:16:45,075 44k INFO Losses: [2.1778204441070557, 2.8961727619171143, 15.223419189453125, 22.25122833251953, 0.834793746471405], step: 64200, lr: 3.8910235221407516e-05, reference_loss: 43.3834342956543 2023-09-17 21:17:30,200 44k INFO ====> Epoch: 2007, cost 60.01 s 2023-09-17 21:18:29,302 44k INFO ====> Epoch: 2008, cost 59.10 s 2023-09-17 21:19:29,150 44k INFO ====> Epoch: 2009, cost 59.85 s 2023-09-17 21:20:29,671 44k INFO ====> Epoch: 2010, cost 60.52 s 2023-09-17 21:21:28,745 44k INFO ====> Epoch: 2011, cost 59.07 s 2023-09-17 21:22:28,808 44k INFO ====> Epoch: 2012, cost 60.06 s 2023-09-17 21:22:59,471 44k INFO Train Epoch: 2013 [47%] 2023-09-17 21:22:59,472 44k INFO Losses: [2.102344036102295, 2.7850217819213867, 14.238021850585938, 21.956443786621094, 0.8045037388801575], step: 64400, lr: 3.888106166305804e-05, reference_loss: 41.88633728027344 2023-09-17 21:23:28,876 44k INFO ====> Epoch: 2013, cost 60.07 s 2023-09-17 21:24:28,082 44k INFO ====> Epoch: 2014, cost 59.21 s 2023-09-17 21:25:27,553 44k INFO ====> Epoch: 2015, cost 59.47 s 2023-09-17 21:26:27,014 44k INFO ====> Epoch: 2016, cost 59.46 s 2023-09-17 21:27:26,620 44k INFO ====> Epoch: 2017, cost 59.61 s 2023-09-17 21:28:25,532 44k INFO ====> Epoch: 2018, cost 58.91 s 2023-09-17 21:29:10,465 44k INFO Train Epoch: 2019 [72%] 2023-09-17 21:29:10,466 44k INFO Losses: [2.197561740875244, 2.6270694732666016, 14.317267417907715, 21.509536743164062, 0.8296830654144287], step: 64600, lr: 3.885190997804091e-05, reference_loss: 41.481117248535156 2023-09-17 21:29:24,503 44k INFO ====> Epoch: 2019, cost 58.97 s 2023-09-17 21:30:23,179 44k INFO ====> Epoch: 2020, cost 58.68 s 2023-09-17 21:31:21,756 44k INFO ====> Epoch: 2021, cost 58.58 s 2023-09-17 21:32:20,168 44k INFO ====> Epoch: 2022, cost 58.41 s 2023-09-17 21:33:18,790 44k INFO ====> Epoch: 2023, cost 58.62 s 2023-09-17 21:34:17,651 44k INFO ====> Epoch: 2024, cost 58.86 s 2023-09-17 21:35:16,677 44k INFO Train Epoch: 2025 [97%] 2023-09-17 21:35:16,678 44k INFO Losses: [1.5746723413467407, 3.3419950008392334, 13.765741348266602, 16.731414794921875, -0.6190206408500671], step: 64800, lr: 3.882278014995626e-05, reference_loss: 34.794803619384766 2023-09-17 21:35:32,477 44k INFO Saving model and optimizer state at iteration 2025 to ./logs/44k/G_64800.pth 2023-09-17 21:35:35,893 44k INFO Saving model and optimizer state at iteration 2025 to ./logs/44k/D_64800.pth 2023-09-17 21:35:36,451 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_60800.pth 2023-09-17 21:35:36,453 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_60800.pth 2023-09-17 21:35:36,453 44k INFO ====> Epoch: 2025, cost 78.80 s 2023-09-17 21:36:34,548 44k INFO ====> Epoch: 2026, cost 58.09 s 2023-09-17 21:37:32,797 44k INFO ====> Epoch: 2027, cost 58.25 s 2023-09-17 21:38:31,853 44k INFO ====> Epoch: 2028, cost 59.06 s 2023-09-17 21:39:31,269 44k INFO ====> Epoch: 2029, cost 59.42 s 2023-09-17 21:40:29,753 44k INFO ====> Epoch: 2030, cost 58.48 s 2023-09-17 21:41:27,995 44k INFO ====> Epoch: 2031, cost 58.24 s 2023-09-17 21:41:42,732 44k INFO Train Epoch: 2032 [22%] 2023-09-17 21:41:42,734 44k INFO Losses: [2.1402041912078857, 3.006481409072876, 14.839184761047363, 22.054134368896484, 0.8133116364479065], step: 65000, lr: 3.878882295339621e-05, reference_loss: 42.85331726074219 2023-09-17 21:42:27,178 44k INFO ====> Epoch: 2032, cost 59.18 s 2023-09-17 21:43:26,192 44k INFO ====> Epoch: 2033, cost 59.01 s 2023-09-17 21:44:24,882 44k INFO ====> Epoch: 2034, cost 58.69 s 2023-09-17 21:45:23,337 44k INFO ====> Epoch: 2035, cost 58.46 s 2023-09-17 21:46:22,455 44k INFO ====> Epoch: 2036, cost 59.12 s 2023-09-17 21:47:21,033 44k INFO ====> Epoch: 2037, cost 58.58 s 2023-09-17 21:47:51,028 44k INFO Train Epoch: 2038 [47%] 2023-09-17 21:47:51,029 44k INFO Losses: [2.146920680999756, 2.7527523040771484, 14.09073543548584, 21.878334045410156, 0.7613791823387146], step: 65200, lr: 3.875974042579649e-05, reference_loss: 41.630123138427734 2023-09-17 21:48:20,682 44k INFO ====> Epoch: 2038, cost 59.65 s 2023-09-17 21:49:19,268 44k INFO ====> Epoch: 2039, cost 58.59 s 2023-09-17 21:50:18,092 44k INFO ====> Epoch: 2040, cost 58.82 s 2023-09-17 21:51:17,094 44k INFO ====> Epoch: 2041, cost 59.00 s 2023-09-17 21:52:16,110 44k INFO ====> Epoch: 2042, cost 59.02 s 2023-09-17 21:53:14,406 44k INFO ====> Epoch: 2043, cost 58.30 s 2023-09-17 21:54:00,049 44k INFO Train Epoch: 2044 [72%] 2023-09-17 21:54:00,050 44k INFO Losses: [2.1304304599761963, 2.7035562992095947, 14.562763214111328, 21.704193115234375, 0.8229109048843384], step: 65400, lr: 3.873067970327738e-05, reference_loss: 41.92385482788086 2023-09-17 21:54:14,210 44k INFO ====> Epoch: 2044, cost 59.80 s 2023-09-17 21:55:12,511 44k INFO ====> Epoch: 2045, cost 58.30 s 2023-09-17 21:56:11,030 44k INFO ====> Epoch: 2046, cost 58.52 s 2023-09-17 21:57:09,510 44k INFO ====> Epoch: 2047, cost 58.48 s 2023-09-17 21:58:08,195 44k INFO ====> Epoch: 2048, cost 58.69 s 2023-09-17 21:59:07,128 44k INFO ====> Epoch: 2049, cost 58.93 s 2023-09-17 22:00:05,822 44k INFO Train Epoch: 2050 [97%] 2023-09-17 22:00:05,823 44k INFO Losses: [1.6418997049331665, 3.097229480743408, 13.5366849899292, 16.18768310546875, -0.613948404788971], step: 65600, lr: 3.8701640769490196e-05, reference_loss: 33.84954833984375 2023-09-17 22:00:21,584 44k INFO Saving model and optimizer state at iteration 2050 to ./logs/44k/G_65600.pth 2023-09-17 22:00:24,650 44k INFO Saving model and optimizer state at iteration 2050 to ./logs/44k/D_65600.pth 2023-09-17 22:00:25,883 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_61600.pth 2023-09-17 22:00:25,888 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_61600.pth 2023-09-17 22:00:25,889 44k INFO ====> Epoch: 2050, cost 78.76 s 2023-09-17 22:01:25,023 44k INFO ====> Epoch: 2051, cost 59.13 s 2023-09-17 22:02:23,705 44k INFO ====> Epoch: 2052, cost 58.68 s 2023-09-17 22:03:22,640 44k INFO ====> Epoch: 2053, cost 58.94 s 2023-09-17 22:04:21,924 44k INFO ====> Epoch: 2054, cost 59.28 s 2023-09-17 22:05:20,491 44k INFO ====> Epoch: 2055, cost 58.57 s 2023-09-17 22:06:19,293 44k INFO ====> Epoch: 2056, cost 58.80 s 2023-09-17 22:06:34,209 44k INFO Train Epoch: 2057 [22%] 2023-09-17 22:06:34,210 44k INFO Losses: [2.073171377182007, 2.8298938274383545, 15.706947326660156, 22.132205963134766, 0.8260788917541504], step: 65800, lr: 3.866778953014746e-05, reference_loss: 43.56829833984375 2023-09-17 22:07:18,503 44k INFO ====> Epoch: 2057, cost 59.21 s 2023-09-17 22:08:17,513 44k INFO ====> Epoch: 2058, cost 59.01 s 2023-09-17 22:09:16,436 44k INFO ====> Epoch: 2059, cost 58.92 s 2023-09-17 22:10:15,232 44k INFO ====> Epoch: 2060, cost 58.80 s 2023-09-17 22:11:14,007 44k INFO ====> Epoch: 2061, cost 58.77 s 2023-09-17 22:12:12,922 44k INFO ====> Epoch: 2062, cost 58.91 s 2023-09-17 22:12:42,757 44k INFO Train Epoch: 2063 [47%] 2023-09-17 22:12:42,758 44k INFO Losses: [2.023326873779297, 2.876437187194824, 15.14238452911377, 21.989839553833008, 0.7630974650382996], step: 66000, lr: 3.863879774925269e-05, reference_loss: 42.79508590698242 2023-09-17 22:13:12,108 44k INFO ====> Epoch: 2063, cost 59.19 s 2023-09-17 22:14:10,739 44k INFO ====> Epoch: 2064, cost 58.63 s 2023-09-17 22:15:09,511 44k INFO ====> Epoch: 2065, cost 58.77 s 2023-09-17 22:16:08,530 44k INFO ====> Epoch: 2066, cost 59.02 s 2023-09-17 22:17:07,099 44k INFO ====> Epoch: 2067, cost 58.57 s 2023-09-17 22:18:06,020 44k INFO ====> Epoch: 2068, cost 58.92 s 2023-09-17 22:18:51,093 44k INFO Train Epoch: 2069 [72%] 2023-09-17 22:18:51,094 44k INFO Losses: [2.1184356212615967, 2.801332950592041, 14.58930492401123, 21.48724937438965, 0.8418854475021362], step: 66200, lr: 3.8609827705399767e-05, reference_loss: 41.83820724487305 2023-09-17 22:19:05,546 44k INFO ====> Epoch: 2069, cost 59.53 s 2023-09-17 22:20:04,833 44k INFO ====> Epoch: 2070, cost 59.29 s 2023-09-17 22:21:03,633 44k INFO ====> Epoch: 2071, cost 58.80 s 2023-09-17 22:22:02,591 44k INFO ====> Epoch: 2072, cost 58.96 s 2023-09-17 22:23:01,289 44k INFO ====> Epoch: 2073, cost 58.70 s 2023-09-17 22:24:00,376 44k INFO ====> Epoch: 2074, cost 59.09 s 2023-09-17 22:24:59,421 44k INFO Train Epoch: 2075 [97%] 2023-09-17 22:24:59,422 44k INFO Losses: [1.6365751028060913, 3.1652255058288574, 13.9056978225708, 15.954188346862793, -0.6863560676574707], step: 66400, lr: 3.858087938229102e-05, reference_loss: 33.9753303527832 2023-09-17 22:25:16,416 44k INFO Saving model and optimizer state at iteration 2075 to ./logs/44k/G_66400.pth 2023-09-17 22:25:19,813 44k INFO Saving model and optimizer state at iteration 2075 to ./logs/44k/D_66400.pth 2023-09-17 22:25:20,922 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_62400.pth 2023-09-17 22:25:20,924 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_62400.pth 2023-09-17 22:25:20,924 44k INFO ====> Epoch: 2075, cost 80.55 s 2023-09-17 22:26:19,974 44k INFO ====> Epoch: 2076, cost 59.05 s 2023-09-17 22:27:18,528 44k INFO ====> Epoch: 2077, cost 58.55 s 2023-09-17 22:28:17,051 44k INFO ====> Epoch: 2078, cost 58.52 s 2023-09-17 22:29:15,654 44k INFO ====> Epoch: 2079, cost 58.60 s 2023-09-17 22:30:14,746 44k INFO ====> Epoch: 2080, cost 59.09 s 2023-09-17 22:31:13,727 44k INFO ====> Epoch: 2081, cost 58.98 s 2023-09-17 22:31:28,431 44k INFO Train Epoch: 2082 [22%] 2023-09-17 22:31:28,432 44k INFO Losses: [2.102818489074707, 2.8562088012695312, 15.814887046813965, 22.044679641723633, 0.8053243160247803], step: 66600, lr: 3.854713376954552e-05, reference_loss: 43.62392044067383 2023-09-17 22:32:12,620 44k INFO ====> Epoch: 2082, cost 58.89 s 2023-09-17 22:33:11,522 44k INFO ====> Epoch: 2083, cost 58.90 s 2023-09-17 22:34:10,655 44k INFO ====> Epoch: 2084, cost 59.13 s 2023-09-17 22:35:08,696 44k INFO ====> Epoch: 2085, cost 58.04 s 2023-09-17 22:36:07,109 44k INFO ====> Epoch: 2086, cost 58.41 s 2023-09-17 22:37:05,773 44k INFO ====> Epoch: 2087, cost 58.66 s 2023-09-17 22:37:35,684 44k INFO Train Epoch: 2088 [47%] 2023-09-17 22:37:35,685 44k INFO Losses: [2.063547372817993, 2.77974009513855, 15.159147262573242, 22.224029541015625, 0.7620710134506226], step: 66800, lr: 3.851823245219722e-05, reference_loss: 42.98853302001953 2023-09-17 22:38:05,134 44k INFO ====> Epoch: 2088, cost 59.36 s 2023-09-17 22:39:04,231 44k INFO ====> Epoch: 2089, cost 59.10 s 2023-09-17 22:40:03,026 44k INFO ====> Epoch: 2090, cost 58.79 s 2023-09-17 22:41:01,436 44k INFO ====> Epoch: 2091, cost 58.41 s 2023-09-17 22:42:00,187 44k INFO ====> Epoch: 2092, cost 58.75 s 2023-09-17 22:42:58,943 44k INFO ====> Epoch: 2093, cost 58.76 s 2023-09-17 22:43:44,060 44k INFO Train Epoch: 2094 [72%] 2023-09-17 22:43:44,061 44k INFO Losses: [2.1207785606384277, 2.6999151706695557, 15.401117324829102, 21.60333251953125, 0.8322130441665649], step: 67000, lr: 3.848935280406431e-05, reference_loss: 42.65735626220703 2023-09-17 22:43:58,311 44k INFO ====> Epoch: 2094, cost 59.37 s 2023-09-17 22:44:56,891 44k INFO ====> Epoch: 2095, cost 58.58 s 2023-09-17 22:45:55,572 44k INFO ====> Epoch: 2096, cost 58.68 s 2023-09-17 22:46:54,252 44k INFO ====> Epoch: 2097, cost 58.68 s 2023-09-17 22:47:53,168 44k INFO ====> Epoch: 2098, cost 58.92 s 2023-09-17 22:48:51,763 44k INFO ====> Epoch: 2099, cost 58.59 s 2023-09-17 22:49:50,072 44k INFO Train Epoch: 2100 [97%] 2023-09-17 22:49:50,073 44k INFO Losses: [1.4977259635925293, 3.2647790908813477, 12.464496612548828, 15.755260467529297, -0.6748818755149841], step: 67200, lr: 3.846049480889997e-05, reference_loss: 32.30738067626953 2023-09-17 22:50:08,178 44k INFO Saving model and optimizer state at iteration 2100 to ./logs/44k/G_67200.pth 2023-09-17 22:50:10,393 44k INFO Saving model and optimizer state at iteration 2100 to ./logs/44k/D_67200.pth 2023-09-17 22:50:10,919 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_63200.pth 2023-09-17 22:50:10,921 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_63200.pth 2023-09-17 22:50:10,921 44k INFO ====> Epoch: 2100, cost 79.16 s 2023-09-17 22:51:09,899 44k INFO ====> Epoch: 2101, cost 58.98 s 2023-09-17 22:52:08,678 44k INFO ====> Epoch: 2102, cost 58.78 s 2023-09-17 22:53:07,376 44k INFO ====> Epoch: 2103, cost 58.70 s 2023-09-17 22:54:06,448 44k INFO ====> Epoch: 2104, cost 59.07 s 2023-09-17 22:55:05,017 44k INFO ====> Epoch: 2105, cost 58.57 s 2023-09-17 22:56:03,411 44k INFO ====> Epoch: 2106, cost 58.39 s 2023-09-17 22:56:18,213 44k INFO Train Epoch: 2107 [22%] 2023-09-17 22:56:18,214 44k INFO Losses: [2.1181538105010986, 2.876126289367676, 15.702056884765625, 21.999046325683594, 0.8139902949333191], step: 67400, lr: 3.8426854493163226e-05, reference_loss: 43.50937271118164 2023-09-17 22:57:02,562 44k INFO ====> Epoch: 2107, cost 59.15 s 2023-09-17 22:58:01,284 44k INFO ====> Epoch: 2108, cost 58.72 s 2023-09-17 22:58:59,847 44k INFO ====> Epoch: 2109, cost 58.56 s 2023-09-17 22:59:59,184 44k INFO ====> Epoch: 2110, cost 59.34 s 2023-09-17 23:00:57,245 44k INFO ====> Epoch: 2111, cost 58.06 s 2023-09-17 23:01:55,741 44k INFO ====> Epoch: 2112, cost 58.50 s 2023-09-17 23:02:25,692 44k INFO Train Epoch: 2113 [47%] 2023-09-17 23:02:25,693 44k INFO Losses: [2.088092088699341, 2.7509920597076416, 15.228827476501465, 22.166738510131836, 0.7825416922569275], step: 67600, lr: 3.839804335708647e-05, reference_loss: 43.01719284057617 2023-09-17 23:02:55,145 44k INFO ====> Epoch: 2113, cost 59.40 s 2023-09-17 23:03:53,731 44k INFO ====> Epoch: 2114, cost 58.59 s 2023-09-17 23:04:52,660 44k INFO ====> Epoch: 2115, cost 58.93 s 2023-09-17 23:05:51,036 44k INFO ====> Epoch: 2116, cost 58.38 s 2023-09-17 23:06:50,451 44k INFO ====> Epoch: 2117, cost 59.42 s 2023-09-17 23:07:49,015 44k INFO ====> Epoch: 2118, cost 58.56 s 2023-09-17 23:08:34,463 44k INFO Train Epoch: 2119 [72%] 2023-09-17 23:08:34,464 44k INFO Losses: [2.0844826698303223, 2.6382296085357666, 14.019429206848145, 21.39937973022461, 0.8050697445869446], step: 67800, lr: 3.8369253822610274e-05, reference_loss: 40.946590423583984 2023-09-17 23:08:49,029 44k INFO ====> Epoch: 2119, cost 60.01 s 2023-09-17 23:09:48,014 44k INFO ====> Epoch: 2120, cost 58.99 s 2023-09-17 23:10:46,976 44k INFO ====> Epoch: 2121, cost 58.96 s 2023-09-17 23:11:45,579 44k INFO ====> Epoch: 2122, cost 58.60 s 2023-09-17 23:12:43,961 44k INFO ====> Epoch: 2123, cost 58.38 s 2023-09-17 23:13:43,069 44k INFO ====> Epoch: 2124, cost 59.11 s 2023-09-17 23:14:41,652 44k INFO Train Epoch: 2125 [97%] 2023-09-17 23:14:41,653 44k INFO Losses: [1.5064473152160645, 3.1874499320983887, 13.363829612731934, 16.04973602294922, -0.6782575845718384], step: 68000, lr: 3.8340485873538515e-05, reference_loss: 33.42920684814453 2023-09-17 23:14:59,113 44k INFO Saving model and optimizer state at iteration 2125 to ./logs/44k/G_68000.pth 2023-09-17 23:15:02,514 44k INFO Saving model and optimizer state at iteration 2125 to ./logs/44k/D_68000.pth 2023-09-17 23:15:03,616 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_64000.pth 2023-09-17 23:15:03,618 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_64000.pth 2023-09-17 23:15:03,618 44k INFO ====> Epoch: 2125, cost 80.55 s 2023-09-17 23:16:02,285 44k INFO ====> Epoch: 2126, cost 58.67 s 2023-09-17 23:17:01,066 44k INFO ====> Epoch: 2127, cost 58.78 s 2023-09-17 23:17:59,526 44k INFO ====> Epoch: 2128, cost 58.46 s 2023-09-17 23:18:58,215 44k INFO ====> Epoch: 2129, cost 58.69 s 2023-09-17 23:19:57,115 44k INFO ====> Epoch: 2130, cost 58.90 s 2023-09-17 23:20:55,952 44k INFO ====> Epoch: 2131, cost 58.84 s 2023-09-17 23:21:10,897 44k INFO Train Epoch: 2132 [22%] 2023-09-17 23:21:10,898 44k INFO Losses: [2.102175235748291, 2.9281680583953857, 14.99245834350586, 22.19490623474121, 0.8205403089523315], step: 68200, lr: 3.830695052625048e-05, reference_loss: 43.038246154785156 2023-09-17 23:21:55,722 44k INFO ====> Epoch: 2132, cost 59.77 s 2023-09-17 23:22:53,767 44k INFO ====> Epoch: 2133, cost 58.04 s 2023-09-17 23:23:52,411 44k INFO ====> Epoch: 2134, cost 58.64 s 2023-09-17 23:24:50,441 44k INFO ====> Epoch: 2135, cost 58.03 s 2023-09-17 23:25:49,156 44k INFO ====> Epoch: 2136, cost 58.71 s 2023-09-17 23:26:48,109 44k INFO ====> Epoch: 2137, cost 58.95 s 2023-09-17 23:27:18,332 44k INFO Train Epoch: 2138 [47%] 2023-09-17 23:27:18,333 44k INFO Losses: [2.0869193077087402, 2.728869915008545, 15.002304077148438, 22.17856216430664, 0.7409511804580688], step: 68400, lr: 3.827822929005109e-05, reference_loss: 42.737606048583984 2023-09-17 23:27:47,478 44k INFO ====> Epoch: 2138, cost 59.37 s 2023-09-17 23:28:45,533 44k INFO ====> Epoch: 2139, cost 58.06 s 2023-09-17 23:29:44,200 44k INFO ====> Epoch: 2140, cost 58.67 s 2023-09-17 23:30:42,545 44k INFO ====> Epoch: 2141, cost 58.34 s 2023-09-17 23:31:41,653 44k INFO ====> Epoch: 2142, cost 59.11 s 2023-09-17 23:32:40,092 44k INFO ====> Epoch: 2143, cost 58.44 s 2023-09-17 23:33:25,183 44k INFO Train Epoch: 2144 [72%] 2023-09-17 23:33:25,184 44k INFO Losses: [2.140948534011841, 2.65208101272583, 13.453506469726562, 21.334199905395508, 0.8097086548805237], step: 68600, lr: 3.824952958804843e-05, reference_loss: 40.39044189453125 2023-09-17 23:33:39,652 44k INFO ====> Epoch: 2144, cost 59.56 s 2023-09-17 23:34:38,284 44k INFO ====> Epoch: 2145, cost 58.63 s 2023-09-17 23:35:36,924 44k INFO ====> Epoch: 2146, cost 58.64 s 2023-09-17 23:36:35,950 44k INFO ====> Epoch: 2147, cost 59.03 s 2023-09-17 23:37:34,848 44k INFO ====> Epoch: 2148, cost 58.90 s 2023-09-17 23:38:33,722 44k INFO ====> Epoch: 2149, cost 58.87 s 2023-09-17 23:39:32,103 44k INFO Train Epoch: 2150 [97%] 2023-09-17 23:39:32,104 44k INFO Losses: [1.844504952430725, 2.615569591522217, 6.567249774932861, 14.879817008972168, -0.588670015335083], step: 68800, lr: 3.822085140409689e-05, reference_loss: 25.318471908569336 2023-09-17 23:39:48,617 44k INFO Saving model and optimizer state at iteration 2150 to ./logs/44k/G_68800.pth 2023-09-17 23:39:52,111 44k INFO Saving model and optimizer state at iteration 2150 to ./logs/44k/D_68800.pth 2023-09-17 23:39:52,825 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_64800.pth 2023-09-17 23:39:52,826 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_64800.pth 2023-09-17 23:39:52,826 44k INFO ====> Epoch: 2150, cost 79.10 s 2023-09-17 23:40:51,479 44k INFO ====> Epoch: 2151, cost 58.65 s 2023-09-17 23:41:49,826 44k INFO ====> Epoch: 2152, cost 58.35 s 2023-09-17 23:42:48,715 44k INFO ====> Epoch: 2153, cost 58.89 s 2023-09-17 23:43:47,512 44k INFO ====> Epoch: 2154, cost 58.80 s 2023-09-17 23:44:46,818 44k INFO ====> Epoch: 2155, cost 59.31 s 2023-09-17 23:45:44,906 44k INFO ====> Epoch: 2156, cost 58.09 s 2023-09-17 23:45:59,884 44k INFO Train Epoch: 2157 [22%] 2023-09-17 23:45:59,885 44k INFO Losses: [2.1476502418518066, 2.922477960586548, 14.767953872680664, 22.203638076782227, 0.8154978156089783], step: 69000, lr: 3.818742069772274e-05, reference_loss: 42.857215881347656 2023-09-17 23:46:44,184 44k INFO ====> Epoch: 2157, cost 59.28 s 2023-09-17 23:47:42,695 44k INFO ====> Epoch: 2158, cost 58.51 s 2023-09-17 23:48:41,478 44k INFO ====> Epoch: 2159, cost 58.78 s 2023-09-17 23:49:40,135 44k INFO ====> Epoch: 2160, cost 58.66 s 2023-09-17 23:50:38,461 44k INFO ====> Epoch: 2161, cost 58.33 s 2023-09-17 23:51:36,730 44k INFO ====> Epoch: 2162, cost 58.27 s 2023-09-17 23:52:06,956 44k INFO Train Epoch: 2163 [47%] 2023-09-17 23:52:06,957 44k INFO Losses: [2.0600523948669434, 2.7789502143859863, 15.126502990722656, 22.0716495513916, 0.7475436329841614], step: 69200, lr: 3.815878908088461e-05, reference_loss: 42.784698486328125 2023-09-17 23:52:36,096 44k INFO ====> Epoch: 2163, cost 59.37 s 2023-09-17 23:53:35,215 44k INFO ====> Epoch: 2164, cost 59.12 s 2023-09-17 23:54:33,995 44k INFO ====> Epoch: 2165, cost 58.78 s 2023-09-17 23:55:32,817 44k INFO ====> Epoch: 2166, cost 58.82 s 2023-09-17 23:56:31,911 44k INFO ====> Epoch: 2167, cost 59.09 s 2023-09-17 23:57:30,662 44k INFO ====> Epoch: 2168, cost 58.75 s 2023-09-17 23:58:15,996 44k INFO Train Epoch: 2169 [72%] 2023-09-17 23:58:15,997 44k INFO Losses: [2.043907403945923, 2.713762044906616, 15.307992935180664, 21.525358200073242, 0.828478217124939], step: 69400, lr: 3.8130178931049694e-05, reference_loss: 42.419498443603516 2023-09-17 23:58:30,253 44k INFO ====> Epoch: 2169, cost 59.59 s 2023-09-17 23:59:28,398 44k INFO ====> Epoch: 2170, cost 58.14 s 2023-09-18 00:00:26,918 44k INFO ====> Epoch: 2171, cost 58.52 s 2023-09-18 00:01:25,914 44k INFO ====> Epoch: 2172, cost 59.00 s 2023-09-18 00:02:24,899 44k INFO ====> Epoch: 2173, cost 58.98 s 2023-09-18 00:03:23,206 44k INFO ====> Epoch: 2174, cost 58.31 s 2023-09-18 00:04:22,353 44k INFO Train Epoch: 2175 [97%] 2023-09-18 00:04:22,354 44k INFO Losses: [1.51272451877594, 3.1689419746398926, 13.455704689025879, 16.37052345275879, -0.6914419531822205], step: 69600, lr: 3.8101590232122766e-05, reference_loss: 33.81645584106445 2023-09-18 00:04:39,833 44k INFO Saving model and optimizer state at iteration 2175 to ./logs/44k/G_69600.pth 2023-09-18 00:04:42,965 44k INFO Saving model and optimizer state at iteration 2175 to ./logs/44k/D_69600.pth 2023-09-18 00:04:43,932 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_65600.pth 2023-09-18 00:04:43,944 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_65600.pth 2023-09-18 00:04:43,960 44k INFO ====> Epoch: 2175, cost 80.75 s 2023-09-18 00:05:42,727 44k INFO ====> Epoch: 2176, cost 58.77 s 2023-09-18 00:06:41,313 44k INFO ====> Epoch: 2177, cost 58.59 s 2023-09-18 00:07:40,400 44k INFO ====> Epoch: 2178, cost 59.09 s 2023-09-18 00:08:39,399 44k INFO ====> Epoch: 2179, cost 59.00 s 2023-09-18 00:09:38,293 44k INFO ====> Epoch: 2180, cost 58.89 s 2023-09-18 00:10:36,785 44k INFO ====> Epoch: 2181, cost 58.49 s 2023-09-18 00:10:51,673 44k INFO Train Epoch: 2182 [22%] 2023-09-18 00:10:51,674 44k INFO Losses: [2.1072654724121094, 2.802136182785034, 14.231714248657227, 21.95248794555664, 0.8021700382232666], step: 69800, lr: 3.806826384014966e-05, reference_loss: 41.895774841308594 2023-09-18 00:11:36,207 44k INFO ====> Epoch: 2182, cost 59.42 s 2023-09-18 00:12:34,296 44k INFO ====> Epoch: 2183, cost 58.09 s 2023-09-18 00:13:32,548 44k INFO ====> Epoch: 2184, cost 58.25 s 2023-09-18 00:14:31,610 44k INFO ====> Epoch: 2185, cost 59.06 s 2023-09-18 00:15:30,892 44k INFO ====> Epoch: 2186, cost 59.28 s 2023-09-18 00:16:29,391 44k INFO ====> Epoch: 2187, cost 58.50 s 2023-09-18 00:16:59,033 44k INFO Train Epoch: 2188 [47%] 2023-09-18 00:16:59,034 44k INFO Losses: [2.0526485443115234, 2.8872926235198975, 15.111408233642578, 22.215614318847656, 0.7553017735481262], step: 70000, lr: 3.8039721563031974e-05, reference_loss: 43.02226638793945 2023-09-18 00:17:28,231 44k INFO ====> Epoch: 2188, cost 58.84 s 2023-09-18 00:18:27,233 44k INFO ====> Epoch: 2189, cost 59.00 s 2023-09-18 00:19:26,286 44k INFO ====> Epoch: 2190, cost 59.05 s 2023-09-18 00:20:25,288 44k INFO ====> Epoch: 2191, cost 59.00 s 2023-09-18 00:21:23,766 44k INFO ====> Epoch: 2192, cost 58.48 s 2023-09-18 00:22:22,576 44k INFO ====> Epoch: 2193, cost 58.81 s 2023-09-18 00:23:07,556 44k INFO Train Epoch: 2194 [72%] 2023-09-18 00:23:07,557 44k INFO Losses: [2.110382318496704, 2.773146390914917, 13.749303817749023, 21.264162063598633, 0.8183205723762512], step: 70200, lr: 3.8011200685933636e-05, reference_loss: 40.71531677246094 2023-09-18 00:23:21,763 44k INFO ====> Epoch: 2194, cost 59.19 s 2023-09-18 00:24:20,850 44k INFO ====> Epoch: 2195, cost 59.09 s 2023-09-18 00:25:19,272 44k INFO ====> Epoch: 2196, cost 58.42 s 2023-09-18 00:26:18,092 44k INFO ====> Epoch: 2197, cost 58.82 s 2023-09-18 00:27:16,817 44k INFO ====> Epoch: 2198, cost 58.72 s 2023-09-18 00:28:16,253 44k INFO ====> Epoch: 2199, cost 59.44 s 2023-09-18 00:29:15,142 44k INFO Train Epoch: 2200 [97%] 2023-09-18 00:29:15,143 44k INFO Losses: [1.478506088256836, 3.3238861560821533, 10.802029609680176, 15.841050148010254, -0.6551035046577454], step: 70400, lr: 3.798270119280966e-05, reference_loss: 30.790369033813477 2023-09-18 00:29:33,332 44k INFO Saving model and optimizer state at iteration 2200 to ./logs/44k/G_70400.pth 2023-09-18 00:29:36,675 44k INFO Saving model and optimizer state at iteration 2200 to ./logs/44k/D_70400.pth 2023-09-18 00:29:37,761 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_66400.pth 2023-09-18 00:29:37,763 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_66400.pth 2023-09-18 00:29:37,763 44k INFO ====> Epoch: 2200, cost 81.51 s 2023-09-18 00:30:36,333 44k INFO ====> Epoch: 2201, cost 58.57 s 2023-09-18 00:31:34,717 44k INFO ====> Epoch: 2202, cost 58.38 s 2023-09-18 00:32:33,764 44k INFO ====> Epoch: 2203, cost 59.05 s 2023-09-18 00:33:32,658 44k INFO ====> Epoch: 2204, cost 58.89 s 2023-09-18 00:34:31,684 44k INFO ====> Epoch: 2205, cost 59.03 s 2023-09-18 00:35:30,573 44k INFO ====> Epoch: 2206, cost 58.89 s 2023-09-18 00:35:45,326 44k INFO Train Epoch: 2207 [22%] 2023-09-18 00:35:45,327 44k INFO Losses: [2.161652088165283, 2.9123263359069824, 13.610086441040039, 21.771150588989258, 0.801672637462616], step: 70600, lr: 3.794947878974363e-05, reference_loss: 41.25688934326172 2023-09-18 00:36:29,719 44k INFO ====> Epoch: 2207, cost 59.15 s 2023-09-18 00:37:28,144 44k INFO ====> Epoch: 2208, cost 58.42 s 2023-09-18 00:38:27,498 44k INFO ====> Epoch: 2209, cost 59.35 s 2023-09-18 00:39:26,321 44k INFO ====> Epoch: 2210, cost 58.82 s 2023-09-18 00:40:24,936 44k INFO ====> Epoch: 2211, cost 58.61 s 2023-09-18 00:41:23,440 44k INFO ====> Epoch: 2212, cost 58.50 s 2023-09-18 00:41:53,379 44k INFO Train Epoch: 2213 [47%] 2023-09-18 00:41:53,380 44k INFO Losses: [2.061187744140625, 2.7574517726898193, 15.386341094970703, 22.0482234954834, 0.7598125338554382], step: 70800, lr: 3.792102557357814e-05, reference_loss: 43.01301574707031 2023-09-18 00:42:22,785 44k INFO ====> Epoch: 2213, cost 59.35 s 2023-09-18 00:43:21,732 44k INFO ====> Epoch: 2214, cost 58.95 s 2023-09-18 00:44:20,069 44k INFO ====> Epoch: 2215, cost 58.34 s 2023-09-18 00:45:18,923 44k INFO ====> Epoch: 2216, cost 58.85 s 2023-09-18 00:46:18,319 44k INFO ====> Epoch: 2217, cost 59.40 s 2023-09-18 00:47:16,846 44k INFO ====> Epoch: 2218, cost 58.53 s 2023-09-18 00:48:01,969 44k INFO Train Epoch: 2219 [72%] 2023-09-18 00:48:01,970 44k INFO Losses: [2.0861024856567383, 2.8371381759643555, 14.360204696655273, 21.212648391723633, 0.8218165636062622], step: 71000, lr: 3.7892593690657165e-05, reference_loss: 41.317909240722656 2023-09-18 00:48:16,091 44k INFO ====> Epoch: 2219, cost 59.24 s 2023-09-18 00:49:15,055 44k INFO ====> Epoch: 2220, cost 58.96 s 2023-09-18 00:50:13,649 44k INFO ====> Epoch: 2221, cost 58.59 s 2023-09-18 00:51:12,455 44k INFO ====> Epoch: 2222, cost 58.81 s 2023-09-18 00:52:11,676 44k INFO ====> Epoch: 2223, cost 59.22 s 2023-09-18 00:53:10,767 44k INFO ====> Epoch: 2224, cost 59.09 s 2023-09-18 00:54:09,398 44k INFO Train Epoch: 2225 [97%] 2023-09-18 00:54:09,399 44k INFO Losses: [1.3722567558288574, 3.1828393936157227, 14.406062126159668, 15.838987350463867, -0.6876747012138367], step: 71200, lr: 3.7864183124985765e-05, reference_loss: 34.11247253417969 2023-09-18 00:54:25,052 44k INFO Saving model and optimizer state at iteration 2225 to ./logs/44k/G_71200.pth 2023-09-18 00:54:27,177 44k INFO Saving model and optimizer state at iteration 2225 to ./logs/44k/D_71200.pth 2023-09-18 00:54:27,716 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_67200.pth 2023-09-18 00:54:27,717 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_67200.pth 2023-09-18 00:54:27,718 44k INFO ====> Epoch: 2225, cost 76.95 s 2023-09-18 00:55:27,052 44k INFO ====> Epoch: 2226, cost 59.33 s 2023-09-18 00:56:25,698 44k INFO ====> Epoch: 2227, cost 58.65 s 2023-09-18 00:57:23,677 44k INFO ====> Epoch: 2228, cost 57.98 s 2023-09-18 00:58:22,747 44k INFO ====> Epoch: 2229, cost 59.07 s 2023-09-18 00:59:21,201 44k INFO ====> Epoch: 2230, cost 58.45 s 2023-09-18 01:00:19,950 44k INFO ====> Epoch: 2231, cost 58.75 s 2023-09-18 01:00:34,736 44k INFO Train Epoch: 2232 [22%] 2023-09-18 01:00:34,737 44k INFO Losses: [2.0774712562561035, 2.9380369186401367, 15.523204803466797, 22.151363372802734, 0.8028131127357483], step: 71400, lr: 3.783106438634842e-05, reference_loss: 43.49289321899414 2023-09-18 01:01:19,843 44k INFO ====> Epoch: 2232, cost 59.89 s 2023-09-18 01:02:18,659 44k INFO ====> Epoch: 2233, cost 58.82 s 2023-09-18 01:03:17,529 44k INFO ====> Epoch: 2234, cost 58.87 s 2023-09-18 01:04:16,268 44k INFO ====> Epoch: 2235, cost 58.74 s 2023-09-18 01:05:15,392 44k INFO ====> Epoch: 2236, cost 59.12 s 2023-09-18 01:06:14,316 44k INFO ====> Epoch: 2237, cost 58.92 s 2023-09-18 01:06:44,551 44k INFO Train Epoch: 2238 [47%] 2023-09-18 01:06:44,552 44k INFO Losses: [2.0218100547790527, 2.8540990352630615, 16.297306060791016, 22.254520416259766, 0.8023523092269897], step: 71600, lr: 3.780269995323673e-05, reference_loss: 44.23008728027344 2023-09-18 01:07:14,245 44k INFO ====> Epoch: 2238, cost 59.93 s 2023-09-18 01:08:12,278 44k INFO ====> Epoch: 2239, cost 58.03 s 2023-09-18 01:09:11,242 44k INFO ====> Epoch: 2240, cost 58.96 s 2023-09-18 01:10:09,456 44k INFO ====> Epoch: 2241, cost 58.21 s 2023-09-18 01:11:08,398 44k INFO ====> Epoch: 2242, cost 58.94 s 2023-09-18 01:12:06,832 44k INFO ====> Epoch: 2243, cost 58.43 s 2023-09-18 01:12:51,841 44k INFO Train Epoch: 2244 [72%] 2023-09-18 01:12:51,842 44k INFO Losses: [2.0666146278381348, 2.695310354232788, 15.500073432922363, 21.43015480041504, 0.8127872347831726], step: 71800, lr: 3.7774356786803065e-05, reference_loss: 42.504940032958984 2023-09-18 01:13:06,121 44k INFO ====> Epoch: 2244, cost 59.29 s 2023-09-18 01:14:05,178 44k INFO ====> Epoch: 2245, cost 59.06 s 2023-09-18 01:15:03,967 44k INFO ====> Epoch: 2246, cost 58.79 s 2023-09-18 01:16:02,275 44k INFO ====> Epoch: 2247, cost 58.31 s 2023-09-18 01:17:01,147 44k INFO ====> Epoch: 2248, cost 58.87 s 2023-09-18 01:17:59,939 44k INFO ====> Epoch: 2249, cost 58.79 s 2023-09-18 01:18:58,790 44k INFO Train Epoch: 2250 [97%] 2023-09-18 01:18:58,791 44k INFO Losses: [1.810563325881958, 2.9991722106933594, 8.13439655303955, 15.750753402709961, -0.6446875333786011], step: 72000, lr: 3.7746034871102405e-05, reference_loss: 28.05019760131836 2023-09-18 01:19:16,810 44k INFO Saving model and optimizer state at iteration 2250 to ./logs/44k/G_72000.pth 2023-09-18 01:19:20,257 44k INFO Saving model and optimizer state at iteration 2250 to ./logs/44k/D_72000.pth 2023-09-18 01:19:20,937 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_68000.pth 2023-09-18 01:19:20,938 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_68000.pth 2023-09-18 01:19:20,938 44k INFO ====> Epoch: 2250, cost 81.00 s 2023-09-18 01:20:20,128 44k INFO ====> Epoch: 2251, cost 59.19 s 2023-09-18 01:21:19,420 44k INFO ====> Epoch: 2252, cost 59.29 s 2023-09-18 01:22:18,697 44k INFO ====> Epoch: 2253, cost 59.28 s 2023-09-18 01:23:18,251 44k INFO ====> Epoch: 2254, cost 59.55 s 2023-09-18 01:24:17,366 44k INFO ====> Epoch: 2255, cost 59.12 s 2023-09-18 01:25:16,645 44k INFO ====> Epoch: 2256, cost 59.28 s 2023-09-18 01:25:31,586 44k INFO Train Epoch: 2257 [22%] 2023-09-18 01:25:31,587 44k INFO Losses: [2.0876541137695312, 2.8272972106933594, 15.212210655212402, 22.054105758666992, 0.8071452379226685], step: 72200, lr: 3.77130194734279e-05, reference_loss: 42.98841094970703 2023-09-18 01:26:16,214 44k INFO ====> Epoch: 2257, cost 59.57 s 2023-09-18 01:27:15,837 44k INFO ====> Epoch: 2258, cost 59.62 s 2023-09-18 01:28:15,177 44k INFO ====> Epoch: 2259, cost 59.34 s 2023-09-18 01:29:14,413 44k INFO ====> Epoch: 2260, cost 59.24 s 2023-09-18 01:30:13,951 44k INFO ====> Epoch: 2261, cost 59.54 s 2023-09-18 01:31:13,608 44k INFO ====> Epoch: 2262, cost 59.66 s 2023-09-18 01:31:44,011 44k INFO Train Epoch: 2263 [47%] 2023-09-18 01:31:44,012 44k INFO Losses: [2.039727210998535, 2.767404794692993, 14.197949409484863, 22.031322479248047, 0.7419703602790833], step: 72400, lr: 3.768474354633873e-05, reference_loss: 41.77837371826172 2023-09-18 01:32:13,795 44k INFO ====> Epoch: 2263, cost 60.19 s 2023-09-18 01:33:13,453 44k INFO ====> Epoch: 2264, cost 59.66 s 2023-09-18 01:34:12,963 44k INFO ====> Epoch: 2265, cost 59.51 s 2023-09-18 01:35:12,201 44k INFO ====> Epoch: 2266, cost 59.24 s 2023-09-18 01:36:12,110 44k INFO ====> Epoch: 2267, cost 59.91 s 2023-09-18 01:37:11,571 44k INFO ====> Epoch: 2268, cost 59.46 s 2023-09-18 01:37:57,160 44k INFO Train Epoch: 2269 [72%] 2023-09-18 01:37:57,161 44k INFO Losses: [2.0510973930358887, 2.811516046524048, 15.597136497497559, 21.499919891357422, 0.8192503452301025], step: 72600, lr: 3.7656488819568814e-05, reference_loss: 42.77892303466797 2023-09-18 01:38:11,767 44k INFO ====> Epoch: 2269, cost 60.20 s 2023-09-18 01:39:11,436 44k INFO ====> Epoch: 2270, cost 59.67 s 2023-09-18 01:40:10,730 44k INFO ====> Epoch: 2271, cost 59.29 s 2023-09-18 01:41:09,965 44k INFO ====> Epoch: 2272, cost 59.24 s 2023-09-18 01:42:09,045 44k INFO ====> Epoch: 2273, cost 59.08 s 2023-09-18 01:43:08,340 44k INFO ====> Epoch: 2274, cost 59.30 s 2023-09-18 01:44:07,933 44k INFO Train Epoch: 2275 [97%] 2023-09-18 01:44:07,934 44k INFO Losses: [1.5788346529006958, 3.028134346008301, 12.944146156311035, 15.840314865112305, -0.7088740468025208], step: 72800, lr: 3.7628255277222876e-05, reference_loss: 32.68255615234375 2023-09-18 01:44:25,550 44k INFO Saving model and optimizer state at iteration 2275 to ./logs/44k/G_72800.pth 2023-09-18 01:44:29,163 44k INFO Saving model and optimizer state at iteration 2275 to ./logs/44k/D_72800.pth 2023-09-18 01:44:29,685 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_68800.pth 2023-09-18 01:44:29,686 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_68800.pth 2023-09-18 01:44:29,686 44k INFO ====> Epoch: 2275, cost 81.35 s 2023-09-18 01:45:29,015 44k INFO ====> Epoch: 2276, cost 59.33 s 2023-09-18 01:46:28,820 44k INFO ====> Epoch: 2277, cost 59.80 s 2023-09-18 01:47:28,909 44k INFO ====> Epoch: 2278, cost 60.09 s 2023-09-18 01:48:28,189 44k INFO ====> Epoch: 2279, cost 59.28 s 2023-09-18 01:49:27,546 44k INFO ====> Epoch: 2280, cost 59.36 s 2023-09-18 01:50:26,325 44k INFO ====> Epoch: 2281, cost 58.78 s 2023-09-18 01:50:41,142 44k INFO Train Epoch: 2282 [22%] 2023-09-18 01:50:41,143 44k INFO Losses: [2.059953451156616, 3.0977399349212646, 16.1200008392334, 22.05389976501465, 0.8061563372612], step: 73000, lr: 3.759534289805464e-05, reference_loss: 44.137752532958984 2023-09-18 01:51:25,764 44k INFO ====> Epoch: 2282, cost 59.44 s 2023-09-18 01:52:24,729 44k INFO ====> Epoch: 2283, cost 58.97 s 2023-09-18 01:53:23,997 44k INFO ====> Epoch: 2284, cost 59.27 s 2023-09-18 01:54:23,503 44k INFO ====> Epoch: 2285, cost 59.51 s 2023-09-18 01:55:23,462 44k INFO ====> Epoch: 2286, cost 59.96 s 2023-09-18 01:56:22,510 44k INFO ====> Epoch: 2287, cost 59.05 s 2023-09-18 01:56:52,979 44k INFO Train Epoch: 2288 [47%] 2023-09-18 01:56:52,980 44k INFO Losses: [2.077582359313965, 2.8157808780670166, 14.404403686523438, 22.07065773010254, 0.726999044418335], step: 73200, lr: 3.756715520082115e-05, reference_loss: 42.095420837402344 2023-09-18 01:57:22,674 44k INFO ====> Epoch: 2288, cost 60.16 s 2023-09-18 01:58:22,313 44k INFO ====> Epoch: 2289, cost 59.64 s 2023-09-18 01:59:21,575 44k INFO ====> Epoch: 2290, cost 59.26 s 2023-09-18 02:00:20,791 44k INFO ====> Epoch: 2291, cost 59.22 s 2023-09-18 02:01:19,749 44k INFO ====> Epoch: 2292, cost 58.96 s 2023-09-18 02:02:19,379 44k INFO ====> Epoch: 2293, cost 59.63 s 2023-09-18 02:03:04,669 44k INFO Train Epoch: 2294 [72%] 2023-09-18 02:03:04,669 44k INFO Losses: [2.0923821926116943, 2.7820322513580322, 14.855237007141113, 21.505104064941406, 0.8115933537483215], step: 73400, lr: 3.753898863775519e-05, reference_loss: 42.046348571777344 2023-09-18 02:03:19,300 44k INFO ====> Epoch: 2294, cost 59.92 s 2023-09-18 02:04:18,586 44k INFO ====> Epoch: 2295, cost 59.29 s 2023-09-18 02:05:17,604 44k INFO ====> Epoch: 2296, cost 59.02 s 2023-09-18 02:06:17,147 44k INFO ====> Epoch: 2297, cost 59.54 s 2023-09-18 02:07:16,532 44k INFO ====> Epoch: 2298, cost 59.38 s 2023-09-18 02:08:15,457 44k INFO ====> Epoch: 2299, cost 58.92 s 2023-09-18 02:09:15,012 44k INFO Train Epoch: 2300 [97%] 2023-09-18 02:09:15,013 44k INFO Losses: [1.4838926792144775, 3.4304051399230957, 14.387245178222656, 16.844345092773438, -0.7175965309143066], step: 73600, lr: 3.75108431930111e-05, reference_loss: 35.42829132080078 2023-09-18 02:09:31,358 44k INFO Saving model and optimizer state at iteration 2300 to ./logs/44k/G_73600.pth 2023-09-18 02:09:34,569 44k INFO Saving model and optimizer state at iteration 2300 to ./logs/44k/D_73600.pth 2023-09-18 02:09:35,514 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_69600.pth 2023-09-18 02:09:35,531 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_69600.pth 2023-09-18 02:09:35,552 44k INFO ====> Epoch: 2300, cost 80.10 s 2023-09-18 02:10:35,034 44k INFO ====> Epoch: 2301, cost 59.48 s 2023-09-18 02:11:34,002 44k INFO ====> Epoch: 2302, cost 58.97 s 2023-09-18 02:12:33,013 44k INFO ====> Epoch: 2303, cost 59.01 s 2023-09-18 02:13:32,098 44k INFO ====> Epoch: 2304, cost 59.08 s 2023-09-18 02:14:31,345 44k INFO ====> Epoch: 2305, cost 59.25 s 2023-09-18 02:15:30,611 44k INFO ====> Epoch: 2306, cost 59.27 s 2023-09-18 02:15:45,543 44k INFO Train Epoch: 2307 [22%] 2023-09-18 02:15:45,544 44k INFO Losses: [2.14731502532959, 2.9630789756774902, 14.320298194885254, 21.794763565063477, 0.7933802008628845], step: 73800, lr: 3.747803351089873e-05, reference_loss: 42.018836975097656 2023-09-18 02:16:30,763 44k INFO ====> Epoch: 2307, cost 60.15 s 2023-09-18 02:17:29,793 44k INFO ====> Epoch: 2308, cost 59.03 s 2023-09-18 02:18:28,333 44k INFO ====> Epoch: 2309, cost 58.54 s 2023-09-18 02:19:27,591 44k INFO ====> Epoch: 2310, cost 59.26 s 2023-09-18 02:20:26,562 44k INFO ====> Epoch: 2311, cost 58.97 s 2023-09-18 02:21:24,989 44k INFO ====> Epoch: 2312, cost 58.43 s 2023-09-18 02:21:55,026 44k INFO Train Epoch: 2313 [47%] 2023-09-18 02:21:55,027 44k INFO Losses: [2.078589916229248, 2.729112148284912, 15.645517349243164, 22.106630325317383, 0.7627485394477844], step: 74000, lr: 3.74499337682158e-05, reference_loss: 43.322601318359375 2023-09-18 02:22:24,749 44k INFO ====> Epoch: 2313, cost 59.76 s 2023-09-18 02:23:24,104 44k INFO ====> Epoch: 2314, cost 59.35 s 2023-09-18 02:24:22,578 44k INFO ====> Epoch: 2315, cost 58.47 s 2023-09-18 02:25:22,084 44k INFO ====> Epoch: 2316, cost 59.51 s 2023-09-18 02:26:20,882 44k INFO ====> Epoch: 2317, cost 58.80 s 2023-09-18 02:27:19,851 44k INFO ====> Epoch: 2318, cost 58.97 s 2023-09-18 02:28:05,340 44k INFO Train Epoch: 2319 [72%] 2023-09-18 02:28:05,341 44k INFO Losses: [2.093186378479004, 2.7345986366271973, 14.582247734069824, 21.513582229614258, 0.8116017580032349], step: 74200, lr: 3.74218550937551e-05, reference_loss: 41.7352180480957 2023-09-18 02:28:19,732 44k INFO ====> Epoch: 2319, cost 59.88 s 2023-09-18 02:29:19,396 44k INFO ====> Epoch: 2320, cost 59.66 s 2023-09-18 02:30:18,717 44k INFO ====> Epoch: 2321, cost 59.32 s 2023-09-18 02:31:18,396 44k INFO ====> Epoch: 2322, cost 59.68 s 2023-09-18 02:32:18,089 44k INFO ====> Epoch: 2323, cost 59.69 s 2023-09-18 02:33:17,332 44k INFO ====> Epoch: 2324, cost 59.24 s 2023-09-18 02:34:16,824 44k INFO Train Epoch: 2325 [97%] 2023-09-18 02:34:16,826 44k INFO Losses: [1.488105297088623, 3.166285514831543, 11.699100494384766, 15.56074333190918, -0.6445828080177307], step: 74400, lr: 3.739379747172041e-05, reference_loss: 31.269651412963867 2023-09-18 02:34:34,001 44k INFO Saving model and optimizer state at iteration 2325 to ./logs/44k/G_74400.pth 2023-09-18 02:34:37,422 44k INFO Saving model and optimizer state at iteration 2325 to ./logs/44k/D_74400.pth 2023-09-18 02:34:38,085 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_70400.pth 2023-09-18 02:34:38,087 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_70400.pth 2023-09-18 02:34:38,087 44k INFO ====> Epoch: 2325, cost 80.75 s 2023-09-18 02:35:37,431 44k INFO ====> Epoch: 2326, cost 59.34 s 2023-09-18 02:36:36,818 44k INFO ====> Epoch: 2327, cost 59.39 s 2023-09-18 02:37:36,185 44k INFO ====> Epoch: 2328, cost 59.37 s 2023-09-18 02:38:36,043 44k INFO ====> Epoch: 2329, cost 59.86 s 2023-09-18 02:39:35,520 44k INFO ====> Epoch: 2330, cost 59.48 s 2023-09-18 02:40:35,049 44k INFO ====> Epoch: 2331, cost 59.53 s 2023-09-18 02:40:49,874 44k INFO Train Epoch: 2332 [22%] 2023-09-18 02:40:49,875 44k INFO Losses: [2.0784451961517334, 2.939589023590088, 15.314668655395508, 21.8812255859375, 0.8010609149932861], step: 74600, lr: 3.7361090166216545e-05, reference_loss: 43.01498794555664 2023-09-18 02:41:34,810 44k INFO ====> Epoch: 2332, cost 59.76 s 2023-09-18 02:42:34,422 44k INFO ====> Epoch: 2333, cost 59.61 s 2023-09-18 02:43:33,774 44k INFO ====> Epoch: 2334, cost 59.35 s 2023-09-18 02:44:33,195 44k INFO ====> Epoch: 2335, cost 59.42 s 2023-09-18 02:45:32,908 44k INFO ====> Epoch: 2336, cost 59.71 s 2023-09-18 02:46:32,057 44k INFO ====> Epoch: 2337, cost 59.15 s 2023-09-18 02:47:02,046 44k INFO Train Epoch: 2338 [47%] 2023-09-18 02:47:02,047 44k INFO Losses: [2.066243886947632, 2.8709421157836914, 15.1210298538208, 21.993389129638672, 0.7457335591316223], step: 74800, lr: 3.7333078103638096e-05, reference_loss: 42.797340393066406 2023-09-18 02:47:31,382 44k INFO ====> Epoch: 2338, cost 59.33 s 2023-09-18 02:48:30,957 44k INFO ====> Epoch: 2339, cost 59.58 s 2023-09-18 02:49:30,794 44k INFO ====> Epoch: 2340, cost 59.84 s 2023-09-18 02:50:30,473 44k INFO ====> Epoch: 2341, cost 59.68 s 2023-09-18 02:51:29,961 44k INFO ====> Epoch: 2342, cost 59.49 s 2023-09-18 02:52:29,655 44k INFO ====> Epoch: 2343, cost 59.69 s 2023-09-18 02:53:14,800 44k INFO Train Epoch: 2344 [72%] 2023-09-18 02:53:14,801 44k INFO Losses: [2.044727325439453, 2.813305377960205, 16.063838958740234, 21.50834083557129, 0.8098564147949219], step: 75000, lr: 3.7305087043542346e-05, reference_loss: 43.24007034301758 2023-09-18 02:53:29,405 44k INFO ====> Epoch: 2344, cost 59.75 s 2023-09-18 02:54:28,622 44k INFO ====> Epoch: 2345, cost 59.22 s 2023-09-18 02:55:28,091 44k INFO ====> Epoch: 2346, cost 59.47 s 2023-09-18 02:56:27,782 44k INFO ====> Epoch: 2347, cost 59.69 s 2023-09-18 02:57:27,277 44k INFO ====> Epoch: 2348, cost 59.50 s 2023-09-18 02:58:27,044 44k INFO ====> Epoch: 2349, cost 59.77 s 2023-09-18 02:59:26,303 44k INFO Train Epoch: 2350 [97%] 2023-09-18 02:59:26,304 44k INFO Losses: [1.6262609958648682, 3.024423599243164, 11.326373100280762, 15.897705078125, -0.735952615737915], step: 75200, lr: 3.727711697018236e-05, reference_loss: 31.138811111450195 2023-09-18 02:59:43,612 44k INFO Saving model and optimizer state at iteration 2350 to ./logs/44k/G_75200.pth 2023-09-18 02:59:45,928 44k INFO Saving model and optimizer state at iteration 2350 to ./logs/44k/D_75200.pth 2023-09-18 02:59:46,502 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_71200.pth 2023-09-18 02:59:46,504 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_71200.pth 2023-09-18 02:59:46,504 44k INFO ====> Epoch: 2350, cost 79.46 s 2023-09-18 03:00:45,954 44k INFO ====> Epoch: 2351, cost 59.45 s 2023-09-18 03:01:45,073 44k INFO ====> Epoch: 2352, cost 59.12 s 2023-09-18 03:02:43,950 44k INFO ====> Epoch: 2353, cost 58.88 s 2023-09-18 03:03:42,682 44k INFO ====> Epoch: 2354, cost 58.73 s 2023-09-18 03:04:41,868 44k INFO ====> Epoch: 2355, cost 59.19 s 2023-09-18 03:05:41,099 44k INFO ====> Epoch: 2356, cost 59.23 s 2023-09-18 03:05:56,292 44k INFO Train Epoch: 2357 [22%] 2023-09-18 03:05:56,292 44k INFO Losses: [2.081005811691284, 2.9542768001556396, 15.8681001663208, 21.97502899169922, 0.7994431853294373], step: 75400, lr: 3.724451172183952e-05, reference_loss: 43.677852630615234 2023-09-18 03:06:40,989 44k INFO ====> Epoch: 2357, cost 59.89 s 2023-09-18 03:07:40,347 44k INFO ====> Epoch: 2358, cost 59.36 s 2023-09-18 03:08:39,699 44k INFO ====> Epoch: 2359, cost 59.35 s 2023-09-18 03:09:38,836 44k INFO ====> Epoch: 2360, cost 59.14 s 2023-09-18 03:10:38,557 44k INFO ====> Epoch: 2361, cost 59.72 s 2023-09-18 03:11:37,707 44k INFO ====> Epoch: 2362, cost 59.15 s 2023-09-18 03:12:07,793 44k INFO Train Epoch: 2363 [47%] 2023-09-18 03:12:07,794 44k INFO Losses: [2.061861991882324, 2.6719183921813965, 14.268061637878418, 21.859529495239258, 0.7711536288261414], step: 75600, lr: 3.7216587065775845e-05, reference_loss: 41.63252639770508 2023-09-18 03:12:37,523 44k INFO ====> Epoch: 2363, cost 59.82 s 2023-09-18 03:13:36,216 44k INFO ====> Epoch: 2364, cost 58.69 s 2023-09-18 03:14:34,192 44k INFO ====> Epoch: 2365, cost 57.98 s 2023-09-18 03:15:33,259 44k INFO ====> Epoch: 2366, cost 59.07 s 2023-09-18 03:16:31,526 44k INFO ====> Epoch: 2367, cost 58.27 s 2023-09-18 03:17:29,933 44k INFO ====> Epoch: 2368, cost 58.41 s 2023-09-18 03:18:14,702 44k INFO Train Epoch: 2369 [72%] 2023-09-18 03:18:14,703 44k INFO Losses: [2.0318799018859863, 2.7973310947418213, 16.16792869567871, 21.639116287231445, 0.8105025887489319], step: 75800, lr: 3.7188683346660464e-05, reference_loss: 43.446754455566406 2023-09-18 03:18:29,122 44k INFO ====> Epoch: 2369, cost 59.19 s 2023-09-18 03:19:27,589 44k INFO ====> Epoch: 2370, cost 58.47 s 2023-09-18 03:20:26,329 44k INFO ====> Epoch: 2371, cost 58.74 s 2023-09-18 03:21:25,254 44k INFO ====> Epoch: 2372, cost 58.93 s 2023-09-18 03:22:23,342 44k INFO ====> Epoch: 2373, cost 58.09 s 2023-09-18 03:23:21,970 44k INFO ====> Epoch: 2374, cost 58.63 s 2023-09-18 03:24:20,960 44k INFO Train Epoch: 2375 [97%] 2023-09-18 03:24:20,961 44k INFO Losses: [1.7670056819915771, 2.7849295139312744, 8.468921661376953, 15.59945011138916, -0.7363102436065674], step: 76000, lr: 3.716080054879558e-05, reference_loss: 27.883996963500977 2023-09-18 03:24:38,396 44k INFO Saving model and optimizer state at iteration 2375 to ./logs/44k/G_76000.pth 2023-09-18 03:24:41,586 44k INFO Saving model and optimizer state at iteration 2375 to ./logs/44k/D_76000.pth 2023-09-18 03:24:42,853 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_72000.pth 2023-09-18 03:24:42,855 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_72000.pth 2023-09-18 03:24:42,855 44k INFO ====> Epoch: 2375, cost 80.89 s 2023-09-18 03:25:40,537 44k INFO ====> Epoch: 2376, cost 57.68 s 2023-09-18 03:26:38,855 44k INFO ====> Epoch: 2377, cost 58.32 s 2023-09-18 03:27:37,634 44k INFO ====> Epoch: 2378, cost 58.78 s 2023-09-18 03:28:35,586 44k INFO ====> Epoch: 2379, cost 57.95 s 2023-09-18 03:29:34,487 44k INFO ====> Epoch: 2380, cost 58.90 s 2023-09-18 03:30:33,208 44k INFO ====> Epoch: 2381, cost 58.72 s 2023-09-18 03:30:48,025 44k INFO Train Epoch: 2382 [22%] 2023-09-18 03:30:48,026 44k INFO Losses: [2.099414587020874, 2.861154079437256, 14.583746910095215, 21.80767059326172, 0.8059393167495728], step: 76200, lr: 3.7128297039163077e-05, reference_loss: 42.15792465209961 2023-09-18 03:31:32,160 44k INFO ====> Epoch: 2382, cost 58.95 s 2023-09-18 03:32:30,696 44k INFO ====> Epoch: 2383, cost 58.54 s 2023-09-18 03:33:29,245 44k INFO ====> Epoch: 2384, cost 58.55 s 2023-09-18 03:34:28,140 44k INFO ====> Epoch: 2385, cost 58.89 s 2023-09-18 03:35:26,697 44k INFO ====> Epoch: 2386, cost 58.56 s 2023-09-18 03:36:25,516 44k INFO ====> Epoch: 2387, cost 58.82 s 2023-09-18 03:36:55,332 44k INFO Train Epoch: 2388 [47%] 2023-09-18 03:36:55,333 44k INFO Losses: [2.0812668800354004, 2.8583831787109375, 16.108890533447266, 22.15748405456543, 0.7461532950401306], step: 76400, lr: 3.7100459516878135e-05, reference_loss: 43.952178955078125 2023-09-18 03:37:24,464 44k INFO ====> Epoch: 2388, cost 58.95 s 2023-09-18 03:38:22,946 44k INFO ====> Epoch: 2389, cost 58.48 s 2023-09-18 03:39:22,096 44k INFO ====> Epoch: 2390, cost 59.15 s 2023-09-18 03:40:20,589 44k INFO ====> Epoch: 2391, cost 58.49 s 2023-09-18 03:41:19,194 44k INFO ====> Epoch: 2392, cost 58.60 s 2023-09-18 03:42:18,225 44k INFO ====> Epoch: 2393, cost 59.03 s 2023-09-18 03:43:02,951 44k INFO Train Epoch: 2394 [72%] 2023-09-18 03:43:02,952 44k INFO Losses: [2.071072578430176, 2.7412376403808594, 14.273017883300781, 21.230783462524414, 0.8103875517845154], step: 76600, lr: 3.7072642866211565e-05, reference_loss: 41.126495361328125 2023-09-18 03:43:17,107 44k INFO ====> Epoch: 2394, cost 58.88 s 2023-09-18 03:44:15,646 44k INFO ====> Epoch: 2395, cost 58.54 s 2023-09-18 03:45:13,995 44k INFO ====> Epoch: 2396, cost 58.35 s 2023-09-18 03:46:12,740 44k INFO ====> Epoch: 2397, cost 58.75 s 2023-09-18 03:47:11,369 44k INFO ====> Epoch: 2398, cost 58.63 s 2023-09-18 03:48:10,355 44k INFO ====> Epoch: 2399, cost 58.99 s 2023-09-18 03:49:09,627 44k INFO Train Epoch: 2400 [97%] 2023-09-18 03:49:09,628 44k INFO Losses: [1.5224963426589966, 3.1720309257507324, 12.010065078735352, 15.348358154296875, -0.8268147706985474], step: 76800, lr: 3.7044847071514555e-05, reference_loss: 31.226133346557617 2023-09-18 03:49:27,210 44k INFO Saving model and optimizer state at iteration 2400 to ./logs/44k/G_76800.pth 2023-09-18 03:49:30,619 44k INFO Saving model and optimizer state at iteration 2400 to ./logs/44k/D_76800.pth 2023-09-18 03:49:31,467 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_72800.pth 2023-09-18 03:49:31,469 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_72800.pth 2023-09-18 03:49:31,469 44k INFO ====> Epoch: 2400, cost 81.11 s 2023-09-18 03:50:30,561 44k INFO ====> Epoch: 2401, cost 59.09 s 2023-09-18 03:51:28,432 44k INFO ====> Epoch: 2402, cost 57.87 s 2023-09-18 03:52:27,277 44k INFO ====> Epoch: 2403, cost 58.84 s 2023-09-18 03:53:25,473 44k INFO ====> Epoch: 2404, cost 58.20 s 2023-09-18 03:54:23,892 44k INFO ====> Epoch: 2405, cost 58.42 s 2023-09-18 03:55:22,153 44k INFO ====> Epoch: 2406, cost 58.26 s 2023-09-18 03:55:36,943 44k INFO Train Epoch: 2407 [22%] 2023-09-18 03:55:36,944 44k INFO Losses: [2.1010677814483643, 3.0140533447265625, 15.01379108428955, 21.800331115722656, 0.800018310546875], step: 77000, lr: 3.701244498313537e-05, reference_loss: 42.72926330566406 2023-09-18 03:56:21,523 44k INFO ====> Epoch: 2407, cost 59.37 s 2023-09-18 03:57:19,714 44k INFO ====> Epoch: 2408, cost 58.19 s 2023-09-18 03:58:18,422 44k INFO ====> Epoch: 2409, cost 58.71 s 2023-09-18 03:59:17,750 44k INFO ====> Epoch: 2410, cost 59.33 s 2023-09-18 04:00:16,572 44k INFO ====> Epoch: 2411, cost 58.82 s 2023-09-18 04:01:15,114 44k INFO ====> Epoch: 2412, cost 58.54 s 2023-09-18 04:01:45,028 44k INFO Train Epoch: 2413 [47%] 2023-09-18 04:01:45,029 44k INFO Losses: [2.038663148880005, 2.8444466590881348, 15.18020248413086, 21.80423355102539, 0.7544267773628235], step: 77200, lr: 3.698469432274414e-05, reference_loss: 42.621971130371094 2023-09-18 04:02:14,712 44k INFO ====> Epoch: 2413, cost 59.60 s 2023-09-18 04:03:13,488 44k INFO ====> Epoch: 2414, cost 58.78 s 2023-09-18 04:04:11,905 44k INFO ====> Epoch: 2415, cost 58.42 s 2023-09-18 04:05:10,387 44k INFO ====> Epoch: 2416, cost 58.48 s 2023-09-18 04:06:09,029 44k INFO ====> Epoch: 2417, cost 58.64 s 2023-09-18 04:07:07,779 44k INFO ====> Epoch: 2418, cost 58.75 s 2023-09-18 04:07:52,702 44k INFO Train Epoch: 2419 [72%] 2023-09-18 04:07:52,703 44k INFO Losses: [2.0769553184509277, 2.6627392768859863, 14.354313850402832, 21.296154022216797, 0.8033899068832397], step: 77400, lr: 3.6956964468845225e-05, reference_loss: 41.19355392456055 2023-09-18 04:08:07,280 44k INFO ====> Epoch: 2419, cost 59.50 s 2023-09-18 04:09:05,635 44k INFO ====> Epoch: 2420, cost 58.35 s 2023-09-18 04:10:04,065 44k INFO ====> Epoch: 2421, cost 58.43 s 2023-09-18 04:11:03,044 44k INFO ====> Epoch: 2422, cost 58.98 s 2023-09-18 04:12:02,091 44k INFO ====> Epoch: 2423, cost 59.05 s 2023-09-18 04:13:00,594 44k INFO ====> Epoch: 2424, cost 58.50 s 2023-09-18 04:13:58,948 44k INFO Train Epoch: 2425 [97%] 2023-09-18 04:13:58,950 44k INFO Losses: [1.5076097249984741, 3.183945655822754, 11.995193481445312, 15.762344360351562, -0.8099625706672668], step: 77600, lr: 3.6929255405838634e-05, reference_loss: 31.639129638671875 2023-09-18 04:14:16,628 44k INFO Saving model and optimizer state at iteration 2425 to ./logs/44k/G_77600.pth 2023-09-18 04:14:20,011 44k INFO Saving model and optimizer state at iteration 2425 to ./logs/44k/D_77600.pth 2023-09-18 04:14:21,013 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_73600.pth 2023-09-18 04:14:21,016 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_73600.pth 2023-09-18 04:14:21,016 44k INFO ====> Epoch: 2425, cost 80.42 s 2023-09-18 04:15:19,204 44k INFO ====> Epoch: 2426, cost 58.19 s 2023-09-18 04:16:17,489 44k INFO ====> Epoch: 2427, cost 58.29 s 2023-09-18 04:17:15,983 44k INFO ====> Epoch: 2428, cost 58.49 s 2023-09-18 04:18:15,173 44k INFO ====> Epoch: 2429, cost 59.19 s 2023-09-18 04:19:14,126 44k INFO ====> Epoch: 2430, cost 58.95 s 2023-09-18 04:20:12,642 44k INFO ====> Epoch: 2431, cost 58.52 s 2023-09-18 04:20:27,344 44k INFO Train Epoch: 2432 [22%] 2023-09-18 04:20:27,344 44k INFO Losses: [2.0312957763671875, 2.9820995330810547, 15.781170845031738, 21.957294464111328, 0.8097060322761536], step: 77800, lr: 3.689695442224631e-05, reference_loss: 43.56156921386719 2023-09-18 04:21:11,777 44k INFO ====> Epoch: 2432, cost 59.14 s 2023-09-18 04:22:09,865 44k INFO ====> Epoch: 2433, cost 58.09 s 2023-09-18 04:23:07,727 44k INFO ====> Epoch: 2434, cost 57.86 s 2023-09-18 04:24:05,981 44k INFO ====> Epoch: 2435, cost 58.25 s 2023-09-18 04:25:04,290 44k INFO ====> Epoch: 2436, cost 58.31 s 2023-09-18 04:26:02,818 44k INFO ====> Epoch: 2437, cost 58.53 s 2023-09-18 04:26:32,282 44k INFO Train Epoch: 2438 [47%] 2023-09-18 04:26:32,284 44k INFO Losses: [2.072788715362549, 2.6455273628234863, 15.27103042602539, 22.231061935424805, 0.7528464794158936], step: 78000, lr: 3.6869290352712145e-05, reference_loss: 42.97325134277344 2023-09-18 04:27:01,839 44k INFO ====> Epoch: 2438, cost 59.02 s 2023-09-18 04:28:00,482 44k INFO ====> Epoch: 2439, cost 58.64 s 2023-09-18 04:28:58,890 44k INFO ====> Epoch: 2440, cost 58.41 s 2023-09-18 04:29:57,302 44k INFO ====> Epoch: 2441, cost 58.41 s 2023-09-18 04:30:55,786 44k INFO ====> Epoch: 2442, cost 58.48 s 2023-09-18 04:31:54,351 44k INFO ====> Epoch: 2443, cost 58.57 s 2023-09-18 04:32:39,179 44k INFO Train Epoch: 2444 [72%] 2023-09-18 04:32:39,180 44k INFO Losses: [2.05265474319458, 2.7753231525421143, 15.27588176727295, 21.55010414123535, 0.8295038938522339], step: 78200, lr: 3.6841647024747455e-05, reference_loss: 42.483463287353516 2023-09-18 04:32:53,180 44k INFO ====> Epoch: 2444, cost 58.83 s 2023-09-18 04:33:51,428 44k INFO ====> Epoch: 2445, cost 58.25 s 2023-09-18 04:34:49,510 44k INFO ====> Epoch: 2446, cost 58.08 s 2023-09-18 04:35:48,332 44k INFO ====> Epoch: 2447, cost 58.82 s 2023-09-18 04:36:46,788 44k INFO ====> Epoch: 2448, cost 58.46 s 2023-09-18 04:37:45,513 44k INFO ====> Epoch: 2449, cost 58.72 s 2023-09-18 04:38:43,977 44k INFO Train Epoch: 2450 [97%] 2023-09-18 04:38:43,978 44k INFO Losses: [1.5215191841125488, 3.2239222526550293, 13.686613082885742, 16.063430786132812, -0.8085801601409912], step: 78400, lr: 3.6814024422800916e-05, reference_loss: 33.68690490722656 2023-09-18 04:39:01,496 44k INFO Saving model and optimizer state at iteration 2450 to ./logs/44k/G_78400.pth 2023-09-18 04:39:04,902 44k INFO Saving model and optimizer state at iteration 2450 to ./logs/44k/D_78400.pth 2023-09-18 04:39:05,580 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_74400.pth 2023-09-18 04:39:05,582 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_74400.pth 2023-09-18 04:39:05,582 44k INFO ====> Epoch: 2450, cost 80.07 s 2023-09-18 04:40:03,362 44k INFO ====> Epoch: 2451, cost 57.78 s 2023-09-18 04:41:02,077 44k INFO ====> Epoch: 2452, cost 58.72 s 2023-09-18 04:42:00,335 44k INFO ====> Epoch: 2453, cost 58.26 s 2023-09-18 04:42:58,064 44k INFO ====> Epoch: 2454, cost 57.73 s 2023-09-18 04:43:56,385 44k INFO ====> Epoch: 2455, cost 58.32 s 2023-09-18 04:44:55,112 44k INFO ====> Epoch: 2456, cost 58.73 s 2023-09-18 04:45:09,871 44k INFO Train Epoch: 2457 [22%] 2023-09-18 04:45:09,872 44k INFO Losses: [2.0961427688598633, 3.032569408416748, 14.92405891418457, 21.906185150146484, 0.81947261095047], step: 78600, lr: 3.6781824228516445e-05, reference_loss: 42.77842712402344 2023-09-18 04:45:53,889 44k INFO ====> Epoch: 2457, cost 58.78 s 2023-09-18 04:46:51,433 44k INFO ====> Epoch: 2458, cost 57.54 s 2023-09-18 04:47:49,908 44k INFO ====> Epoch: 2459, cost 58.47 s 2023-09-18 04:48:48,151 44k INFO ====> Epoch: 2460, cost 58.24 s 2023-09-18 04:49:45,874 44k INFO ====> Epoch: 2461, cost 57.72 s 2023-09-18 04:50:44,478 44k INFO ====> Epoch: 2462, cost 58.60 s 2023-09-18 04:51:13,941 44k INFO Train Epoch: 2463 [47%] 2023-09-18 04:51:13,941 44k INFO Losses: [2.0509467124938965, 2.9153237342834473, 15.108786582946777, 21.923852920532227, 0.7460740208625793], step: 78800, lr: 3.675424647964844e-05, reference_loss: 42.7449836730957 2023-09-18 04:51:42,906 44k INFO ====> Epoch: 2463, cost 58.43 s 2023-09-18 04:52:41,160 44k INFO ====> Epoch: 2464, cost 58.25 s 2023-09-18 04:53:39,135 44k INFO ====> Epoch: 2465, cost 57.98 s 2023-09-18 04:54:37,565 44k INFO ====> Epoch: 2466, cost 58.43 s 2023-09-18 04:55:35,993 44k INFO ====> Epoch: 2467, cost 58.43 s 2023-09-18 04:56:34,285 44k INFO ====> Epoch: 2468, cost 58.29 s 2023-09-18 04:57:19,628 44k INFO Train Epoch: 2469 [72%] 2023-09-18 04:57:19,629 44k INFO Losses: [2.0817325115203857, 2.802929162979126, 14.461831092834473, 21.27838706970215, 0.8208445906639099], step: 79000, lr: 3.6726689407629644e-05, reference_loss: 41.44572067260742 2023-09-18 04:57:34,130 44k INFO ====> Epoch: 2469, cost 59.85 s 2023-09-18 04:58:32,251 44k INFO ====> Epoch: 2470, cost 58.12 s 2023-09-18 04:59:30,460 44k INFO ====> Epoch: 2471, cost 58.21 s 2023-09-18 05:00:28,285 44k INFO ====> Epoch: 2472, cost 57.83 s 2023-09-18 05:01:26,472 44k INFO ====> Epoch: 2473, cost 58.19 s 2023-09-18 05:02:24,978 44k INFO ====> Epoch: 2474, cost 58.51 s 2023-09-18 05:03:23,201 44k INFO Train Epoch: 2475 [97%] 2023-09-18 05:03:23,202 44k INFO Losses: [1.547266960144043, 3.161890983581543, 15.040534019470215, 15.710104942321777, -0.7888275384902954], step: 79200, lr: 3.669915299695725e-05, reference_loss: 34.67097091674805 2023-09-18 05:03:39,680 44k INFO Saving model and optimizer state at iteration 2475 to ./logs/44k/G_79200.pth 2023-09-18 05:03:41,830 44k INFO Saving model and optimizer state at iteration 2475 to ./logs/44k/D_79200.pth 2023-09-18 05:03:42,356 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_75200.pth 2023-09-18 05:03:42,357 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_75200.pth 2023-09-18 05:03:42,358 44k INFO ====> Epoch: 2475, cost 77.38 s 2023-09-18 05:04:40,612 44k INFO ====> Epoch: 2476, cost 58.25 s 2023-09-18 05:05:39,287 44k INFO ====> Epoch: 2477, cost 58.67 s 2023-09-18 05:06:37,949 44k INFO ====> Epoch: 2478, cost 58.66 s 2023-09-18 05:07:36,347 44k INFO ====> Epoch: 2479, cost 58.40 s 2023-09-18 05:08:34,869 44k INFO ====> Epoch: 2480, cost 58.52 s 2023-09-18 05:09:33,523 44k INFO ====> Epoch: 2481, cost 58.65 s 2023-09-18 05:09:48,233 44k INFO Train Epoch: 2482 [22%] 2023-09-18 05:09:48,234 44k INFO Losses: [2.0351004600524902, 3.1571755409240723, 15.379192352294922, 21.901779174804688, 0.7984345555305481], step: 79400, lr: 3.666705327748606e-05, reference_loss: 43.27168273925781 2023-09-18 05:10:32,478 44k INFO ====> Epoch: 2482, cost 58.95 s 2023-09-18 05:11:31,193 44k INFO ====> Epoch: 2483, cost 58.72 s 2023-09-18 05:12:29,278 44k INFO ====> Epoch: 2484, cost 58.09 s 2023-09-18 05:13:27,649 44k INFO ====> Epoch: 2485, cost 58.37 s 2023-09-18 05:14:25,742 44k INFO ====> Epoch: 2486, cost 58.09 s 2023-09-18 05:15:23,970 44k INFO ====> Epoch: 2487, cost 58.23 s 2023-09-18 05:15:53,999 44k INFO Train Epoch: 2488 [47%] 2023-09-18 05:15:54,000 44k INFO Losses: [2.092184543609619, 2.70405650138855, 14.594996452331543, 21.774749755859375, 0.740095853805542], step: 79600, lr: 3.6639561579936375e-05, reference_loss: 41.90608215332031 2023-09-18 05:16:23,252 44k INFO ====> Epoch: 2488, cost 59.28 s 2023-09-18 05:17:21,145 44k INFO ====> Epoch: 2489, cost 57.89 s 2023-09-18 05:18:19,784 44k INFO ====> Epoch: 2490, cost 58.64 s 2023-09-18 05:19:18,825 44k INFO ====> Epoch: 2491, cost 59.04 s 2023-09-18 05:20:17,491 44k INFO ====> Epoch: 2492, cost 58.67 s 2023-09-18 05:21:16,036 44k INFO ====> Epoch: 2493, cost 58.54 s 2023-09-18 05:22:00,920 44k INFO Train Epoch: 2494 [72%] 2023-09-18 05:22:00,921 44k INFO Losses: [2.093907594680786, 2.790473222732544, 14.481528282165527, 21.377939224243164, 0.7887418270111084], step: 79800, lr: 3.661209049471756e-05, reference_loss: 41.5325927734375 2023-09-18 05:22:15,032 44k INFO ====> Epoch: 2494, cost 59.00 s 2023-09-18 05:23:13,123 44k INFO ====> Epoch: 2495, cost 58.09 s 2023-09-18 05:24:11,070 44k INFO ====> Epoch: 2496, cost 57.95 s 2023-09-18 05:25:09,521 44k INFO ====> Epoch: 2497, cost 58.45 s 2023-09-18 05:26:07,585 44k INFO ====> Epoch: 2498, cost 58.06 s 2023-09-18 05:27:06,168 44k INFO ====> Epoch: 2499, cost 58.58 s 2023-09-18 05:28:04,019 44k INFO Train Epoch: 2500 [97%] 2023-09-18 05:28:04,020 44k INFO Losses: [1.5427887439727783, 3.407294511795044, 14.330493927001953, 16.856714248657227, -0.7952144145965576], step: 80000, lr: 3.6584640006375195e-05, reference_loss: 35.34207534790039 2023-09-18 05:28:20,818 44k INFO Saving model and optimizer state at iteration 2500 to ./logs/44k/G_80000.pth 2023-09-18 05:28:23,978 44k INFO Saving model and optimizer state at iteration 2500 to ./logs/44k/D_80000.pth 2023-09-18 05:28:25,130 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_76000.pth 2023-09-18 05:28:25,136 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_76000.pth 2023-09-18 05:28:25,136 44k INFO ====> Epoch: 2500, cost 78.97 s 2023-09-18 05:29:24,508 44k INFO ====> Epoch: 2501, cost 59.37 s 2023-09-18 05:30:22,937 44k INFO ====> Epoch: 2502, cost 58.43 s 2023-09-18 05:31:21,430 44k INFO ====> Epoch: 2503, cost 58.49 s 2023-09-18 05:32:20,159 44k INFO ====> Epoch: 2504, cost 58.73 s 2023-09-18 05:33:18,465 44k INFO ====> Epoch: 2505, cost 58.31 s 2023-09-18 05:34:17,433 44k INFO ====> Epoch: 2506, cost 58.97 s 2023-09-18 05:34:32,252 44k INFO Train Epoch: 2507 [22%] 2023-09-18 05:34:32,253 44k INFO Losses: [2.0900027751922607, 2.9762003421783447, 14.765908241271973, 21.719161987304688, 0.7930758595466614], step: 80200, lr: 3.655264044820401e-05, reference_loss: 42.3443489074707 2023-09-18 05:35:16,740 44k INFO ====> Epoch: 2507, cost 59.31 s 2023-09-18 05:36:15,488 44k INFO ====> Epoch: 2508, cost 58.75 s 2023-09-18 05:37:13,925 44k INFO ====> Epoch: 2509, cost 58.44 s 2023-09-18 05:38:12,017 44k INFO ====> Epoch: 2510, cost 58.09 s 2023-09-18 05:39:10,235 44k INFO ====> Epoch: 2511, cost 58.22 s 2023-09-18 05:40:08,890 44k INFO ====> Epoch: 2512, cost 58.66 s 2023-09-18 05:40:38,527 44k INFO Train Epoch: 2513 [47%] 2023-09-18 05:40:38,528 44k INFO Losses: [1.9849591255187988, 2.9876492023468018, 16.021007537841797, 22.097286224365234, 0.7603310346603394], step: 80400, lr: 3.652523453346525e-05, reference_loss: 43.851234436035156 2023-09-18 05:41:07,772 44k INFO ====> Epoch: 2513, cost 58.88 s 2023-09-18 05:42:05,992 44k INFO ====> Epoch: 2514, cost 58.22 s 2023-09-18 05:43:04,287 44k INFO ====> Epoch: 2515, cost 58.30 s 2023-09-18 05:44:02,841 44k INFO ====> Epoch: 2516, cost 58.55 s 2023-09-18 05:45:01,495 44k INFO ====> Epoch: 2517, cost 58.65 s 2023-09-18 05:45:59,690 44k INFO ====> Epoch: 2518, cost 58.19 s 2023-09-18 05:46:44,583 44k INFO Train Epoch: 2519 [72%] 2023-09-18 05:46:44,584 44k INFO Losses: [2.041508674621582, 2.8094840049743652, 14.887720108032227, 21.325969696044922, 0.797172486782074], step: 80600, lr: 3.649784916674035e-05, reference_loss: 41.861854553222656 2023-09-18 05:46:58,822 44k INFO ====> Epoch: 2519, cost 59.13 s 2023-09-18 05:47:56,668 44k INFO ====> Epoch: 2520, cost 57.85 s 2023-09-18 05:48:55,073 44k INFO ====> Epoch: 2521, cost 58.40 s 2023-09-18 05:49:53,740 44k INFO ====> Epoch: 2522, cost 58.67 s 2023-09-18 05:50:51,942 44k INFO ====> Epoch: 2523, cost 58.20 s 2023-09-18 05:51:50,118 44k INFO ====> Epoch: 2524, cost 58.18 s 2023-09-18 05:52:48,542 44k INFO Train Epoch: 2525 [97%] 2023-09-18 05:52:48,543 44k INFO Losses: [1.873676061630249, 2.8087964057922363, 9.257454872131348, 15.907007217407227, -0.7286337018013], step: 80800, lr: 3.6470484332623115e-05, reference_loss: 29.118301391601562 2023-09-18 05:53:05,902 44k INFO Saving model and optimizer state at iteration 2525 to ./logs/44k/G_80800.pth 2023-09-18 05:53:09,297 44k INFO Saving model and optimizer state at iteration 2525 to ./logs/44k/D_80800.pth 2023-09-18 05:53:09,951 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_76800.pth 2023-09-18 05:53:09,952 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_76800.pth 2023-09-18 05:53:09,953 44k INFO ====> Epoch: 2525, cost 79.83 s 2023-09-18 05:54:08,485 44k INFO ====> Epoch: 2526, cost 58.53 s 2023-09-18 05:55:06,432 44k INFO ====> Epoch: 2527, cost 57.95 s 2023-09-18 05:56:05,205 44k INFO ====> Epoch: 2528, cost 58.77 s 2023-09-18 05:57:03,466 44k INFO ====> Epoch: 2529, cost 58.26 s 2023-09-18 05:58:01,972 44k INFO ====> Epoch: 2530, cost 58.51 s 2023-09-18 05:59:00,124 44k INFO ====> Epoch: 2531, cost 58.15 s 2023-09-18 05:59:14,845 44k INFO Train Epoch: 2532 [22%] 2023-09-18 05:59:14,846 44k INFO Losses: [2.0758299827575684, 2.9109416007995605, 14.755376815795898, 21.64443588256836, 0.8156006336212158], step: 81000, lr: 3.6438584623216936e-05, reference_loss: 42.202186584472656 2023-09-18 05:59:58,795 44k INFO ====> Epoch: 2532, cost 58.67 s 2023-09-18 06:00:57,336 44k INFO ====> Epoch: 2533, cost 58.54 s 2023-09-18 06:01:55,564 44k INFO ====> Epoch: 2534, cost 58.23 s 2023-09-18 06:02:53,882 44k INFO ====> Epoch: 2535, cost 58.32 s 2023-09-18 06:03:52,554 44k INFO ====> Epoch: 2536, cost 58.67 s 2023-09-18 06:04:51,297 44k INFO ====> Epoch: 2537, cost 58.74 s 2023-09-18 06:05:21,075 44k INFO Train Epoch: 2538 [47%] 2023-09-18 06:05:21,076 44k INFO Losses: [2.0271384716033936, 2.838885545730591, 15.847275733947754, 22.03387451171875, 0.744698166847229], step: 81200, lr: 3.641126422361954e-05, reference_loss: 43.49187088012695 2023-09-18 06:05:50,535 44k INFO ====> Epoch: 2538, cost 59.24 s 2023-09-18 06:06:48,719 44k INFO ====> Epoch: 2539, cost 58.18 s 2023-09-18 06:07:47,908 44k INFO ====> Epoch: 2540, cost 59.19 s 2023-09-18 06:08:46,067 44k INFO ====> Epoch: 2541, cost 58.16 s 2023-09-18 06:09:44,711 44k INFO ====> Epoch: 2542, cost 58.64 s 2023-09-18 06:10:43,466 44k INFO ====> Epoch: 2543, cost 58.76 s 2023-09-18 06:11:28,308 44k INFO Train Epoch: 2544 [72%] 2023-09-18 06:11:28,309 44k INFO Losses: [2.0323586463928223, 2.8412818908691406, 15.355264663696289, 21.460771560668945, 0.8057379722595215], step: 81400, lr: 3.638396430791969e-05, reference_loss: 42.49541091918945 2023-09-18 06:11:42,664 44k INFO ====> Epoch: 2544, cost 59.20 s 2023-09-18 06:12:41,055 44k INFO ====> Epoch: 2545, cost 58.39 s 2023-09-18 06:13:39,484 44k INFO ====> Epoch: 2546, cost 58.43 s 2023-09-18 06:14:38,016 44k INFO ====> Epoch: 2547, cost 58.53 s 2023-09-18 06:15:36,594 44k INFO ====> Epoch: 2548, cost 58.58 s 2023-09-18 06:16:35,515 44k INFO ====> Epoch: 2549, cost 58.92 s 2023-09-18 06:17:34,315 44k INFO Train Epoch: 2550 [97%] 2023-09-18 06:17:34,316 44k INFO Losses: [1.6357882022857666, 3.3336620330810547, 15.242067337036133, 15.98670482635498, -0.8036141991615295], step: 81600, lr: 3.635668486075926e-05, reference_loss: 35.39460754394531 2023-09-18 06:17:51,322 44k INFO Saving model and optimizer state at iteration 2550 to ./logs/44k/G_81600.pth 2023-09-18 06:17:54,803 44k INFO Saving model and optimizer state at iteration 2550 to ./logs/44k/D_81600.pth 2023-09-18 06:17:55,356 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_77600.pth 2023-09-18 06:17:55,357 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_77600.pth 2023-09-18 06:17:55,357 44k INFO ====> Epoch: 2550, cost 79.84 s 2023-09-18 06:18:54,107 44k INFO ====> Epoch: 2551, cost 58.75 s 2023-09-18 06:19:52,651 44k INFO ====> Epoch: 2552, cost 58.54 s 2023-09-18 06:20:51,023 44k INFO ====> Epoch: 2553, cost 58.37 s 2023-09-18 06:21:49,354 44k INFO ====> Epoch: 2554, cost 58.33 s 2023-09-18 06:22:48,322 44k INFO ====> Epoch: 2555, cost 58.97 s 2023-09-18 06:23:46,668 44k INFO ====> Epoch: 2556, cost 58.35 s 2023-09-18 06:24:01,288 44k INFO Train Epoch: 2557 [22%] 2023-09-18 06:24:01,289 44k INFO Losses: [2.0847747325897217, 3.0798068046569824, 16.310632705688477, 22.032468795776367, 0.803991973400116], step: 81800, lr: 3.632488468855829e-05, reference_loss: 44.311676025390625 2023-09-18 06:24:45,076 44k INFO ====> Epoch: 2557, cost 58.41 s 2023-09-18 06:25:43,757 44k INFO ====> Epoch: 2558, cost 58.68 s 2023-09-18 06:26:42,020 44k INFO ====> Epoch: 2559, cost 58.26 s 2023-09-18 06:27:40,443 44k INFO ====> Epoch: 2560, cost 58.42 s 2023-09-18 06:28:39,071 44k INFO ====> Epoch: 2561, cost 58.63 s 2023-09-18 06:29:37,552 44k INFO ====> Epoch: 2562, cost 58.48 s 2023-09-18 06:30:07,389 44k INFO Train Epoch: 2563 [47%] 2023-09-18 06:30:07,390 44k INFO Losses: [2.034182548522949, 2.889552116394043, 15.251166343688965, 21.85786247253418, 0.7454743981361389], step: 82000, lr: 3.6297649537267906e-05, reference_loss: 42.778236389160156 2023-09-18 06:30:36,885 44k INFO ====> Epoch: 2563, cost 59.33 s 2023-09-18 06:31:35,342 44k INFO ====> Epoch: 2564, cost 58.46 s 2023-09-18 06:32:33,294 44k INFO ====> Epoch: 2565, cost 57.95 s 2023-09-18 06:33:31,883 44k INFO ====> Epoch: 2566, cost 58.59 s 2023-09-18 06:34:30,415 44k INFO ====> Epoch: 2567, cost 58.53 s 2023-09-18 06:35:28,736 44k INFO ====> Epoch: 2568, cost 58.32 s 2023-09-18 06:36:14,138 44k INFO Train Epoch: 2569 [72%] 2023-09-18 06:36:14,139 44k INFO Losses: [2.0142147541046143, 2.7714855670928955, 15.37228775024414, 21.46529769897461, 0.8333816528320312], step: 82200, lr: 3.627043480595881e-05, reference_loss: 42.4566650390625 2023-09-18 06:36:28,449 44k INFO ====> Epoch: 2569, cost 59.71 s 2023-09-18 06:37:27,630 44k INFO ====> Epoch: 2570, cost 59.18 s 2023-09-18 06:38:26,470 44k INFO ====> Epoch: 2571, cost 58.84 s 2023-09-18 06:39:24,751 44k INFO ====> Epoch: 2572, cost 58.28 s 2023-09-18 06:40:23,125 44k INFO ====> Epoch: 2573, cost 58.37 s 2023-09-18 06:41:21,009 44k INFO ====> Epoch: 2574, cost 57.88 s 2023-09-18 06:42:19,391 44k INFO Train Epoch: 2575 [97%] 2023-09-18 06:42:19,392 44k INFO Losses: [1.3797744512557983, 3.567558765411377, 14.54659652709961, 16.30454444885254, -0.8684641718864441], step: 82400, lr: 3.624324047932081e-05, reference_loss: 34.93001174926758 2023-09-18 06:42:36,187 44k INFO Saving model and optimizer state at iteration 2575 to ./logs/44k/G_82400.pth 2023-09-18 06:42:39,624 44k INFO Saving model and optimizer state at iteration 2575 to ./logs/44k/D_82400.pth 2023-09-18 06:42:40,675 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_78400.pth 2023-09-18 06:42:40,677 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_78400.pth 2023-09-18 06:42:40,677 44k INFO ====> Epoch: 2575, cost 79.67 s 2023-09-18 06:43:39,238 44k INFO ====> Epoch: 2576, cost 58.56 s 2023-09-18 06:44:38,186 44k INFO ====> Epoch: 2577, cost 58.95 s 2023-09-18 06:45:37,064 44k INFO ====> Epoch: 2578, cost 58.88 s 2023-09-18 06:46:35,108 44k INFO ====> Epoch: 2579, cost 58.04 s 2023-09-18 06:47:33,484 44k INFO ====> Epoch: 2580, cost 58.38 s 2023-09-18 06:48:31,894 44k INFO ====> Epoch: 2581, cost 58.41 s 2023-09-18 06:48:46,727 44k INFO Train Epoch: 2582 [22%] 2023-09-18 06:48:46,728 44k INFO Losses: [2.024264335632324, 3.025566816329956, 16.081636428833008, 22.0011043548584, 0.776301920413971], step: 82600, lr: 3.621153953373742e-05, reference_loss: 43.90887451171875 2023-09-18 06:49:31,425 44k INFO ====> Epoch: 2582, cost 59.53 s 2023-09-18 06:50:30,200 44k INFO ====> Epoch: 2583, cost 58.77 s 2023-09-18 06:51:28,677 44k INFO ====> Epoch: 2584, cost 58.48 s 2023-09-18 06:52:27,246 44k INFO ====> Epoch: 2585, cost 58.57 s 2023-09-18 06:53:25,719 44k INFO ====> Epoch: 2586, cost 58.47 s 2023-09-18 06:54:24,364 44k INFO ====> Epoch: 2587, cost 58.64 s 2023-09-18 06:54:54,318 44k INFO Train Epoch: 2588 [47%] 2023-09-18 06:54:54,319 44k INFO Losses: [1.983131766319275, 2.892728805541992, 14.838948249816895, 21.8153133392334, 0.7239136695861816], step: 82800, lr: 3.6184389364752304e-05, reference_loss: 42.25403594970703 2023-09-18 06:55:23,282 44k INFO ====> Epoch: 2588, cost 58.92 s 2023-09-18 06:56:22,047 44k INFO ====> Epoch: 2589, cost 58.76 s 2023-09-18 06:57:20,671 44k INFO ====> Epoch: 2590, cost 58.62 s 2023-09-18 06:58:19,757 44k INFO ====> Epoch: 2591, cost 59.09 s 2023-09-18 06:59:18,109 44k INFO ====> Epoch: 2592, cost 58.35 s 2023-09-18 07:00:17,079 44k INFO ====> Epoch: 2593, cost 58.97 s 2023-09-18 07:01:01,773 44k INFO Train Epoch: 2594 [72%] 2023-09-18 07:01:01,774 44k INFO Losses: [2.078009843826294, 2.7368927001953125, 14.992169380187988, 21.299320220947266, 0.7978481650352478], step: 83000, lr: 3.615725955203167e-05, reference_loss: 41.904239654541016 2023-09-18 07:01:16,000 44k INFO ====> Epoch: 2594, cost 58.92 s 2023-09-18 07:02:14,222 44k INFO ====> Epoch: 2595, cost 58.22 s 2023-09-18 07:03:12,307 44k INFO ====> Epoch: 2596, cost 58.09 s 2023-09-18 07:04:10,790 44k INFO ====> Epoch: 2597, cost 58.48 s 2023-09-18 07:05:08,966 44k INFO ====> Epoch: 2598, cost 58.18 s 2023-09-18 07:06:07,469 44k INFO ====> Epoch: 2599, cost 58.50 s 2023-09-18 07:07:05,503 44k INFO Train Epoch: 2600 [97%] 2023-09-18 07:07:05,504 44k INFO Losses: [1.4703433513641357, 3.2396225929260254, 15.064845085144043, 16.29201889038086, -0.8787950277328491], step: 83200, lr: 3.613015008031308e-05, reference_loss: 35.18803405761719 2023-09-18 07:07:22,212 44k INFO Saving model and optimizer state at iteration 2600 to ./logs/44k/G_83200.pth 2023-09-18 07:07:24,958 44k INFO Saving model and optimizer state at iteration 2600 to ./logs/44k/D_83200.pth 2023-09-18 07:07:25,476 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_79200.pth 2023-09-18 07:07:25,479 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_79200.pth 2023-09-18 07:07:25,479 44k INFO ====> Epoch: 2600, cost 78.01 s 2023-09-18 07:08:23,632 44k INFO ====> Epoch: 2601, cost 58.15 s 2023-09-18 07:09:22,141 44k INFO ====> Epoch: 2602, cost 58.51 s 2023-09-18 07:10:20,888 44k INFO ====> Epoch: 2603, cost 58.75 s 2023-09-18 07:11:20,075 44k INFO ====> Epoch: 2604, cost 59.19 s 2023-09-18 07:12:17,937 44k INFO ====> Epoch: 2605, cost 57.86 s 2023-09-18 07:13:16,265 44k INFO ====> Epoch: 2606, cost 58.33 s 2023-09-18 07:13:31,063 44k INFO Train Epoch: 2607 [22%] 2023-09-18 07:13:31,064 44k INFO Losses: [2.0953731536865234, 3.088273763656616, 15.639832496643066, 22.002809524536133, 0.8001629114151001], step: 83400, lr: 3.609854805172877e-05, reference_loss: 43.62644958496094 2023-09-18 07:14:15,668 44k INFO ====> Epoch: 2607, cost 59.40 s 2023-09-18 07:15:14,719 44k INFO ====> Epoch: 2608, cost 59.05 s 2023-09-18 07:16:12,958 44k INFO ====> Epoch: 2609, cost 58.24 s 2023-09-18 07:17:11,563 44k INFO ====> Epoch: 2610, cost 58.60 s 2023-09-18 07:18:09,836 44k INFO ====> Epoch: 2611, cost 58.27 s 2023-09-18 07:19:08,127 44k INFO ====> Epoch: 2612, cost 58.29 s 2023-09-18 07:19:38,179 44k INFO Train Epoch: 2613 [47%] 2023-09-18 07:19:38,180 44k INFO Losses: [2.0264649391174316, 2.7188830375671387, 15.883180618286133, 21.962493896484375, 0.7447996735572815], step: 83600, lr: 3.6071482599877196e-05, reference_loss: 43.33582305908203 2023-09-18 07:20:07,280 44k INFO ====> Epoch: 2613, cost 59.15 s 2023-09-18 07:21:05,838 44k INFO ====> Epoch: 2614, cost 58.56 s 2023-09-18 07:22:04,347 44k INFO ====> Epoch: 2615, cost 58.51 s 2023-09-18 07:23:02,704 44k INFO ====> Epoch: 2616, cost 58.36 s 2023-09-18 07:24:01,808 44k INFO ====> Epoch: 2617, cost 59.10 s 2023-09-18 07:25:00,395 44k INFO ====> Epoch: 2618, cost 58.59 s 2023-09-18 07:25:45,018 44k INFO Train Epoch: 2619 [72%] 2023-09-18 07:25:45,019 44k INFO Losses: [2.0429160594940186, 2.834721326828003, 15.224031448364258, 21.146203994750977, 0.7811703681945801], step: 83800, lr: 3.604443744077211e-05, reference_loss: 42.0290412902832 2023-09-18 07:25:59,458 44k INFO ====> Epoch: 2619, cost 59.06 s 2023-09-18 07:26:58,516 44k INFO ====> Epoch: 2620, cost 59.06 s 2023-09-18 07:27:57,297 44k INFO ====> Epoch: 2621, cost 58.78 s 2023-09-18 07:28:56,584 44k INFO ====> Epoch: 2622, cost 59.29 s 2023-09-18 07:29:56,453 44k INFO ====> Epoch: 2623, cost 59.87 s 2023-09-18 07:30:56,348 44k INFO ====> Epoch: 2624, cost 59.90 s 2023-09-18 07:31:55,763 44k INFO Train Epoch: 2625 [97%] 2023-09-18 07:31:55,764 44k INFO Losses: [1.392472267150879, 3.509364128112793, 14.9949951171875, 16.006071090698242, -0.8865563869476318], step: 84000, lr: 3.601741255919869e-05, reference_loss: 35.0163459777832 2023-09-18 07:32:12,865 44k INFO Saving model and optimizer state at iteration 2625 to ./logs/44k/G_84000.pth 2023-09-18 07:32:16,013 44k INFO Saving model and optimizer state at iteration 2625 to ./logs/44k/D_84000.pth 2023-09-18 07:32:17,134 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_80000.pth 2023-09-18 07:32:17,142 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_80000.pth 2023-09-18 07:32:17,153 44k INFO ====> Epoch: 2625, cost 80.80 s 2023-09-18 07:33:16,928 44k INFO ====> Epoch: 2626, cost 59.78 s 2023-09-18 07:34:16,703 44k INFO ====> Epoch: 2627, cost 59.77 s 2023-09-18 07:35:15,399 44k INFO ====> Epoch: 2628, cost 58.70 s 2023-09-18 07:36:14,938 44k INFO ====> Epoch: 2629, cost 59.54 s 2023-09-18 07:37:14,747 44k INFO ====> Epoch: 2630, cost 59.81 s 2023-09-18 07:38:14,460 44k INFO ====> Epoch: 2631, cost 59.71 s 2023-09-18 07:38:29,660 44k INFO Train Epoch: 2632 [22%] 2023-09-18 07:38:29,662 44k INFO Losses: [2.0595521926879883, 2.995741605758667, 14.964737892150879, 21.82304573059082, 0.7752523422241211], step: 84200, lr: 3.5985909138961056e-05, reference_loss: 42.61833190917969 2023-09-18 07:39:14,657 44k INFO ====> Epoch: 2632, cost 60.20 s 2023-09-18 07:40:14,525 44k INFO ====> Epoch: 2633, cost 59.87 s 2023-09-18 07:41:14,185 44k INFO ====> Epoch: 2634, cost 59.66 s 2023-09-18 07:42:13,643 44k INFO ====> Epoch: 2635, cost 59.46 s 2023-09-18 07:43:13,110 44k INFO ====> Epoch: 2636, cost 59.47 s 2023-09-18 07:44:12,811 44k INFO ====> Epoch: 2637, cost 59.70 s 2023-09-18 07:44:43,327 44k INFO Train Epoch: 2638 [47%] 2023-09-18 07:44:43,329 44k INFO Losses: [2.019850254058838, 2.8949806690216064, 15.764652252197266, 21.879594802856445, 0.7252859473228455], step: 84400, lr: 3.595892813989871e-05, reference_loss: 43.28436279296875 2023-09-18 07:45:12,835 44k INFO ====> Epoch: 2638, cost 60.02 s 2023-09-18 07:46:12,090 44k INFO ====> Epoch: 2639, cost 59.26 s 2023-09-18 07:47:12,142 44k INFO ====> Epoch: 2640, cost 60.05 s 2023-09-18 07:48:12,018 44k INFO ====> Epoch: 2641, cost 59.88 s 2023-09-18 07:49:11,807 44k INFO ====> Epoch: 2642, cost 59.79 s 2023-09-18 07:50:10,743 44k INFO ====> Epoch: 2643, cost 58.94 s 2023-09-18 07:50:56,243 44k INFO Train Epoch: 2644 [72%] 2023-09-18 07:50:56,244 44k INFO Losses: [2.0566205978393555, 2.833756446838379, 15.611577987670898, 21.50995635986328, 0.801095187664032], step: 84600, lr: 3.593196737026304e-05, reference_loss: 42.81300354003906 2023-09-18 07:51:10,846 44k INFO ====> Epoch: 2644, cost 60.10 s 2023-09-18 07:52:10,359 44k INFO ====> Epoch: 2645, cost 59.51 s 2023-09-18 07:53:09,941 44k INFO ====> Epoch: 2646, cost 59.58 s 2023-09-18 07:54:08,988 44k INFO ====> Epoch: 2647, cost 59.05 s 2023-09-18 07:55:09,097 44k INFO ====> Epoch: 2648, cost 60.11 s 2023-09-18 07:56:09,511 44k INFO ====> Epoch: 2649, cost 60.41 s 2023-09-18 07:57:09,248 44k INFO Train Epoch: 2650 [97%] 2023-09-18 07:57:09,249 44k INFO Losses: [1.433654546737671, 3.375859498977661, 16.799449920654297, 16.17645263671875, -0.8043685555458069], step: 84800, lr: 3.590502681488673e-05, reference_loss: 36.981048583984375 2023-09-18 07:57:25,975 44k INFO Saving model and optimizer state at iteration 2650 to ./logs/44k/G_84800.pth 2023-09-18 07:57:29,418 44k INFO Saving model and optimizer state at iteration 2650 to ./logs/44k/D_84800.pth 2023-09-18 07:57:30,134 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_80800.pth 2023-09-18 07:57:30,135 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_80800.pth 2023-09-18 07:57:30,135 44k INFO ====> Epoch: 2650, cost 80.62 s 2023-09-18 07:58:29,260 44k INFO ====> Epoch: 2651, cost 59.13 s 2023-09-18 07:59:28,724 44k INFO ====> Epoch: 2652, cost 59.46 s 2023-09-18 08:00:28,124 44k INFO ====> Epoch: 2653, cost 59.40 s 2023-09-18 08:01:27,872 44k INFO ====> Epoch: 2654, cost 59.75 s 2023-09-18 08:02:27,225 44k INFO ====> Epoch: 2655, cost 59.35 s 2023-09-18 08:03:26,946 44k INFO ====> Epoch: 2656, cost 59.72 s 2023-09-18 08:03:42,060 44k INFO Train Epoch: 2657 [22%] 2023-09-18 08:03:42,061 44k INFO Losses: [2.071089506149292, 2.882488250732422, 15.197637557983398, 21.874624252319336, 0.7844153642654419], step: 85000, lr: 3.587362169530648e-05, reference_loss: 42.81025695800781 2023-09-18 08:04:26,849 44k INFO ====> Epoch: 2657, cost 59.90 s 2023-09-18 08:05:26,278 44k INFO ====> Epoch: 2658, cost 59.43 s 2023-09-18 08:06:25,450 44k INFO ====> Epoch: 2659, cost 59.17 s 2023-09-18 08:07:24,681 44k INFO ====> Epoch: 2660, cost 59.23 s 2023-09-18 08:08:24,504 44k INFO ====> Epoch: 2661, cost 59.82 s 2023-09-18 08:09:24,055 44k INFO ====> Epoch: 2662, cost 59.55 s 2023-09-18 08:09:54,778 44k INFO Train Epoch: 2663 [47%] 2023-09-18 08:09:54,779 44k INFO Losses: [2.0188002586364746, 2.9244673252105713, 15.410161972045898, 21.94852066040039, 0.7293481230735779], step: 85200, lr: 3.5846724885513894e-05, reference_loss: 43.03129577636719 2023-09-18 08:10:24,392 44k INFO ====> Epoch: 2663, cost 60.34 s 2023-09-18 08:11:24,236 44k INFO ====> Epoch: 2664, cost 59.84 s 2023-09-18 08:12:23,825 44k INFO ====> Epoch: 2665, cost 59.59 s 2023-09-18 08:13:23,192 44k INFO ====> Epoch: 2666, cost 59.37 s 2023-09-18 08:14:22,552 44k INFO ====> Epoch: 2667, cost 59.36 s 2023-09-18 08:15:22,302 44k INFO ====> Epoch: 2668, cost 59.75 s 2023-09-18 08:16:07,821 44k INFO Train Epoch: 2669 [72%] 2023-09-18 08:16:07,823 44k INFO Losses: [2.040451765060425, 2.8291175365448, 16.238895416259766, 21.543636322021484, 0.8139227032661438], step: 85400, lr: 3.581984824202576e-05, reference_loss: 43.46602249145508 2023-09-18 08:16:22,151 44k INFO ====> Epoch: 2669, cost 59.85 s 2023-09-18 08:17:21,877 44k INFO ====> Epoch: 2670, cost 59.73 s 2023-09-18 08:18:21,926 44k INFO ====> Epoch: 2671, cost 60.05 s 2023-09-18 08:19:21,527 44k INFO ====> Epoch: 2672, cost 59.60 s 2023-09-18 08:20:21,189 44k INFO ====> Epoch: 2673, cost 59.66 s 2023-09-18 08:21:20,679 44k INFO ====> Epoch: 2674, cost 59.49 s 2023-09-18 08:22:19,511 44k INFO Train Epoch: 2675 [97%] 2023-09-18 08:22:19,512 44k INFO Losses: [1.5036730766296387, 3.0545244216918945, 12.104247093200684, 15.030006408691406, -0.871816098690033], step: 85600, lr: 3.579299174972209e-05, reference_loss: 30.820634841918945 2023-09-18 08:22:36,996 44k INFO Saving model and optimizer state at iteration 2675 to ./logs/44k/G_85600.pth 2023-09-18 08:22:39,704 44k INFO Saving model and optimizer state at iteration 2675 to ./logs/44k/D_85600.pth 2023-09-18 08:22:40,256 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_81600.pth 2023-09-18 08:22:40,258 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_81600.pth 2023-09-18 08:22:40,258 44k INFO ====> Epoch: 2675, cost 79.58 s 2023-09-18 08:23:39,325 44k INFO ====> Epoch: 2676, cost 59.07 s 2023-09-18 08:24:38,420 44k INFO ====> Epoch: 2677, cost 59.09 s 2023-09-18 08:25:38,337 44k INFO ====> Epoch: 2678, cost 59.92 s 2023-09-18 08:26:37,299 44k INFO ====> Epoch: 2679, cost 58.96 s 2023-09-18 08:27:36,609 44k INFO ====> Epoch: 2680, cost 59.31 s 2023-09-18 08:28:35,571 44k INFO ====> Epoch: 2681, cost 58.96 s 2023-09-18 08:28:50,588 44k INFO Train Epoch: 2682 [22%] 2023-09-18 08:28:50,589 44k INFO Losses: [2.0152738094329834, 2.950700283050537, 15.985217094421387, 21.853147506713867, 0.7791374325752258], step: 85800, lr: 3.5761684624070006e-05, reference_loss: 43.583473205566406 2023-09-18 08:29:35,541 44k INFO ====> Epoch: 2682, cost 59.97 s 2023-09-18 08:30:34,595 44k INFO ====> Epoch: 2683, cost 59.05 s 2023-09-18 08:31:34,692 44k INFO ====> Epoch: 2684, cost 60.10 s 2023-09-18 08:32:33,653 44k INFO ====> Epoch: 2685, cost 58.96 s 2023-09-18 08:33:33,292 44k INFO ====> Epoch: 2686, cost 59.64 s 2023-09-18 08:34:32,783 44k INFO ====> Epoch: 2687, cost 59.49 s 2023-09-18 08:35:02,965 44k INFO Train Epoch: 2688 [47%] 2023-09-18 08:35:02,966 44k INFO Losses: [1.9908055067062378, 2.9876742362976074, 15.080568313598633, 21.921661376953125, 0.7232112884521484], step: 86000, lr: 3.5734871740849964e-05, reference_loss: 42.70391845703125 2023-09-18 08:35:32,889 44k INFO ====> Epoch: 2688, cost 60.11 s 2023-09-18 08:36:32,989 44k INFO ====> Epoch: 2689, cost 60.10 s 2023-09-18 08:37:32,519 44k INFO ====> Epoch: 2690, cost 59.53 s 2023-09-18 08:38:31,935 44k INFO ====> Epoch: 2691, cost 59.42 s 2023-09-18 08:39:31,266 44k INFO ====> Epoch: 2692, cost 59.33 s 2023-09-18 08:40:30,860 44k INFO ====> Epoch: 2693, cost 59.59 s 2023-09-18 08:41:16,495 44k INFO Train Epoch: 2694 [72%] 2023-09-18 08:41:16,496 44k INFO Losses: [2.049130439758301, 2.833765983581543, 16.06048011779785, 21.59493637084961, 0.7949966788291931], step: 86200, lr: 3.570807896100912e-05, reference_loss: 43.33331298828125 2023-09-18 08:41:31,295 44k INFO ====> Epoch: 2694, cost 60.43 s 2023-09-18 08:42:30,792 44k INFO ====> Epoch: 2695, cost 59.50 s 2023-09-18 08:43:29,904 44k INFO ====> Epoch: 2696, cost 59.11 s 2023-09-18 08:44:29,491 44k INFO ====> Epoch: 2697, cost 59.59 s 2023-09-18 08:45:28,653 44k INFO ====> Epoch: 2698, cost 59.16 s 2023-09-18 08:46:28,305 44k INFO ====> Epoch: 2699, cost 59.65 s 2023-09-18 08:47:28,078 44k INFO Train Epoch: 2700 [97%] 2023-09-18 08:47:28,079 44k INFO Losses: [1.4296119213104248, 3.4631106853485107, 11.264873504638672, 15.882285118103027, -0.8697938919067383], step: 86400, lr: 3.5681306269474645e-05, reference_loss: 31.170089721679688 2023-09-18 08:47:45,565 44k INFO Saving model and optimizer state at iteration 2700 to ./logs/44k/G_86400.pth 2023-09-18 08:47:48,910 44k INFO Saving model and optimizer state at iteration 2700 to ./logs/44k/D_86400.pth 2023-09-18 08:47:50,007 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_82400.pth 2023-09-18 08:47:50,009 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_82400.pth 2023-09-18 08:47:50,010 44k INFO ====> Epoch: 2700, cost 81.70 s 2023-09-18 08:48:49,937 44k INFO ====> Epoch: 2701, cost 59.93 s 2023-09-18 08:49:49,687 44k INFO ====> Epoch: 2702, cost 59.75 s 2023-09-18 08:50:49,751 44k INFO ====> Epoch: 2703, cost 60.06 s 2023-09-18 08:51:49,319 44k INFO ====> Epoch: 2704, cost 59.57 s 2023-09-18 08:52:48,694 44k INFO ====> Epoch: 2705, cost 59.38 s 2023-09-18 08:53:48,356 44k INFO ====> Epoch: 2706, cost 59.66 s 2023-09-18 08:54:03,683 44k INFO Train Epoch: 2707 [22%] 2023-09-18 08:54:03,684 44k INFO Losses: [2.064453601837158, 2.9582619667053223, 14.60705280303955, 21.869640350341797, 0.7732359766960144], step: 86600, lr: 3.565009683197862e-05, reference_loss: 42.27264404296875 2023-09-18 08:54:48,979 44k INFO ====> Epoch: 2707, cost 60.62 s 2023-09-18 08:55:49,013 44k INFO ====> Epoch: 2708, cost 60.03 s 2023-09-18 08:56:47,916 44k INFO ====> Epoch: 2709, cost 58.90 s 2023-09-18 08:57:46,831 44k INFO ====> Epoch: 2710, cost 58.92 s 2023-09-18 08:58:45,168 44k INFO ====> Epoch: 2711, cost 58.34 s 2023-09-18 08:59:43,993 44k INFO ====> Epoch: 2712, cost 58.83 s 2023-09-18 09:00:13,819 44k INFO Train Epoch: 2713 [47%] 2023-09-18 09:00:13,820 44k INFO Losses: [2.0419628620147705, 2.854191780090332, 15.0718994140625, 21.864519119262695, 0.7563018202781677], step: 86800, lr: 3.562336761345362e-05, reference_loss: 42.58887481689453 2023-09-18 09:00:43,303 44k INFO ====> Epoch: 2713, cost 59.31 s 2023-09-18 09:01:42,484 44k INFO ====> Epoch: 2714, cost 59.18 s 2023-09-18 09:02:40,959 44k INFO ====> Epoch: 2715, cost 58.48 s 2023-09-18 09:03:39,905 44k INFO ====> Epoch: 2716, cost 58.95 s 2023-09-18 09:04:38,673 44k INFO ====> Epoch: 2717, cost 58.77 s 2023-09-18 09:05:37,449 44k INFO ====> Epoch: 2718, cost 58.78 s 2023-09-18 09:06:22,727 44k INFO Train Epoch: 2719 [72%] 2023-09-18 09:06:22,728 44k INFO Losses: [2.0394163131713867, 2.738412857055664, 15.150118827819824, 21.254364013671875, 0.7792297601699829], step: 87000, lr: 3.5596658435578896e-05, reference_loss: 41.96154022216797 2023-09-18 09:06:36,973 44k INFO ====> Epoch: 2719, cost 59.52 s 2023-09-18 09:07:35,581 44k INFO ====> Epoch: 2720, cost 58.61 s 2023-09-18 09:08:34,093 44k INFO ====> Epoch: 2721, cost 58.51 s 2023-09-18 09:09:32,454 44k INFO ====> Epoch: 2722, cost 58.36 s 2023-09-18 09:10:30,942 44k INFO ====> Epoch: 2723, cost 58.49 s 2023-09-18 09:11:29,094 44k INFO ====> Epoch: 2724, cost 58.15 s 2023-09-18 09:12:27,760 44k INFO Train Epoch: 2725 [97%] 2023-09-18 09:12:27,761 44k INFO Losses: [1.2440898418426514, 3.771162748336792, 14.897324562072754, 16.0648193359375, -0.8865500092506409], step: 87200, lr: 3.5569969283328653e-05, reference_loss: 35.090843200683594 2023-09-18 09:12:45,390 44k INFO Saving model and optimizer state at iteration 2725 to ./logs/44k/G_87200.pth 2023-09-18 09:12:47,985 44k INFO Saving model and optimizer state at iteration 2725 to ./logs/44k/D_87200.pth 2023-09-18 09:12:48,495 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_83200.pth 2023-09-18 09:12:48,496 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_83200.pth 2023-09-18 09:12:48,496 44k INFO ====> Epoch: 2725, cost 79.40 s 2023-09-18 09:13:47,049 44k INFO ====> Epoch: 2726, cost 58.55 s 2023-09-18 09:14:45,562 44k INFO ====> Epoch: 2727, cost 58.51 s 2023-09-18 09:15:44,329 44k INFO ====> Epoch: 2728, cost 58.77 s 2023-09-18 09:16:42,954 44k INFO ====> Epoch: 2729, cost 58.62 s 2023-09-18 09:17:41,536 44k INFO ====> Epoch: 2730, cost 58.58 s 2023-09-18 09:18:39,960 44k INFO ====> Epoch: 2731, cost 58.42 s 2023-09-18 09:18:54,765 44k INFO Train Epoch: 2732 [22%] 2023-09-18 09:18:54,766 44k INFO Losses: [2.0353472232818604, 3.2028043270111084, 15.589210510253906, 21.73322296142578, 0.7933074235916138], step: 87400, lr: 3.5538857229170675e-05, reference_loss: 43.3538932800293 2023-09-18 09:19:39,374 44k INFO ====> Epoch: 2732, cost 59.41 s 2023-09-18 09:20:38,153 44k INFO ====> Epoch: 2733, cost 58.78 s 2023-09-18 09:21:36,756 44k INFO ====> Epoch: 2734, cost 58.60 s 2023-09-18 09:22:35,106 44k INFO ====> Epoch: 2735, cost 58.35 s 2023-09-18 09:23:34,192 44k INFO ====> Epoch: 2736, cost 59.09 s 2023-09-18 09:24:33,163 44k INFO ====> Epoch: 2737, cost 58.97 s 2023-09-18 09:25:02,931 44k INFO Train Epoch: 2738 [47%] 2023-09-18 09:25:02,932 44k INFO Losses: [2.0789542198181152, 2.7972590923309326, 14.699854850769043, 21.77562713623047, 0.7354401350021362], step: 87600, lr: 3.551221141428035e-05, reference_loss: 42.087135314941406 2023-09-18 09:25:32,554 44k INFO ====> Epoch: 2738, cost 59.39 s 2023-09-18 09:26:30,948 44k INFO ====> Epoch: 2739, cost 58.39 s 2023-09-18 09:27:29,547 44k INFO ====> Epoch: 2740, cost 58.60 s 2023-09-18 09:28:28,447 44k INFO ====> Epoch: 2741, cost 58.90 s 2023-09-18 09:29:27,403 44k INFO ====> Epoch: 2742, cost 58.96 s 2023-09-18 09:30:26,233 44k INFO ====> Epoch: 2743, cost 58.83 s 2023-09-18 09:31:11,342 44k INFO Train Epoch: 2744 [72%] 2023-09-18 09:31:11,343 44k INFO Losses: [2.0475993156433105, 2.8013112545013428, 15.839824676513672, 21.340286254882812, 0.7849003672599792], step: 87800, lr: 3.548558557750712e-05, reference_loss: 42.81392288208008 2023-09-18 09:31:25,708 44k INFO ====> Epoch: 2744, cost 59.47 s 2023-09-18 09:32:24,157 44k INFO ====> Epoch: 2745, cost 58.45 s 2023-09-18 09:33:23,145 44k INFO ====> Epoch: 2746, cost 58.99 s 2023-09-18 09:34:21,651 44k INFO ====> Epoch: 2747, cost 58.51 s 2023-09-18 09:35:20,217 44k INFO ====> Epoch: 2748, cost 58.57 s 2023-09-18 09:36:18,803 44k INFO ====> Epoch: 2749, cost 58.59 s 2023-09-18 09:37:17,632 44k INFO Train Epoch: 2750 [97%] 2023-09-18 09:37:17,633 44k INFO Losses: [1.6522622108459473, 2.940011501312256, 11.583182334899902, 15.950223922729492, -0.8564541339874268], step: 88000, lr: 3.545897970387207e-05, reference_loss: 31.269224166870117 2023-09-18 09:37:34,879 44k INFO Saving model and optimizer state at iteration 2750 to ./logs/44k/G_88000.pth 2023-09-18 09:37:37,901 44k INFO Saving model and optimizer state at iteration 2750 to ./logs/44k/D_88000.pth 2023-09-18 09:37:38,923 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_84000.pth 2023-09-18 09:37:38,930 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_84000.pth 2023-09-18 09:37:38,941 44k INFO ====> Epoch: 2750, cost 80.14 s 2023-09-18 09:38:37,566 44k INFO ====> Epoch: 2751, cost 58.63 s 2023-09-18 09:39:36,202 44k INFO ====> Epoch: 2752, cost 58.64 s 2023-09-18 09:40:34,959 44k INFO ====> Epoch: 2753, cost 58.76 s 2023-09-18 09:41:33,452 44k INFO ====> Epoch: 2754, cost 58.49 s 2023-09-18 09:42:32,044 44k INFO ====> Epoch: 2755, cost 58.59 s 2023-09-18 09:43:30,414 44k INFO ====> Epoch: 2756, cost 58.37 s 2023-09-18 09:43:45,342 44k INFO Train Epoch: 2757 [22%] 2023-09-18 09:43:45,343 44k INFO Losses: [2.0185604095458984, 3.0232858657836914, 16.48313331604004, 21.78895378112793, 0.7800450325012207], step: 88200, lr: 3.5427964729185235e-05, reference_loss: 44.09397888183594 2023-09-18 09:44:29,458 44k INFO ====> Epoch: 2757, cost 59.04 s 2023-09-18 09:45:28,324 44k INFO ====> Epoch: 2758, cost 58.87 s 2023-09-18 09:46:26,986 44k INFO ====> Epoch: 2759, cost 58.66 s 2023-09-18 09:47:25,459 44k INFO ====> Epoch: 2760, cost 58.47 s 2023-09-18 09:48:23,885 44k INFO ====> Epoch: 2761, cost 58.43 s 2023-09-18 09:49:22,227 44k INFO ====> Epoch: 2762, cost 58.34 s 2023-09-18 09:49:52,246 44k INFO Train Epoch: 2763 [47%] 2023-09-18 09:49:52,247 44k INFO Losses: [2.0351104736328125, 2.943814516067505, 15.209818840026855, 21.767765045166016, 0.718167245388031], step: 88400, lr: 3.54014020576838e-05, reference_loss: 42.67467498779297 2023-09-18 09:50:21,644 44k INFO ====> Epoch: 2763, cost 59.42 s 2023-09-18 09:51:19,811 44k INFO ====> Epoch: 2764, cost 58.17 s 2023-09-18 09:52:18,823 44k INFO ====> Epoch: 2765, cost 59.01 s 2023-09-18 09:53:17,447 44k INFO ====> Epoch: 2766, cost 58.62 s 2023-09-18 09:54:16,019 44k INFO ====> Epoch: 2767, cost 58.57 s 2023-09-18 09:55:14,418 44k INFO ====> Epoch: 2768, cost 58.40 s 2023-09-18 09:55:59,226 44k INFO Train Epoch: 2769 [72%] 2023-09-18 09:55:59,227 44k INFO Losses: [2.065485954284668, 2.768345355987549, 14.753839492797852, 21.237943649291992, 0.7948131561279297], step: 88600, lr: 3.5374859301961405e-05, reference_loss: 41.62042999267578 2023-09-18 09:56:13,723 44k INFO ====> Epoch: 2769, cost 59.30 s 2023-09-18 09:57:12,469 44k INFO ====> Epoch: 2770, cost 58.75 s 2023-09-18 09:58:10,964 44k INFO ====> Epoch: 2771, cost 58.50 s 2023-09-18 09:59:09,546 44k INFO ====> Epoch: 2772, cost 58.58 s 2023-09-18 10:00:07,977 44k INFO ====> Epoch: 2773, cost 58.43 s 2023-09-18 10:01:06,596 44k INFO ====> Epoch: 2774, cost 58.62 s 2023-09-18 10:02:05,238 44k INFO Train Epoch: 2775 [97%] 2023-09-18 10:02:05,239 44k INFO Losses: [1.3533713817596436, 3.3746891021728516, 11.662631034851074, 15.71069622039795, -0.8855734467506409], step: 88800, lr: 3.534833644708588e-05, reference_loss: 31.21581268310547 2023-09-18 10:02:22,048 44k INFO Saving model and optimizer state at iteration 2775 to ./logs/44k/G_88800.pth 2023-09-18 10:02:25,522 44k INFO Saving model and optimizer state at iteration 2775 to ./logs/44k/D_88800.pth 2023-09-18 10:02:26,050 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_84800.pth 2023-09-18 10:02:26,052 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_84800.pth 2023-09-18 10:02:26,052 44k INFO ====> Epoch: 2775, cost 79.46 s 2023-09-18 10:03:24,563 44k INFO ====> Epoch: 2776, cost 58.51 s 2023-09-18 10:04:22,984 44k INFO ====> Epoch: 2777, cost 58.42 s 2023-09-18 10:05:21,437 44k INFO ====> Epoch: 2778, cost 58.45 s 2023-09-18 10:06:20,166 44k INFO ====> Epoch: 2779, cost 58.73 s 2023-09-18 10:07:18,283 44k INFO ====> Epoch: 2780, cost 58.12 s 2023-09-18 10:08:16,667 44k INFO ====> Epoch: 2781, cost 58.38 s 2023-09-18 10:08:31,680 44k INFO Train Epoch: 2782 [22%] 2023-09-18 10:08:31,681 44k INFO Losses: [2.082911252975464, 2.964639663696289, 15.069674491882324, 21.63878059387207, 0.7895351648330688], step: 89000, lr: 3.531741824895147e-05, reference_loss: 42.54553985595703 2023-09-18 10:09:16,294 44k INFO ====> Epoch: 2782, cost 59.63 s 2023-09-18 10:10:14,939 44k INFO ====> Epoch: 2783, cost 58.64 s 2023-09-18 10:11:13,318 44k INFO ====> Epoch: 2784, cost 58.38 s 2023-09-18 10:12:11,276 44k INFO ====> Epoch: 2785, cost 57.96 s 2023-09-18 10:13:09,574 44k INFO ====> Epoch: 2786, cost 58.30 s 2023-09-18 10:14:08,250 44k INFO ====> Epoch: 2787, cost 58.68 s 2023-09-18 10:14:38,095 44k INFO Train Epoch: 2788 [47%] 2023-09-18 10:14:38,097 44k INFO Losses: [1.9792323112487793, 2.961763381958008, 15.931412696838379, 21.943845748901367, 0.7166988253593445], step: 89200, lr: 3.5290938461405194e-05, reference_loss: 43.53295135498047 2023-09-18 10:15:07,369 44k INFO ====> Epoch: 2788, cost 59.12 s 2023-09-18 10:16:06,148 44k INFO ====> Epoch: 2789, cost 58.78 s 2023-09-18 10:17:04,259 44k INFO ====> Epoch: 2790, cost 58.11 s 2023-09-18 10:18:02,517 44k INFO ====> Epoch: 2791, cost 58.26 s 2023-09-18 10:19:01,477 44k INFO ====> Epoch: 2792, cost 58.96 s 2023-09-18 10:20:00,078 44k INFO ====> Epoch: 2793, cost 58.60 s 2023-09-18 10:20:44,338 44k INFO Train Epoch: 2794 [72%] 2023-09-18 10:20:44,339 44k INFO Losses: [2.0289628505706787, 2.7792797088623047, 15.113940238952637, 21.162349700927734, 0.7825740575790405], step: 89400, lr: 3.526447852749441e-05, reference_loss: 41.86710739135742 2023-09-18 10:20:58,690 44k INFO ====> Epoch: 2794, cost 58.61 s 2023-09-18 10:21:57,045 44k INFO ====> Epoch: 2795, cost 58.36 s 2023-09-18 10:22:55,975 44k INFO ====> Epoch: 2796, cost 58.93 s 2023-09-18 10:23:54,184 44k INFO ====> Epoch: 2797, cost 58.21 s 2023-09-18 10:24:53,143 44k INFO ====> Epoch: 2798, cost 58.96 s 2023-09-18 10:25:52,015 44k INFO ====> Epoch: 2799, cost 58.87 s 2023-09-18 10:26:50,663 44k INFO Train Epoch: 2800 [97%] 2023-09-18 10:26:50,664 44k INFO Losses: [1.3693991899490356, 3.492889881134033, 13.570584297180176, 15.286778450012207, -0.9478369355201721], step: 89600, lr: 3.523803843233355e-05, reference_loss: 32.771812438964844 2023-09-18 10:27:08,264 44k INFO Saving model and optimizer state at iteration 2800 to ./logs/44k/G_89600.pth 2023-09-18 10:27:10,503 44k INFO Saving model and optimizer state at iteration 2800 to ./logs/44k/D_89600.pth 2023-09-18 10:27:11,022 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_85600.pth 2023-09-18 10:27:11,023 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_85600.pth 2023-09-18 10:27:11,023 44k INFO ====> Epoch: 2800, cost 79.01 s 2023-09-18 10:28:09,138 44k INFO ====> Epoch: 2801, cost 58.11 s 2023-09-18 10:29:08,017 44k INFO ====> Epoch: 2802, cost 58.88 s 2023-09-18 10:30:06,786 44k INFO ====> Epoch: 2803, cost 58.77 s 2023-09-18 10:31:05,039 44k INFO ====> Epoch: 2804, cost 58.25 s 2023-09-18 10:32:03,430 44k INFO ====> Epoch: 2805, cost 58.39 s 2023-09-18 10:33:01,535 44k INFO ====> Epoch: 2806, cost 58.10 s 2023-09-18 10:33:16,360 44k INFO Train Epoch: 2807 [22%] 2023-09-18 10:33:16,361 44k INFO Losses: [1.990992784500122, 3.1523475646972656, 17.096614837646484, 21.94095230102539, 0.7919902801513672], step: 89800, lr: 3.520721670877806e-05, reference_loss: 44.972900390625 2023-09-18 10:34:00,978 44k INFO ====> Epoch: 2807, cost 59.44 s 2023-09-18 10:34:59,602 44k INFO ====> Epoch: 2808, cost 58.62 s 2023-09-18 10:35:57,743 44k INFO ====> Epoch: 2809, cost 58.14 s 2023-09-18 10:36:56,505 44k INFO ====> Epoch: 2810, cost 58.76 s 2023-09-18 10:37:54,599 44k INFO ====> Epoch: 2811, cost 58.09 s 2023-09-18 10:38:52,950 44k INFO ====> Epoch: 2812, cost 58.35 s 2023-09-18 10:39:23,256 44k INFO Train Epoch: 2813 [47%] 2023-09-18 10:39:23,257 44k INFO Losses: [2.0235540866851807, 2.8597607612609863, 16.05812644958496, 21.859420776367188, 0.7339729070663452], step: 90000, lr: 3.518081954656274e-05, reference_loss: 43.53483581542969 2023-09-18 10:39:52,730 44k INFO ====> Epoch: 2813, cost 59.78 s 2023-09-18 10:40:51,305 44k INFO ====> Epoch: 2814, cost 58.57 s 2023-09-18 10:41:49,474 44k INFO ====> Epoch: 2815, cost 58.17 s 2023-09-18 10:42:48,150 44k INFO ====> Epoch: 2816, cost 58.68 s 2023-09-18 10:43:46,430 44k INFO ====> Epoch: 2817, cost 58.28 s 2023-09-18 10:44:45,381 44k INFO ====> Epoch: 2818, cost 58.95 s 2023-09-18 10:45:30,244 44k INFO Train Epoch: 2819 [72%] 2023-09-18 10:45:30,245 44k INFO Losses: [2.009913682937622, 2.806490421295166, 14.488445281982422, 21.16152572631836, 0.7800108194351196], step: 90200, lr: 3.5154442176033265e-05, reference_loss: 41.24638748168945 2023-09-18 10:45:44,580 44k INFO ====> Epoch: 2819, cost 59.20 s 2023-09-18 10:46:42,990 44k INFO ====> Epoch: 2820, cost 58.41 s 2023-09-18 10:47:42,052 44k INFO ====> Epoch: 2821, cost 59.06 s 2023-09-18 10:48:40,809 44k INFO ====> Epoch: 2822, cost 58.76 s 2023-09-18 10:49:39,288 44k INFO ====> Epoch: 2823, cost 58.48 s 2023-09-18 10:50:37,745 44k INFO ====> Epoch: 2824, cost 58.46 s 2023-09-18 10:51:35,982 44k INFO Train Epoch: 2825 [97%] 2023-09-18 10:51:35,983 44k INFO Losses: [1.5851938724517822, 3.076608896255493, 11.30295181274414, 14.845932006835938, -0.876884400844574], step: 90400, lr: 3.5128084582350516e-05, reference_loss: 29.933801651000977 2023-09-18 10:51:53,312 44k INFO Saving model and optimizer state at iteration 2825 to ./logs/44k/G_90400.pth 2023-09-18 10:51:56,753 44k INFO Saving model and optimizer state at iteration 2825 to ./logs/44k/D_90400.pth 2023-09-18 10:51:57,854 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_86400.pth 2023-09-18 10:51:57,856 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_86400.pth 2023-09-18 10:51:57,856 44k INFO ====> Epoch: 2825, cost 80.11 s 2023-09-18 10:52:56,280 44k INFO ====> Epoch: 2826, cost 58.42 s 2023-09-18 10:53:54,465 44k INFO ====> Epoch: 2827, cost 58.19 s 2023-09-18 10:54:53,301 44k INFO ====> Epoch: 2828, cost 58.84 s 2023-09-18 10:55:51,935 44k INFO ====> Epoch: 2829, cost 58.63 s 2023-09-18 10:56:50,227 44k INFO ====> Epoch: 2830, cost 58.29 s 2023-09-18 10:57:48,331 44k INFO ====> Epoch: 2831, cost 58.10 s 2023-09-18 10:58:02,927 44k INFO Train Epoch: 2832 [22%] 2023-09-18 10:58:02,928 44k INFO Losses: [2.029489040374756, 3.086263656616211, 15.379423141479492, 21.882848739624023, 0.7751601934432983], step: 90600, lr: 3.509735903234268e-05, reference_loss: 43.1531867980957 2023-09-18 10:58:47,629 44k INFO ====> Epoch: 2832, cost 59.30 s 2023-09-18 10:59:46,492 44k INFO ====> Epoch: 2833, cost 58.86 s 2023-09-18 11:00:44,742 44k INFO ====> Epoch: 2834, cost 58.25 s 2023-09-18 11:01:42,619 44k INFO ====> Epoch: 2835, cost 57.88 s 2023-09-18 11:02:41,798 44k INFO ====> Epoch: 2836, cost 59.18 s 2023-09-18 11:03:40,212 44k INFO ====> Epoch: 2837, cost 58.41 s 2023-09-18 11:04:10,282 44k INFO Train Epoch: 2838 [47%] 2023-09-18 11:04:10,283 44k INFO Losses: [2.0109364986419678, 2.9301459789276123, 14.71748161315918, 21.73241424560547, 0.7086659073829651], step: 90800, lr: 3.5071044237641074e-05, reference_loss: 42.09964370727539 2023-09-18 11:04:39,468 44k INFO ====> Epoch: 2838, cost 59.26 s 2023-09-18 11:05:38,546 44k INFO ====> Epoch: 2839, cost 59.08 s 2023-09-18 11:06:37,326 44k INFO ====> Epoch: 2840, cost 58.78 s 2023-09-18 11:07:35,425 44k INFO ====> Epoch: 2841, cost 58.10 s 2023-09-18 11:08:34,173 44k INFO ====> Epoch: 2842, cost 58.75 s 2023-09-18 11:09:32,927 44k INFO ====> Epoch: 2843, cost 58.75 s 2023-09-18 11:10:18,080 44k INFO Train Epoch: 2844 [72%] 2023-09-18 11:10:18,081 44k INFO Losses: [2.035248041152954, 2.7477879524230957, 14.872174263000488, 21.21975326538086, 0.7819084525108337], step: 91000, lr: 3.504474917286898e-05, reference_loss: 41.6568717956543 2023-09-18 11:10:32,448 44k INFO ====> Epoch: 2844, cost 59.52 s 2023-09-18 11:11:30,551 44k INFO ====> Epoch: 2845, cost 58.10 s 2023-09-18 11:12:29,495 44k INFO ====> Epoch: 2846, cost 58.94 s 2023-09-18 11:13:28,116 44k INFO ====> Epoch: 2847, cost 58.62 s 2023-09-18 11:14:26,214 44k INFO ====> Epoch: 2848, cost 58.10 s 2023-09-18 11:15:24,661 44k INFO ====> Epoch: 2849, cost 58.45 s 2023-09-18 11:16:23,533 44k INFO Train Epoch: 2850 [97%] 2023-09-18 11:16:23,534 44k INFO Losses: [1.5385748147964478, 3.2416234016418457, 14.006020545959473, 15.213170051574707, -0.9446448683738708], step: 91200, lr: 3.50184738232336e-05, reference_loss: 33.054744720458984 2023-09-18 11:16:40,069 44k INFO Saving model and optimizer state at iteration 2850 to ./logs/44k/G_91200.pth 2023-09-18 11:16:43,557 44k INFO Saving model and optimizer state at iteration 2850 to ./logs/44k/D_91200.pth 2023-09-18 11:16:44,084 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_87200.pth 2023-09-18 11:16:44,085 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_87200.pth 2023-09-18 11:16:44,085 44k INFO ====> Epoch: 2850, cost 79.42 s 2023-09-18 11:17:42,005 44k INFO ====> Epoch: 2851, cost 57.92 s 2023-09-18 11:18:39,879 44k INFO ====> Epoch: 2852, cost 57.87 s 2023-09-18 11:19:38,412 44k INFO ====> Epoch: 2853, cost 58.53 s 2023-09-18 11:20:36,496 44k INFO ====> Epoch: 2854, cost 58.08 s 2023-09-18 11:21:35,596 44k INFO ====> Epoch: 2855, cost 59.10 s 2023-09-18 11:22:34,091 44k INFO ====> Epoch: 2856, cost 58.49 s 2023-09-18 11:22:48,562 44k INFO Train Epoch: 2857 [22%] 2023-09-18 11:22:48,563 44k INFO Losses: [2.068697452545166, 2.9857542514801025, 14.48666000366211, 21.58393096923828, 0.7804219126701355], step: 91400, lr: 3.4987844146681445e-05, reference_loss: 41.90546417236328 2023-09-18 11:23:33,040 44k INFO ====> Epoch: 2857, cost 58.95 s 2023-09-18 11:24:31,723 44k INFO ====> Epoch: 2858, cost 58.68 s 2023-09-18 11:25:30,035 44k INFO ====> Epoch: 2859, cost 58.31 s 2023-09-18 11:26:28,593 44k INFO ====> Epoch: 2860, cost 58.56 s 2023-09-18 11:27:28,349 44k INFO ====> Epoch: 2861, cost 59.76 s 2023-09-18 11:28:26,983 44k INFO ====> Epoch: 2862, cost 58.63 s 2023-09-18 11:28:56,792 44k INFO Train Epoch: 2863 [47%] 2023-09-18 11:28:56,793 44k INFO Losses: [2.005837917327881, 2.6412291526794434, 16.47013282775879, 22.028453826904297, 0.729698657989502], step: 91600, lr: 3.496161146248081e-05, reference_loss: 43.87535095214844 2023-09-18 11:29:26,248 44k INFO ====> Epoch: 2863, cost 59.27 s 2023-09-18 11:30:24,832 44k INFO ====> Epoch: 2864, cost 58.58 s 2023-09-18 11:31:23,668 44k INFO ====> Epoch: 2865, cost 58.84 s 2023-09-18 11:32:22,350 44k INFO ====> Epoch: 2866, cost 58.68 s 2023-09-18 11:33:21,117 44k INFO ====> Epoch: 2867, cost 58.77 s 2023-09-18 11:34:19,675 44k INFO ====> Epoch: 2868, cost 58.56 s 2023-09-18 11:35:04,697 44k INFO Train Epoch: 2869 [72%] 2023-09-18 11:35:04,698 44k INFO Losses: [2.083564043045044, 2.76425838470459, 15.227760314941406, 21.31001091003418, 0.7955904006958008], step: 91800, lr: 3.493539844664606e-05, reference_loss: 42.181182861328125 2023-09-18 11:35:18,733 44k INFO ====> Epoch: 2869, cost 59.06 s 2023-09-18 11:36:17,509 44k INFO ====> Epoch: 2870, cost 58.78 s 2023-09-18 11:37:15,668 44k INFO ====> Epoch: 2871, cost 58.16 s 2023-09-18 11:38:14,597 44k INFO ====> Epoch: 2872, cost 58.93 s 2023-09-18 11:39:13,774 44k INFO ====> Epoch: 2873, cost 59.18 s 2023-09-18 11:40:12,436 44k INFO ====> Epoch: 2874, cost 58.66 s 2023-09-18 11:41:11,076 44k INFO Train Epoch: 2875 [97%] 2023-09-18 11:41:11,077 44k INFO Losses: [1.2704576253890991, 3.360994338989258, 14.041566848754883, 15.575831413269043, -0.9795694947242737], step: 92000, lr: 3.4909205084430545e-05, reference_loss: 33.269283294677734 2023-09-18 11:41:27,694 44k INFO Saving model and optimizer state at iteration 2875 to ./logs/44k/G_92000.pth 2023-09-18 11:41:30,776 44k INFO Saving model and optimizer state at iteration 2875 to ./logs/44k/D_92000.pth 2023-09-18 11:41:31,752 44k INFO .. Free up space by deleting ckpt ./logs/44k/G_88000.pth 2023-09-18 11:41:31,764 44k INFO .. Free up space by deleting ckpt ./logs/44k/D_88000.pth 2023-09-18 11:41:31,775 44k INFO ====> Epoch: 2875, cost 79.34 s 2023-09-18 11:42:30,384 44k INFO ====> Epoch: 2876, cost 58.61 s 2023-09-18 11:43:29,088 44k INFO ====> Epoch: 2877, cost 58.70 s 2023-09-18 11:44:27,289 44k INFO ====> Epoch: 2878, cost 58.20 s 2023-09-18 11:45:25,794 44k INFO ====> Epoch: 2879, cost 58.50 s 2023-09-18 11:46:24,117 44k INFO ====> Epoch: 2880, cost 58.32 s 2023-09-18 11:47:22,980 44k INFO ====> Epoch: 2881, cost 58.86 s 2023-09-18 11:47:37,692 44k INFO Train Epoch: 2882 [22%] 2023-09-18 11:47:37,693 44k INFO Losses: [1.979581594467163, 3.139124870300293, 16.09514045715332, 21.797178268432617, 0.7800350785255432], step: 92200, lr: 3.4878670982178504e-05, reference_loss: 43.79106140136719 2023-09-18 11:48:21,791 44k INFO ====> Epoch: 2882, cost 58.81 s 2023-09-18 11:49:20,314 44k INFO ====> Epoch: 2883, cost 58.52 s 2023-09-18 11:50:18,722 44k INFO ====> Epoch: 2884, cost 58.41 s 2023-09-18 11:51:17,394 44k INFO ====> Epoch: 2885, cost 58.67 s 2023-09-18 11:52:15,927 44k INFO ====> Epoch: 2886, cost 58.53 s 2023-09-18 11:53:14,080 44k INFO ====> Epoch: 2887, cost 58.15 s 2023-09-18 11:53:44,199 44k INFO Train Epoch: 2888 [47%] 2023-09-18 11:53:44,200 44k INFO Losses: [2.039576530456543, 2.8340580463409424, 14.949982643127441, 21.63873863220215, 0.7057287096977234], step: 92400, lr: 3.485252015226805e-05, reference_loss: 42.168087005615234 2023-09-18 11:54:13,289 44k INFO ====> Epoch: 2888, cost 59.21 s 2023-09-18 11:55:11,715 44k INFO ====> Epoch: 2889, cost 58.43 s 2023-09-18 11:56:10,421 44k INFO ====> Epoch: 2890, cost 58.71 s 2023-09-18 11:57:09,000 44k INFO ====> Epoch: 2891, cost 58.58 s 2023-09-18 11:58:07,715 44k INFO ====> Epoch: 2892, cost 58.72 s 2023-09-18 11:59:06,581 44k INFO ====> Epoch: 2893, cost 58.87 s 2023-09-18 11:59:51,630 44k INFO Train Epoch: 2894 [72%] 2023-09-18 11:59:51,632 44k INFO Losses: [2.03602933883667, 2.7920217514038086, 14.423547744750977, 21.110742568969727, 0.7783050537109375], step: 92600, lr: 3.4826388929351956e-05, reference_loss: 41.140647888183594 2023-09-18 12:00:06,238 44k INFO ====> Epoch: 2894, cost 59.66 s