2024-02-27 15:10:59,120 - INFO: Problem Type: text_causal_classification_modeling
2024-02-27 15:10:59,120 - INFO: Global random seed: 550978
2024-02-27 15:10:59,120 - INFO: Preparing the data...
2024-02-27 15:10:59,120 - INFO: Setting up automatic validation split...
2024-02-27 15:10:59,161 - INFO: Preparing train and validation data
2024-02-27 15:10:59,161 - INFO: Loading train dataset...
2024-02-27 15:10:59,884 - INFO: Loading validation dataset...
2024-02-27 15:11:00,030 - INFO: Number of observations in train dataset: 10800
2024-02-27 15:11:00,030 - INFO: Number of observations in validation dataset: 1200
2024-02-27 15:11:00,338 - WARNING: 
WARNING: You are currently loading Falcon using legacy code contained in the model repository. Falcon has now been fully ported into the Hugging Face transformers library. For the most up-to-date and high-performance version of the Falcon model code, please update to the latest version of transformers and then load the model without the trust_remote_code=True argument.

2024-02-27 15:11:00,476 - WARNING: PAD token id not matching between config and tokenizer. Overwriting with tokenizer id.
2024-02-27 15:11:00,483 - INFO: Using bfloat16 for backbone
2024-02-27 15:11:00,483 - INFO: Loading tiiuae/falcon-7b. This may take a while.
2024-02-27 15:11:48,531 - INFO: Loaded tiiuae/falcon-7b.
2024-02-27 15:11:48,533 - WARNING: PAD token id not matching between generation config and tokenizer. Overwriting with tokenizer id.
2024-02-27 15:11:48,533 - INFO: Lora module names: ['query_key_value', 'dense', 'dense_h_to_4h', 'dense_4h_to_h']
2024-02-27 15:11:48,673 - INFO: Enough space available for saving model weights.Required space: 13632.81MB, Available space: 991138.32MB.
2024-02-27 15:11:48,818 - INFO: Training Epoch: 1 / 15
2024-02-27 15:11:48,818 - INFO: train loss:   0%|          | 0/1350 [00:00<?, ?it/s]
2024-02-27 15:11:48,897 - INFO: Evaluation step: 1350
2024-02-27 15:12:38,005 - INFO: train loss: 5.39:   5%|4         | 67/1350 [00:49<15:41,  1.36it/s]
2024-02-27 15:12:48,819 - INFO: train loss: 5.39:   5%|4         | 67/1350 [01:00<15:41,  1.36it/s]
2024-02-27 15:13:27,421 - INFO: train loss: 4.75:  10%|9         | 134/1350 [01:38<14:55,  1.36it/s]
2024-02-27 15:13:38,821 - INFO: train loss: 4.75:  10%|9         | 134/1350 [01:50<14:55,  1.36it/s]
2024-02-27 15:14:16,304 - INFO: train loss: 4.33:  15%|#4        | 201/1350 [02:27<14:02,  1.36it/s]
2024-02-27 15:14:28,822 - INFO: train loss: 4.33:  15%|#4        | 201/1350 [02:40<14:02,  1.36it/s]
2024-02-27 15:15:04,428 - INFO: train loss: 2.26:  20%|#9        | 268/1350 [03:15<13:06,  1.37it/s]
2024-02-27 15:15:18,823 - INFO: train loss: 2.26:  20%|#9        | 268/1350 [03:30<13:06,  1.37it/s]
2024-02-27 15:15:52,987 - INFO: train loss: 1.87:  25%|##4       | 335/1350 [04:04<12:17,  1.38it/s]
2024-02-27 15:16:08,825 - INFO: train loss: 1.87:  25%|##4       | 335/1350 [04:20<12:17,  1.38it/s]
2024-02-27 15:16:40,862 - INFO: train loss: 2.10:  30%|##9       | 402/1350 [04:52<11:24,  1.38it/s]
2024-02-27 15:16:58,826 - INFO: train loss: 2.10:  30%|##9       | 402/1350 [05:10<11:24,  1.38it/s]
2024-02-27 15:17:29,975 - INFO: train loss: 1.72:  35%|###4      | 469/1350 [05:41<10:39,  1.38it/s]
2024-02-27 15:17:40,723 - INFO: train loss: 1.72:  35%|###4      | 469/1350 [05:51<10:39,  1.38it/s]
2024-02-27 15:18:18,537 - INFO: train loss: 1.97:  40%|###9      | 536/1350 [06:29<09:50,  1.38it/s]
2024-02-27 15:18:28,828 - INFO: train loss: 1.97:  40%|###9      | 536/1350 [06:40<09:50,  1.38it/s]
2024-02-27 15:19:07,071 - INFO: train loss: 1.56:  45%|####4     | 603/1350 [07:18<09:01,  1.38it/s]
2024-02-27 15:19:18,829 - INFO: train loss: 1.56:  45%|####4     | 603/1350 [07:30<09:01,  1.38it/s]
2024-02-27 15:19:56,260 - INFO: train loss: 1.73:  50%|####9     | 670/1350 [08:07<08:14,  1.37it/s]
2024-02-27 15:20:08,830 - INFO: train loss: 1.73:  50%|####9     | 670/1350 [08:20<08:14,  1.37it/s]
2024-02-27 15:20:45,822 - INFO: train loss: 1.89:  55%|#####4    | 737/1350 [08:57<07:28,  1.37it/s]
2024-02-27 15:20:58,832 - INFO: train loss: 1.89:  55%|#####4    | 737/1350 [09:10<07:28,  1.37it/s]
2024-02-27 15:21:34,932 - INFO: train loss: 1.68:  60%|#####9    | 804/1350 [09:46<06:39,  1.37it/s]
2024-02-27 15:21:48,833 - INFO: train loss: 1.68:  60%|#####9    | 804/1350 [10:00<06:39,  1.37it/s]
2024-02-27 15:22:24,373 - INFO: train loss: 1.53:  65%|######4   | 871/1350 [10:35<05:51,  1.36it/s]
2024-02-27 15:22:38,834 - INFO: train loss: 1.53:  65%|######4   | 871/1350 [10:50<05:51,  1.36it/s]
2024-02-27 15:23:11,765 - INFO: train loss: 1.50:  69%|######9   | 938/1350 [11:22<04:59,  1.38it/s]
2024-02-27 15:23:28,835 - INFO: train loss: 1.50:  69%|######9   | 938/1350 [11:40<04:59,  1.38it/s]
2024-02-27 15:24:01,645 - INFO: train loss: 1.55:  74%|#######4  | 1005/1350 [12:12<04:12,  1.37it/s]
2024-02-27 15:24:18,836 - INFO: train loss: 1.55:  74%|#######4  | 1005/1350 [12:30<04:12,  1.37it/s]
2024-02-27 15:24:50,764 - INFO: train loss: 1.31:  79%|#######9  | 1072/1350 [13:01<03:23,  1.37it/s]
2024-02-27 15:25:08,837 - INFO: train loss: 1.31:  79%|#######9  | 1072/1350 [13:20<03:23,  1.37it/s]
2024-02-27 15:25:41,093 - INFO: train loss: 1.22:  84%|########4 | 1139/1350 [13:52<02:35,  1.36it/s]
2024-02-27 15:25:58,839 - INFO: train loss: 1.22:  84%|########4 | 1139/1350 [14:10<02:35,  1.36it/s]
2024-02-27 15:26:30,846 - INFO: train loss: 1.10:  89%|########9 | 1206/1350 [14:42<01:46,  1.35it/s]
2024-02-27 15:26:48,840 - INFO: train loss: 1.10:  89%|########9 | 1206/1350 [15:00<01:46,  1.35it/s]
2024-02-27 15:27:20,319 - INFO: train loss: 1.06:  94%|#########4| 1273/1350 [15:31<00:56,  1.35it/s]
2024-02-27 15:27:30,735 - INFO: train loss: 1.06:  94%|#########4| 1273/1350 [15:41<00:56,  1.35it/s]
2024-02-27 15:28:10,080 - INFO: train loss: 1.05:  99%|#########9| 1340/1350 [16:21<00:07,  1.35it/s]
2024-02-27 15:28:17,574 - INFO: train loss: 1.04: 100%|##########| 1350/1350 [16:28<00:00,  1.37it/s]
2024-02-27 15:28:17,578 - INFO: Starting validation inference
2024-02-27 15:28:17,578 - INFO: validation progress:   0%|          | 0/1200 [00:00<?, ?it/s]
2024-02-27 15:28:21,713 - INFO: validation progress:   5%|5         | 60/1200 [00:04<01:18, 14.51it/s]
2024-02-27 15:28:25,795 - INFO: validation progress:  10%|#         | 120/1200 [00:08<01:13, 14.62it/s]
2024-02-27 15:28:29,801 - INFO: validation progress:  15%|#5        | 180/1200 [00:12<01:09, 14.78it/s]
2024-02-27 15:28:33,844 - INFO: validation progress:  20%|##        | 240/1200 [00:16<01:04, 14.80it/s]
2024-02-27 15:28:37,910 - INFO: validation progress:  25%|##5       | 300/1200 [00:20<01:00, 14.79it/s]
2024-02-27 15:28:41,956 - INFO: validation progress:  30%|###       | 360/1200 [00:24<00:56, 14.80it/s]
2024-02-27 15:28:46,043 - INFO: validation progress:  35%|###5      | 420/1200 [00:28<00:52, 14.76it/s]
2024-02-27 15:28:50,095 - INFO: validation progress:  40%|####      | 480/1200 [00:32<00:48, 14.78it/s]
2024-02-27 15:28:54,158 - INFO: validation progress:  45%|####5     | 540/1200 [00:36<00:44, 14.77it/s]
2024-02-27 15:28:58,232 - INFO: validation progress:  50%|#####     | 600/1200 [00:40<00:40, 14.76it/s]
2024-02-27 15:29:02,307 - INFO: validation progress:  55%|#####5    | 660/1200 [00:44<00:36, 14.75it/s]
2024-02-27 15:29:06,379 - INFO: validation progress:  60%|######    | 720/1200 [00:48<00:32, 14.74it/s]
2024-02-27 15:29:10,386 - INFO: validation progress:  65%|######5   | 780/1200 [00:52<00:28, 14.81it/s]
2024-02-27 15:29:14,485 - INFO: validation progress:  70%|#######   | 840/1200 [00:56<00:24, 14.76it/s]
2024-02-27 15:29:18,557 - INFO: validation progress:  75%|#######5  | 900/1200 [01:00<00:20, 14.75it/s]
2024-02-27 15:29:22,629 - INFO: validation progress:  80%|########  | 960/1200 [01:05<00:16, 14.75it/s]
2024-02-27 15:29:26,707 - INFO: validation progress:  85%|########5 | 1020/1200 [01:09<00:12, 14.74it/s]
2024-02-27 15:29:30,801 - INFO: validation progress:  90%|######### | 1080/1200 [01:13<00:08, 14.71it/s]
2024-02-27 15:29:34,843 - INFO: validation progress:  95%|#########5| 1140/1200 [01:17<00:04, 14.75it/s]
2024-02-27 15:29:38,837 - INFO: validation progress: 100%|##########| 1200/1200 [01:21<00:00, 14.83it/s]
2024-02-27 15:29:38,839 - INFO: validation progress: 100%|##########| 1200/1200 [01:21<00:00, 14.77it/s]
2024-02-27 15:29:38,864 - INFO: Mean validation loss: 0.90334
2024-02-27 15:29:38,866 - INFO: Validation AUC: 0.79197
2024-02-27 15:29:38,903 - INFO: Saving best model checkpoint: val_AUC -inf -> 0.79197 to /app/output
2024-02-27 15:29:52,269 - INFO: Training Epoch: 2 / 15
2024-02-27 15:29:52,270 - INFO: train loss:   0%|          | 0/1350 [00:00<?, ?it/s]
2024-02-27 15:29:52,294 - INFO: Evaluation step: 1350
2024-02-27 15:30:40,267 - INFO: train loss: 0.98:   5%|4         | 67/1350 [00:47<15:19,  1.40it/s]
2024-02-27 15:30:50,846 - INFO: train loss: 0.98:   5%|4         | 67/1350 [00:58<15:19,  1.40it/s]
2024-02-27 15:31:28,420 - INFO: train loss: 0.76:  10%|9         | 134/1350 [01:36<14:32,  1.39it/s]
2024-02-27 15:31:38,853 - INFO: train loss: 0.76:  10%|9         | 134/1350 [01:46<14:32,  1.39it/s]
2024-02-27 15:32:17,535 - INFO: train loss: 0.92:  15%|#4        | 201/1350 [02:25<13:52,  1.38it/s]
2024-02-27 15:32:28,855 - INFO: train loss: 0.92:  15%|#4        | 201/1350 [02:36<13:52,  1.38it/s]
2024-02-27 15:33:07,429 - INFO: train loss: 0.52:  20%|#9        | 268/1350 [03:15<13:12,  1.36it/s]
2024-02-27 15:33:18,856 - INFO: train loss: 0.52:  20%|#9        | 268/1350 [03:26<13:12,  1.36it/s]
2024-02-27 15:33:56,664 - INFO: train loss: 0.59:  25%|##4       | 335/1350 [04:04<12:24,  1.36it/s]
2024-02-27 15:34:08,857 - INFO: train loss: 0.59:  25%|##4       | 335/1350 [04:16<12:24,  1.36it/s]
2024-02-27 15:34:46,245 - INFO: train loss: 0.31:  30%|##9       | 402/1350 [04:53<11:37,  1.36it/s]
2024-02-27 15:34:58,858 - INFO: train loss: 0.31:  30%|##9       | 402/1350 [05:06<11:37,  1.36it/s]
2024-02-27 15:35:36,310 - INFO: train loss: 0.47:  35%|###4      | 469/1350 [05:44<10:51,  1.35it/s]
2024-02-27 15:35:48,860 - INFO: train loss: 0.47:  35%|###4      | 469/1350 [05:56<10:51,  1.35it/s]
2024-02-27 15:36:26,152 - INFO: train loss: 0.66:  40%|###9      | 536/1350 [06:33<10:03,  1.35it/s]
2024-02-27 15:36:38,861 - INFO: train loss: 0.66:  40%|###9      | 536/1350 [06:46<10:03,  1.35it/s]
2024-02-27 15:37:14,471 - INFO: train loss: 0.40:  45%|####4     | 603/1350 [07:22<09:08,  1.36it/s]
2024-02-27 15:37:28,862 - INFO: train loss: 0.40:  45%|####4     | 603/1350 [07:36<09:08,  1.36it/s]
2024-02-27 15:38:02,866 - INFO: train loss: 0.56:  50%|####9     | 670/1350 [08:10<08:17,  1.37it/s]
2024-02-27 15:38:18,864 - INFO: train loss: 0.56:  50%|####9     | 670/1350 [08:26<08:17,  1.37it/s]
2024-02-27 15:38:52,699 - INFO: train loss: 0.33:  55%|#####4    | 737/1350 [09:00<07:30,  1.36it/s]
2024-02-27 15:39:08,865 - INFO: train loss: 0.33:  55%|#####4    | 737/1350 [09:16<07:30,  1.36it/s]
2024-02-27 15:39:42,366 - INFO: train loss: 0.48:  60%|#####9    | 804/1350 [09:50<06:42,  1.36it/s]
2024-02-27 15:39:58,866 - INFO: train loss: 0.48:  60%|#####9    | 804/1350 [10:06<06:42,  1.36it/s]
2024-02-27 15:40:30,326 - INFO: train loss: 0.37:  65%|######4   | 871/1350 [10:38<05:49,  1.37it/s]
2024-02-27 15:40:40,857 - INFO: train loss: 0.37:  65%|######4   | 871/1350 [10:48<05:49,  1.37it/s]
2024-02-27 15:41:19,361 - INFO: train loss: 0.56:  69%|######9   | 938/1350 [11:27<05:01,  1.37it/s]
2024-02-27 15:41:29,499 - INFO: train loss: 0.56:  69%|######9   | 938/1350 [11:37<05:01,  1.37it/s]
2024-02-27 15:42:08,067 - INFO: train loss: 0.32:  74%|#######4  | 1005/1350 [12:15<04:11,  1.37it/s]
2024-02-27 15:42:18,869 - INFO: train loss: 0.32:  74%|#######4  | 1005/1350 [12:26<04:11,  1.37it/s]
2024-02-27 15:42:57,081 - INFO: train loss: 0.57:  79%|#######9  | 1072/1350 [13:04<03:23,  1.37it/s]
2024-02-27 15:43:08,871 - INFO: train loss: 0.57:  79%|#######9  | 1072/1350 [13:16<03:23,  1.37it/s]
2024-02-27 15:43:46,836 - INFO: train loss: 0.39:  84%|########4 | 1139/1350 [13:54<02:34,  1.36it/s]
2024-02-27 15:43:58,872 - INFO: train loss: 0.39:  84%|########4 | 1139/1350 [14:06<02:34,  1.36it/s]
2024-02-27 15:44:34,530 - INFO: train loss: 0.47:  89%|########9 | 1206/1350 [14:42<01:44,  1.37it/s]
2024-02-27 15:44:48,873 - INFO: train loss: 0.47:  89%|########9 | 1206/1350 [14:56<01:44,  1.37it/s]
2024-02-27 15:45:24,188 - INFO: train loss: 0.38:  94%|#########4| 1273/1350 [15:31<00:56,  1.37it/s]
2024-02-27 15:45:38,875 - INFO: train loss: 0.38:  94%|#########4| 1273/1350 [15:46<00:56,  1.37it/s]
2024-02-27 15:46:12,578 - INFO: train loss: 0.26:  99%|#########9| 1340/1350 [16:20<00:07,  1.37it/s]
2024-02-27 15:46:19,467 - INFO: train loss: 0.29: 100%|##########| 1350/1350 [16:27<00:00,  1.37it/s]
2024-02-27 15:46:19,472 - INFO: Starting validation inference
2024-02-27 15:46:19,472 - INFO: validation progress:   0%|          | 0/1200 [00:00<?, ?it/s]
2024-02-27 15:46:23,636 - INFO: validation progress:   5%|5         | 60/1200 [00:04<01:19, 14.41it/s]
2024-02-27 15:46:27,760 - INFO: validation progress:  10%|#         | 120/1200 [00:08<01:14, 14.49it/s]
2024-02-27 15:46:31,812 - INFO: validation progress:  15%|#5        | 180/1200 [00:12<01:09, 14.63it/s]
2024-02-27 15:46:35,897 - INFO: validation progress:  20%|##        | 240/1200 [00:16<01:05, 14.66it/s]
2024-02-27 15:46:40,004 - INFO: validation progress:  25%|##5       | 300/1200 [00:20<01:01, 14.64it/s]
2024-02-27 15:46:44,081 - INFO: validation progress:  30%|###       | 360/1200 [00:24<00:57, 14.67it/s]
2024-02-27 15:46:48,203 - INFO: validation progress:  35%|###5      | 420/1200 [00:28<00:53, 14.63it/s]
2024-02-27 15:46:52,276 - INFO: validation progress:  40%|####      | 480/1200 [00:32<00:49, 14.66it/s]
2024-02-27 15:46:56,415 - INFO: validation progress:  45%|####5     | 540/1200 [00:36<00:45, 14.61it/s]
2024-02-27 15:47:00,536 - INFO: validation progress:  50%|#####     | 600/1200 [00:41<00:41, 14.59it/s]
2024-02-27 15:47:04,650 - INFO: validation progress:  55%|#####5    | 660/1200 [00:45<00:37, 14.59it/s]
2024-02-27 15:47:08,748 - INFO: validation progress:  60%|######    | 720/1200 [00:49<00:32, 14.61it/s]
2024-02-27 15:47:12,801 - INFO: validation progress:  65%|######5   | 780/1200 [00:53<00:28, 14.67it/s]
2024-02-27 15:47:16,908 - INFO: validation progress:  70%|#######   | 840/1200 [00:57<00:24, 14.65it/s]
2024-02-27 15:47:21,016 - INFO: validation progress:  75%|#######5  | 900/1200 [01:01<00:20, 14.64it/s]
2024-02-27 15:47:25,091 - INFO: validation progress:  80%|########  | 960/1200 [01:05<00:16, 14.66it/s]
2024-02-27 15:47:29,177 - INFO: validation progress:  85%|########5 | 1020/1200 [01:09<00:12, 14.67it/s]
2024-02-27 15:47:33,287 - INFO: validation progress:  90%|######### | 1080/1200 [01:13<00:08, 14.65it/s]
2024-02-27 15:47:37,347 - INFO: validation progress:  95%|#########5| 1140/1200 [01:17<00:04, 14.69it/s]
2024-02-27 15:47:41,364 - INFO: validation progress: 100%|##########| 1200/1200 [01:21<00:00, 14.76it/s]
2024-02-27 15:47:41,366 - INFO: validation progress: 100%|##########| 1200/1200 [01:21<00:00, 14.65it/s]
2024-02-27 15:47:41,393 - INFO: Mean validation loss: 0.34724
2024-02-27 15:47:41,394 - INFO: Validation AUC: 0.97231
2024-02-27 15:47:41,433 - INFO: Saving best model checkpoint: val_AUC 0.79197 -> 0.97231 to /app/output
2024-02-27 15:47:55,634 - INFO: Training Epoch: 3 / 15
2024-02-27 15:47:55,634 - INFO: train loss:   0%|          | 0/1350 [00:00<?, ?it/s]
2024-02-27 15:47:55,659 - INFO: Evaluation step: 1350
2024-02-27 15:48:44,330 - INFO: train loss: 0.24:   5%|4         | 67/1350 [00:48<15:32,  1.38it/s]
2024-02-27 15:48:58,962 - INFO: train loss: 0.24:   5%|4         | 67/1350 [01:03<15:32,  1.38it/s]
2024-02-27 15:49:33,947 - INFO: train loss: 0.24:  10%|9         | 134/1350 [01:38<14:53,  1.36it/s]
2024-02-27 15:49:48,963 - INFO: train loss: 0.24:  10%|9         | 134/1350 [01:53<14:53,  1.36it/s]
2024-02-27 15:50:24,061 - INFO: train loss: 0.45:  15%|#4        | 201/1350 [02:28<14:11,  1.35it/s]
2024-02-27 15:50:38,964 - INFO: train loss: 0.45:  15%|#4        | 201/1350 [02:43<14:11,  1.35it/s]
2024-02-27 15:51:15,137 - INFO: train loss: 0.35:  20%|#9        | 268/1350 [03:19<13:30,  1.33it/s]
2024-02-27 15:51:28,966 - INFO: train loss: 0.35:  20%|#9        | 268/1350 [03:33<13:30,  1.33it/s]
2024-02-27 15:52:04,065 - INFO: train loss: 0.29:  25%|##4       | 335/1350 [04:08<12:33,  1.35it/s]
2024-02-27 15:52:18,967 - INFO: train loss: 0.29:  25%|##4       | 335/1350 [04:23<12:33,  1.35it/s]
2024-02-27 15:52:54,392 - INFO: train loss: 0.27:  30%|##9       | 402/1350 [04:58<11:46,  1.34it/s]
2024-02-27 15:53:08,968 - INFO: train loss: 0.27:  30%|##9       | 402/1350 [05:13<11:46,  1.34it/s]
2024-02-27 15:53:42,207 - INFO: train loss: 0.39:  35%|###4      | 469/1350 [05:46<10:47,  1.36it/s]
2024-02-27 15:53:58,970 - INFO: train loss: 0.39:  35%|###4      | 469/1350 [06:03<10:47,  1.36it/s]
2024-02-27 15:54:31,674 - INFO: train loss: 0.41:  40%|###9      | 536/1350 [06:36<09:59,  1.36it/s]
2024-02-27 15:54:48,971 - INFO: train loss: 0.41:  40%|###9      | 536/1350 [06:53<09:59,  1.36it/s]
2024-02-27 15:55:19,071 - INFO: train loss: 0.30:  45%|####4     | 603/1350 [07:23<09:03,  1.38it/s]
2024-02-27 15:55:29,613 - INFO: train loss: 0.30:  45%|####4     | 603/1350 [07:33<09:03,  1.38it/s]
2024-02-27 15:56:08,097 - INFO: train loss: 0.38:  50%|####9     | 670/1350 [08:12<08:15,  1.37it/s]
2024-02-27 15:56:18,973 - INFO: train loss: 0.38:  50%|####9     | 670/1350 [08:23<08:15,  1.37it/s]
2024-02-27 15:56:57,480 - INFO: train loss: 0.30:  55%|#####4    | 737/1350 [09:01<07:28,  1.37it/s]
2024-02-27 15:57:08,974 - INFO: train loss: 0.30:  55%|#####4    | 737/1350 [09:13<07:28,  1.37it/s]
2024-02-27 15:57:46,055 - INFO: train loss: 0.40:  60%|#####9    | 804/1350 [09:50<06:38,  1.37it/s]
2024-02-27 15:57:58,976 - INFO: train loss: 0.40:  60%|#####9    | 804/1350 [10:03<06:38,  1.37it/s]
2024-02-27 15:58:35,524 - INFO: train loss: 0.27:  65%|######4   | 871/1350 [10:39<05:50,  1.37it/s]
2024-02-27 15:58:48,977 - INFO: train loss: 0.27:  65%|######4   | 871/1350 [10:53<05:50,  1.37it/s]
2024-02-27 15:59:24,087 - INFO: train loss: 0.41:  69%|######9   | 938/1350 [11:28<05:00,  1.37it/s]
2024-02-27 15:59:38,978 - INFO: train loss: 0.41:  69%|######9   | 938/1350 [11:43<05:00,  1.37it/s]
2024-02-27 16:00:13,900 - INFO: train loss: 0.54:  74%|#######4  | 1005/1350 [12:18<04:13,  1.36it/s]
2024-02-27 16:00:28,980 - INFO: train loss: 0.54:  74%|#######4  | 1005/1350 [12:33<04:13,  1.36it/s]
2024-02-27 16:01:02,605 - INFO: train loss: 0.31:  79%|#######9  | 1072/1350 [13:06<03:23,  1.37it/s]
2024-02-27 16:01:18,981 - INFO: train loss: 0.31:  79%|#######9  | 1072/1350 [13:23<03:23,  1.37it/s]
2024-02-27 16:01:52,569 - INFO: train loss: 0.44:  84%|########4 | 1139/1350 [13:56<02:35,  1.36it/s]
2024-02-27 16:02:08,982 - INFO: train loss: 0.44:  84%|########4 | 1139/1350 [14:13<02:35,  1.36it/s]
2024-02-27 16:02:40,755 - INFO: train loss: 0.17:  89%|########9 | 1206/1350 [14:45<01:45,  1.37it/s]
2024-02-27 16:02:50,929 - INFO: train loss: 0.17:  89%|########9 | 1206/1350 [14:55<01:45,  1.37it/s]
2024-02-27 16:03:28,505 - INFO: train loss: 0.27:  94%|#########4| 1273/1350 [15:32<00:55,  1.38it/s]
2024-02-27 16:03:38,984 - INFO: train loss: 0.27:  94%|#########4| 1273/1350 [15:43<00:55,  1.38it/s]
2024-02-27 16:04:17,776 - INFO: train loss: 0.32:  99%|#########9| 1340/1350 [16:22<00:07,  1.37it/s]
2024-02-27 16:04:24,908 - INFO: train loss: 0.25: 100%|##########| 1350/1350 [16:29<00:00,  1.36it/s]
2024-02-27 16:04:24,912 - INFO: Starting validation inference
2024-02-27 16:04:24,912 - INFO: validation progress:   0%|          | 0/1200 [00:00<?, ?it/s]
2024-02-27 16:04:29,109 - INFO: validation progress:   5%|5         | 60/1200 [00:04<01:19, 14.30it/s]
2024-02-27 16:04:33,272 - INFO: validation progress:  10%|#         | 120/1200 [00:08<01:15, 14.36it/s]
2024-02-27 16:04:37,356 - INFO: validation progress:  15%|#5        | 180/1200 [00:12<01:10, 14.51it/s]
2024-02-27 16:04:41,494 - INFO: validation progress:  20%|##        | 240/1200 [00:16<01:06, 14.51it/s]
2024-02-27 16:04:45,651 - INFO: validation progress:  25%|##5       | 300/1200 [00:20<01:02, 14.48it/s]
2024-02-27 16:04:49,770 - INFO: validation progress:  30%|###       | 360/1200 [00:24<00:57, 14.51it/s]
2024-02-27 16:04:53,936 - INFO: validation progress:  35%|###5      | 420/1200 [00:29<00:53, 14.47it/s]
2024-02-27 16:04:58,040 - INFO: validation progress:  40%|####      | 480/1200 [00:33<00:49, 14.52it/s]
2024-02-27 16:05:02,194 - INFO: validation progress:  45%|####5     | 540/1200 [00:37<00:45, 14.50it/s]
2024-02-27 16:05:06,361 - INFO: validation progress:  50%|#####     | 600/1200 [00:41<00:41, 14.47it/s]
2024-02-27 16:05:10,516 - INFO: validation progress:  55%|#####5    | 660/1200 [00:45<00:37, 14.46it/s]
2024-02-27 16:05:14,669 - INFO: validation progress:  60%|######    | 720/1200 [00:49<00:33, 14.46it/s]
2024-02-27 16:05:18,765 - INFO: validation progress:  65%|######5   | 780/1200 [00:53<00:28, 14.51it/s]
2024-02-27 16:05:22,909 - INFO: validation progress:  70%|#######   | 840/1200 [00:57<00:24, 14.50it/s]
2024-02-27 16:05:27,077 - INFO: validation progress:  75%|#######5  | 900/1200 [01:02<00:20, 14.47it/s]
2024-02-27 16:05:31,204 - INFO: validation progress:  80%|########  | 960/1200 [01:06<00:16, 14.49it/s]
2024-02-27 16:05:35,351 - INFO: validation progress:  85%|########5 | 1020/1200 [01:10<00:12, 14.48it/s]
2024-02-27 16:05:39,503 - INFO: validation progress:  90%|######### | 1080/1200 [01:14<00:08, 14.47it/s]
2024-02-27 16:05:43,621 - INFO: validation progress:  95%|#########5| 1140/1200 [01:18<00:04, 14.50it/s]
2024-02-27 16:05:47,698 - INFO: validation progress: 100%|##########| 1200/1200 [01:22<00:00, 14.57it/s]
2024-02-27 16:05:47,700 - INFO: validation progress: 100%|##########| 1200/1200 [01:22<00:00, 14.49it/s]
2024-02-27 16:05:47,727 - INFO: Mean validation loss: 0.31232
2024-02-27 16:05:47,729 - INFO: Validation AUC: 0.97734
2024-02-27 16:05:47,767 - INFO: Saving best model checkpoint: val_AUC 0.97231 -> 0.97734 to /app/output
2024-02-27 16:06:01,990 - INFO: Training Epoch: 4 / 15
2024-02-27 16:06:01,990 - INFO: train loss:   0%|          | 0/1350 [00:00<?, ?it/s]
2024-02-27 16:06:02,016 - INFO: Evaluation step: 1350
2024-02-27 16:06:51,334 - INFO: train loss: 0.21:   5%|4         | 67/1350 [00:49<15:44,  1.36it/s]
2024-02-27 16:07:09,168 - INFO: train loss: 0.21:   5%|4         | 67/1350 [01:07<15:44,  1.36it/s]
2024-02-27 16:07:40,541 - INFO: train loss: 0.09:  10%|9         | 134/1350 [01:38<14:54,  1.36it/s]
2024-02-27 16:07:50,950 - INFO: train loss: 0.09:  10%|9         | 134/1350 [01:48<14:54,  1.36it/s]
2024-02-27 16:08:30,081 - INFO: train loss: 0.19:  15%|#4        | 201/1350 [02:28<14:06,  1.36it/s]
2024-02-27 16:08:40,951 - INFO: train loss: 0.19:  15%|#4        | 201/1350 [02:38<14:06,  1.36it/s]
2024-02-27 16:09:18,496 - INFO: train loss: 0.24:  20%|#9        | 268/1350 [03:16<13:11,  1.37it/s]
2024-02-27 16:09:29,171 - INFO: train loss: 0.24:  20%|#9        | 268/1350 [03:27<13:11,  1.37it/s]
2024-02-27 16:10:08,931 - INFO: train loss: 0.14:  25%|##4       | 335/1350 [04:06<12:30,  1.35it/s]
2024-02-27 16:10:19,173 - INFO: train loss: 0.14:  25%|##4       | 335/1350 [04:17<12:30,  1.35it/s]
2024-02-27 16:10:57,913 - INFO: train loss: 0.22:  30%|##9       | 402/1350 [04:55<11:38,  1.36it/s]
2024-02-27 16:11:09,174 - INFO: train loss: 0.22:  30%|##9       | 402/1350 [05:07<11:38,  1.36it/s]
2024-02-27 16:11:46,772 - INFO: train loss: 0.20:  35%|###4      | 469/1350 [05:44<10:46,  1.36it/s]
2024-02-27 16:11:59,175 - INFO: train loss: 0.20:  35%|###4      | 469/1350 [05:57<10:46,  1.36it/s]
2024-02-27 16:12:35,227 - INFO: train loss: 0.14:  40%|###9      | 536/1350 [06:33<09:54,  1.37it/s]
2024-02-27 16:12:49,176 - INFO: train loss: 0.14:  40%|###9      | 536/1350 [06:47<09:54,  1.37it/s]
2024-02-27 16:13:24,332 - INFO: train loss: 0.12:  45%|####4     | 603/1350 [07:22<09:06,  1.37it/s]
2024-02-27 16:13:39,178 - INFO: train loss: 0.12:  45%|####4     | 603/1350 [07:37<09:06,  1.37it/s]
2024-02-27 16:14:12,476 - INFO: train loss: 0.17:  50%|####9     | 670/1350 [08:10<08:14,  1.37it/s]
2024-02-27 16:14:29,179 - INFO: train loss: 0.17:  50%|####9     | 670/1350 [08:27<08:14,  1.37it/s]
2024-02-27 16:15:01,820 - INFO: train loss: 0.18:  55%|#####4    | 737/1350 [08:59<07:27,  1.37it/s]
2024-02-27 16:15:19,180 - INFO: train loss: 0.18:  55%|#####4    | 737/1350 [09:17<07:27,  1.37it/s]
2024-02-27 16:15:52,248 - INFO: train loss: 0.14:  60%|#####9    | 804/1350 [09:50<06:42,  1.36it/s]
2024-02-27 16:16:09,182 - INFO: train loss: 0.14:  60%|#####9    | 804/1350 [10:07<06:42,  1.36it/s]
2024-02-27 16:16:40,738 - INFO: train loss: 0.21:  65%|######4   | 871/1350 [10:38<05:51,  1.36it/s]
2024-02-27 16:16:50,962 - INFO: train loss: 0.21:  65%|######4   | 871/1350 [10:48<05:51,  1.36it/s]
2024-02-27 16:17:31,430 - INFO: train loss: 0.28:  69%|######9   | 938/1350 [11:29<05:04,  1.35it/s]
2024-02-27 16:17:49,184 - INFO: train loss: 0.28:  69%|######9   | 938/1350 [11:47<05:04,  1.35it/s]
2024-02-27 16:18:19,938 - INFO: train loss: 0.27:  74%|#######4  | 1005/1350 [12:17<04:13,  1.36it/s]
2024-02-27 16:18:30,964 - INFO: train loss: 0.27:  74%|#######4  | 1005/1350 [12:28<04:13,  1.36it/s]
2024-02-27 16:19:07,292 - INFO: train loss: 0.27:  79%|#######9  | 1072/1350 [13:05<03:22,  1.38it/s]
2024-02-27 16:19:19,186 - INFO: train loss: 0.27:  79%|#######9  | 1072/1350 [13:17<03:22,  1.38it/s]
2024-02-27 16:19:57,865 - INFO: train loss: 0.37:  84%|########4 | 1139/1350 [13:55<02:35,  1.36it/s]
2024-02-27 16:20:09,188 - INFO: train loss: 0.37:  84%|########4 | 1139/1350 [14:07<02:35,  1.36it/s]
2024-02-27 16:20:47,020 - INFO: train loss: 0.27:  89%|########9 | 1206/1350 [14:45<01:45,  1.36it/s]
2024-02-27 16:20:59,189 - INFO: train loss: 0.27:  89%|########9 | 1206/1350 [14:57<01:45,  1.36it/s]
2024-02-27 16:21:35,850 - INFO: train loss: 0.16:  94%|#########4| 1273/1350 [15:33<00:56,  1.36it/s]
2024-02-27 16:21:49,190 - INFO: train loss: 0.16:  94%|#########4| 1273/1350 [15:47<00:56,  1.36it/s]
2024-02-27 16:22:25,717 - INFO: train loss: 0.25:  99%|#########9| 1340/1350 [16:23<00:07,  1.36it/s]
2024-02-27 16:22:33,016 - INFO: train loss: 0.23: 100%|##########| 1350/1350 [16:31<00:00,  1.36it/s]
2024-02-27 16:22:33,020 - INFO: Starting validation inference
2024-02-27 16:22:33,021 - INFO: validation progress:   0%|          | 0/1200 [00:00<?, ?it/s]
2024-02-27 16:22:37,246 - INFO: validation progress:   5%|5         | 60/1200 [00:04<01:20, 14.20it/s]
2024-02-27 16:22:41,458 - INFO: validation progress:  10%|#         | 120/1200 [00:08<01:15, 14.23it/s]
2024-02-27 16:22:45,584 - INFO: validation progress:  15%|#5        | 180/1200 [00:12<01:10, 14.37it/s]
2024-02-27 16:22:49,733 - INFO: validation progress:  20%|##        | 240/1200 [00:16<01:06, 14.41it/s]
2024-02-27 16:22:53,907 - INFO: validation progress:  25%|##5       | 300/1200 [00:20<01:02, 14.39it/s]
2024-02-27 16:22:58,054 - INFO: validation progress:  30%|###       | 360/1200 [00:25<00:58, 14.42it/s]
2024-02-27 16:23:02,243 - INFO: validation progress:  35%|###5      | 420/1200 [00:29<00:54, 14.39it/s]
2024-02-27 16:23:06,387 - INFO: validation progress:  40%|####      | 480/1200 [00:33<00:49, 14.42it/s]
2024-02-27 16:23:10,552 - INFO: validation progress:  45%|####5     | 540/1200 [00:37<00:45, 14.41it/s]
2024-02-27 16:23:14,718 - INFO: validation progress:  50%|#####     | 600/1200 [00:41<00:41, 14.41it/s]
2024-02-27 16:23:18,878 - INFO: validation progress:  55%|#####5    | 660/1200 [00:45<00:37, 14.41it/s]
2024-02-27 16:23:23,033 - INFO: validation progress:  60%|######    | 720/1200 [00:50<00:33, 14.42it/s]
2024-02-27 16:23:27,102 - INFO: validation progress:  65%|######5   | 780/1200 [00:54<00:28, 14.52it/s]
2024-02-27 16:23:31,237 - INFO: validation progress:  70%|#######   | 840/1200 [00:58<00:24, 14.52it/s]
2024-02-27 16:23:35,394 - INFO: validation progress:  75%|#######5  | 900/1200 [01:02<00:20, 14.49it/s]
2024-02-27 16:23:39,511 - INFO: validation progress:  80%|########  | 960/1200 [01:06<00:16, 14.52it/s]
2024-02-27 16:23:43,677 - INFO: validation progress:  85%|########5 | 1020/1200 [01:10<00:12, 14.48it/s]
2024-02-27 16:23:47,838 - INFO: validation progress:  90%|######### | 1080/1200 [01:14<00:08, 14.46it/s]
2024-02-27 16:23:52,005 - INFO: validation progress:  95%|#########5| 1140/1200 [01:18<00:04, 14.44it/s]
2024-02-27 16:23:56,078 - INFO: validation progress: 100%|##########| 1200/1200 [01:23<00:00, 14.53it/s]
2024-02-27 16:23:56,081 - INFO: validation progress: 100%|##########| 1200/1200 [01:23<00:00, 14.45it/s]
2024-02-27 16:23:56,107 - INFO: Mean validation loss: 0.32958
2024-02-27 16:23:56,109 - INFO: Validation AUC: 0.97797
2024-02-27 16:23:56,147 - INFO: Saving best model checkpoint: val_AUC 0.97734 -> 0.97797 to /app/output
2024-02-27 16:24:10,405 - INFO: Training Epoch: 5 / 15
2024-02-27 16:24:10,405 - INFO: train loss:   0%|          | 0/1350 [00:00<?, ?it/s]
2024-02-27 16:24:10,432 - INFO: Evaluation step: 1350
2024-02-27 16:24:59,544 - INFO: train loss: 0.10:   5%|4         | 67/1350 [00:49<15:40,  1.36it/s]
2024-02-27 16:25:09,742 - INFO: train loss: 0.10:   5%|4         | 67/1350 [00:59<15:40,  1.36it/s]
2024-02-27 16:25:48,261 - INFO: train loss: 0.14:  10%|9         | 134/1350 [01:37<14:47,  1.37it/s]
2024-02-27 16:25:59,335 - INFO: train loss: 0.14:  10%|9         | 134/1350 [01:48<14:47,  1.37it/s]
2024-02-27 16:26:36,364 - INFO: train loss: 0.05:  15%|#4        | 201/1350 [02:25<13:52,  1.38it/s]
2024-02-27 16:26:49,337 - INFO: train loss: 0.05:  15%|#4        | 201/1350 [02:38<13:52,  1.38it/s]
2024-02-27 16:27:24,644 - INFO: train loss: 0.15:  20%|#9        | 268/1350 [03:14<13:02,  1.38it/s]
2024-02-27 16:27:39,338 - INFO: train loss: 0.15:  20%|#9        | 268/1350 [03:28<13:02,  1.38it/s]
2024-02-27 16:28:13,602 - INFO: train loss: 0.17:  25%|##4       | 335/1350 [04:03<12:16,  1.38it/s]
2024-02-27 16:28:29,339 - INFO: train loss: 0.17:  25%|##4       | 335/1350 [04:18<12:16,  1.38it/s]
2024-02-27 16:29:02,658 - INFO: train loss: 0.13:  30%|##9       | 402/1350 [04:52<11:30,  1.37it/s]
2024-02-27 16:29:19,341 - INFO: train loss: 0.13:  30%|##9       | 402/1350 [05:08<11:30,  1.37it/s]
2024-02-27 16:29:50,139 - INFO: train loss: 0.12:  35%|###4      | 469/1350 [05:39<10:35,  1.39it/s]
2024-02-27 16:30:01,076 - INFO: train loss: 0.12:  35%|###4      | 469/1350 [05:50<10:35,  1.39it/s]
2024-02-27 16:30:39,323 - INFO: train loss: 0.10:  40%|###9      | 536/1350 [06:28<09:50,  1.38it/s]
2024-02-27 16:30:49,343 - INFO: train loss: 0.10:  40%|###9      | 536/1350 [06:38<09:50,  1.38it/s]
2024-02-27 16:31:27,602 - INFO: train loss: 0.31:  45%|####4     | 603/1350 [07:17<09:00,  1.38it/s]
2024-02-27 16:31:39,344 - INFO: train loss: 0.31:  45%|####4     | 603/1350 [07:28<09:00,  1.38it/s]
2024-02-27 16:32:17,637 - INFO: train loss: 0.16:  50%|####9     | 670/1350 [08:07<08:17,  1.37it/s]
2024-02-27 16:32:29,345 - INFO: train loss: 0.16:  50%|####9     | 670/1350 [08:18<08:17,  1.37it/s]
2024-02-27 16:33:06,454 - INFO: train loss: 0.14:  55%|#####4    | 737/1350 [08:56<07:27,  1.37it/s]
2024-02-27 16:33:19,346 - INFO: train loss: 0.14:  55%|#####4    | 737/1350 [09:08<07:27,  1.37it/s]
2024-02-27 16:33:56,015 - INFO: train loss: 0.15:  60%|#####9    | 804/1350 [09:45<06:40,  1.36it/s]
2024-02-27 16:34:09,347 - INFO: train loss: 0.15:  60%|#####9    | 804/1350 [09:58<06:40,  1.36it/s]
2024-02-27 16:34:44,988 - INFO: train loss: 0.07:  65%|######4   | 871/1350 [10:34<05:50,  1.37it/s]
2024-02-27 16:34:59,349 - INFO: train loss: 0.07:  65%|######4   | 871/1350 [10:48<05:50,  1.37it/s]
2024-02-27 16:35:32,609 - INFO: train loss: 0.20:  69%|######9   | 938/1350 [11:22<04:59,  1.38it/s]
2024-02-27 16:35:49,350 - INFO: train loss: 0.20:  69%|######9   | 938/1350 [11:38<04:59,  1.38it/s]
2024-02-27 16:36:22,715 - INFO: train loss: 0.20:  74%|#######4  | 1005/1350 [12:12<04:12,  1.37it/s]
2024-02-27 16:36:39,351 - INFO: train loss: 0.20:  74%|#######4  | 1005/1350 [12:28<04:12,  1.37it/s]
2024-02-27 16:37:11,405 - INFO: train loss: 0.15:  79%|#######9  | 1072/1350 [13:00<03:23,  1.37it/s]
2024-02-27 16:37:29,352 - INFO: train loss: 0.15:  79%|#######9  | 1072/1350 [13:18<03:23,  1.37it/s]
2024-02-27 16:38:01,672 - INFO: train loss: 0.18:  84%|########4 | 1139/1350 [13:51<02:35,  1.36it/s]
2024-02-27 16:38:19,354 - INFO: train loss: 0.18:  84%|########4 | 1139/1350 [14:08<02:35,  1.36it/s]
2024-02-27 16:38:51,447 - INFO: train loss: 0.11:  89%|########9 | 1206/1350 [14:41<01:46,  1.35it/s]
2024-02-27 16:39:09,355 - INFO: train loss: 0.11:  89%|########9 | 1206/1350 [14:58<01:46,  1.35it/s]
2024-02-27 16:39:39,914 - INFO: train loss: 0.27:  94%|#########4| 1273/1350 [15:29<00:56,  1.36it/s]
2024-02-27 16:39:51,088 - INFO: train loss: 0.27:  94%|#########4| 1273/1350 [15:40<00:56,  1.36it/s]
2024-02-27 16:40:29,266 - INFO: train loss: 0.19:  99%|#########9| 1340/1350 [16:18<00:07,  1.36it/s]
2024-02-27 16:40:36,395 - INFO: train loss: 0.12: 100%|##########| 1350/1350 [16:25<00:00,  1.37it/s]
2024-02-27 16:40:36,399 - INFO: Starting validation inference
2024-02-27 16:40:36,400 - INFO: validation progress:   0%|          | 0/1200 [00:00<?, ?it/s]
2024-02-27 16:40:40,614 - INFO: validation progress:   5%|5         | 60/1200 [00:04<01:20, 14.24it/s]
2024-02-27 16:40:44,788 - INFO: validation progress:  10%|#         | 120/1200 [00:08<01:15, 14.32it/s]
2024-02-27 16:40:48,902 - INFO: validation progress:  15%|#5        | 180/1200 [00:12<01:10, 14.44it/s]
2024-02-27 16:40:53,062 - INFO: validation progress:  20%|##        | 240/1200 [00:16<01:06, 14.43it/s]
2024-02-27 16:40:57,260 - INFO: validation progress:  25%|##5       | 300/1200 [00:20<01:02, 14.38it/s]
2024-02-27 16:41:01,451 - INFO: validation progress:  30%|###       | 360/1200 [00:25<00:58, 14.36it/s]
2024-02-27 16:41:05,664 - INFO: validation progress:  35%|###5      | 420/1200 [00:29<00:54, 14.32it/s]
2024-02-27 16:41:09,828 - INFO: validation progress:  40%|####      | 480/1200 [00:33<00:50, 14.35it/s]
2024-02-27 16:41:14,059 - INFO: validation progress:  45%|####5     | 540/1200 [00:37<00:46, 14.30it/s]
2024-02-27 16:41:18,271 - INFO: validation progress:  50%|#####     | 600/1200 [00:41<00:42, 14.28it/s]
2024-02-27 16:41:22,464 - INFO: validation progress:  55%|#####5    | 660/1200 [00:46<00:37, 14.29it/s]
2024-02-27 16:41:26,652 - INFO: validation progress:  60%|######    | 720/1200 [00:50<00:33, 14.30it/s]
2024-02-27 16:41:30,779 - INFO: validation progress:  65%|######5   | 780/1200 [00:54<00:29, 14.37it/s]
2024-02-27 16:41:34,974 - INFO: validation progress:  70%|#######   | 840/1200 [00:58<00:25, 14.35it/s]
2024-02-27 16:41:39,160 - INFO: validation progress:  75%|#######5  | 900/1200 [01:02<00:20, 14.35it/s]
2024-02-27 16:41:43,322 - INFO: validation progress:  80%|########  | 960/1200 [01:06<00:16, 14.37it/s]
2024-02-27 16:41:47,499 - INFO: validation progress:  85%|########5 | 1020/1200 [01:11<00:12, 14.37it/s]
2024-02-27 16:41:51,688 - INFO: validation progress:  90%|######### | 1080/1200 [01:15<00:08, 14.35it/s]
2024-02-27 16:41:55,833 - INFO: validation progress:  95%|#########5| 1140/1200 [01:19<00:04, 14.39it/s]
2024-02-27 16:41:59,938 - INFO: validation progress: 100%|##########| 1200/1200 [01:23<00:00, 14.46it/s]
2024-02-27 16:41:59,941 - INFO: validation progress: 100%|##########| 1200/1200 [01:23<00:00, 14.36it/s]
2024-02-27 16:41:59,968 - INFO: Mean validation loss: 0.42099
2024-02-27 16:41:59,970 - INFO: Validation AUC: 0.97725
2024-02-27 16:42:00,034 - INFO: Training Epoch: 6 / 15
2024-02-27 16:42:00,034 - INFO: train loss:   0%|          | 0/1350 [00:00<?, ?it/s]
2024-02-27 16:42:00,061 - INFO: Evaluation step: 1350
2024-02-27 16:42:49,227 - INFO: train loss: 0.08:   5%|4         | 67/1350 [00:49<15:41,  1.36it/s]
2024-02-27 16:42:59,360 - INFO: train loss: 0.08:   5%|4         | 67/1350 [00:59<15:41,  1.36it/s]
2024-02-27 16:43:37,853 - INFO: train loss: 0.07:  10%|9         | 134/1350 [01:37<14:46,  1.37it/s]
2024-02-27 16:43:49,361 - INFO: train loss: 0.07:  10%|9         | 134/1350 [01:49<14:46,  1.37it/s]
2024-02-27 16:44:28,092 - INFO: train loss: 0.07:  15%|#4        | 201/1350 [02:28<14:08,  1.35it/s]
2024-02-27 16:44:39,362 - INFO: train loss: 0.07:  15%|#4        | 201/1350 [02:39<14:08,  1.35it/s]
2024-02-27 16:45:19,117 - INFO: train loss: 0.07:  20%|#9        | 268/1350 [03:19<13:29,  1.34it/s]
2024-02-27 16:45:29,363 - INFO: train loss: 0.07:  20%|#9        | 268/1350 [03:29<13:29,  1.34it/s]
2024-02-27 16:46:08,582 - INFO: train loss: 0.10:  25%|##4       | 335/1350 [04:08<12:35,  1.34it/s]
2024-02-27 16:46:19,364 - INFO: train loss: 0.10:  25%|##4       | 335/1350 [04:19<12:35,  1.34it/s]
2024-02-27 16:46:56,973 - INFO: train loss: 0.03:  30%|##9       | 402/1350 [04:56<11:38,  1.36it/s]
2024-02-27 16:47:09,366 - INFO: train loss: 0.03:  30%|##9       | 402/1350 [05:09<11:38,  1.36it/s]
2024-02-27 16:47:45,197 - INFO: train loss: 0.05:  35%|###4      | 469/1350 [05:45<10:44,  1.37it/s]
2024-02-27 16:47:59,367 - INFO: train loss: 0.05:  35%|###4      | 469/1350 [05:59<10:44,  1.37it/s]
2024-02-27 16:48:32,037 - INFO: train loss: 0.04:  40%|###9      | 536/1350 [06:32<09:46,  1.39it/s]
2024-02-27 16:48:49,368 - INFO: train loss: 0.04:  40%|###9      | 536/1350 [06:49<09:46,  1.39it/s]
2024-02-27 16:49:22,511 - INFO: train loss: 0.05:  45%|####4     | 603/1350 [07:22<09:06,  1.37it/s]
2024-02-27 16:49:39,369 - INFO: train loss: 0.05:  45%|####4     | 603/1350 [07:39<09:06,  1.37it/s]
2024-02-27 16:50:11,618 - INFO: train loss: 0.06:  50%|####9     | 670/1350 [08:11<08:17,  1.37it/s]
2024-02-27 16:50:29,370 - INFO: train loss: 0.06:  50%|####9     | 670/1350 [08:29<08:17,  1.37it/s]
2024-02-27 16:51:00,061 - INFO: train loss: 0.04:  55%|#####4    | 737/1350 [09:00<07:26,  1.37it/s]
2024-02-27 16:51:11,102 - INFO: train loss: 0.04:  55%|#####4    | 737/1350 [09:11<07:26,  1.37it/s]
2024-02-27 16:51:50,850 - INFO: train loss: 0.12:  60%|#####9    | 804/1350 [09:50<06:42,  1.36it/s]
2024-02-27 16:52:01,104 - INFO: train loss: 0.12:  60%|#####9    | 804/1350 [10:01<06:42,  1.36it/s]
2024-02-27 16:52:40,355 - INFO: train loss: 0.04:  65%|######4   | 871/1350 [10:40<05:53,  1.35it/s]
2024-02-27 16:52:51,105 - INFO: train loss: 0.04:  65%|######4   | 871/1350 [10:51<05:53,  1.35it/s]
2024-02-27 16:53:28,731 - INFO: train loss: 0.11:  69%|######9   | 938/1350 [11:28<05:02,  1.36it/s]
2024-02-27 16:53:39,374 - INFO: train loss: 0.11:  69%|######9   | 938/1350 [11:39<05:02,  1.36it/s]
2024-02-27 16:54:18,008 - INFO: train loss: 0.09:  74%|#######4  | 1005/1350 [12:17<04:13,  1.36it/s]
2024-02-27 16:54:29,376 - INFO: train loss: 0.09:  74%|#######4  | 1005/1350 [12:29<04:13,  1.36it/s]
2024-02-27 16:55:06,585 - INFO: train loss: 0.09:  79%|#######9  | 1072/1350 [13:06<03:23,  1.37it/s]
2024-02-27 16:55:19,377 - INFO: train loss: 0.09:  79%|#######9  | 1072/1350 [13:19<03:23,  1.37it/s]
2024-02-27 16:55:56,984 - INFO: train loss: 0.04:  84%|########4 | 1139/1350 [13:56<02:35,  1.36it/s]
2024-02-27 16:56:09,378 - INFO: train loss: 0.04:  84%|########4 | 1139/1350 [14:09<02:35,  1.36it/s]
2024-02-27 16:56:45,946 - INFO: train loss: 0.11:  89%|########9 | 1206/1350 [14:45<01:45,  1.36it/s]
2024-02-27 16:56:59,380 - INFO: train loss: 0.11:  89%|########9 | 1206/1350 [14:59<01:45,  1.36it/s]
2024-02-27 16:57:35,033 - INFO: train loss: 0.14:  94%|#########4| 1273/1350 [15:34<00:56,  1.36it/s]
2024-02-27 16:57:49,381 - INFO: train loss: 0.14:  94%|#########4| 1273/1350 [15:49<00:56,  1.36it/s]
2024-02-27 16:58:24,675 - INFO: train loss: 0.12:  99%|#########9| 1340/1350 [16:24<00:07,  1.36it/s]
2024-02-27 16:58:31,829 - INFO: train loss: 0.11: 100%|##########| 1350/1350 [16:31<00:00,  1.36it/s]
2024-02-27 16:58:31,834 - INFO: Starting validation inference
2024-02-27 16:58:31,834 - INFO: validation progress:   0%|          | 0/1200 [00:00<?, ?it/s]
2024-02-27 16:58:36,109 - INFO: validation progress:   5%|5         | 60/1200 [00:04<01:21, 14.03it/s]
2024-02-27 16:58:40,332 - INFO: validation progress:  10%|#         | 120/1200 [00:08<01:16, 14.14it/s]
2024-02-27 16:58:44,481 - INFO: validation progress:  15%|#5        | 180/1200 [00:12<01:11, 14.28it/s]
2024-02-27 16:58:48,657 - INFO: validation progress:  20%|##        | 240/1200 [00:16<01:07, 14.32it/s]
2024-02-27 16:58:52,875 - INFO: validation progress:  25%|##5       | 300/1200 [00:21<01:03, 14.28it/s]
2024-02-27 16:58:57,054 - INFO: validation progress:  30%|###       | 360/1200 [00:25<00:58, 14.31it/s]
2024-02-27 16:59:01,279 - INFO: validation progress:  35%|###5      | 420/1200 [00:29<00:54, 14.27it/s]
2024-02-27 16:59:05,453 - INFO: validation progress:  40%|####      | 480/1200 [00:33<00:50, 14.31it/s]
2024-02-27 16:59:09,657 - INFO: validation progress:  45%|####5     | 540/1200 [00:37<00:46, 14.29it/s]
2024-02-27 16:59:13,870 - INFO: validation progress:  50%|#####     | 600/1200 [00:42<00:42, 14.28it/s]
2024-02-27 16:59:18,072 - INFO: validation progress:  55%|#####5    | 660/1200 [00:46<00:37, 14.28it/s]
2024-02-27 16:59:22,276 - INFO: validation progress:  60%|######    | 720/1200 [00:50<00:33, 14.28it/s]
2024-02-27 16:59:26,439 - INFO: validation progress:  65%|######5   | 780/1200 [00:54<00:29, 14.32it/s]
2024-02-27 16:59:30,653 - INFO: validation progress:  70%|#######   | 840/1200 [00:58<00:25, 14.29it/s]
2024-02-27 16:59:34,874 - INFO: validation progress:  75%|#######5  | 900/1200 [01:03<00:21, 14.27it/s]
2024-02-27 16:59:39,056 - INFO: validation progress:  80%|########  | 960/1200 [01:07<00:16, 14.29it/s]
2024-02-27 16:59:43,268 - INFO: validation progress:  85%|########5 | 1020/1200 [01:11<00:12, 14.28it/s]
2024-02-27 16:59:47,480 - INFO: validation progress:  90%|######### | 1080/1200 [01:15<00:08, 14.27it/s]
2024-02-27 16:59:51,675 - INFO: validation progress:  95%|#########5| 1140/1200 [01:19<00:04, 14.28it/s]
2024-02-27 16:59:55,799 - INFO: validation progress: 100%|##########| 1200/1200 [01:23<00:00, 14.36it/s]
2024-02-27 16:59:55,802 - INFO: validation progress: 100%|##########| 1200/1200 [01:23<00:00, 14.29it/s]
2024-02-27 16:59:55,829 - INFO: Mean validation loss: 0.35996
2024-02-27 16:59:55,831 - INFO: Validation AUC: 0.98081
2024-02-27 16:59:55,869 - INFO: Saving best model checkpoint: val_AUC 0.97797 -> 0.98081 to /app/output
2024-02-27 17:00:10,151 - INFO: Training Epoch: 7 / 15
2024-02-27 17:00:10,151 - INFO: train loss:   0%|          | 0/1350 [00:00<?, ?it/s]
2024-02-27 17:00:10,178 - INFO: Evaluation step: 1350
2024-02-27 17:00:59,169 - INFO: train loss: 0.04:   5%|4         | 67/1350 [00:49<15:38,  1.37it/s]
2024-02-27 17:01:09,481 - INFO: train loss: 0.04:   5%|4         | 67/1350 [00:59<15:38,  1.37it/s]
2024-02-27 17:01:48,570 - INFO: train loss: 0.07:  10%|9         | 134/1350 [01:38<14:53,  1.36it/s]
2024-02-27 17:01:59,483 - INFO: train loss: 0.07:  10%|9         | 134/1350 [01:49<14:53,  1.36it/s]
2024-02-27 17:02:37,008 - INFO: train loss: 0.07:  15%|#4        | 201/1350 [02:26<13:58,  1.37it/s]
2024-02-27 17:02:49,484 - INFO: train loss: 0.07:  15%|#4        | 201/1350 [02:39<13:58,  1.37it/s]
2024-02-27 17:03:26,520 - INFO: train loss: 0.01:  20%|#9        | 268/1350 [03:16<13:13,  1.36it/s]
2024-02-27 17:03:39,485 - INFO: train loss: 0.01:  20%|#9        | 268/1350 [03:29<13:13,  1.36it/s]
2024-02-27 17:04:15,557 - INFO: train loss: 0.03:  25%|##4       | 335/1350 [04:05<12:23,  1.36it/s]
2024-02-27 17:04:29,487 - INFO: train loss: 0.03:  25%|##4       | 335/1350 [04:19<12:23,  1.36it/s]
2024-02-27 17:05:03,749 - INFO: train loss: 0.02:  30%|##9       | 402/1350 [04:53<11:30,  1.37it/s]
2024-02-27 17:05:19,488 - INFO: train loss: 0.02:  30%|##9       | 402/1350 [05:09<11:30,  1.37it/s]
2024-02-27 17:05:52,427 - INFO: train loss: 0.10:  35%|###4      | 469/1350 [05:42<10:41,  1.37it/s]
2024-02-27 17:06:09,489 - INFO: train loss: 0.10:  35%|###4      | 469/1350 [05:59<10:41,  1.37it/s]
2024-02-27 17:06:40,530 - INFO: train loss: 0.21:  40%|###9      | 536/1350 [06:30<09:49,  1.38it/s]
2024-02-27 17:06:51,200 - INFO: train loss: 0.21:  40%|###9      | 536/1350 [06:41<09:49,  1.38it/s]
2024-02-27 17:07:29,889 - INFO: train loss: 0.03:  45%|####4     | 603/1350 [07:19<09:04,  1.37it/s]
2024-02-27 17:07:39,895 - INFO: train loss: 0.03:  45%|####4     | 603/1350 [07:29<09:04,  1.37it/s]
2024-02-27 17:08:19,218 - INFO: train loss: 0.03:  50%|####9     | 670/1350 [08:09<08:16,  1.37it/s]
2024-02-27 17:08:29,492 - INFO: train loss: 0.03:  50%|####9     | 670/1350 [08:19<08:16,  1.37it/s]
2024-02-27 17:09:09,175 - INFO: train loss: 0.12:  55%|#####4    | 737/1350 [08:59<07:30,  1.36it/s]
2024-02-27 17:09:19,493 - INFO: train loss: 0.12:  55%|#####4    | 737/1350 [09:09<07:30,  1.36it/s]
2024-02-27 17:09:58,540 - INFO: train loss: 0.04:  60%|#####9    | 804/1350 [09:48<06:41,  1.36it/s]
2024-02-27 17:10:09,495 - INFO: train loss: 0.04:  60%|#####9    | 804/1350 [09:59<06:41,  1.36it/s]
2024-02-27 17:10:46,803 - INFO: train loss: 0.11:  65%|######4   | 871/1350 [10:36<05:50,  1.37it/s]
2024-02-27 17:10:59,496 - INFO: train loss: 0.11:  65%|######4   | 871/1350 [10:49<05:50,  1.37it/s]
2024-02-27 17:11:35,507 - INFO: train loss: 0.01:  69%|######9   | 938/1350 [11:25<05:00,  1.37it/s]
2024-02-27 17:11:49,497 - INFO: train loss: 0.01:  69%|######9   | 938/1350 [11:39<05:00,  1.37it/s]
2024-02-27 17:12:25,717 - INFO: train loss: 0.03:  74%|#######4  | 1005/1350 [12:15<04:13,  1.36it/s]
2024-02-27 17:12:39,498 - INFO: train loss: 0.03:  74%|#######4  | 1005/1350 [12:29<04:13,  1.36it/s]
2024-02-27 17:13:14,554 - INFO: train loss: 0.05:  79%|#######9  | 1072/1350 [13:04<03:23,  1.36it/s]
2024-02-27 17:13:29,499 - INFO: train loss: 0.05:  79%|#######9  | 1072/1350 [13:19<03:23,  1.36it/s]
2024-02-27 17:14:03,633 - INFO: train loss: 0.12:  84%|########4 | 1139/1350 [13:53<02:34,  1.36it/s]
2024-02-27 17:14:19,500 - INFO: train loss: 0.12:  84%|########4 | 1139/1350 [14:09<02:34,  1.36it/s]
2024-02-27 17:14:53,167 - INFO: train loss: 0.10:  89%|########9 | 1206/1350 [14:43<01:45,  1.36it/s]
2024-02-27 17:15:09,502 - INFO: train loss: 0.10:  89%|########9 | 1206/1350 [14:59<01:45,  1.36it/s]
2024-02-27 17:15:42,796 - INFO: train loss: 0.05:  94%|#########4| 1273/1350 [15:32<00:56,  1.36it/s]
2024-02-27 17:15:59,503 - INFO: train loss: 0.05:  94%|#########4| 1273/1350 [15:49<00:56,  1.36it/s]
2024-02-27 17:16:32,697 - INFO: train loss: 0.14:  99%|#########9| 1340/1350 [16:22<00:07,  1.35it/s]
2024-02-27 17:16:40,047 - INFO: train loss: 0.06: 100%|##########| 1350/1350 [16:29<00:00,  1.36it/s]
2024-02-27 17:16:40,051 - INFO: Starting validation inference
2024-02-27 17:16:40,051 - INFO: validation progress:   0%|          | 0/1200 [00:00<?, ?it/s]
2024-02-27 17:16:44,321 - INFO: validation progress:   5%|5         | 60/1200 [00:04<01:21, 14.05it/s]
2024-02-27 17:16:48,551 - INFO: validation progress:  10%|#         | 120/1200 [00:08<01:16, 14.13it/s]
2024-02-27 17:16:52,701 - INFO: validation progress:  15%|#5        | 180/1200 [00:12<01:11, 14.28it/s]
2024-02-27 17:16:56,882 - INFO: validation progress:  20%|##        | 240/1200 [00:16<01:07, 14.31it/s]
2024-02-27 17:17:01,089 - INFO: validation progress:  25%|##5       | 300/1200 [00:21<01:02, 14.29it/s]
2024-02-27 17:17:05,291 - INFO: validation progress:  30%|###       | 360/1200 [00:25<00:58, 14.29it/s]
2024-02-27 17:17:09,499 - INFO: validation progress:  35%|###5      | 420/1200 [00:29<00:54, 14.28it/s]
2024-02-27 17:17:13,685 - INFO: validation progress:  40%|####      | 480/1200 [00:33<00:50, 14.30it/s]
2024-02-27 17:17:17,876 - INFO: validation progress:  45%|####5     | 540/1200 [00:37<00:46, 14.30it/s]
2024-02-27 17:17:22,089 - INFO: validation progress:  50%|#####     | 600/1200 [00:42<00:42, 14.28it/s]
2024-02-27 17:17:26,301 - INFO: validation progress:  55%|#####5    | 660/1200 [00:46<00:37, 14.27it/s]
2024-02-27 17:17:30,509 - INFO: validation progress:  60%|######    | 720/1200 [00:50<00:33, 14.27it/s]
2024-02-27 17:17:34,664 - INFO: validation progress:  65%|######5   | 780/1200 [00:54<00:29, 14.32it/s]
2024-02-27 17:17:38,873 - INFO: validation progress:  70%|#######   | 840/1200 [00:58<00:25, 14.30it/s]
2024-02-27 17:17:43,093 - INFO: validation progress:  75%|#######5  | 900/1200 [01:03<00:21, 14.27it/s]
2024-02-27 17:17:47,273 - INFO: validation progress:  80%|########  | 960/1200 [01:07<00:16, 14.30it/s]
2024-02-27 17:17:51,495 - INFO: validation progress:  85%|########5 | 1020/1200 [01:11<00:12, 14.27it/s]
2024-02-27 17:17:55,696 - INFO: validation progress:  90%|######### | 1080/1200 [01:15<00:08, 14.28it/s]
2024-02-27 17:17:59,877 - INFO: validation progress:  95%|#########5| 1140/1200 [01:19<00:04, 14.30it/s]
2024-02-27 17:18:04,011 - INFO: validation progress: 100%|##########| 1200/1200 [01:23<00:00, 14.36it/s]
2024-02-27 17:18:04,015 - INFO: validation progress: 100%|##########| 1200/1200 [01:23<00:00, 14.29it/s]
2024-02-27 17:18:04,042 - INFO: Mean validation loss: 0.52982
2024-02-27 17:18:04,044 - INFO: Validation AUC: 0.97502
2024-02-27 17:18:04,249 - INFO: Training Epoch: 8 / 15
2024-02-27 17:18:04,249 - INFO: train loss:   0%|          | 0/1350 [00:00<?, ?it/s]
2024-02-27 17:18:04,277 - INFO: Evaluation step: 1350
2024-02-27 17:18:54,074 - INFO: train loss: 0.05:   5%|4         | 67/1350 [00:49<15:54,  1.34it/s]
2024-02-27 17:19:09,507 - INFO: train loss: 0.05:   5%|4         | 67/1350 [01:05<15:54,  1.34it/s]
2024-02-27 17:19:43,628 - INFO: train loss: 0.01:  10%|9         | 134/1350 [01:39<15:01,  1.35it/s]
2024-02-27 17:19:59,508 - INFO: train loss: 0.01:  10%|9         | 134/1350 [01:55<15:01,  1.35it/s]
2024-02-27 17:20:32,566 - INFO: train loss: 0.03:  15%|#4        | 201/1350 [02:28<14:06,  1.36it/s]
2024-02-27 17:20:49,509 - INFO: train loss: 0.03:  15%|#4        | 201/1350 [02:45<14:06,  1.36it/s]
2024-02-27 17:21:23,691 - INFO: train loss: 0.13:  20%|#9        | 268/1350 [03:19<13:28,  1.34it/s]
2024-02-27 17:21:39,511 - INFO: train loss: 0.13:  20%|#9        | 268/1350 [03:35<13:28,  1.34it/s]
2024-02-27 17:22:13,406 - INFO: train loss: 0.08:  25%|##4       | 335/1350 [04:09<12:36,  1.34it/s]
2024-02-27 17:22:29,512 - INFO: train loss: 0.08:  25%|##4       | 335/1350 [04:25<12:36,  1.34it/s]
2024-02-27 17:23:01,748 - INFO: train loss: 0.04:  30%|##9       | 402/1350 [04:57<11:38,  1.36it/s]
2024-02-27 17:23:19,514 - INFO: train loss: 0.04:  30%|##9       | 402/1350 [05:15<11:38,  1.36it/s]
2024-02-27 17:23:51,493 - INFO: train loss: 0.02:  35%|###4      | 469/1350 [05:47<10:50,  1.35it/s]
2024-02-27 17:24:09,515 - INFO: train loss: 0.02:  35%|###4      | 469/1350 [06:05<10:50,  1.35it/s]
2024-02-27 17:24:39,130 - INFO: train loss: 0.05:  40%|###9      | 536/1350 [06:34<09:54,  1.37it/s]
2024-02-27 17:24:49,516 - INFO: train loss: 0.05:  40%|###9      | 536/1350 [06:45<09:54,  1.37it/s]
2024-02-27 17:25:28,234 - INFO: train loss: 0.11:  45%|####4     | 603/1350 [07:23<09:05,  1.37it/s]
2024-02-27 17:25:39,517 - INFO: train loss: 0.11:  45%|####4     | 603/1350 [07:35<09:05,  1.37it/s]
2024-02-27 17:26:18,704 - INFO: train loss: 0.03:  50%|####9     | 670/1350 [08:14<08:21,  1.36it/s]
2024-02-27 17:26:29,518 - INFO: train loss: 0.03:  50%|####9     | 670/1350 [08:25<08:21,  1.36it/s]
2024-02-27 17:27:08,194 - INFO: train loss: 0.03:  55%|#####4    | 737/1350 [09:03<07:32,  1.35it/s]
2024-02-27 17:27:19,520 - INFO: train loss: 0.03:  55%|#####4    | 737/1350 [09:15<07:32,  1.35it/s]
2024-02-27 17:27:56,263 - INFO: train loss: 0.07:  60%|#####9    | 804/1350 [09:52<06:39,  1.37it/s]
2024-02-27 17:28:09,521 - INFO: train loss: 0.07:  60%|#####9    | 804/1350 [10:05<06:39,  1.37it/s]
2024-02-27 17:28:47,023 - INFO: train loss: 0.02:  65%|######4   | 871/1350 [10:42<05:54,  1.35it/s]
2024-02-27 17:28:59,522 - INFO: train loss: 0.02:  65%|######4   | 871/1350 [10:55<05:54,  1.35it/s]
2024-02-27 17:29:37,288 - INFO: train loss: 0.03:  69%|######9   | 938/1350 [11:33<05:06,  1.35it/s]
2024-02-27 17:29:49,524 - INFO: train loss: 0.03:  69%|######9   | 938/1350 [11:45<05:06,  1.35it/s]
2024-02-27 17:30:25,864 - INFO: train loss: 0.01:  74%|#######4  | 1005/1350 [12:21<04:14,  1.36it/s]
2024-02-27 17:30:39,525 - INFO: train loss: 0.01:  74%|#######4  | 1005/1350 [12:35<04:14,  1.36it/s]
2024-02-27 17:31:13,197 - INFO: train loss: 0.12:  79%|#######9  | 1072/1350 [13:08<03:22,  1.37it/s]
2024-02-27 17:31:29,526 - INFO: train loss: 0.12:  79%|#######9  | 1072/1350 [13:25<03:22,  1.37it/s]
2024-02-27 17:32:03,012 - INFO: train loss: 0.08:  84%|########4 | 1139/1350 [13:58<02:34,  1.36it/s]
2024-02-27 17:32:19,528 - INFO: train loss: 0.08:  84%|########4 | 1139/1350 [14:15<02:34,  1.36it/s]
2024-02-27 17:32:52,710 - INFO: train loss: 0.07:  89%|########9 | 1206/1350 [14:48<01:45,  1.36it/s]
2024-02-27 17:33:09,531 - INFO: train loss: 0.07:  89%|########9 | 1206/1350 [15:05<01:45,  1.36it/s]
2024-02-27 17:33:41,708 - INFO: train loss: 0.08:  94%|#########4| 1273/1350 [15:37<00:56,  1.36it/s]
2024-02-27 17:33:59,532 - INFO: train loss: 0.08:  94%|#########4| 1273/1350 [15:55<00:56,  1.36it/s]
2024-02-27 17:34:30,578 - INFO: train loss: 0.10:  99%|#########9| 1340/1350 [16:26<00:07,  1.36it/s]
2024-02-27 17:34:37,264 - INFO: train loss: 0.20: 100%|##########| 1350/1350 [16:33<00:00,  1.36it/s]
2024-02-27 17:34:37,268 - INFO: Starting validation inference
2024-02-27 17:34:37,269 - INFO: validation progress:   0%|          | 0/1200 [00:00<?, ?it/s]
2024-02-27 17:34:41,588 - INFO: validation progress:   5%|5         | 60/1200 [00:04<01:22, 13.89it/s]
2024-02-27 17:34:45,889 - INFO: validation progress:  10%|#         | 120/1200 [00:08<01:17, 13.93it/s]
2024-02-27 17:34:50,070 - INFO: validation progress:  15%|#5        | 180/1200 [00:12<01:12, 14.12it/s]
2024-02-27 17:34:54,292 - INFO: validation progress:  20%|##        | 240/1200 [00:17<01:07, 14.15it/s]
2024-02-27 17:34:58,557 - INFO: validation progress:  25%|##5       | 300/1200 [00:21<01:03, 14.12it/s]
2024-02-27 17:35:02,799 - INFO: validation progress:  30%|###       | 360/1200 [00:25<00:59, 14.13it/s]
2024-02-27 17:35:07,075 - INFO: validation progress:  35%|###5      | 420/1200 [00:29<00:55, 14.10it/s]
2024-02-27 17:35:11,305 - INFO: validation progress:  40%|####      | 480/1200 [00:34<00:50, 14.13it/s]
2024-02-27 17:35:15,556 - INFO: validation progress:  45%|####5     | 540/1200 [00:38<00:46, 14.12it/s]
2024-02-27 17:35:19,820 - INFO: validation progress:  50%|#####     | 600/1200 [00:42<00:42, 14.11it/s]
2024-02-27 17:35:24,117 - INFO: validation progress:  55%|#####5    | 660/1200 [00:46<00:38, 14.06it/s]
2024-02-27 17:35:28,369 - INFO: validation progress:  60%|######    | 720/1200 [00:51<00:34, 14.08it/s]
2024-02-27 17:35:32,566 - INFO: validation progress:  65%|######5   | 780/1200 [00:55<00:29, 14.14it/s]
2024-02-27 17:35:36,825 - INFO: validation progress:  70%|#######   | 840/1200 [00:59<00:25, 14.13it/s]
2024-02-27 17:35:41,096 - INFO: validation progress:  75%|#######5  | 900/1200 [01:03<00:21, 14.10it/s]
2024-02-27 17:35:45,324 - INFO: validation progress:  80%|########  | 960/1200 [01:08<00:16, 14.13it/s]
2024-02-27 17:35:49,577 - INFO: validation progress:  85%|########5 | 1020/1200 [01:12<00:12, 14.12it/s]
2024-02-27 17:35:53,830 - INFO: validation progress:  90%|######### | 1080/1200 [01:16<00:08, 14.12it/s]
2024-02-27 17:35:58,078 - INFO: validation progress:  95%|#########5| 1140/1200 [01:20<00:04, 14.12it/s]
2024-02-27 17:36:02,289 - INFO: validation progress: 100%|##########| 1200/1200 [01:25<00:00, 14.16it/s]
2024-02-27 17:36:02,293 - INFO: validation progress: 100%|##########| 1200/1200 [01:25<00:00, 14.11it/s]
2024-02-27 17:36:02,321 - INFO: Mean validation loss: 0.46195
2024-02-27 17:36:02,323 - INFO: Validation AUC: 0.98454
2024-02-27 17:36:02,361 - INFO: Saving best model checkpoint: val_AUC 0.98081 -> 0.98454 to /app/output
2024-02-27 17:36:16,570 - INFO: Training Epoch: 9 / 15
2024-02-27 17:36:16,571 - INFO: train loss:   0%|          | 0/1350 [00:00<?, ?it/s]
2024-02-27 17:36:16,597 - INFO: Evaluation step: 1350
2024-02-27 17:37:06,880 - INFO: train loss: 0.10:   5%|4         | 67/1350 [00:50<16:03,  1.33it/s]
2024-02-27 17:37:19,544 - INFO: train loss: 0.10:   5%|4         | 67/1350 [01:02<16:03,  1.33it/s]
2024-02-27 17:37:55,697 - INFO: train loss: 0.04:  10%|9         | 134/1350 [01:39<14:57,  1.36it/s]
2024-02-27 17:38:09,546 - INFO: train loss: 0.04:  10%|9         | 134/1350 [01:52<14:57,  1.36it/s]
2024-02-27 17:38:45,033 - INFO: train loss: 0.01:  15%|#4        | 201/1350 [02:28<14:06,  1.36it/s]
2024-02-27 17:38:59,547 - INFO: train loss: 0.01:  15%|#4        | 201/1350 [02:42<14:06,  1.36it/s]
2024-02-27 17:39:35,036 - INFO: train loss: 0.00:  20%|#9        | 268/1350 [03:18<13:21,  1.35it/s]
2024-02-27 17:39:49,548 - INFO: train loss: 0.00:  20%|#9        | 268/1350 [03:32<13:21,  1.35it/s]
2024-02-27 17:40:24,263 - INFO: train loss: 0.02:  25%|##4       | 335/1350 [04:07<12:29,  1.35it/s]
2024-02-27 17:40:39,549 - INFO: train loss: 0.02:  25%|##4       | 335/1350 [04:22<12:29,  1.35it/s]
2024-02-27 17:41:13,948 - INFO: train loss: 0.08:  30%|##9       | 402/1350 [04:57<11:41,  1.35it/s]
2024-02-27 17:41:29,551 - INFO: train loss: 0.08:  30%|##9       | 402/1350 [05:12<11:41,  1.35it/s]
2024-02-27 17:42:02,899 - INFO: train loss: 0.05:  35%|###4      | 469/1350 [05:46<10:48,  1.36it/s]
2024-02-27 17:42:19,552 - INFO: train loss: 0.05:  35%|###4      | 469/1350 [06:02<10:48,  1.36it/s]
2024-02-27 17:42:51,035 - INFO: train loss: 0.06:  40%|###9      | 536/1350 [06:34<09:54,  1.37it/s]
2024-02-27 17:43:01,296 - INFO: train loss: 0.06:  40%|###9      | 536/1350 [06:44<09:54,  1.37it/s]
2024-02-27 17:43:40,623 - INFO: train loss: 0.03:  45%|####4     | 603/1350 [07:24<09:08,  1.36it/s]
2024-02-27 17:43:51,298 - INFO: train loss: 0.03:  45%|####4     | 603/1350 [07:34<09:08,  1.36it/s]
2024-02-27 17:44:29,100 - INFO: train loss: 0.03:  50%|####9     | 670/1350 [08:12<08:16,  1.37it/s]
2024-02-27 17:44:39,555 - INFO: train loss: 0.03:  50%|####9     | 670/1350 [08:22<08:16,  1.37it/s]
2024-02-27 17:45:18,795 - INFO: train loss: 0.01:  55%|#####4    | 737/1350 [09:02<07:29,  1.36it/s]
2024-02-27 17:45:29,557 - INFO: train loss: 0.01:  55%|#####4    | 737/1350 [09:12<07:29,  1.36it/s]
2024-02-27 17:46:09,755 - INFO: train loss: 0.03:  60%|#####9    | 804/1350 [09:53<06:45,  1.35it/s]
2024-02-27 17:46:19,953 - INFO: train loss: 0.03:  60%|#####9    | 804/1350 [10:03<06:45,  1.35it/s]
2024-02-27 17:46:59,813 - INFO: train loss: 0.01:  65%|######4   | 871/1350 [10:43<05:56,  1.34it/s]
2024-02-27 17:47:09,955 - INFO: train loss: 0.01:  65%|######4   | 871/1350 [10:53<05:56,  1.34it/s]
2024-02-27 17:47:49,508 - INFO: train loss: 0.06:  69%|######9   | 938/1350 [11:32<05:06,  1.35it/s]
2024-02-27 17:47:59,561 - INFO: train loss: 0.06:  69%|######9   | 938/1350 [11:42<05:06,  1.35it/s]
2024-02-27 17:48:38,156 - INFO: train loss: 0.03:  74%|#######4  | 1005/1350 [12:21<04:14,  1.36it/s]
2024-02-27 17:48:49,562 - INFO: train loss: 0.03:  74%|#######4  | 1005/1350 [12:32<04:14,  1.36it/s]
2024-02-27 17:49:27,757 - INFO: train loss: 0.07:  79%|#######9  | 1072/1350 [13:11<03:25,  1.35it/s]
2024-02-27 17:49:39,563 - INFO: train loss: 0.07:  79%|#######9  | 1072/1350 [13:22<03:25,  1.35it/s]
2024-02-27 17:50:17,103 - INFO: train loss: 0.02:  84%|########4 | 1139/1350 [14:00<02:35,  1.36it/s]
2024-02-27 17:50:29,565 - INFO: train loss: 0.02:  84%|########4 | 1139/1350 [14:12<02:35,  1.36it/s]
2024-02-27 17:51:03,868 - INFO: train loss: 0.10:  89%|########9 | 1206/1350 [14:47<01:44,  1.38it/s]
2024-02-27 17:51:19,566 - INFO: train loss: 0.10:  89%|########9 | 1206/1350 [15:02<01:44,  1.38it/s]
2024-02-27 17:51:53,738 - INFO: train loss: 0.07:  94%|#########4| 1273/1350 [15:37<00:56,  1.37it/s]
2024-02-27 17:52:09,568 - INFO: train loss: 0.07:  94%|#########4| 1273/1350 [15:52<00:56,  1.37it/s]
2024-02-27 17:52:42,157 - INFO: train loss: 0.08:  99%|#########9| 1340/1350 [16:25<00:07,  1.37it/s]
2024-02-27 17:52:49,906 - INFO: train loss: 0.04: 100%|##########| 1350/1350 [16:33<00:00,  1.36it/s]
2024-02-27 17:52:49,911 - INFO: Starting validation inference
2024-02-27 17:52:49,911 - INFO: validation progress:   0%|          | 0/1200 [00:00<?, ?it/s]
2024-02-27 17:52:54,209 - INFO: validation progress:   5%|5         | 60/1200 [00:04<01:21, 13.96it/s]
2024-02-27 17:52:58,475 - INFO: validation progress:  10%|#         | 120/1200 [00:08<01:17, 14.02it/s]
2024-02-27 17:53:02,676 - INFO: validation progress:  15%|#5        | 180/1200 [00:12<01:12, 14.14it/s]
2024-02-27 17:53:06,916 - INFO: validation progress:  20%|##        | 240/1200 [00:17<01:07, 14.14it/s]
2024-02-27 17:53:11,176 - INFO: validation progress:  25%|##5       | 300/1200 [00:21<01:03, 14.12it/s]
2024-02-27 17:53:15,399 - INFO: validation progress:  30%|###       | 360/1200 [00:25<00:59, 14.15it/s]
2024-02-27 17:53:19,642 - INFO: validation progress:  35%|###5      | 420/1200 [00:29<00:55, 14.15it/s]
2024-02-27 17:53:23,864 - INFO: validation progress:  40%|####      | 480/1200 [00:33<00:50, 14.17it/s]
2024-02-27 17:53:28,136 - INFO: validation progress:  45%|####5     | 540/1200 [00:38<00:46, 14.13it/s]
2024-02-27 17:53:32,398 - INFO: validation progress:  50%|#####     | 600/1200 [00:42<00:42, 14.11it/s]
2024-02-27 17:53:36,669 - INFO: validation progress:  55%|#####5    | 660/1200 [00:46<00:38, 14.09it/s]
2024-02-27 17:53:40,923 - INFO: validation progress:  60%|######    | 720/1200 [00:51<00:34, 14.10it/s]
2024-02-27 17:53:45,101 - INFO: validation progress:  65%|######5   | 780/1200 [00:55<00:29, 14.18it/s]
2024-02-27 17:53:49,334 - INFO: validation progress:  70%|#######   | 840/1200 [00:59<00:25, 14.17it/s]
2024-02-27 17:53:53,603 - INFO: validation progress:  75%|#######5  | 900/1200 [01:03<00:21, 14.14it/s]
2024-02-27 17:53:57,853 - INFO: validation progress:  80%|########  | 960/1200 [01:07<00:16, 14.13it/s]
2024-02-27 17:54:02,109 - INFO: validation progress:  85%|########5 | 1020/1200 [01:12<00:12, 14.12it/s]
2024-02-27 17:54:06,373 - INFO: validation progress:  90%|######### | 1080/1200 [01:16<00:08, 14.11it/s]
2024-02-27 17:54:10,624 - INFO: validation progress:  95%|#########5| 1140/1200 [01:20<00:04, 14.11it/s]
2024-02-27 17:54:14,806 - INFO: validation progress: 100%|##########| 1200/1200 [01:24<00:00, 14.18it/s]
2024-02-27 17:54:14,811 - INFO: validation progress: 100%|##########| 1200/1200 [01:24<00:00, 14.13it/s]
2024-02-27 17:54:14,839 - INFO: Mean validation loss: 0.55605
2024-02-27 17:54:14,841 - INFO: Validation AUC: 0.98378
2024-02-27 17:54:14,908 - INFO: Training Epoch: 10 / 15
2024-02-27 17:54:14,908 - INFO: train loss:   0%|          | 0/1350 [00:00<?, ?it/s]
2024-02-27 17:54:14,937 - INFO: Evaluation step: 1350
2024-02-27 17:55:04,346 - INFO: train loss: 0.03:   5%|4         | 67/1350 [00:49<15:46,  1.36it/s]
2024-02-27 17:55:19,574 - INFO: train loss: 0.03:   5%|4         | 67/1350 [01:04<15:46,  1.36it/s]
2024-02-27 17:55:53,948 - INFO: train loss: 0.09:  10%|9         | 134/1350 [01:39<14:59,  1.35it/s]
2024-02-27 17:56:09,576 - INFO: train loss: 0.09:  10%|9         | 134/1350 [01:54<14:59,  1.35it/s]
2024-02-27 17:56:43,757 - INFO: train loss: 0.06:  15%|#4        | 201/1350 [02:28<14:11,  1.35it/s]
2024-02-27 17:56:59,577 - INFO: train loss: 0.06:  15%|#4        | 201/1350 [02:44<14:11,  1.35it/s]
2024-02-27 17:57:33,782 - INFO: train loss: 0.06:  20%|#9        | 268/1350 [03:18<13:24,  1.35it/s]
2024-02-27 17:57:49,578 - INFO: train loss: 0.06:  20%|#9        | 268/1350 [03:34<13:24,  1.35it/s]
2024-02-27 17:58:22,901 - INFO: train loss: 0.01:  25%|##4       | 335/1350 [04:07<12:30,  1.35it/s]
2024-02-27 17:58:39,579 - INFO: train loss: 0.01:  25%|##4       | 335/1350 [04:24<12:30,  1.35it/s]
2024-02-27 17:59:13,756 - INFO: train loss: 0.08:  30%|##9       | 402/1350 [04:58<11:47,  1.34it/s]
2024-02-27 17:59:29,580 - INFO: train loss: 0.08:  30%|##9       | 402/1350 [05:14<11:47,  1.34it/s]
2024-02-27 18:00:03,506 - INFO: train loss: 0.04:  35%|###4      | 469/1350 [05:48<10:56,  1.34it/s]
2024-02-27 18:00:19,582 - INFO: train loss: 0.04:  35%|###4      | 469/1350 [06:04<10:56,  1.34it/s]
2024-02-27 18:00:52,786 - INFO: train loss: 0.13:  40%|###9      | 536/1350 [06:37<10:03,  1.35it/s]
2024-02-27 18:01:09,583 - INFO: train loss: 0.13:  40%|###9      | 536/1350 [06:54<10:03,  1.35it/s]
2024-02-27 18:01:41,699 - INFO: train loss: 0.07:  45%|####4     | 603/1350 [07:26<09:11,  1.35it/s]
2024-02-27 18:01:59,584 - INFO: train loss: 0.07:  45%|####4     | 603/1350 [07:44<09:11,  1.35it/s]
2024-02-27 18:02:31,773 - INFO: train loss: 0.20:  50%|####9     | 670/1350 [08:16<08:23,  1.35it/s]
2024-02-27 18:02:49,585 - INFO: train loss: 0.20:  50%|####9     | 670/1350 [08:34<08:23,  1.35it/s]
2024-02-27 18:03:21,265 - INFO: train loss: 0.14:  55%|#####4    | 737/1350 [09:06<07:33,  1.35it/s]
2024-02-27 18:03:31,324 - INFO: train loss: 0.14:  55%|#####4    | 737/1350 [09:16<07:33,  1.35it/s]
2024-02-27 18:04:11,276 - INFO: train loss: 0.09:  60%|#####9    | 804/1350 [09:56<06:45,  1.35it/s]
2024-02-27 18:04:21,326 - INFO: train loss: 0.09:  60%|#####9    | 804/1350 [10:06<06:45,  1.35it/s]
2024-02-27 18:05:02,239 - INFO: train loss: 0.04:  65%|######4   | 871/1350 [10:47<05:58,  1.34it/s]
2024-02-27 18:05:19,588 - INFO: train loss: 0.04:  65%|######4   | 871/1350 [11:04<05:58,  1.34it/s]
2024-02-27 18:05:51,313 - INFO: train loss: 0.01:  69%|######9   | 938/1350 [11:36<05:06,  1.35it/s]
2024-02-27 18:06:01,328 - INFO: train loss: 0.01:  69%|######9   | 938/1350 [11:46<05:06,  1.35it/s]
2024-02-27 18:06:39,474 - INFO: train loss: 0.00:  74%|#######4  | 1005/1350 [12:24<04:13,  1.36it/s]
2024-02-27 18:06:49,590 - INFO: train loss: 0.00:  74%|#######4  | 1005/1350 [12:34<04:13,  1.36it/s]
2024-02-27 18:07:29,181 - INFO: train loss: 0.01:  79%|#######9  | 1072/1350 [13:14<03:25,  1.36it/s]
2024-02-27 18:07:39,591 - INFO: train loss: 0.01:  79%|#######9  | 1072/1350 [13:24<03:25,  1.36it/s]
2024-02-27 18:08:17,728 - INFO: train loss: 0.01:  84%|########4 | 1139/1350 [14:02<02:34,  1.36it/s]
2024-02-27 18:08:29,592 - INFO: train loss: 0.01:  84%|########4 | 1139/1350 [14:14<02:34,  1.36it/s]
2024-02-27 18:09:06,737 - INFO: train loss: 0.05:  89%|########9 | 1206/1350 [14:51<01:45,  1.36it/s]
2024-02-27 18:09:19,594 - INFO: train loss: 0.05:  89%|########9 | 1206/1350 [15:04<01:45,  1.36it/s]
2024-02-27 18:09:54,878 - INFO: train loss: 0.01:  94%|#########4| 1273/1350 [15:39<00:56,  1.37it/s]
2024-02-27 18:10:09,595 - INFO: train loss: 0.01:  94%|#########4| 1273/1350 [15:54<00:56,  1.37it/s]
2024-02-27 18:10:44,764 - INFO: train loss: 0.07:  99%|#########9| 1340/1350 [16:29<00:07,  1.36it/s]
2024-02-27 18:10:51,812 - INFO: train loss: 0.17: 100%|##########| 1350/1350 [16:36<00:00,  1.35it/s]
2024-02-27 18:10:51,817 - INFO: Starting validation inference
2024-02-27 18:10:51,817 - INFO: validation progress:   0%|          | 0/1200 [00:00<?, ?it/s]
2024-02-27 18:10:56,173 - INFO: validation progress:   5%|5         | 60/1200 [00:04<01:22, 13.78it/s]
2024-02-27 18:11:00,504 - INFO: validation progress:  10%|#         | 120/1200 [00:08<01:18, 13.82it/s]
2024-02-27 18:11:04,750 - INFO: validation progress:  15%|#5        | 180/1200 [00:12<01:13, 13.96it/s]
2024-02-27 18:11:09,020 - INFO: validation progress:  20%|##        | 240/1200 [00:17<01:08, 14.00it/s]
2024-02-27 18:11:13,373 - INFO: validation progress:  25%|##5       | 300/1200 [00:21<01:04, 13.92it/s]
2024-02-27 18:11:17,664 - INFO: validation progress:  30%|###       | 360/1200 [00:25<01:00, 13.94it/s]
2024-02-27 18:11:21,982 - INFO: validation progress:  35%|###5      | 420/1200 [00:30<00:56, 13.93it/s]
2024-02-27 18:11:26,261 - INFO: validation progress:  40%|####      | 480/1200 [00:34<00:51, 13.96it/s]
2024-02-27 18:11:30,556 - INFO: validation progress:  45%|####5     | 540/1200 [00:38<00:47, 13.96it/s]
2024-02-27 18:11:34,873 - INFO: validation progress:  50%|#####     | 600/1200 [00:43<00:43, 13.94it/s]
2024-02-27 18:11:39,182 - INFO: validation progress:  55%|#####5    | 660/1200 [00:47<00:38, 13.94it/s]
2024-02-27 18:11:43,490 - INFO: validation progress:  60%|######    | 720/1200 [00:51<00:34, 13.93it/s]
2024-02-27 18:11:47,738 - INFO: validation progress:  65%|######5   | 780/1200 [00:55<00:30, 13.99it/s]
2024-02-27 18:11:52,070 - INFO: validation progress:  70%|#######   | 840/1200 [01:00<00:25, 13.95it/s]
2024-02-27 18:11:56,406 - INFO: validation progress:  75%|#######5  | 900/1200 [01:04<00:21, 13.91it/s]
2024-02-27 18:12:00,707 - INFO: validation progress:  80%|########  | 960/1200 [01:08<00:17, 13.93it/s]
2024-02-27 18:12:05,023 - INFO: validation progress:  85%|########5 | 1020/1200 [01:13<00:12, 13.92it/s]
2024-02-27 18:12:09,336 - INFO: validation progress:  90%|######### | 1080/1200 [01:17<00:08, 13.92it/s]
2024-02-27 18:12:13,653 - INFO: validation progress:  95%|#########5| 1140/1200 [01:21<00:04, 13.91it/s]
2024-02-27 18:12:17,888 - INFO: validation progress: 100%|##########| 1200/1200 [01:26<00:00, 13.99it/s]
2024-02-27 18:12:17,892 - INFO: validation progress: 100%|##########| 1200/1200 [01:26<00:00, 13.94it/s]
2024-02-27 18:12:17,921 - INFO: Mean validation loss: 0.53012
2024-02-27 18:12:17,923 - INFO: Validation AUC: 0.98495
2024-02-27 18:12:17,961 - INFO: Saving best model checkpoint: val_AUC 0.98454 -> 0.98495 to /app/output
2024-02-27 18:12:32,189 - INFO: Training Epoch: 11 / 15
2024-02-27 18:12:32,190 - INFO: train loss:   0%|          | 0/1350 [00:00<?, ?it/s]
2024-02-27 18:12:32,216 - INFO: Evaluation step: 1350
2024-02-27 18:13:22,554 - INFO: train loss: 0.00:   5%|4         | 67/1350 [00:50<16:04,  1.33it/s]
2024-02-27 18:13:39,767 - INFO: train loss: 0.00:   5%|4         | 67/1350 [01:07<16:04,  1.33it/s]
2024-02-27 18:14:12,877 - INFO: train loss: 0.09:  10%|9         | 134/1350 [01:40<15:13,  1.33it/s]
2024-02-27 18:14:29,768 - INFO: train loss: 0.09:  10%|9         | 134/1350 [01:57<15:13,  1.33it/s]
2024-02-27 18:15:01,022 - INFO: train loss: 0.00:  15%|#4        | 201/1350 [02:28<14:06,  1.36it/s]
2024-02-27 18:15:11,450 - INFO: train loss: 0.00:  15%|#4        | 201/1350 [02:39<14:06,  1.36it/s]
2024-02-27 18:15:51,010 - INFO: train loss: 0.02:  20%|#9        | 268/1350 [03:18<13:20,  1.35it/s]
2024-02-27 18:16:01,451 - INFO: train loss: 0.02:  20%|#9        | 268/1350 [03:29<13:20,  1.35it/s]
2024-02-27 18:16:41,968 - INFO: train loss: 0.00:  25%|##4       | 335/1350 [04:09<12:38,  1.34it/s]
2024-02-27 18:16:59,772 - INFO: train loss: 0.00:  25%|##4       | 335/1350 [04:27<12:38,  1.34it/s]
2024-02-27 18:17:33,353 - INFO: train loss: 0.13:  30%|##9       | 402/1350 [05:01<11:54,  1.33it/s]
2024-02-27 18:17:49,773 - INFO: train loss: 0.13:  30%|##9       | 402/1350 [05:17<11:54,  1.33it/s]
2024-02-27 18:18:22,506 - INFO: train loss: 0.05:  35%|###4      | 469/1350 [05:50<10:58,  1.34it/s]
2024-02-27 18:18:39,775 - INFO: train loss: 0.05:  35%|###4      | 469/1350 [06:07<10:58,  1.34it/s]
2024-02-27 18:19:13,150 - INFO: train loss: 0.00:  40%|###9      | 536/1350 [06:40<10:10,  1.33it/s]
2024-02-27 18:19:29,776 - INFO: train loss: 0.00:  40%|###9      | 536/1350 [06:57<10:10,  1.33it/s]
2024-02-27 18:20:01,579 - INFO: train loss: 0.06:  45%|####4     | 603/1350 [07:29<09:13,  1.35it/s]
2024-02-27 18:20:19,777 - INFO: train loss: 0.06:  45%|####4     | 603/1350 [07:47<09:13,  1.35it/s]
2024-02-27 18:20:50,464 - INFO: train loss: 0.09:  50%|####9     | 670/1350 [08:18<08:21,  1.36it/s]
2024-02-27 18:21:01,457 - INFO: train loss: 0.09:  50%|####9     | 670/1350 [08:29<08:21,  1.36it/s]
2024-02-27 18:21:39,993 - INFO: train loss: 0.02:  55%|#####4    | 737/1350 [09:07<07:32,  1.35it/s]
2024-02-27 18:21:50,136 - INFO: train loss: 0.02:  55%|#####4    | 737/1350 [09:17<07:32,  1.35it/s]
2024-02-27 18:22:30,102 - INFO: train loss: 0.06:  60%|#####9    | 804/1350 [09:57<06:44,  1.35it/s]
2024-02-27 18:22:40,137 - INFO: train loss: 0.06:  60%|#####9    | 804/1350 [10:07<06:44,  1.35it/s]
2024-02-27 18:23:18,425 - INFO: train loss: 0.03:  65%|######4   | 871/1350 [10:46<05:52,  1.36it/s]
2024-02-27 18:23:29,781 - INFO: train loss: 0.03:  65%|######4   | 871/1350 [10:57<05:52,  1.36it/s]
2024-02-27 18:24:08,584 - INFO: train loss: 0.01:  69%|######9   | 938/1350 [11:36<05:04,  1.35it/s]
2024-02-27 18:24:19,783 - INFO: train loss: 0.01:  69%|######9   | 938/1350 [11:47<05:04,  1.35it/s]
2024-02-27 18:24:56,998 - INFO: train loss: 0.02:  74%|#######4  | 1005/1350 [12:24<04:13,  1.36it/s]
2024-02-27 18:25:09,784 - INFO: train loss: 0.02:  74%|#######4  | 1005/1350 [12:37<04:13,  1.36it/s]
2024-02-27 18:25:46,476 - INFO: train loss: 0.04:  79%|#######9  | 1072/1350 [13:14<03:24,  1.36it/s]
2024-02-27 18:25:59,786 - INFO: train loss: 0.04:  79%|#######9  | 1072/1350 [13:27<03:24,  1.36it/s]
2024-02-27 18:26:35,201 - INFO: train loss: 0.04:  84%|########4 | 1139/1350 [14:03<02:34,  1.36it/s]
2024-02-27 18:26:49,787 - INFO: train loss: 0.04:  84%|########4 | 1139/1350 [14:17<02:34,  1.36it/s]
2024-02-27 18:27:26,327 - INFO: train loss: 0.01:  89%|########9 | 1206/1350 [14:54<01:46,  1.35it/s]
2024-02-27 18:27:39,788 - INFO: train loss: 0.01:  89%|########9 | 1206/1350 [15:07<01:46,  1.35it/s]
2024-02-27 18:28:16,264 - INFO: train loss: 0.04:  94%|#########4| 1273/1350 [15:44<00:57,  1.35it/s]
2024-02-27 18:28:29,789 - INFO: train loss: 0.04:  94%|#########4| 1273/1350 [15:57<00:57,  1.35it/s]
2024-02-27 18:29:05,376 - INFO: train loss: 0.04:  99%|#########9| 1340/1350 [16:33<00:07,  1.35it/s]
2024-02-27 18:29:13,213 - INFO: train loss: 0.11: 100%|##########| 1350/1350 [16:41<00:00,  1.35it/s]
2024-02-27 18:29:13,218 - INFO: Starting validation inference
2024-02-27 18:29:13,218 - INFO: validation progress:   0%|          | 0/1200 [00:00<?, ?it/s]
2024-02-27 18:29:17,580 - INFO: validation progress:   5%|5         | 60/1200 [00:04<01:22, 13.76it/s]
2024-02-27 18:29:21,919 - INFO: validation progress:  10%|#         | 120/1200 [00:08<01:18, 13.80it/s]
2024-02-27 18:29:26,163 - INFO: validation progress:  15%|#5        | 180/1200 [00:12<01:13, 13.95it/s]
2024-02-27 18:29:30,462 - INFO: validation progress:  20%|##        | 240/1200 [00:17<01:08, 13.95it/s]
2024-02-27 18:29:34,805 - INFO: validation progress:  25%|##5       | 300/1200 [00:21<01:04, 13.90it/s]
2024-02-27 18:29:39,094 - INFO: validation progress:  30%|###       | 360/1200 [00:25<01:00, 13.93it/s]
2024-02-27 18:29:43,442 - INFO: validation progress:  35%|###5      | 420/1200 [00:30<00:56, 13.89it/s]
2024-02-27 18:29:47,736 - INFO: validation progress:  40%|####      | 480/1200 [00:34<00:51, 13.92it/s]
2024-02-27 18:29:52,079 - INFO: validation progress:  45%|####5     | 540/1200 [00:38<00:47, 13.88it/s]
2024-02-27 18:29:56,392 - INFO: validation progress:  50%|#####     | 600/1200 [00:43<00:43, 13.89it/s]
2024-02-27 18:30:00,720 - INFO: validation progress:  55%|#####5    | 660/1200 [00:47<00:38, 13.88it/s]
2024-02-27 18:30:05,044 - INFO: validation progress:  60%|######    | 720/1200 [00:51<00:34, 13.88it/s]
2024-02-27 18:30:09,306 - INFO: validation progress:  65%|######5   | 780/1200 [00:56<00:30, 13.94it/s]
2024-02-27 18:30:13,631 - INFO: validation progress:  70%|#######   | 840/1200 [01:00<00:25, 13.92it/s]
2024-02-27 18:30:17,962 - INFO: validation progress:  75%|#######5  | 900/1200 [01:04<00:21, 13.90it/s]
2024-02-27 18:30:22,256 - INFO: validation progress:  80%|########  | 960/1200 [01:09<00:17, 13.92it/s]
2024-02-27 18:30:26,572 - INFO: validation progress:  85%|########5 | 1020/1200 [01:13<00:12, 13.92it/s]
2024-02-27 18:30:30,900 - INFO: validation progress:  90%|######### | 1080/1200 [01:17<00:08, 13.90it/s]
2024-02-27 18:30:35,189 - INFO: validation progress:  95%|#########5| 1140/1200 [01:21<00:04, 13.93it/s]
2024-02-27 18:30:39,433 - INFO: validation progress: 100%|##########| 1200/1200 [01:26<00:00, 13.99it/s]
2024-02-27 18:30:39,438 - INFO: validation progress: 100%|##########| 1200/1200 [01:26<00:00, 13.92it/s]
2024-02-27 18:30:39,467 - INFO: Mean validation loss: 0.50891
2024-02-27 18:30:39,469 - INFO: Validation AUC: 0.98125
2024-02-27 18:30:39,538 - INFO: Training Epoch: 12 / 15
2024-02-27 18:30:39,538 - INFO: train loss:   0%|          | 0/1350 [00:00<?, ?it/s]
2024-02-27 18:30:39,567 - INFO: Evaluation step: 1350
2024-02-27 18:31:29,498 - INFO: train loss: 0.02:   5%|4         | 67/1350 [00:49<15:56,  1.34it/s]
2024-02-27 18:31:39,794 - INFO: train loss: 0.02:   5%|4         | 67/1350 [01:00<15:56,  1.34it/s]
2024-02-27 18:32:17,877 - INFO: train loss: 0.07:  10%|9         | 134/1350 [01:38<14:49,  1.37it/s]
2024-02-27 18:32:29,795 - INFO: train loss: 0.07:  10%|9         | 134/1350 [01:50<14:49,  1.37it/s]
2024-02-27 18:33:07,379 - INFO: train loss: 0.02:  15%|#4        | 201/1350 [02:27<14:04,  1.36it/s]
2024-02-27 18:33:19,796 - INFO: train loss: 0.02:  15%|#4        | 201/1350 [02:40<14:04,  1.36it/s]
2024-02-27 18:33:56,250 - INFO: train loss: 0.01:  20%|#9        | 268/1350 [03:16<13:12,  1.36it/s]
2024-02-27 18:34:09,798 - INFO: train loss: 0.01:  20%|#9        | 268/1350 [03:30<13:12,  1.36it/s]
2024-02-27 18:34:44,070 - INFO: train loss: 0.02:  25%|##4       | 335/1350 [04:04<12:16,  1.38it/s]
2024-02-27 18:34:59,799 - INFO: train loss: 0.02:  25%|##4       | 335/1350 [04:20<12:16,  1.38it/s]
2024-02-27 18:35:35,083 - INFO: train loss: 0.02:  30%|##9       | 402/1350 [04:55<11:39,  1.36it/s]
2024-02-27 18:35:49,800 - INFO: train loss: 0.02:  30%|##9       | 402/1350 [05:10<11:39,  1.36it/s]
2024-02-27 18:36:24,126 - INFO: train loss: 0.01:  35%|###4      | 469/1350 [05:44<10:48,  1.36it/s]
2024-02-27 18:36:39,801 - INFO: train loss: 0.01:  35%|###4      | 469/1350 [06:00<10:48,  1.36it/s]
2024-02-27 18:37:13,111 - INFO: train loss: 0.05:  40%|###9      | 536/1350 [06:33<09:57,  1.36it/s]
2024-02-27 18:37:29,803 - INFO: train loss: 0.05:  40%|###9      | 536/1350 [06:50<09:57,  1.36it/s]
2024-02-27 18:38:03,364 - INFO: train loss: 0.06:  45%|####4     | 603/1350 [07:23<09:12,  1.35it/s]
2024-02-27 18:38:19,804 - INFO: train loss: 0.06:  45%|####4     | 603/1350 [07:40<09:12,  1.35it/s]
2024-02-27 18:38:53,347 - INFO: train loss: 0.02:  50%|####9     | 670/1350 [08:13<08:24,  1.35it/s]
2024-02-27 18:39:09,805 - INFO: train loss: 0.02:  50%|####9     | 670/1350 [08:30<08:24,  1.35it/s]
2024-02-27 18:39:42,089 - INFO: train loss: 0.02:  55%|#####4    | 737/1350 [09:02<07:31,  1.36it/s]
2024-02-27 18:39:59,807 - INFO: train loss: 0.02:  55%|#####4    | 737/1350 [09:20<07:31,  1.36it/s]
2024-02-27 18:40:33,112 - INFO: train loss: 0.06:  60%|#####9    | 804/1350 [09:53<06:46,  1.34it/s]
2024-02-27 18:40:49,808 - INFO: train loss: 0.06:  60%|#####9    | 804/1350 [10:10<06:46,  1.34it/s]
2024-02-27 18:41:22,702 - INFO: train loss: 0.07:  65%|######4   | 871/1350 [10:43<05:56,  1.35it/s]
2024-02-27 18:41:39,809 - INFO: train loss: 0.07:  65%|######4   | 871/1350 [11:00<05:56,  1.35it/s]
2024-02-27 18:42:12,320 - INFO: train loss: 0.03:  69%|######9   | 938/1350 [11:32<05:05,  1.35it/s]
2024-02-27 18:42:29,811 - INFO: train loss: 0.03:  69%|######9   | 938/1350 [11:50<05:05,  1.35it/s]
2024-02-27 18:43:02,020 - INFO: train loss: 0.13:  74%|#######4  | 1005/1350 [12:22<04:16,  1.35it/s]
2024-02-27 18:43:19,812 - INFO: train loss: 0.13:  74%|#######4  | 1005/1350 [12:40<04:16,  1.35it/s]
2024-02-27 18:43:51,520 - INFO: train loss: 0.03:  79%|#######9  | 1072/1350 [13:11<03:26,  1.35it/s]
2024-02-27 18:44:09,813 - INFO: train loss: 0.03:  79%|#######9  | 1072/1350 [13:30<03:26,  1.35it/s]
2024-02-27 18:44:41,522 - INFO: train loss: 0.14:  84%|########4 | 1139/1350 [14:01<02:36,  1.35it/s]
2024-02-27 18:44:59,815 - INFO: train loss: 0.14:  84%|########4 | 1139/1350 [14:20<02:36,  1.35it/s]
2024-02-27 18:45:32,127 - INFO: train loss: 0.06:  89%|########9 | 1206/1350 [14:52<01:47,  1.34it/s]
2024-02-27 18:45:49,816 - INFO: train loss: 0.06:  89%|########9 | 1206/1350 [15:10<01:47,  1.34it/s]
2024-02-27 18:46:20,702 - INFO: train loss: 0.05:  94%|#########4| 1273/1350 [15:41<00:56,  1.35it/s]
2024-02-27 18:46:31,490 - INFO: train loss: 0.05:  94%|#########4| 1273/1350 [15:51<00:56,  1.35it/s]
2024-02-27 18:47:08,501 - INFO: train loss: 0.06:  99%|#########9| 1340/1350 [16:28<00:07,  1.37it/s]
2024-02-27 18:47:16,192 - INFO: train loss: 0.11: 100%|##########| 1350/1350 [16:36<00:00,  1.35it/s]
2024-02-27 18:47:16,196 - INFO: Starting validation inference
2024-02-27 18:47:16,196 - INFO: validation progress:   0%|          | 0/1200 [00:00<?, ?it/s]
2024-02-27 18:47:20,591 - INFO: validation progress:   5%|5         | 60/1200 [00:04<01:23, 13.65it/s]
2024-02-27 18:47:24,958 - INFO: validation progress:  10%|#         | 120/1200 [00:08<01:18, 13.70it/s]
2024-02-27 18:47:29,240 - INFO: validation progress:  15%|#5        | 180/1200 [00:13<01:13, 13.84it/s]
2024-02-27 18:47:33,560 - INFO: validation progress:  20%|##        | 240/1200 [00:17<01:09, 13.86it/s]
2024-02-27 18:47:37,910 - INFO: validation progress:  25%|##5       | 300/1200 [00:21<01:05, 13.84it/s]
2024-02-27 18:47:42,216 - INFO: validation progress:  30%|###       | 360/1200 [00:26<01:00, 13.87it/s]
2024-02-27 18:47:46,563 - INFO: validation progress:  35%|###5      | 420/1200 [00:30<00:56, 13.85it/s]
2024-02-27 18:47:50,871 - INFO: validation progress:  40%|####      | 480/1200 [00:34<00:51, 13.87it/s]
2024-02-27 18:47:55,201 - INFO: validation progress:  45%|####5     | 540/1200 [00:39<00:47, 13.87it/s]
2024-02-27 18:47:59,537 - INFO: validation progress:  50%|#####     | 600/1200 [00:43<00:43, 13.86it/s]
2024-02-27 18:48:03,877 - INFO: validation progress:  55%|#####5    | 660/1200 [00:47<00:38, 13.85it/s]
2024-02-27 18:48:08,203 - INFO: validation progress:  60%|######    | 720/1200 [00:52<00:34, 13.85it/s]
2024-02-27 18:48:12,485 - INFO: validation progress:  65%|######5   | 780/1200 [00:56<00:30, 13.90it/s]
2024-02-27 18:48:16,848 - INFO: validation progress:  70%|#######   | 840/1200 [01:00<00:25, 13.86it/s]
2024-02-27 18:48:21,212 - INFO: validation progress:  75%|#######5  | 900/1200 [01:05<00:21, 13.82it/s]
2024-02-27 18:48:25,520 - INFO: validation progress:  80%|########  | 960/1200 [01:09<00:17, 13.86it/s]
2024-02-27 18:48:29,843 - INFO: validation progress:  85%|########5 | 1020/1200 [01:13<00:12, 13.86it/s]
2024-02-27 18:48:34,176 - INFO: validation progress:  90%|######### | 1080/1200 [01:17<00:08, 13.86it/s]
2024-02-27 18:48:38,477 - INFO: validation progress:  95%|#########5| 1140/1200 [01:22<00:04, 13.88it/s]
2024-02-27 18:48:42,743 - INFO: validation progress: 100%|##########| 1200/1200 [01:26<00:00, 13.94it/s]
2024-02-27 18:48:42,748 - INFO: validation progress: 100%|##########| 1200/1200 [01:26<00:00, 13.86it/s]
2024-02-27 18:48:42,777 - INFO: Mean validation loss: 0.64564
2024-02-27 18:48:42,778 - INFO: Validation AUC: 0.98292
2024-02-27 18:48:42,844 - INFO: Training Epoch: 13 / 15
2024-02-27 18:48:42,845 - INFO: train loss:   0%|          | 0/1350 [00:00<?, ?it/s]
2024-02-27 18:48:42,873 - INFO: Evaluation step: 1350
2024-02-27 18:49:33,111 - INFO: train loss: 0.01:   5%|4         | 67/1350 [00:50<16:02,  1.33it/s]
2024-02-27 18:49:49,823 - INFO: train loss: 0.01:   5%|4         | 67/1350 [01:06<16:02,  1.33it/s]
2024-02-27 18:50:22,185 - INFO: train loss: 0.07:  10%|9         | 134/1350 [01:39<14:59,  1.35it/s]
2024-02-27 18:50:39,825 - INFO: train loss: 0.07:  10%|9         | 134/1350 [01:56<14:59,  1.35it/s]
2024-02-27 18:51:12,158 - INFO: train loss: 0.01:  15%|#4        | 201/1350 [02:29<14:13,  1.35it/s]
2024-02-27 18:51:29,826 - INFO: train loss: 0.01:  15%|#4        | 201/1350 [02:46<14:13,  1.35it/s]
2024-02-27 18:52:00,799 - INFO: train loss: 0.03:  20%|#9        | 268/1350 [03:17<13:16,  1.36it/s]
2024-02-27 18:52:11,498 - INFO: train loss: 0.03:  20%|#9        | 268/1350 [03:28<13:16,  1.36it/s]
2024-02-27 18:52:49,577 - INFO: train loss: 0.01:  25%|##4       | 335/1350 [04:06<12:24,  1.36it/s]
2024-02-27 18:52:59,828 - INFO: train loss: 0.01:  25%|##4       | 335/1350 [04:16<12:24,  1.36it/s]
2024-02-27 18:53:39,799 - INFO: train loss: 0.00:  30%|##9       | 402/1350 [04:56<11:40,  1.35it/s]
2024-02-27 18:53:49,829 - INFO: train loss: 0.00:  30%|##9       | 402/1350 [05:06<11:40,  1.35it/s]
2024-02-27 18:54:29,946 - INFO: train loss: 0.01:  35%|###4      | 469/1350 [05:47<10:53,  1.35it/s]
2024-02-27 18:54:40,180 - INFO: train loss: 0.01:  35%|###4      | 469/1350 [05:57<10:53,  1.35it/s]
2024-02-27 18:55:20,555 - INFO: train loss: 0.00:  40%|###9      | 536/1350 [06:37<10:07,  1.34it/s]
2024-02-27 18:55:31,502 - INFO: train loss: 0.00:  40%|###9      | 536/1350 [06:48<10:07,  1.34it/s]
2024-02-27 18:56:10,102 - INFO: train loss: 0.06:  45%|####4     | 603/1350 [07:27<09:15,  1.34it/s]
2024-02-27 18:56:20,182 - INFO: train loss: 0.06:  45%|####4     | 603/1350 [07:37<09:15,  1.34it/s]
2024-02-27 18:56:59,482 - INFO: train loss: 0.08:  50%|####9     | 670/1350 [08:16<08:24,  1.35it/s]
2024-02-27 18:57:09,833 - INFO: train loss: 0.08:  50%|####9     | 670/1350 [08:26<08:24,  1.35it/s]
2024-02-27 18:57:48,396 - INFO: train loss: 0.02:  55%|#####4    | 737/1350 [09:05<07:32,  1.35it/s]
2024-02-27 18:57:59,835 - INFO: train loss: 0.02:  55%|#####4    | 737/1350 [09:16<07:32,  1.35it/s]
2024-02-27 18:58:38,726 - INFO: train loss: 0.10:  60%|#####9    | 804/1350 [09:55<06:45,  1.35it/s]
2024-02-27 18:58:49,836 - INFO: train loss: 0.10:  60%|#####9    | 804/1350 [10:06<06:45,  1.35it/s]
2024-02-27 18:59:28,246 - INFO: train loss: 0.08:  65%|######4   | 871/1350 [10:45<05:55,  1.35it/s]
2024-02-27 18:59:39,837 - INFO: train loss: 0.08:  65%|######4   | 871/1350 [10:56<05:55,  1.35it/s]
2024-02-27 19:00:17,268 - INFO: train loss: 0.00:  69%|######9   | 938/1350 [11:34<05:04,  1.35it/s]
2024-02-27 19:00:29,838 - INFO: train loss: 0.00:  69%|######9   | 938/1350 [11:46<05:04,  1.35it/s]
2024-02-27 19:01:06,503 - INFO: train loss: 0.03:  74%|#######4  | 1005/1350 [12:23<04:14,  1.36it/s]
2024-02-27 19:01:19,839 - INFO: train loss: 0.03:  74%|#######4  | 1005/1350 [12:36<04:14,  1.36it/s]
2024-02-27 19:01:56,986 - INFO: train loss: 0.02:  79%|#######9  | 1072/1350 [13:14<03:26,  1.35it/s]
2024-02-27 19:02:09,841 - INFO: train loss: 0.02:  79%|#######9  | 1072/1350 [13:26<03:26,  1.35it/s]
2024-02-27 19:02:48,419 - INFO: train loss: 0.04:  84%|########4 | 1139/1350 [14:05<02:38,  1.33it/s]
2024-02-27 19:02:59,842 - INFO: train loss: 0.04:  84%|########4 | 1139/1350 [14:16<02:38,  1.33it/s]
2024-02-27 19:03:37,619 - INFO: train loss: 0.03:  89%|########9 | 1206/1350 [14:54<01:47,  1.34it/s]
2024-02-27 19:03:49,843 - INFO: train loss: 0.03:  89%|########9 | 1206/1350 [15:06<01:47,  1.34it/s]
2024-02-27 19:04:27,425 - INFO: train loss: 0.02:  94%|#########4| 1273/1350 [15:44<00:57,  1.34it/s]
2024-02-27 19:04:39,844 - INFO: train loss: 0.02:  94%|#########4| 1273/1350 [15:56<00:57,  1.34it/s]
2024-02-27 19:05:17,844 - INFO: train loss: 0.01:  99%|#########9| 1340/1350 [16:34<00:07,  1.34it/s]
2024-02-27 19:05:25,189 - INFO: train loss: 0.02: 100%|##########| 1350/1350 [16:42<00:00,  1.35it/s]
2024-02-27 19:05:25,194 - INFO: Starting validation inference
2024-02-27 19:05:25,194 - INFO: validation progress:   0%|          | 0/1200 [00:00<?, ?it/s]
2024-02-27 19:05:29,633 - INFO: validation progress:   5%|5         | 60/1200 [00:04<01:24, 13.52it/s]
2024-02-27 19:05:34,046 - INFO: validation progress:  10%|#         | 120/1200 [00:08<01:19, 13.56it/s]
2024-02-27 19:05:38,366 - INFO: validation progress:  15%|#5        | 180/1200 [00:13<01:14, 13.71it/s]
2024-02-27 19:05:42,722 - INFO: validation progress:  20%|##        | 240/1200 [00:17<01:09, 13.74it/s]
2024-02-27 19:05:47,112 - INFO: validation progress:  25%|##5       | 300/1200 [00:21<01:05, 13.71it/s]
2024-02-27 19:05:51,480 - INFO: validation progress:  30%|###       | 360/1200 [00:26<01:01, 13.72it/s]
2024-02-27 19:05:55,861 - INFO: validation progress:  35%|###5      | 420/1200 [00:30<00:56, 13.71it/s]
2024-02-27 19:06:00,201 - INFO: validation progress:  40%|####      | 480/1200 [00:35<00:52, 13.75it/s]
2024-02-27 19:06:04,576 - INFO: validation progress:  45%|####5     | 540/1200 [00:39<00:48, 13.74it/s]
2024-02-27 19:06:08,957 - INFO: validation progress:  50%|#####     | 600/1200 [00:43<00:43, 13.72it/s]
2024-02-27 19:06:13,348 - INFO: validation progress:  55%|#####5    | 660/1200 [00:48<00:39, 13.71it/s]
2024-02-27 19:06:17,711 - INFO: validation progress:  60%|######    | 720/1200 [00:52<00:34, 13.72it/s]
2024-02-27 19:06:22,021 - INFO: validation progress:  65%|######5   | 780/1200 [00:56<00:30, 13.78it/s]
2024-02-27 19:06:26,404 - INFO: validation progress:  70%|#######   | 840/1200 [01:01<00:26, 13.75it/s]
2024-02-27 19:06:30,790 - INFO: validation progress:  75%|#######5  | 900/1200 [01:05<00:21, 13.73it/s]
2024-02-27 19:06:35,170 - INFO: validation progress:  80%|########  | 960/1200 [01:09<00:17, 13.72it/s]
2024-02-27 19:06:39,540 - INFO: validation progress:  85%|########5 | 1020/1200 [01:14<00:13, 13.72it/s]
2024-02-27 19:06:43,911 - INFO: validation progress:  90%|######### | 1080/1200 [01:18<00:08, 13.73it/s]
2024-02-27 19:06:48,256 - INFO: validation progress:  95%|#########5| 1140/1200 [01:23<00:04, 13.75it/s]
2024-02-27 19:06:52,554 - INFO: validation progress: 100%|##########| 1200/1200 [01:27<00:00, 13.81it/s]
2024-02-27 19:06:52,559 - INFO: validation progress: 100%|##########| 1200/1200 [01:27<00:00, 13.74it/s]
2024-02-27 19:06:52,589 - INFO: Mean validation loss: 0.62987
2024-02-27 19:06:52,590 - INFO: Validation AUC: 0.98378
2024-02-27 19:06:52,685 - INFO: Training Epoch: 14 / 15
2024-02-27 19:06:52,685 - INFO: train loss:   0%|          | 0/1350 [00:00<?, ?it/s]
2024-02-27 19:06:52,714 - INFO: Evaluation step: 1350
2024-02-27 19:07:42,678 - INFO: train loss: 0.00:   5%|4         | 67/1350 [00:49<15:57,  1.34it/s]
2024-02-27 19:07:59,850 - INFO: train loss: 0.00:   5%|4         | 67/1350 [01:07<15:57,  1.34it/s]
2024-02-27 19:08:33,409 - INFO: train loss: 0.04:  10%|9         | 134/1350 [01:40<15:15,  1.33it/s]
2024-02-27 19:08:49,851 - INFO: train loss: 0.04:  10%|9         | 134/1350 [01:57<15:15,  1.33it/s]
2024-02-27 19:09:23,499 - INFO: train loss: 0.10:  15%|#4        | 201/1350 [02:30<14:22,  1.33it/s]
2024-02-27 19:09:39,853 - INFO: train loss: 0.10:  15%|#4        | 201/1350 [02:47<14:22,  1.33it/s]
2024-02-27 19:10:12,451 - INFO: train loss: 0.06:  20%|#9        | 268/1350 [03:19<13:23,  1.35it/s]
2024-02-27 19:10:29,854 - INFO: train loss: 0.06:  20%|#9        | 268/1350 [03:37<13:23,  1.35it/s]
2024-02-27 19:11:00,794 - INFO: train loss: 0.02:  25%|##4       | 335/1350 [04:08<12:26,  1.36it/s]
2024-02-27 19:11:11,524 - INFO: train loss: 0.02:  25%|##4       | 335/1350 [04:18<12:26,  1.36it/s]
2024-02-27 19:11:50,589 - INFO: train loss: 0.00:  30%|##9       | 402/1350 [04:57<11:39,  1.36it/s]
2024-02-27 19:12:01,525 - INFO: train loss: 0.00:  30%|##9       | 402/1350 [05:08<11:39,  1.36it/s]
2024-02-27 19:12:41,244 - INFO: train loss: 0.02:  35%|###4      | 469/1350 [05:48<10:55,  1.34it/s]
2024-02-27 19:12:51,526 - INFO: train loss: 0.02:  35%|###4      | 469/1350 [05:58<10:55,  1.34it/s]
2024-02-27 19:13:29,947 - INFO: train loss: 0.00:  40%|###9      | 536/1350 [06:37<10:01,  1.35it/s]
2024-02-27 19:13:40,205 - INFO: train loss: 0.00:  40%|###9      | 536/1350 [06:47<10:01,  1.35it/s]
2024-02-27 19:14:19,407 - INFO: train loss: 0.02:  45%|####4     | 603/1350 [07:26<09:11,  1.35it/s]
2024-02-27 19:14:29,859 - INFO: train loss: 0.02:  45%|####4     | 603/1350 [07:37<09:11,  1.35it/s]
2024-02-27 19:15:10,147 - INFO: train loss: 0.14:  50%|####9     | 670/1350 [08:17<08:26,  1.34it/s]
2024-02-27 19:15:20,207 - INFO: train loss: 0.14:  50%|####9     | 670/1350 [08:27<08:26,  1.34it/s]
2024-02-27 19:15:58,135 - INFO: train loss: 0.02:  55%|#####4    | 737/1350 [09:05<07:30,  1.36it/s]
2024-02-27 19:16:09,862 - INFO: train loss: 0.02:  55%|#####4    | 737/1350 [09:17<07:30,  1.36it/s]
2024-02-27 19:16:48,384 - INFO: train loss: 0.00:  60%|#####9    | 804/1350 [09:55<06:44,  1.35it/s]
2024-02-27 19:16:59,863 - INFO: train loss: 0.00:  60%|#####9    | 804/1350 [10:07<06:44,  1.35it/s]
2024-02-27 19:17:38,519 - INFO: train loss: 0.08:  65%|######4   | 871/1350 [10:45<05:55,  1.35it/s]
2024-02-27 19:17:49,865 - INFO: train loss: 0.08:  65%|######4   | 871/1350 [10:57<05:55,  1.35it/s]
2024-02-27 19:18:27,336 - INFO: train loss: 0.04:  69%|######9   | 938/1350 [11:34<05:04,  1.35it/s]
2024-02-27 19:18:39,868 - INFO: train loss: 0.04:  69%|######9   | 938/1350 [11:47<05:04,  1.35it/s]
2024-02-27 19:19:19,118 - INFO: train loss: 0.04:  74%|#######4  | 1005/1350 [12:26<04:18,  1.34it/s]
2024-02-27 19:19:29,870 - INFO: train loss: 0.04:  74%|#######4  | 1005/1350 [12:37<04:18,  1.34it/s]
2024-02-27 19:20:09,058 - INFO: train loss: 0.06:  79%|#######9  | 1072/1350 [13:16<03:27,  1.34it/s]
2024-02-27 19:20:19,871 - INFO: train loss: 0.06:  79%|#######9  | 1072/1350 [13:27<03:27,  1.34it/s]
2024-02-27 19:20:58,304 - INFO: train loss: 0.01:  84%|########4 | 1139/1350 [14:05<02:36,  1.34it/s]
2024-02-27 19:21:09,872 - INFO: train loss: 0.01:  84%|########4 | 1139/1350 [14:17<02:36,  1.34it/s]
2024-02-27 19:21:47,297 - INFO: train loss: 0.05:  89%|########9 | 1206/1350 [14:54<01:46,  1.35it/s]
2024-02-27 19:21:59,873 - INFO: train loss: 0.05:  89%|########9 | 1206/1350 [15:07<01:46,  1.35it/s]
2024-02-27 19:22:37,073 - INFO: train loss: 0.03:  94%|#########4| 1273/1350 [15:44<00:57,  1.35it/s]
2024-02-27 19:22:49,875 - INFO: train loss: 0.03:  94%|#########4| 1273/1350 [15:57<00:57,  1.35it/s]
2024-02-27 19:23:26,790 - INFO: train loss: 0.01:  99%|#########9| 1340/1350 [16:34<00:07,  1.35it/s]
2024-02-27 19:23:34,256 - INFO: train loss: 0.01: 100%|##########| 1350/1350 [16:41<00:00,  1.35it/s]
2024-02-27 19:23:34,260 - INFO: Starting validation inference
2024-02-27 19:23:34,261 - INFO: validation progress:   0%|          | 0/1200 [00:00<?, ?it/s]
2024-02-27 19:23:38,716 - INFO: validation progress:   5%|5         | 60/1200 [00:04<01:24, 13.47it/s]
2024-02-27 19:23:43,151 - INFO: validation progress:  10%|#         | 120/1200 [00:08<01:19, 13.50it/s]
2024-02-27 19:23:47,507 - INFO: validation progress:  15%|#5        | 180/1200 [00:13<01:14, 13.63it/s]
2024-02-27 19:23:51,895 - INFO: validation progress:  20%|##        | 240/1200 [00:17<01:10, 13.64it/s]
2024-02-27 19:23:56,324 - INFO: validation progress:  25%|##5       | 300/1200 [00:22<01:06, 13.61it/s]
2024-02-27 19:24:00,710 - INFO: validation progress:  30%|###       | 360/1200 [00:26<01:01, 13.63it/s]
2024-02-27 19:24:05,134 - INFO: validation progress:  35%|###5      | 420/1200 [00:30<00:57, 13.61it/s]
2024-02-27 19:24:09,515 - INFO: validation progress:  40%|####      | 480/1200 [00:35<00:52, 13.64it/s]
2024-02-27 19:24:13,927 - INFO: validation progress:  45%|####5     | 540/1200 [00:39<00:48, 13.62it/s]
2024-02-27 19:24:18,336 - INFO: validation progress:  50%|#####     | 600/1200 [00:44<00:44, 13.62it/s]
2024-02-27 19:24:22,752 - INFO: validation progress:  55%|#####5    | 660/1200 [00:48<00:39, 13.61it/s]
2024-02-27 19:24:27,156 - INFO: validation progress:  60%|######    | 720/1200 [00:52<00:35, 13.61it/s]
2024-02-27 19:24:31,513 - INFO: validation progress:  65%|######5   | 780/1200 [00:57<00:30, 13.66it/s]
2024-02-27 19:24:35,914 - INFO: validation progress:  70%|#######   | 840/1200 [01:01<00:26, 13.65it/s]
2024-02-27 19:24:40,325 - INFO: validation progress:  75%|#######5  | 900/1200 [01:06<00:21, 13.64it/s]
2024-02-27 19:24:44,716 - INFO: validation progress:  80%|########  | 960/1200 [01:10<00:17, 13.65it/s]
2024-02-27 19:24:49,117 - INFO: validation progress:  85%|########5 | 1020/1200 [01:14<00:13, 13.64it/s]
2024-02-27 19:24:53,534 - INFO: validation progress:  90%|######### | 1080/1200 [01:19<00:08, 13.62it/s]
2024-02-27 19:24:57,899 - INFO: validation progress:  95%|#########5| 1140/1200 [01:23<00:04, 13.66it/s]
2024-02-27 19:25:02,223 - INFO: validation progress: 100%|##########| 1200/1200 [01:27<00:00, 13.73it/s]
2024-02-27 19:25:02,229 - INFO: validation progress: 100%|##########| 1200/1200 [01:27<00:00, 13.64it/s]
2024-02-27 19:25:02,257 - INFO: Mean validation loss: 0.74412
2024-02-27 19:25:02,259 - INFO: Validation AUC: 0.98246
2024-02-27 19:25:02,327 - INFO: Training Epoch: 15 / 15
2024-02-27 19:25:02,327 - INFO: train loss:   0%|          | 0/1350 [00:00<?, ?it/s]
2024-02-27 19:25:02,355 - INFO: Evaluation step: 1350
2024-02-27 19:25:52,461 - INFO: train loss: 0.00:   5%|4         | 67/1350 [00:50<16:00,  1.34it/s]
2024-02-27 19:26:09,879 - INFO: train loss: 0.00:   5%|4         | 67/1350 [01:07<16:00,  1.34it/s]
2024-02-27 19:26:42,728 - INFO: train loss: 0.08:  10%|9         | 134/1350 [01:40<15:11,  1.33it/s]
2024-02-27 19:26:59,880 - INFO: train loss: 0.08:  10%|9         | 134/1350 [01:57<15:11,  1.33it/s]
2024-02-27 19:27:31,814 - INFO: train loss: 0.00:  15%|#4        | 201/1350 [02:29<14:12,  1.35it/s]
2024-02-27 19:27:49,881 - INFO: train loss: 0.00:  15%|#4        | 201/1350 [02:47<14:12,  1.35it/s]
2024-02-27 19:28:20,956 - INFO: train loss: 0.03:  20%|#9        | 268/1350 [03:18<13:19,  1.35it/s]
2024-02-27 19:28:31,548 - INFO: train loss: 0.03:  20%|#9        | 268/1350 [03:29<13:19,  1.35it/s]
2024-02-27 19:29:10,893 - INFO: train loss: 0.23:  25%|##4       | 335/1350 [04:08<12:32,  1.35it/s]
2024-02-27 19:29:21,549 - INFO: train loss: 0.23:  25%|##4       | 335/1350 [04:19<12:32,  1.35it/s]
2024-02-27 19:30:01,674 - INFO: train loss: 0.03:  30%|##9       | 402/1350 [04:59<11:47,  1.34it/s]
2024-02-27 19:30:19,884 - INFO: train loss: 0.03:  30%|##9       | 402/1350 [05:17<11:47,  1.34it/s]
2024-02-27 19:30:49,522 - INFO: train loss: 0.14:  35%|###4      | 469/1350 [05:47<10:48,  1.36it/s]
2024-02-27 19:30:59,885 - INFO: train loss: 0.14:  35%|###4      | 469/1350 [05:57<10:48,  1.36it/s]
2024-02-27 19:31:39,371 - INFO: train loss: 0.01:  40%|###9      | 536/1350 [06:37<10:01,  1.35it/s]
2024-02-27 19:31:49,886 - INFO: train loss: 0.01:  40%|###9      | 536/1350 [06:47<10:01,  1.35it/s]
2024-02-27 19:32:29,545 - INFO: train loss: 0.01:  45%|####4     | 603/1350 [07:27<09:14,  1.35it/s]
2024-02-27 19:32:39,887 - INFO: train loss: 0.01:  45%|####4     | 603/1350 [07:37<09:14,  1.35it/s]
2024-02-27 19:33:19,844 - INFO: train loss: 0.12:  50%|####9     | 670/1350 [08:17<08:26,  1.34it/s]
2024-02-27 19:33:29,888 - INFO: train loss: 0.12:  50%|####9     | 670/1350 [08:27<08:26,  1.34it/s]
2024-02-27 19:34:08,989 - INFO: train loss: 0.03:  55%|#####4    | 737/1350 [09:06<07:34,  1.35it/s]
2024-02-27 19:34:19,889 - INFO: train loss: 0.03:  55%|#####4    | 737/1350 [09:17<07:34,  1.35it/s]
2024-02-27 19:34:58,941 - INFO: train loss: 0.01:  60%|#####9    | 804/1350 [09:56<06:45,  1.35it/s]
2024-02-27 19:35:09,891 - INFO: train loss: 0.01:  60%|#####9    | 804/1350 [10:07<06:45,  1.35it/s]
2024-02-27 19:35:49,846 - INFO: train loss: 0.01:  65%|######4   | 871/1350 [10:47<05:58,  1.34it/s]
2024-02-27 19:35:59,892 - INFO: train loss: 0.01:  65%|######4   | 871/1350 [10:57<05:58,  1.34it/s]
2024-02-27 19:36:37,800 - INFO: train loss: 0.02:  69%|######9   | 938/1350 [11:35<05:04,  1.35it/s]
2024-02-27 19:36:49,893 - INFO: train loss: 0.02:  69%|######9   | 938/1350 [11:47<05:04,  1.35it/s]
2024-02-27 19:37:27,177 - INFO: train loss: 0.02:  74%|#######4  | 1005/1350 [12:24<04:14,  1.36it/s]
2024-02-27 19:37:39,894 - INFO: train loss: 0.02:  74%|#######4  | 1005/1350 [12:37<04:14,  1.36it/s]
2024-02-27 19:38:18,232 - INFO: train loss: 0.02:  79%|#######9  | 1072/1350 [13:15<03:27,  1.34it/s]
2024-02-27 19:38:29,895 - INFO: train loss: 0.02:  79%|#######9  | 1072/1350 [13:27<03:27,  1.34it/s]
2024-02-27 19:39:07,223 - INFO: train loss: 0.13:  84%|########4 | 1139/1350 [14:04<02:36,  1.35it/s]
2024-02-27 19:39:19,896 - INFO: train loss: 0.13:  84%|########4 | 1139/1350 [14:17<02:36,  1.35it/s]
2024-02-27 19:39:58,424 - INFO: train loss: 0.00:  89%|########9 | 1206/1350 [14:56<01:47,  1.34it/s]
2024-02-27 19:40:09,897 - INFO: train loss: 0.00:  89%|########9 | 1206/1350 [15:07<01:47,  1.34it/s]
2024-02-27 19:40:48,328 - INFO: train loss: 0.01:  94%|#########4| 1273/1350 [15:46<00:57,  1.34it/s]
2024-02-27 19:40:59,899 - INFO: train loss: 0.01:  94%|#########4| 1273/1350 [15:57<00:57,  1.34it/s]
2024-02-27 19:41:38,376 - INFO: train loss: 0.08:  99%|#########9| 1340/1350 [16:36<00:07,  1.34it/s]
2024-02-27 19:41:45,770 - INFO: train loss: 0.05: 100%|##########| 1350/1350 [16:43<00:00,  1.35it/s]
2024-02-27 19:41:45,775 - INFO: Starting validation inference
2024-02-27 19:41:45,775 - INFO: validation progress:   0%|          | 0/1200 [00:00<?, ?it/s]
2024-02-27 19:41:50,265 - INFO: validation progress:   5%|5         | 60/1200 [00:04<01:25, 13.36it/s]
2024-02-27 19:41:54,699 - INFO: validation progress:  10%|#         | 120/1200 [00:08<01:20, 13.46it/s]
2024-02-27 19:41:59,047 - INFO: validation progress:  15%|#5        | 180/1200 [00:13<01:14, 13.61it/s]
2024-02-27 19:42:03,453 - INFO: validation progress:  20%|##        | 240/1200 [00:17<01:10, 13.62it/s]
2024-02-27 19:42:07,886 - INFO: validation progress:  25%|##5       | 300/1200 [00:22<01:06, 13.59it/s]
2024-02-27 19:42:12,291 - INFO: validation progress:  30%|###       | 360/1200 [00:26<01:01, 13.60it/s]
2024-02-27 19:42:16,735 - INFO: validation progress:  35%|###5      | 420/1200 [00:30<00:57, 13.57it/s]
2024-02-27 19:42:21,142 - INFO: validation progress:  40%|####      | 480/1200 [00:35<00:53, 13.58it/s]
2024-02-27 19:42:25,564 - INFO: validation progress:  45%|####5     | 540/1200 [00:39<00:48, 13.58it/s]
2024-02-27 19:42:29,989 - INFO: validation progress:  50%|#####     | 600/1200 [00:44<00:44, 13.57it/s]
2024-02-27 19:42:34,421 - INFO: validation progress:  55%|#####5    | 660/1200 [00:48<00:39, 13.56it/s]
2024-02-27 19:42:38,848 - INFO: validation progress:  60%|######    | 720/1200 [00:53<00:35, 13.56it/s]
2024-02-27 19:42:43,219 - INFO: validation progress:  65%|######5   | 780/1200 [00:57<00:30, 13.61it/s]
2024-02-27 19:42:47,653 - INFO: validation progress:  70%|#######   | 840/1200 [01:01<00:26, 13.59it/s]
2024-02-27 19:42:52,078 - INFO: validation progress:  75%|#######5  | 900/1200 [01:06<00:22, 13.58it/s]
2024-02-27 19:42:56,490 - INFO: validation progress:  80%|########  | 960/1200 [01:10<00:17, 13.58it/s]
2024-02-27 19:43:00,920 - INFO: validation progress:  85%|########5 | 1020/1200 [01:15<00:13, 13.57it/s]
2024-02-27 19:43:05,362 - INFO: validation progress:  90%|######### | 1080/1200 [01:19<00:08, 13.55it/s]
2024-02-27 19:43:09,758 - INFO: validation progress:  95%|#########5| 1140/1200 [01:23<00:04, 13.58it/s]
2024-02-27 19:43:14,101 - INFO: validation progress: 100%|##########| 1200/1200 [01:28<00:00, 13.65it/s]
2024-02-27 19:43:14,108 - INFO: validation progress: 100%|##########| 1200/1200 [01:28<00:00, 13.58it/s]
2024-02-27 19:43:14,136 - INFO: Mean validation loss: 0.74600
2024-02-27 19:43:14,138 - INFO: Validation AUC: 0.98278