[2024-11-17 07:02:34,030][__main__][INFO] - Using GPU for training. Main device : cuda:2
[2024-11-17 07:02:34,030][__main__][INFO] - Using multiple GPUs : [2, 3, 0, 1]
[2024-11-17 07:02:40,066][__main__][INFO] - PSF shape : torch.Size([1, 135, 240, 3])
[2024-11-17 07:02:40,085][__main__][INFO] - PSF min : -0.06443563848733902
[2024-11-17 07:02:40,098][__main__][INFO] - PSF max : 0.1306159645318985
[2024-11-17 07:02:40,098][__main__][INFO] - PSF dtype : torch.float32
[2024-11-17 07:02:40,132][__main__][INFO] - PSF norm : 5.099710464477539
[2024-11-17 07:02:40,132][__main__][INFO] - Reconstruction a few images with ADMM...
[2024-11-17 07:02:40,714][__main__][INFO] - Cropped shape :  (135, 240, 3)
[2024-11-17 07:02:41,936][__main__][INFO] - Train test size : 24000
[2024-11-17 07:02:41,936][__main__][INFO] - Test test size : 999
[2024-11-17 07:02:42,031][__main__][INFO] - Training model with 8161269 parameters
[2024-11-17 07:02:42,031][__main__][INFO] - -- Post-process model with 8161248 parameters
[2024-11-17 07:02:42,032][__main__][INFO] - Setup time : 0.09506869316101074 s
[2024-11-17 07:02:42,032][__main__][INFO] - PSF shape : torch.Size([1, 135, 240, 3])
[2024-11-17 07:02:42,032][__main__][INFO] - Results saved in /root/LenslessPiCam/outputs/2024-11-17/07-02-27
[2024-11-17 07:03:53,649][__main__][INFO] - Epoch 0 with learning rate [0.0001]
[2024-11-17 07:48:55,871][__main__][INFO] - loss : 0.4277596519837768
[2024-11-17 07:50:03,425][__main__][INFO] - Epoch 1 with learning rate [0.0001]
[2024-11-17 08:35:15,220][__main__][INFO] - loss : 0.3326950888807565
[2024-11-17 08:36:22,413][__main__][INFO] - Epoch 2 with learning rate [0.0001]
[2024-11-17 09:21:01,956][__main__][INFO] - loss : 0.3084536240374061
[2024-11-17 09:22:09,981][__main__][INFO] - Epoch 3 with learning rate [0.0001]
[2024-11-17 10:07:20,327][__main__][INFO] - loss : 0.2960763678848734
[2024-11-17 10:08:27,616][__main__][INFO] - Epoch 4 with learning rate [0.0001]
[2024-11-17 10:53:26,288][__main__][INFO] - loss : 0.2863141925235577
[2024-11-17 10:54:33,590][__main__][INFO] - Epoch 5 with learning rate [0.0001]
[2024-11-17 11:39:24,301][__main__][INFO] - loss : 0.27999862652023627
[2024-11-17 11:40:32,131][__main__][INFO] - Epoch 6 with learning rate [0.0001]
[2024-11-17 12:25:49,224][__main__][INFO] - loss : 0.27391758232066915
[2024-11-17 12:26:56,785][__main__][INFO] - Epoch 7 with learning rate [0.0001]
[2024-11-17 13:11:55,610][__main__][INFO] - loss : 0.2688059678326054
[2024-11-17 13:13:03,809][__main__][INFO] - Epoch 8 with learning rate [0.0001]
[2024-11-17 13:57:47,319][__main__][INFO] - loss : 0.2643992865184947
[2024-11-17 13:58:55,303][__main__][INFO] - Epoch 9 with learning rate [0.0001]
[2024-11-17 14:44:14,270][__main__][INFO] - loss : 0.26014367011437856
[2024-11-17 14:45:21,997][__main__][INFO] - Epoch 10 with learning rate [0.0001]
[2024-11-17 15:30:32,955][__main__][INFO] - loss : 0.2571629213814936
[2024-11-17 15:31:40,299][__main__][INFO] - Epoch 11 with learning rate [0.0001]
[2024-11-17 16:16:36,553][__main__][INFO] - loss : 0.25356949408600643
[2024-11-17 16:17:44,271][__main__][INFO] - Epoch 12 with learning rate [0.0001]
[2024-11-17 17:02:56,234][__main__][INFO] - loss : 0.2506390900537375
[2024-11-17 17:04:03,227][__main__][INFO] - Epoch 13 with learning rate [0.0001]
[2024-11-17 17:48:58,476][__main__][INFO] - loss : 0.24791494862983576
[2024-11-17 17:50:04,929][__main__][INFO] - Epoch 14 with learning rate [0.0001]
[2024-11-17 18:34:44,075][__main__][INFO] - loss : 0.24492231217026683
[2024-11-17 18:35:51,714][__main__][INFO] - Epoch 15 with learning rate [0.0001]
[2024-11-17 19:20:57,726][__main__][INFO] - loss : 0.24242764688531546
[2024-11-17 19:22:04,707][__main__][INFO] - Epoch 16 with learning rate [0.0001]
[2024-11-17 20:07:12,075][__main__][INFO] - loss : 0.24043737865611886
[2024-11-17 20:08:18,682][__main__][INFO] - Epoch 17 with learning rate [0.0001]
[2024-11-17 20:52:58,799][__main__][INFO] - loss : 0.23768051471809523
[2024-11-17 20:54:06,008][__main__][INFO] - Epoch 18 with learning rate [0.0001]
[2024-11-17 21:39:12,051][__main__][INFO] - loss : 0.23580654998992917
[2024-11-17 21:40:19,040][__main__][INFO] - Epoch 19 with learning rate [0.0001]
[2024-11-17 22:25:33,404][__main__][INFO] - loss : 0.2340585375018418
[2024-11-17 22:26:40,780][__main__][INFO] - Epoch 20 with learning rate [0.0001]
[2024-11-17 23:11:39,701][__main__][INFO] - loss : 0.23190293252592312
[2024-11-17 23:12:46,886][__main__][INFO] - Epoch 21 with learning rate [0.0001]
[2024-11-17 23:58:02,224][__main__][INFO] - loss : 0.229818337461601
[2024-11-17 23:59:09,289][__main__][INFO] - Epoch 22 with learning rate [0.0001]
[2024-11-18 00:44:23,338][__main__][INFO] - loss : 0.22798466260855405
[2024-11-18 00:45:30,982][__main__][INFO] - Epoch 23 with learning rate [0.0001]
[2024-11-18 01:30:07,871][__main__][INFO] - loss : 0.2263250241403775
[2024-11-18 01:31:14,673][__main__][INFO] - Epoch 24 with learning rate [0.0001]
[2024-11-18 02:16:16,467][__main__][INFO] - loss : 0.22485770126680488
[2024-11-18 02:17:24,086][__main__][INFO] - Train time [hour] : 19.244581156836617 h
[2024-11-18 02:17:24,087][__main__][INFO] - Results saved in /root/LenslessPiCam/outputs/2024-11-17/07-02-27