2023-07-11 18:22:38,572 INFO StreamThr :2302110 [internal.py:wandb_internal():86] W&B internal server running at pid: 2302110, started at: 2023-07-11 18:22:38.571275 2023-07-11 18:22:38,574 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status 2023-07-11 18:22:38,576 INFO WriterThread:2302110 [datastore.py:open_for_write():85] open: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/run-k29fqnbt.wandb 2023-07-11 18:22:38,581 DEBUG SenderThread:2302110 [sender.py:send():369] send: header 2023-07-11 18:22:38,581 DEBUG SenderThread:2302110 [sender.py:send():369] send: run 2023-07-11 18:22:38,749 ERROR SenderThread:2302110 [internal_api.py:execute():323] 500 response executing GraphQL. 2023-07-11 18:22:38,749 ERROR SenderThread:2302110 [internal_api.py:execute():324] {"errors":[{"message":"An internal error occurred. Please contact support.","path":["upsertBucket"]}],"data":{"upsertBucket":null}} 2023-07-11 18:22:40,225 INFO SenderThread:2302110 [dir_watcher.py:__init__():211] watching files in: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files 2023-07-11 18:22:40,225 INFO SenderThread:2302110 [sender.py:_start_run_threads():1100] run started: k29fqnbt with start time 1689114158.56903 2023-07-11 18:22:40,225 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:22:40,226 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:22:40,231 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: check_version 2023-07-11 18:22:40,232 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: check_version 2023-07-11 18:22:40,309 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: run_start 2023-07-11 18:22:40,342 DEBUG HandlerThread:2302110 [system_info.py:__init__():31] System info init 2023-07-11 18:22:40,342 DEBUG HandlerThread:2302110 [system_info.py:__init__():46] System info init done 2023-07-11 18:22:40,342 INFO HandlerThread:2302110 [system_monitor.py:start():181] Starting system monitor 2023-07-11 18:22:40,342 INFO SystemMonitor:2302110 [system_monitor.py:_start():145] Starting system asset monitoring threads 2023-07-11 18:22:40,343 INFO HandlerThread:2302110 [system_monitor.py:probe():201] Collecting system info 2023-07-11 18:22:40,344 INFO SystemMonitor:2302110 [interfaces.py:start():190] Started cpu monitoring 2023-07-11 18:22:40,345 INFO SystemMonitor:2302110 [interfaces.py:start():190] Started disk monitoring 2023-07-11 18:22:40,346 INFO SystemMonitor:2302110 [interfaces.py:start():190] Started gpu monitoring 2023-07-11 18:22:40,347 INFO SystemMonitor:2302110 [interfaces.py:start():190] Started memory monitoring 2023-07-11 18:22:40,350 INFO SystemMonitor:2302110 [interfaces.py:start():190] Started network monitoring 2023-07-11 18:22:40,400 DEBUG HandlerThread:2302110 [system_info.py:probe():195] Probing system 2023-07-11 18:22:40,410 DEBUG HandlerThread:2302110 [gitlib.py:_init_repo():53] git repository is invalid 2023-07-11 18:22:40,411 DEBUG HandlerThread:2302110 [system_info.py:probe():240] Probing system done 2023-07-11 18:22:40,411 DEBUG HandlerThread:2302110 [system_monitor.py:probe():210] {'os': 'Linux-5.15.0-52-generic-x86_64-with-glibc2.29', 'python': '3.8.10', 'heartbeatAt': '2023-07-11T22:22:40.400597', 'startedAt': '2023-07-11T22:22:38.535512', 'docker': None, 'cuda': None, 'args': ('--pretrained_model_name_or_path=runwayml/stable-diffusion-v1-5', '--train_data_dir=/l/vision/v5/sragas/easel_ai/thumbs_up_plain_dataset/', '--resolution=512', '--center_crop', '--random_flip', '--train_batch_size=2', '--gradient_accumulation_steps=4', '--num_train_epochs=300', '--learning_rate=1e-5', '--max_grad_norm=1', '--lr_scheduler=cosine', '--lr_warmup_steps=500', '--output_dir=/l/vision/v5/sragas/easel_ai/models_plain/', '--checkpointing_steps=500', '--report_to=wandb', '--validation_prompt=tom cruise thumbs up', '--seed=15', '--push_to_hub', '--hub_model_id=person-thumbs-up-plain-lora'), 'state': 'running', 'program': 'train_text_to_image_lora.py', 'codePath': 'train_text_to_image_lora.py', 'host': 'snorlax', 'username': 'sragas', 'executable': '/nfs/blitzle/home/data/vision5/sragas/easel_venv/bin/python3', 'cpu_count': 36, 'cpu_count_logical': 72, 'cpu_freq': {'current': 1183.020486111111, 'min': 1000.0, 'max': 3900.0}, 'cpu_freq_per_core': [{'current': 1000.0, 'min': 1000.0, 'max': 3900.0}, {'current': 1000.0, 'min': 1000.0, 'max': 3900.0}, {'current': 1000.0, 'min': 1000.0, 'max': 3900.0}, {'current': 1000.0, 'min': 1000.0, 'max': 3900.0}, {'current': 2700.0, 'min': 1000.0, 'max': 3900.0}, {'current': 1000.0, 'min': 1000.0, 'max': 3900.0}, {'current': 1000.0, 'min': 1000.0, 'max': 3900.0}, {'current': 1100.0, 'min': 1000.0, 'max': 3900.0}, {'current': 1000.0, 'min': 1000.0, 'max': 3900.0}, {'current': 1000.0, 'min': 1000.0, 'max': 3900.0}, {'current': 1000.0, 'min': 1000.0, 'max': 3900.0}, {'current': 1000.0, 'min': 1000.0, 'max': 3900.0}, {'current': 1000.0, 'min': 1000.0, 'max': 3900.0}, {'current': 1000.0, 'min': 1000.0, 'max': 3900.0}, {'current': 1000.0, 'min': 1000.0, 'max': 3900.0}, {'current': 1000.0, 'min': 1000.0, 'max': 3900.0}, {'current': 1000.0, 'min': 1000.0, 'max': 3900.0}, {'current': 1000.0, 'min': 1000.0, 'max': 3900.0}, {'current': 3490.996, 'min': 1000.0, 'max': 3900.0}, {'current': 1000.0, 'min': 1000.0, 'max': 3900.0}, {'current': 1000.0, 'min': 1000.0, 'max': 3900.0}, {'current': 2000.0, 'min': 1000.0, 'max': 3900.0}, {'current': 1100.0, 'min': 1000.0, 'max': 3900.0}, {'current': 1000.0, 'min': 1000.0, 'max': 3900.0}, {'current': 1100.0, 'min': 1000.0, 'max': 3900.0}, {'current': 2500.0, 'min': 1000.0, 'max': 3900.0}, {'current': 1000.0, 'min': 1000.0, 'max': 3900.0}, {'current': 1000.0, 'min': 1000.0, 'max': 3900.0}, {'current': 1000.0, 'min': 1000.0, 'max': 3900.0}, {'current': 1000.0, 'min': 1000.0, 'max': 3900.0}, {'current': 1000.0, 'min': 1000.0, 'max': 3900.0}, {'current': 1000.0, 'min': 1000.0, 'max': 3900.0}, {'current': 1000.0, 'min': 1000.0, 'max': 3900.0}, {'current': 2043.382, 'min': 1000.0, 'max': 3900.0}, {'current': 1000.0, 'min': 1000.0, 'max': 3900.0}, {'current': 1000.0, 'min': 1000.0, 'max': 3900.0}, {'current': 1000.0, 'min': 1000.0, 'max': 3900.0}, {'current': 1000.0, 'min': 1000.0, 'max': 3900.0}, {'current': 1000.0, 'min': 1000.0, 'max': 3900.0}, {'current': 1000.0, 'min': 1000.0, 'max': 3900.0}, {'current': 1000.0, 'min': 1000.0, 'max': 3900.0}, {'current': 1000.0, 'min': 1000.0, 'max': 3900.0}, {'current': 1000.0, 'min': 1000.0, 'max': 3900.0}, {'current': 1000.0, 'min': 1000.0, 'max': 3900.0}, {'current': 1000.0, 'min': 1000.0, 'max': 3900.0}, {'current': 1100.0, 'min': 1000.0, 'max': 3900.0}, {'current': 1000.0, 'min': 1000.0, 'max': 3900.0}, {'current': 1100.0, 'min': 1000.0, 'max': 3900.0}, {'current': 1721.668, 'min': 1000.0, 'max': 3900.0}, {'current': 1000.0, 'min': 1000.0, 'max': 3900.0}, {'current': 1000.0, 'min': 1000.0, 'max': 3900.0}, {'current': 1000.0, 'min': 1000.0, 'max': 3900.0}, {'current': 1000.0, 'min': 1000.0, 'max': 3900.0}, {'current': 3900.0, 'min': 1000.0, 'max': 3900.0}, {'current': 1000.0, 'min': 1000.0, 'max': 3900.0}, {'current': 1000.0, 'min': 1000.0, 'max': 3900.0}, {'current': 1000.0, 'min': 1000.0, 'max': 3900.0}, {'current': 1000.0, 'min': 1000.0, 'max': 3900.0}, {'current': 1000.0, 'min': 1000.0, 'max': 3900.0}, {'current': 1000.0, 'min': 1000.0, 'max': 3900.0}, {'current': 1000.0, 'min': 1000.0, 'max': 3900.0}, {'current': 1000.0, 'min': 1000.0, 'max': 3900.0}, {'current': 1000.0, 'min': 1000.0, 'max': 3900.0}, {'current': 1000.0, 'min': 1000.0, 'max': 3900.0}, {'current': 1000.0, 'min': 1000.0, 'max': 3900.0}, {'current': 1000.0, 'min': 1000.0, 'max': 3900.0}, {'current': 1000.0, 'min': 1000.0, 'max': 3900.0}, {'current': 1000.0, 'min': 1000.0, 'max': 3900.0}, {'current': 2400.0, 'min': 1000.0, 'max': 3900.0}, {'current': 1000.0, 'min': 1000.0, 'max': 3900.0}, {'current': 1000.0, 'min': 1000.0, 'max': 3900.0}, {'current': 3700.0, 'min': 1000.0, 'max': 3900.0}], 'disk': {'total': 490.5871887207031, 'used': 102.94281387329102}, 'gpu': 'Quadro RTX 6000', 'gpu_count': 4, 'gpu_devices': [{'name': 'Quadro RTX 6000', 'memory_total': 25769803776}, {'name': 'Quadro RTX 6000', 'memory_total': 25769803776}, {'name': 'Quadro RTX 6000', 'memory_total': 25769803776}, {'name': 'Quadro RTX 6000', 'memory_total': 25769803776}], 'memory': {'total': 502.5611343383789}} 2023-07-11 18:22:40,411 INFO HandlerThread:2302110 [system_monitor.py:probe():211] Finished collecting system info 2023-07-11 18:22:40,411 INFO HandlerThread:2302110 [system_monitor.py:probe():214] Publishing system info 2023-07-11 18:22:40,411 DEBUG HandlerThread:2302110 [system_info.py:_save_pip():51] Saving list of pip packages installed into the current environment 2023-07-11 18:22:40,412 DEBUG HandlerThread:2302110 [system_info.py:_save_pip():67] Saving pip packages done 2023-07-11 18:22:40,415 INFO HandlerThread:2302110 [system_monitor.py:probe():216] Finished publishing system info 2023-07-11 18:22:40,424 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:22:40,424 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-metadata.json with policy now 2023-07-11 18:22:40,438 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:22:40,439 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:22:40,517 DEBUG SenderThread:2302110 [sender.py:send():369] send: telemetry 2023-07-11 18:22:40,798 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/alyew3jz-wandb-metadata.json 2023-07-11 18:22:41,230 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-metadata.json 2023-07-11 18:22:41,231 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:22:41,231 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/requirements.txt 2023-07-11 18:22:41,231 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:22:43,229 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:22:44,238 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:22:45,232 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:22:47,234 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:22:48,010 DEBUG SenderThread:2302110 [sender.py:send():369] send: telemetry 2023-07-11 18:22:48,014 DEBUG SenderThread:2302110 [sender.py:send():369] send: config 2023-07-11 18:22:49,236 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:22:49,750 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:22:51,239 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:22:52,379 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:22:53,241 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:22:54,775 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:22:55,244 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:22:55,438 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:22:55,439 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:22:55,686 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:22:55,689 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:22:55,689 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:22:55,691 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:22:56,245 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:22:57,247 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:22:59,080 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:22:59,082 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:22:59,083 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:22:59,084 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:22:59,248 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:22:59,249 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:22:59,880 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:23:01,251 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:23:02,672 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:23:02,674 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:23:02,674 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:23:02,676 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:23:03,254 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:23:03,254 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:23:05,256 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:23:05,750 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:23:06,453 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:23:06,455 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:23:06,456 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:23:06,457 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:23:07,258 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:23:07,259 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:23:09,261 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:23:10,187 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:23:10,189 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:23:10,189 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:23:10,191 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:23:10,262 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:23:10,439 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:23:10,440 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:23:11,069 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:23:11,264 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:23:12,265 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/config.yaml 2023-07-11 18:23:13,266 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:23:13,546 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:23:13,548 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:23:13,548 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:23:13,550 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:23:13,831 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:23:13,833 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:23:13,834 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:23:13,835 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:23:14,267 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:23:15,272 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:23:16,323 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:23:19,284 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:23:21,542 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:23:25,439 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:23:25,440 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:23:27,511 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:23:32,513 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:23:37,513 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:23:40,350 DEBUG SystemMonitor:2302110 [system_monitor.py:_start():159] Starting system metrics aggregation loop 2023-07-11 18:23:40,355 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 18:23:40,471 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:23:40,472 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:23:43,510 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:23:44,593 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:23:44,594 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_8_4f431957962c49867234.png with policy now 2023-07-11 18:23:44,637 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:23:44,637 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_8_43f961633a45946caf61.png with policy now 2023-07-11 18:23:44,638 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:23:44,638 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_8_018b5a46c57a4e62ee91.png with policy now 2023-07-11 18:23:44,657 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:23:44,657 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_8_b1af5834223b938d847b.png with policy now 2023-07-11 18:23:44,700 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:23:44,702 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:23:44,703 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:23:44,708 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:23:44,961 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/b0rpvnla-media/images/validation_8_4f431957962c49867234.png 2023-07-11 18:23:45,075 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/xsgv00jf-media/images/validation_8_018b5a46c57a4e62ee91.png 2023-07-11 18:23:45,083 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/v3owrnbm-media/images/validation_8_b1af5834223b938d847b.png 2023-07-11 18:23:45,159 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/zan9ncu2-media/images/validation_8_43f961633a45946caf61.png 2023-07-11 18:23:45,327 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:23:45,328 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_8_018b5a46c57a4e62ee91.png 2023-07-11 18:23:45,329 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_8_4f431957962c49867234.png 2023-07-11 18:23:45,329 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_8_b1af5834223b938d847b.png 2023-07-11 18:23:45,330 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_8_43f961633a45946caf61.png 2023-07-11 18:23:45,330 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 18:23:45,331 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media 2023-07-11 18:23:45,690 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/2wz0pjtr-media/images/validation_8_018b5a46c57a4e62ee91.png 2023-07-11 18:23:45,713 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/cfujfuqr-media/images/validation_8_b1af5834223b938d847b.png 2023-07-11 18:23:45,717 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/9ahowvtu-media/images/validation_8_4f431957962c49867234.png 2023-07-11 18:23:45,723 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/v5virbff-media/images/validation_8_43f961633a45946caf61.png 2023-07-11 18:23:47,331 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:23:48,650 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:23:49,201 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:23:49,335 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:23:51,338 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:23:53,130 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:23:53,132 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:23:53,132 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:23:53,134 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:23:53,341 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:23:53,342 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:23:54,140 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:23:55,345 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:23:55,469 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:23:55,470 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:23:55,534 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:23:55,536 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:23:55,537 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:23:55,538 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:23:56,346 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:23:57,348 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:23:59,244 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:23:59,246 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:23:59,246 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:23:59,247 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:23:59,248 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:23:59,351 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:23:59,352 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:24:01,354 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:24:02,512 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:24:02,514 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:24:02,514 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:24:02,516 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:24:03,358 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:24:03,358 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:24:05,024 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:24:05,361 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:24:06,377 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:24:06,378 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:24:06,379 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:24:06,381 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:24:07,365 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:24:07,365 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:24:09,368 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:24:09,599 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:24:09,601 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:24:09,601 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:24:09,603 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:24:09,885 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:24:09,887 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:24:09,887 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:24:09,889 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:24:10,358 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 18:24:10,360 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:24:10,369 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:24:10,470 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:24:10,470 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:24:11,371 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:24:13,375 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:24:15,619 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:24:20,620 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:24:22,727 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:24:22,727 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_16_0cf3546cb22517953a42.png with policy now 2023-07-11 18:24:22,737 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:24:22,738 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_16_ba6f49bf4d359df02f2b.png with policy now 2023-07-11 18:24:22,748 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:24:22,748 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_16_53e698e35096606e0dae.png with policy now 2023-07-11 18:24:22,760 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:24:22,760 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_16_e935b3df8a409038c0e2.png with policy now 2023-07-11 18:24:22,797 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:24:22,799 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:24:22,799 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:24:22,802 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:24:23,051 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/sankrbib-media/images/validation_16_0cf3546cb22517953a42.png 2023-07-11 18:24:23,063 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/4lj6hq6v-media/images/validation_16_53e698e35096606e0dae.png 2023-07-11 18:24:23,066 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ugk2e55b-media/images/validation_16_ba6f49bf4d359df02f2b.png 2023-07-11 18:24:23,075 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/xaajhf6l-media/images/validation_16_e935b3df8a409038c0e2.png 2023-07-11 18:24:23,388 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:24:23,389 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_16_ba6f49bf4d359df02f2b.png 2023-07-11 18:24:23,389 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_16_e935b3df8a409038c0e2.png 2023-07-11 18:24:23,390 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_16_0cf3546cb22517953a42.png 2023-07-11 18:24:23,391 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_16_53e698e35096606e0dae.png 2023-07-11 18:24:23,391 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 18:24:23,731 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/jvl75i0n-media/images/validation_16_ba6f49bf4d359df02f2b.png 2023-07-11 18:24:23,748 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ftrfo1x5-media/images/validation_16_e935b3df8a409038c0e2.png 2023-07-11 18:24:23,751 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ib5gis1f-media/images/validation_16_53e698e35096606e0dae.png 2023-07-11 18:24:23,768 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/nq5ayjdt-media/images/validation_16_0cf3546cb22517953a42.png 2023-07-11 18:24:25,392 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:24:25,572 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:24:25,573 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:24:25,673 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:24:26,857 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:24:27,395 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:24:29,399 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:24:30,722 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:24:30,724 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:24:30,724 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:24:30,725 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:24:30,727 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:24:31,402 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:24:31,403 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:24:33,406 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:24:33,938 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:24:33,939 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:24:33,940 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:24:33,941 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:24:34,408 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:24:35,410 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:24:36,119 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:24:36,955 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:24:36,957 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:24:36,957 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:24:36,958 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:24:37,413 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:24:37,414 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:24:39,417 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:24:39,911 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:24:39,913 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:24:39,913 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:24:39,915 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:24:40,361 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 18:24:40,418 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:24:40,572 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:24:40,574 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:24:41,420 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:24:41,424 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:24:43,341 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:24:43,343 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:24:43,343 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:24:43,345 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:24:43,424 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:24:43,424 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:24:45,428 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:24:47,031 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:24:47,033 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:24:47,033 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:24:47,033 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:24:47,035 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:24:47,311 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:24:47,313 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:24:47,314 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:24:47,316 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:24:47,432 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:24:47,432 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:24:49,435 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:24:52,994 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:24:55,586 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:24:55,586 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:24:58,642 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:25:00,050 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:25:00,050 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_24_5a5b54a523e7e8a202d4.png with policy now 2023-07-11 18:25:00,102 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:25:00,102 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_24_68fac8d4cdcb87c0351c.png with policy now 2023-07-11 18:25:00,114 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:25:00,115 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_24_e0d04b7701f5a5d35a98.png with policy now 2023-07-11 18:25:00,123 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:25:00,124 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_24_0684dbe8f6eb82a2da7f.png with policy now 2023-07-11 18:25:00,158 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:25:00,160 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:25:00,160 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:25:00,162 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:25:00,420 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/meqlmbyd-media/images/validation_24_5a5b54a523e7e8a202d4.png 2023-07-11 18:25:00,451 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:25:00,452 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_24_5a5b54a523e7e8a202d4.png 2023-07-11 18:25:00,453 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_24_e0d04b7701f5a5d35a98.png 2023-07-11 18:25:00,453 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_24_0684dbe8f6eb82a2da7f.png 2023-07-11 18:25:00,454 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_24_68fac8d4cdcb87c0351c.png 2023-07-11 18:25:00,454 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 18:25:00,468 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/7tcc814f-media/images/validation_24_0684dbe8f6eb82a2da7f.png 2023-07-11 18:25:00,471 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/blt4053o-media/images/validation_24_68fac8d4cdcb87c0351c.png 2023-07-11 18:25:00,483 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/2im2idl0-media/images/validation_24_e0d04b7701f5a5d35a98.png 2023-07-11 18:25:00,832 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/nnk0bjzp-media/images/validation_24_0684dbe8f6eb82a2da7f.png 2023-07-11 18:25:00,839 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/sa0ywc4q-media/images/validation_24_68fac8d4cdcb87c0351c.png 2023-07-11 18:25:00,841 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/cdwdbw5p-media/images/validation_24_5a5b54a523e7e8a202d4.png 2023-07-11 18:25:01,183 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/cbb4l1u0-media/images/validation_24_e0d04b7701f5a5d35a98.png 2023-07-11 18:25:03,457 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:25:04,114 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:25:04,205 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:25:05,461 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:25:07,465 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:25:07,752 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:25:07,754 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:25:07,755 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:25:07,756 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:25:08,466 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:25:09,206 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:25:09,469 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:25:10,364 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 18:25:10,587 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:25:10,588 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:25:11,340 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:25:11,341 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:25:11,342 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:25:11,343 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:25:11,473 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:25:11,474 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:25:13,477 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:25:14,394 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:25:15,384 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:25:15,386 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:25:15,387 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:25:15,388 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:25:15,481 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:25:15,481 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:25:17,485 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:25:18,198 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:25:18,200 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:25:18,200 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:25:18,202 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:25:18,486 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:25:19,488 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:25:20,080 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:25:20,968 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:25:20,970 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:25:20,970 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:25:20,971 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:25:21,492 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:25:21,493 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:25:23,496 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:25:24,049 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:25:24,051 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:25:24,051 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:25:24,053 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:25:24,333 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:25:24,335 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:25:24,335 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:25:24,336 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:25:24,499 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:25:25,501 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:25:25,588 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:25:25,588 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:25:25,589 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:25:27,505 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:25:30,704 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:25:35,706 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:25:36,966 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:25:36,966 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_32_caeb7e11117ca44da4c8.png with policy now 2023-07-11 18:25:37,009 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:25:37,009 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_32_7ece7c36041231bd9830.png with policy now 2023-07-11 18:25:37,010 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:25:37,010 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_32_603e1acd8de551001b1b.png with policy now 2023-07-11 18:25:37,011 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:25:37,011 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_32_d076df0f0d397a793093.png with policy now 2023-07-11 18:25:37,052 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:25:37,053 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:25:37,053 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:25:37,055 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:25:37,298 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/9qef7ofr-media/images/validation_32_caeb7e11117ca44da4c8.png 2023-07-11 18:25:37,325 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/6l6lvhgg-media/images/validation_32_d076df0f0d397a793093.png 2023-07-11 18:25:37,331 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/xqpxktnx-media/images/validation_32_7ece7c36041231bd9830.png 2023-07-11 18:25:37,339 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/4nkzplpd-media/images/validation_32_603e1acd8de551001b1b.png 2023-07-11 18:25:37,520 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:25:37,521 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_32_caeb7e11117ca44da4c8.png 2023-07-11 18:25:37,521 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_32_603e1acd8de551001b1b.png 2023-07-11 18:25:37,522 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_32_d076df0f0d397a793093.png 2023-07-11 18:25:37,522 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_32_7ece7c36041231bd9830.png 2023-07-11 18:25:37,523 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 18:25:37,850 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/8tj47r17-media/images/validation_32_603e1acd8de551001b1b.png 2023-07-11 18:25:37,881 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/gxnx3vp5-media/images/validation_32_d076df0f0d397a793093.png 2023-07-11 18:25:37,905 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/plzox2j7-media/images/validation_32_7ece7c36041231bd9830.png 2023-07-11 18:25:38,063 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ceyfq4l5-media/images/validation_32_caeb7e11117ca44da4c8.png 2023-07-11 18:25:39,524 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:25:40,367 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 18:25:40,587 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:25:40,588 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:25:41,080 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:25:41,120 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:25:41,528 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:25:43,532 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:25:44,232 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:25:44,234 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:25:44,234 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:25:44,236 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:25:44,534 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:25:45,536 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:25:46,800 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:25:46,802 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:25:46,802 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:25:46,803 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:25:46,805 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:25:47,541 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:25:47,542 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:25:49,545 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:25:50,953 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:25:50,955 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:25:50,955 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:25:50,957 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:25:51,549 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:25:51,550 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:25:52,505 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:25:53,553 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:25:53,819 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:25:53,821 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:25:53,821 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:25:53,823 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:25:54,555 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:25:55,557 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:25:55,588 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:25:55,589 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:25:56,938 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:25:56,940 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:25:56,941 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:25:56,942 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:25:57,561 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:25:57,562 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:25:57,773 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:25:59,566 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:26:01,258 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:26:01,260 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:26:01,261 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:26:01,262 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:26:01,539 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:26:01,541 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:26:01,542 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:26:01,543 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:26:01,575 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:26:01,576 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:26:03,183 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:26:03,582 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:26:08,185 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:26:10,370 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 18:26:10,588 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:26:10,589 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:26:13,683 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:26:14,247 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:26:14,247 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_40_18d5333919a6d64e6d60.png with policy now 2023-07-11 18:26:14,289 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:26:14,289 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_40_86b79bd8a5f65f5b321b.png with policy now 2023-07-11 18:26:14,290 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:26:14,290 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_40_549de1ccb1807d11a4b8.png with policy now 2023-07-11 18:26:14,310 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:26:14,310 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_40_144fef68c2f0b44f2e84.png with policy now 2023-07-11 18:26:14,350 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:26:14,352 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:26:14,352 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:26:14,353 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:26:14,571 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/3rwbexjz-media/images/validation_40_18d5333919a6d64e6d60.png 2023-07-11 18:26:14,601 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:26:14,602 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_40_549de1ccb1807d11a4b8.png 2023-07-11 18:26:14,602 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_40_18d5333919a6d64e6d60.png 2023-07-11 18:26:14,603 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_40_144fef68c2f0b44f2e84.png 2023-07-11 18:26:14,604 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_40_86b79bd8a5f65f5b321b.png 2023-07-11 18:26:14,605 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 18:26:14,608 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/xt4vd8jx-media/images/validation_40_86b79bd8a5f65f5b321b.png 2023-07-11 18:26:14,627 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/wg7eto75-media/images/validation_40_549de1ccb1807d11a4b8.png 2023-07-11 18:26:14,645 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/eqsx3ech-media/images/validation_40_144fef68c2f0b44f2e84.png 2023-07-11 18:26:14,959 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/3syncce5-media/images/validation_40_18d5333919a6d64e6d60.png 2023-07-11 18:26:14,972 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/2b5blmts-media/images/validation_40_549de1ccb1807d11a4b8.png 2023-07-11 18:26:15,006 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/nlj63rqj-media/images/validation_40_144fef68c2f0b44f2e84.png 2023-07-11 18:26:15,215 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/q5zwa3d1-media/images/validation_40_86b79bd8a5f65f5b321b.png 2023-07-11 18:26:17,607 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:26:18,243 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:26:19,245 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:26:19,611 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:26:21,615 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:26:21,803 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:26:21,805 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:26:21,805 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:26:21,807 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:26:22,619 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:26:23,623 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:26:24,354 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:26:25,335 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:26:25,337 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:26:25,337 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:26:25,339 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:26:25,630 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:26:25,630 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:26:25,650 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:26:25,651 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:26:27,636 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:26:27,947 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:26:27,949 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:26:27,949 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:26:27,951 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:26:28,638 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:26:29,366 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:26:29,640 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:26:31,339 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:26:31,341 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:26:31,341 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:26:31,343 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:26:31,644 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:26:31,644 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:26:33,649 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:26:34,271 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:26:34,273 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:26:34,274 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:26:34,275 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:26:34,650 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:26:35,276 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:26:35,653 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:26:37,657 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:26:37,675 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:26:37,677 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:26:37,677 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:26:37,679 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:26:37,958 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:26:37,981 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:26:37,982 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:26:37,983 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:26:38,660 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:26:39,662 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:26:40,373 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 18:26:40,375 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:26:40,650 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:26:40,651 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:26:45,719 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:26:50,570 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:26:50,570 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_48_9c214adc0d7f41183109.png with policy now 2023-07-11 18:26:50,581 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:26:50,581 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_48_d2e4462cea1aed1f2937.png with policy now 2023-07-11 18:26:50,604 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:26:50,604 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_48_a907fc518966b66deb3c.png with policy now 2023-07-11 18:26:50,614 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:26:50,614 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_48_0262af6aae483d433db7.png with policy now 2023-07-11 18:26:50,678 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:26:50,680 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:26:50,681 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:26:50,682 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:26:50,685 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_48_9c214adc0d7f41183109.png 2023-07-11 18:26:50,685 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_48_0262af6aae483d433db7.png 2023-07-11 18:26:50,686 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_48_a907fc518966b66deb3c.png 2023-07-11 18:26:50,686 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_48_d2e4462cea1aed1f2937.png 2023-07-11 18:26:50,687 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 18:26:50,888 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/d4ftggv8-media/images/validation_48_d2e4462cea1aed1f2937.png 2023-07-11 18:26:50,901 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ymw7fov9-media/images/validation_48_a907fc518966b66deb3c.png 2023-07-11 18:26:50,906 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/b1d67w5k-media/images/validation_48_9c214adc0d7f41183109.png 2023-07-11 18:26:51,020 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/9qhfg0xb-media/images/validation_48_0262af6aae483d433db7.png 2023-07-11 18:26:51,230 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/k6npmz08-media/images/validation_48_a907fc518966b66deb3c.png 2023-07-11 18:26:51,405 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/9z1hozj3-media/images/validation_48_d2e4462cea1aed1f2937.png 2023-07-11 18:26:51,607 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/0mk7kx2s-media/images/validation_48_9c214adc0d7f41183109.png 2023-07-11 18:26:51,684 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:26:51,687 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:26:51,727 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/9jhwx7r7-media/images/validation_48_0262af6aae483d433db7.png 2023-07-11 18:26:53,692 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:26:54,225 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:26:55,651 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:26:55,652 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:26:55,697 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:26:57,482 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:26:57,488 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:26:57,489 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:26:57,490 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:26:57,491 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:26:57,702 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:26:57,702 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:26:59,707 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:27:01,320 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:27:01,322 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:27:01,322 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:27:01,324 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:27:01,711 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:27:01,712 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:27:02,608 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:27:03,716 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:27:04,346 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:27:04,347 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:27:04,347 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:27:04,349 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:27:04,717 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:27:05,720 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:27:07,450 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:27:07,452 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:27:07,452 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:27:07,454 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:27:07,724 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:27:07,725 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:27:08,114 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:27:09,728 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:27:10,376 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 18:27:10,825 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:27:10,826 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:27:10,952 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:27:10,954 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:27:10,955 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:27:10,956 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:27:11,733 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:27:11,734 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:27:13,648 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:27:13,738 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:27:14,440 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:27:14,442 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:27:14,443 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:27:14,444 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:27:14,725 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:27:14,727 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:27:14,727 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:27:14,729 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:27:14,743 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:27:15,746 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:27:17,750 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:27:19,351 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:27:24,352 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:27:25,862 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:27:25,862 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:27:27,390 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:27:27,390 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_56_664030382ee576a5c9b2.png with policy now 2023-07-11 18:27:27,405 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:27:27,405 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_56_1536eefdcfae05b945b9.png with policy now 2023-07-11 18:27:27,423 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:27:27,423 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_56_12bc42303d2e7628fd3f.png with policy now 2023-07-11 18:27:27,431 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:27:27,431 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_56_ef4ee0a1a460a56f4751.png with policy now 2023-07-11 18:27:27,470 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:27:27,472 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:27:27,472 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:27:27,473 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:27:27,741 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/7o31ocw7-media/images/validation_56_1536eefdcfae05b945b9.png 2023-07-11 18:27:27,746 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/pr5td6te-media/images/validation_56_664030382ee576a5c9b2.png 2023-07-11 18:27:27,768 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:27:27,769 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_56_664030382ee576a5c9b2.png 2023-07-11 18:27:27,769 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_56_12bc42303d2e7628fd3f.png 2023-07-11 18:27:27,771 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/91523anm-media/images/validation_56_12bc42303d2e7628fd3f.png 2023-07-11 18:27:27,771 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_56_1536eefdcfae05b945b9.png 2023-07-11 18:27:27,772 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_56_ef4ee0a1a460a56f4751.png 2023-07-11 18:27:27,773 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 18:27:27,795 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/smzh12ps-media/images/validation_56_ef4ee0a1a460a56f4751.png 2023-07-11 18:27:28,148 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/dldthc7l-media/images/validation_56_664030382ee576a5c9b2.png 2023-07-11 18:27:28,192 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/coqz8hqx-media/images/validation_56_1536eefdcfae05b945b9.png 2023-07-11 18:27:28,348 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/0cwm101u-media/images/validation_56_ef4ee0a1a460a56f4751.png 2023-07-11 18:27:28,349 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/s82rltkh-media/images/validation_56_12bc42303d2e7628fd3f.png 2023-07-11 18:27:29,395 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:27:29,773 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:27:31,172 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:27:31,777 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:27:33,782 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:27:34,611 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:27:34,613 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:27:34,613 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:27:34,614 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:27:34,616 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:27:34,784 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:27:35,787 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:27:37,775 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:27:37,777 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:27:37,777 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:27:37,779 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:27:37,792 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:27:37,793 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:27:39,773 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:27:39,796 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:27:40,379 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 18:27:40,641 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:27:40,642 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:27:40,643 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:27:40,644 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:27:40,798 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:27:40,861 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:27:40,862 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:27:41,801 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:27:43,806 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:27:44,313 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:27:44,315 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:27:44,315 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:27:44,317 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:27:44,808 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:27:45,318 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:27:45,810 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:27:47,815 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:27:47,870 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:27:47,873 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:27:47,874 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:27:47,875 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:27:48,817 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:27:49,819 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:27:50,829 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:27:50,831 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:27:50,831 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:27:50,832 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:27:50,833 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:27:51,112 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:27:51,114 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:27:51,114 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:27:51,116 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:27:51,824 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:27:51,825 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:27:53,829 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:27:55,862 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:27:55,863 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:27:55,906 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:28:00,907 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:28:03,706 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:28:03,707 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_64_3e87611fd915ec96e580.png with policy now 2023-07-11 18:28:03,721 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:28:03,721 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_64_2537a2652dff56481399.png with policy now 2023-07-11 18:28:03,734 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:28:03,734 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_64_9bd0267d729986b95a5d.png with policy now 2023-07-11 18:28:03,743 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:28:03,743 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_64_75bb28ff057271dedef7.png with policy now 2023-07-11 18:28:03,782 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:28:03,784 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:28:03,788 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:28:03,791 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:28:03,848 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:28:03,849 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_64_2537a2652dff56481399.png 2023-07-11 18:28:03,849 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_64_3e87611fd915ec96e580.png 2023-07-11 18:28:03,850 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_64_9bd0267d729986b95a5d.png 2023-07-11 18:28:03,850 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_64_75bb28ff057271dedef7.png 2023-07-11 18:28:03,851 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 18:28:04,037 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/57cclazn-media/images/validation_64_2537a2652dff56481399.png 2023-07-11 18:28:04,045 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/jrc91jyh-media/images/validation_64_75bb28ff057271dedef7.png 2023-07-11 18:28:04,073 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/l9yvik5a-media/images/validation_64_3e87611fd915ec96e580.png 2023-07-11 18:28:04,102 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/dulkn71c-media/images/validation_64_9bd0267d729986b95a5d.png 2023-07-11 18:28:04,442 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ibrcuw00-media/images/validation_64_2537a2652dff56481399.png 2023-07-11 18:28:04,468 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/186tmbex-media/images/validation_64_75bb28ff057271dedef7.png 2023-07-11 18:28:04,791 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/87woxba5-media/images/validation_64_3e87611fd915ec96e580.png 2023-07-11 18:28:04,827 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/7e1v472l-media/images/validation_64_9bd0267d729986b95a5d.png 2023-07-11 18:28:05,853 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:28:06,232 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:28:07,095 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:28:07,858 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:28:09,862 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:28:09,965 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:28:09,967 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:28:09,967 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:28:09,969 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:28:10,382 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 18:28:10,864 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:28:10,905 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:28:10,906 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:28:11,867 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:28:12,025 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:28:13,438 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:28:13,440 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:28:13,440 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:28:13,441 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:28:13,872 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:28:13,872 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:28:15,876 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:28:16,429 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:28:16,431 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:28:16,432 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:28:16,433 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:28:16,879 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:28:17,289 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:28:17,882 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:28:19,886 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:28:20,294 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:28:20,296 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:28:20,296 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:28:20,298 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:28:20,888 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:28:21,891 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:28:22,772 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:28:23,895 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:28:24,186 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:28:24,188 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:28:24,188 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:28:24,189 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:28:24,898 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:28:25,901 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:28:25,906 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:28:25,907 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:28:27,088 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:28:27,090 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:28:27,091 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:28:27,093 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:28:27,374 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:28:27,376 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:28:27,376 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:28:27,378 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:28:27,840 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:28:27,906 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:28:27,906 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:28:29,910 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:28:32,979 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:28:37,980 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:28:39,988 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:28:39,989 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_72_8f41b5c8c4403cc49f21.png with policy now 2023-07-11 18:28:40,012 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:28:40,012 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_72_8f7cadaa97753cdcfd66.png with policy now 2023-07-11 18:28:40,022 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:28:40,022 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_72_c4f043b99e28a96d715b.png with policy now 2023-07-11 18:28:40,035 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:28:40,035 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_72_9b536587af12e637b86b.png with policy now 2023-07-11 18:28:40,100 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:28:40,105 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:28:40,107 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:28:40,111 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:28:40,308 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/mspqh6fx-media/images/validation_72_8f41b5c8c4403cc49f21.png 2023-07-11 18:28:40,325 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/p68xi20s-media/images/validation_72_8f7cadaa97753cdcfd66.png 2023-07-11 18:28:40,352 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/u1o6r1bs-media/images/validation_72_c4f043b99e28a96d715b.png 2023-07-11 18:28:40,358 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/jajxoaup-media/images/validation_72_9b536587af12e637b86b.png 2023-07-11 18:28:40,385 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 18:28:40,906 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:28:40,907 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:28:40,933 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:28:40,933 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_72_c4f043b99e28a96d715b.png 2023-07-11 18:28:40,934 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_72_8f7cadaa97753cdcfd66.png 2023-07-11 18:28:40,934 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_72_9b536587af12e637b86b.png 2023-07-11 18:28:40,935 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_72_8f41b5c8c4403cc49f21.png 2023-07-11 18:28:40,935 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 18:28:41,256 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/3unvvhbt-media/images/validation_72_c4f043b99e28a96d715b.png 2023-07-11 18:28:41,288 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/wsu0qv2d-media/images/validation_72_8f7cadaa97753cdcfd66.png 2023-07-11 18:28:41,288 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/m92btzm5-media/images/validation_72_9b536587af12e637b86b.png 2023-07-11 18:28:41,303 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/f7x7l1dc-media/images/validation_72_8f41b5c8c4403cc49f21.png 2023-07-11 18:28:43,462 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:28:43,940 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:28:44,156 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:28:45,945 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:28:47,193 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:28:47,195 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:28:47,195 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:28:47,197 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:28:47,950 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:28:47,950 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:28:49,137 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:28:49,955 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:28:50,989 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:28:50,990 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:28:50,991 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:28:50,992 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:28:51,960 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:28:51,960 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:28:53,635 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:28:53,637 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:28:53,637 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:28:53,639 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:28:53,965 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:28:53,965 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:28:54,640 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:28:55,907 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:28:55,907 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:28:55,969 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:28:57,213 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:28:57,215 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:28:57,215 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:28:57,217 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:28:57,974 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:28:57,974 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:28:59,979 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:29:00,029 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:29:00,162 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:29:00,164 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:29:00,165 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:29:00,166 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:29:00,981 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:29:01,983 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:29:03,953 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:29:03,956 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:29:03,956 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:29:03,958 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:29:03,988 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:29:03,989 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:29:04,235 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:29:04,237 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:29:04,238 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:29:04,239 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:29:04,991 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:29:05,760 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:29:05,994 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:29:10,388 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 18:29:10,914 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:29:10,915 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:29:10,915 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:29:15,962 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:29:16,792 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:29:16,792 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_80_642de8ca4837518430ae.png with policy now 2023-07-11 18:29:16,833 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:29:16,833 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_80_002146d58462dcf1138f.png with policy now 2023-07-11 18:29:16,841 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:29:16,842 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_80_7fce802a62c03f349904.png with policy now 2023-07-11 18:29:16,860 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:29:16,860 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_80_4b64689fc286b148b39a.png with policy now 2023-07-11 18:29:16,901 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:29:16,904 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:29:16,907 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:29:16,909 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:29:17,016 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:29:17,017 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_80_642de8ca4837518430ae.png 2023-07-11 18:29:17,017 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_80_7fce802a62c03f349904.png 2023-07-11 18:29:17,018 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_80_4b64689fc286b148b39a.png 2023-07-11 18:29:17,018 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_80_002146d58462dcf1138f.png 2023-07-11 18:29:17,019 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 18:29:17,109 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/9ukguhzl-media/images/validation_80_642de8ca4837518430ae.png 2023-07-11 18:29:17,146 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/v6pd6l2i-media/images/validation_80_002146d58462dcf1138f.png 2023-07-11 18:29:17,166 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/k3eh4k4u-media/images/validation_80_7fce802a62c03f349904.png 2023-07-11 18:29:17,169 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/nghcdp4z-media/images/validation_80_4b64689fc286b148b39a.png 2023-07-11 18:29:17,494 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/3tmyg982-media/images/validation_80_642de8ca4837518430ae.png 2023-07-11 18:29:17,537 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/yeabugn5-media/images/validation_80_002146d58462dcf1138f.png 2023-07-11 18:29:17,840 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/gjhnu7e5-media/images/validation_80_7fce802a62c03f349904.png 2023-07-11 18:29:17,982 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/q6vfjsg2-media/images/validation_80_4b64689fc286b148b39a.png 2023-07-11 18:29:20,024 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:29:21,374 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:29:21,470 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:29:22,029 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:29:24,034 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:29:24,599 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:29:24,601 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:29:24,602 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:29:24,603 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:29:25,036 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:29:26,040 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:29:26,285 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:29:26,286 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:29:26,408 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:29:28,045 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:29:28,186 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:29:28,187 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:29:28,188 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:29:28,190 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:29:29,047 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:29:30,050 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:29:31,789 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:29:32,055 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:29:32,146 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:29:32,148 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:29:32,148 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:29:32,150 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:29:33,057 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:29:34,060 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:29:35,189 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:29:35,191 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:29:35,191 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:29:35,193 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:29:36,065 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:29:36,066 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:29:36,921 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:29:38,071 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:29:38,622 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:29:38,624 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:29:38,624 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:29:38,626 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:29:39,073 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:29:40,076 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:29:40,390 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 18:29:41,286 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:29:41,287 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:29:41,415 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:29:41,416 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:29:41,417 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:29:41,418 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:29:41,697 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:29:41,699 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:29:41,699 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:29:41,701 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:29:42,081 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:29:42,082 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:29:42,101 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:29:44,085 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:29:47,238 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:29:52,240 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:29:54,311 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:29:54,312 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_88_b86a7ba09e3469d620dc.png with policy now 2023-07-11 18:29:54,328 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:29:54,328 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_88_a4422b8ecb356c655d3a.png with policy now 2023-07-11 18:29:54,330 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:29:54,330 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_88_d1da96f2064c4210048a.png with policy now 2023-07-11 18:29:54,339 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:29:54,339 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_88_88384511781630fcde4e.png with policy now 2023-07-11 18:29:54,383 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:29:54,385 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:29:54,385 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:29:54,387 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:29:54,643 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/53465b7d-media/images/validation_88_a4422b8ecb356c655d3a.png 2023-07-11 18:29:54,643 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/8smq2hmt-media/images/validation_88_d1da96f2064c4210048a.png 2023-07-11 18:29:54,646 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/6f3ahzdy-media/images/validation_88_88384511781630fcde4e.png 2023-07-11 18:29:54,649 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/rhvwo1mv-media/images/validation_88_b86a7ba09e3469d620dc.png 2023-07-11 18:29:55,108 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:29:55,108 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_88_d1da96f2064c4210048a.png 2023-07-11 18:29:55,108 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_88_a4422b8ecb356c655d3a.png 2023-07-11 18:29:55,109 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_88_88384511781630fcde4e.png 2023-07-11 18:29:55,110 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_88_b86a7ba09e3469d620dc.png 2023-07-11 18:29:55,110 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 18:29:55,428 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/rw95jcyk-media/images/validation_88_d1da96f2064c4210048a.png 2023-07-11 18:29:55,462 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/20628anx-media/images/validation_88_88384511781630fcde4e.png 2023-07-11 18:29:55,493 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/6v7jfh67-media/images/validation_88_b86a7ba09e3469d620dc.png 2023-07-11 18:29:55,699 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/xcwihc18-media/images/validation_88_a4422b8ecb356c655d3a.png 2023-07-11 18:29:56,287 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:29:56,288 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:29:57,824 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:29:58,116 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:29:58,790 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:30:00,120 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:30:02,125 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:30:02,964 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:30:02,966 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:30:02,966 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:30:02,967 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:30:02,969 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:30:03,127 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:30:04,130 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:30:06,135 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:30:06,398 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:30:06,400 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:30:06,400 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:30:06,402 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:30:07,138 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:30:08,000 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:30:08,140 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:30:09,390 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:30:09,392 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:30:09,393 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:30:09,394 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:30:10,145 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:30:10,145 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:30:10,393 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 18:30:11,300 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:30:11,301 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:30:12,150 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:30:12,774 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:30:12,776 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:30:12,776 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:30:12,778 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:30:13,152 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:30:13,485 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:30:14,155 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:30:16,118 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:30:16,120 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:30:16,120 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:30:16,122 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:30:16,160 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:30:16,160 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:30:18,165 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:30:18,940 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:30:19,740 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:30:19,742 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:30:19,742 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:30:19,744 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:30:20,030 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:30:20,032 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:30:20,033 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:30:20,034 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:30:20,172 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:30:20,172 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:30:22,177 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:30:24,578 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:30:26,300 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:30:26,300 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:30:30,361 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:30:32,572 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:30:32,572 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_96_0d776d95eff027dba77c.png with policy now 2023-07-11 18:30:32,613 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:30:32,613 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_96_f4754be0e0fb99c57d60.png with policy now 2023-07-11 18:30:32,614 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:30:32,614 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_96_298b674dbb75f0e2d399.png with policy now 2023-07-11 18:30:32,615 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:30:32,615 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_96_632ab0608582a5d13e60.png with policy now 2023-07-11 18:30:32,651 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:30:32,653 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:30:32,653 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:30:32,655 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:30:32,894 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ln3k3nyz-media/images/validation_96_0d776d95eff027dba77c.png 2023-07-11 18:30:32,920 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/u867kj4f-media/images/validation_96_f4754be0e0fb99c57d60.png 2023-07-11 18:30:32,944 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/csqrwowj-media/images/validation_96_298b674dbb75f0e2d399.png 2023-07-11 18:30:32,952 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/7a8wmc0p-media/images/validation_96_632ab0608582a5d13e60.png 2023-07-11 18:30:33,203 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:30:33,203 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_96_298b674dbb75f0e2d399.png 2023-07-11 18:30:33,204 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_96_632ab0608582a5d13e60.png 2023-07-11 18:30:33,205 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_96_f4754be0e0fb99c57d60.png 2023-07-11 18:30:33,205 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_96_0d776d95eff027dba77c.png 2023-07-11 18:30:33,206 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 18:30:33,549 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/mi166cu8-media/images/validation_96_f4754be0e0fb99c57d60.png 2023-07-11 18:30:33,571 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/26qcd3lk-media/images/validation_96_298b674dbb75f0e2d399.png 2023-07-11 18:30:33,575 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/u03jgyhn-media/images/validation_96_0d776d95eff027dba77c.png 2023-07-11 18:30:33,607 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/l4ti6ns6-media/images/validation_96_632ab0608582a5d13e60.png 2023-07-11 18:30:35,384 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:30:36,143 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:30:36,211 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:30:38,217 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:30:39,522 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:30:39,523 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:30:39,524 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:30:39,526 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:30:40,222 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:30:40,223 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:30:40,396 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 18:30:40,401 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:30:41,301 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:30:41,303 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:30:42,177 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:30:42,179 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:30:42,180 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:30:42,181 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:30:42,228 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:30:42,228 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:30:44,233 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:30:45,055 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:30:45,057 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:30:45,058 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:30:45,059 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:30:45,236 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:30:46,060 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:30:46,239 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:30:48,244 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:30:48,571 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:30:48,573 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:30:48,573 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:30:48,575 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:30:49,247 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:30:50,250 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:30:51,584 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:30:52,255 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:30:52,891 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:30:52,893 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:30:52,894 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:30:52,895 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:30:53,258 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:30:54,261 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:30:55,944 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:30:55,946 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:30:55,946 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:30:55,948 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:30:56,225 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:30:56,227 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:30:56,227 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:30:56,229 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:30:56,266 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:30:56,267 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:30:56,300 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:30:56,301 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:30:56,644 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:30:58,272 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:31:01,782 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:31:06,783 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:31:08,838 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:31:08,838 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_104_34b936bfe047d298afd0.png with policy now 2023-07-11 18:31:08,848 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:31:08,848 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_104_a3c6437c2be1111726ba.png with policy now 2023-07-11 18:31:08,858 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:31:08,858 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_104_d7a07b12de591dfc784b.png with policy now 2023-07-11 18:31:08,880 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:31:08,880 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_104_85d0efb91c0e48ceb8a6.png with policy now 2023-07-11 18:31:08,922 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:31:08,924 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:31:08,924 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:31:08,929 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:31:09,175 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/hf5q4yx7-media/images/validation_104_a3c6437c2be1111726ba.png 2023-07-11 18:31:09,179 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/kdltpfgo-media/images/validation_104_34b936bfe047d298afd0.png 2023-07-11 18:31:09,203 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/svao3m0y-media/images/validation_104_85d0efb91c0e48ceb8a6.png 2023-07-11 18:31:09,203 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/4w8bmi1q-media/images/validation_104_d7a07b12de591dfc784b.png 2023-07-11 18:31:09,300 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:31:09,300 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_104_34b936bfe047d298afd0.png 2023-07-11 18:31:09,301 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_104_d7a07b12de591dfc784b.png 2023-07-11 18:31:09,301 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_104_85d0efb91c0e48ceb8a6.png 2023-07-11 18:31:09,302 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_104_a3c6437c2be1111726ba.png 2023-07-11 18:31:09,302 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 18:31:09,650 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/t4n7bkjo-media/images/validation_104_d7a07b12de591dfc784b.png 2023-07-11 18:31:09,666 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/lra6v1y6-media/images/validation_104_85d0efb91c0e48ceb8a6.png 2023-07-11 18:31:09,715 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/9cs8y80e-media/images/validation_104_a3c6437c2be1111726ba.png 2023-07-11 18:31:09,805 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/exdi2n5x-media/images/validation_104_34b936bfe047d298afd0.png 2023-07-11 18:31:10,399 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 18:31:11,768 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:31:11,769 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:31:11,804 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:31:12,308 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:31:12,777 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:31:14,314 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:31:15,895 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:31:15,897 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:31:15,897 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:31:15,899 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:31:16,320 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:31:16,320 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:31:17,373 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:31:18,325 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:31:18,739 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:31:18,741 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:31:18,742 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:31:18,743 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:31:19,328 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:31:20,332 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:31:22,335 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:31:22,337 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:31:22,337 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:31:22,339 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:31:22,341 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:31:23,046 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:31:23,344 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:31:24,346 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:31:26,097 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:31:26,099 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:31:26,099 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:31:26,101 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:31:26,352 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:31:26,352 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:31:26,768 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:31:26,770 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:31:28,234 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:31:28,357 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:31:28,840 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:31:28,842 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:31:28,842 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:31:28,844 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:31:29,360 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:31:30,363 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:31:32,309 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:31:32,311 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:31:32,311 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:31:32,313 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:31:32,368 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:31:32,368 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:31:32,590 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:31:32,592 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:31:32,593 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:31:32,594 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:31:33,371 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:31:34,164 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:31:34,374 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:31:39,165 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:31:40,402 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 18:31:41,775 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:31:41,775 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:31:44,835 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:31:45,192 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:31:45,192 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_112_77f47ee5ca4647b53e91.png with policy now 2023-07-11 18:31:45,233 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:31:45,233 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_112_58efbe7f01b6a5ce6ada.png with policy now 2023-07-11 18:31:45,234 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:31:45,234 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_112_37039f7d8b61bffb22b3.png with policy now 2023-07-11 18:31:45,235 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:31:45,235 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_112_3994094417c23d4c4a21.png with policy now 2023-07-11 18:31:45,264 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:31:45,266 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:31:45,266 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:31:45,268 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:31:45,401 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:31:45,402 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_112_37039f7d8b61bffb22b3.png 2023-07-11 18:31:45,402 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_112_58efbe7f01b6a5ce6ada.png 2023-07-11 18:31:45,403 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_112_77f47ee5ca4647b53e91.png 2023-07-11 18:31:45,403 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_112_3994094417c23d4c4a21.png 2023-07-11 18:31:45,404 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 18:31:45,565 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/9f51v34p-media/images/validation_112_58efbe7f01b6a5ce6ada.png 2023-07-11 18:31:45,567 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/0gqhitao-media/images/validation_112_77f47ee5ca4647b53e91.png 2023-07-11 18:31:45,578 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/9bmxtxir-media/images/validation_112_3994094417c23d4c4a21.png 2023-07-11 18:31:45,594 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/m0kjm7wk-media/images/validation_112_37039f7d8b61bffb22b3.png 2023-07-11 18:31:46,035 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/hvql2mib-media/images/validation_112_77f47ee5ca4647b53e91.png 2023-07-11 18:31:46,102 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/2csp5z2j-media/images/validation_112_58efbe7f01b6a5ce6ada.png 2023-07-11 18:31:46,109 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/i5wyubpb-media/images/validation_112_3994094417c23d4c4a21.png 2023-07-11 18:31:46,366 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/sq9pk5kf-media/images/validation_112_37039f7d8b61bffb22b3.png 2023-07-11 18:31:48,410 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:31:48,798 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:31:50,415 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:31:50,778 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:31:52,421 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:31:52,545 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:31:52,547 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:31:52,547 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:31:52,549 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:31:53,424 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:31:54,427 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:31:55,566 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:31:55,568 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:31:55,569 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:31:55,570 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:31:56,189 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:31:56,433 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:31:56,433 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:31:56,774 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:31:56,775 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:31:58,439 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:31:58,672 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:32:01,315 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:32:01,316 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:32:01,316 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:32:01,319 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:32:01,446 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:32:02,010 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:32:02,012 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:32:02,012 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:32:02,014 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:32:02,449 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:32:02,450 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:32:04,455 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:32:05,139 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:32:05,141 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:32:05,142 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:32:05,143 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:32:05,458 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:32:06,400 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:32:06,461 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:32:08,189 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:32:08,191 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:32:08,191 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:32:08,193 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:32:08,467 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:32:08,467 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:32:08,470 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:32:08,472 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:32:08,472 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:32:08,474 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:32:09,470 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:32:10,405 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 18:32:10,473 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:32:11,407 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:32:11,792 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:32:11,793 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:32:16,844 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:32:21,122 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:32:21,122 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_120_e11e426abf94dc089311.png with policy now 2023-07-11 18:32:21,137 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:32:21,137 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_120_70875a697c0992f9c209.png with policy now 2023-07-11 18:32:21,173 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:32:21,173 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_120_10e52d49b55c29d4c70a.png with policy now 2023-07-11 18:32:21,194 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:32:21,194 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_120_bf2536cdbcf9021e038a.png with policy now 2023-07-11 18:32:21,235 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:32:21,237 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:32:21,238 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:32:21,239 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:32:21,432 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/gm8c3dty-media/images/validation_120_e11e426abf94dc089311.png 2023-07-11 18:32:21,475 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/d00c3qwd-media/images/validation_120_70875a697c0992f9c209.png 2023-07-11 18:32:21,497 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/zdiu6abk-media/images/validation_120_10e52d49b55c29d4c70a.png 2023-07-11 18:32:21,502 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:32:21,502 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_120_70875a697c0992f9c209.png 2023-07-11 18:32:21,503 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_120_bf2536cdbcf9021e038a.png 2023-07-11 18:32:21,504 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_120_10e52d49b55c29d4c70a.png 2023-07-11 18:32:21,505 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_120_e11e426abf94dc089311.png 2023-07-11 18:32:21,505 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 18:32:21,512 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/kxiw2d17-media/images/validation_120_bf2536cdbcf9021e038a.png 2023-07-11 18:32:21,859 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/b3icgyj1-media/images/validation_120_bf2536cdbcf9021e038a.png 2023-07-11 18:32:21,861 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/e1oq5jay-media/images/validation_120_70875a697c0992f9c209.png 2023-07-11 18:32:22,044 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/1uawy59s-media/images/validation_120_e11e426abf94dc089311.png 2023-07-11 18:32:22,102 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/rghlcxrj-media/images/validation_120_10e52d49b55c29d4c70a.png 2023-07-11 18:32:22,240 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:32:24,511 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:32:25,079 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:32:26,517 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:32:26,803 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:32:26,804 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:32:27,380 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:32:28,139 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:32:28,141 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:32:28,141 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:32:28,143 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:32:28,522 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:32:28,523 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:32:30,528 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:32:31,150 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:32:31,152 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:32:31,152 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:32:31,154 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:32:31,531 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:32:32,461 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:32:32,535 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:32:34,084 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:32:34,086 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:32:34,087 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:32:34,088 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:32:34,541 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:32:34,542 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:32:36,547 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:32:37,367 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:32:37,369 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:32:37,369 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:32:37,371 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:32:37,550 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:32:38,372 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:32:38,553 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:32:40,410 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 18:32:40,559 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:32:40,730 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:32:40,731 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:32:40,732 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:32:40,733 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:32:41,561 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:32:41,803 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:32:41,804 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:32:42,565 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:32:44,048 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:32:44,050 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:32:44,050 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:32:44,051 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:32:44,053 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:32:44,330 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:32:44,332 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:32:44,332 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:32:44,333 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:32:44,571 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:32:44,571 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:32:46,577 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:32:49,862 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:32:54,863 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:32:56,803 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:32:56,804 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:32:56,925 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:32:56,925 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_128_f12975048cb9e24609f6.png with policy now 2023-07-11 18:32:56,969 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:32:56,969 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_128_c6d1e00286ea24ea39ba.png with policy now 2023-07-11 18:32:56,970 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:32:56,970 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_128_f4dfbac2cbc6682335d5.png with policy now 2023-07-11 18:32:56,970 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:32:56,970 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_128_c9fb58eadba85e11521d.png with policy now 2023-07-11 18:32:57,010 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:32:57,011 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:32:57,012 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:32:57,013 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:32:57,244 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/390pnykg-media/images/validation_128_f12975048cb9e24609f6.png 2023-07-11 18:32:57,330 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/u3qqsw91-media/images/validation_128_f4dfbac2cbc6682335d5.png 2023-07-11 18:32:57,358 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/q1kbcnts-media/images/validation_128_c6d1e00286ea24ea39ba.png 2023-07-11 18:32:57,358 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/9h4o1ok8-media/images/validation_128_c9fb58eadba85e11521d.png 2023-07-11 18:32:57,606 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:32:57,607 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_128_f4dfbac2cbc6682335d5.png 2023-07-11 18:32:57,607 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_128_f12975048cb9e24609f6.png 2023-07-11 18:32:57,608 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_128_c9fb58eadba85e11521d.png 2023-07-11 18:32:57,608 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_128_c6d1e00286ea24ea39ba.png 2023-07-11 18:32:57,609 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 18:32:57,992 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/jbn94ox4-media/images/validation_128_f4dfbac2cbc6682335d5.png 2023-07-11 18:32:58,009 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/qvdn9v2w-media/images/validation_128_f12975048cb9e24609f6.png 2023-07-11 18:32:58,017 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/swe4x6kp-media/images/validation_128_c9fb58eadba85e11521d.png 2023-07-11 18:32:58,032 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/yudo6t5b-media/images/validation_128_c6d1e00286ea24ea39ba.png 2023-07-11 18:32:59,864 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:33:00,617 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:33:00,641 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:33:02,623 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:33:03,657 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:33:03,659 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:33:03,659 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:33:03,660 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:33:04,629 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:33:04,629 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:33:05,407 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:33:06,635 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:33:07,252 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:33:07,254 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:33:07,255 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:33:07,256 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:33:07,638 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:33:08,642 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:33:10,411 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 18:33:10,415 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:33:10,647 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:33:11,150 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:33:11,152 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:33:11,153 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:33:11,154 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:33:11,650 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:33:12,466 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:33:12,467 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:33:12,654 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:33:14,659 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:33:14,923 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:33:14,925 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:33:14,925 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:33:14,927 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:33:15,593 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:33:15,662 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:33:16,665 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:33:18,604 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:33:18,606 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:33:18,607 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:33:18,608 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:33:18,671 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:33:18,672 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:33:20,678 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:33:20,996 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:33:21,809 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:33:21,811 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:33:21,811 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:33:21,813 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:33:22,090 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:33:22,092 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:33:22,093 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:33:22,094 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:33:22,685 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:33:22,685 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:33:24,691 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:33:26,678 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:33:27,467 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:33:27,468 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:33:32,512 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:33:34,696 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:33:34,696 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_136_4a6f2aa08c542f4f3d3b.png with policy now 2023-07-11 18:33:34,718 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_136_e3e73b728661ef994342.png 2023-07-11 18:33:34,719 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_136_36bb53c164993a8cf8d8.png 2023-07-11 18:33:34,719 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_136_4a6f2aa08c542f4f3d3b.png 2023-07-11 18:33:34,720 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 18:33:34,737 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:33:34,737 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_136_e3e73b728661ef994342.png with policy now 2023-07-11 18:33:34,738 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:33:34,738 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_136_36bb53c164993a8cf8d8.png with policy now 2023-07-11 18:33:34,739 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:33:34,739 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_136_3b8a8548b12f4fe561b0.png with policy now 2023-07-11 18:33:34,765 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:33:34,766 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:33:34,767 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:33:34,768 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:33:35,100 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/5aduawoc-media/images/validation_136_4a6f2aa08c542f4f3d3b.png 2023-07-11 18:33:35,109 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/7esexqt3-media/images/validation_136_36bb53c164993a8cf8d8.png 2023-07-11 18:33:35,129 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/grf276zb-media/images/validation_136_e3e73b728661ef994342.png 2023-07-11 18:33:35,149 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/gi9he7t0-media/images/validation_136_3b8a8548b12f4fe561b0.png 2023-07-11 18:33:35,529 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/rhaas1b6-media/images/validation_136_4a6f2aa08c542f4f3d3b.png 2023-07-11 18:33:35,562 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/5n0f5761-media/images/validation_136_36bb53c164993a8cf8d8.png 2023-07-11 18:33:35,722 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:33:35,722 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_136_3b8a8548b12f4fe561b0.png 2023-07-11 18:33:35,723 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 18:33:35,780 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/9rjks106-media/images/validation_136_e3e73b728661ef994342.png 2023-07-11 18:33:36,129 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/86x8t7bj-media/images/validation_136_3b8a8548b12f4fe561b0.png 2023-07-11 18:33:37,547 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:33:38,163 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:33:38,732 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:33:40,414 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 18:33:40,738 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:33:41,377 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:33:41,379 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:33:41,380 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:33:41,381 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:33:41,741 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:33:42,744 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:33:42,993 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:33:42,994 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:33:42,994 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:33:44,751 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:33:45,025 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:33:45,026 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:33:45,027 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:33:45,028 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:33:45,754 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:33:46,757 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:33:48,678 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:33:48,748 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:33:48,749 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:33:48,750 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:33:48,751 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:33:48,763 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:33:48,764 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:33:50,769 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:33:51,986 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:33:51,989 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:33:51,989 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:33:51,991 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:33:52,776 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:33:52,776 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:33:53,689 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:33:54,782 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:33:55,429 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:33:55,431 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:33:55,432 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:33:55,433 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:33:55,784 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:33:56,788 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:33:57,993 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:33:57,994 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:33:58,300 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:33:58,302 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:33:58,302 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:33:58,304 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:33:58,535 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:33:58,537 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:33:58,538 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:33:58,539 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:33:58,794 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:33:58,795 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:33:58,930 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:34:00,801 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:34:04,071 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:34:09,072 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:34:10,417 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 18:34:11,143 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:34:11,143 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_144_bf1d4866e27ceb3b9a7c.png with policy now 2023-07-11 18:34:11,158 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:34:11,158 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_144_2a98bd32e667f5d3ffa0.png with policy now 2023-07-11 18:34:11,167 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:34:11,168 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_144_9571275e76adeef75d31.png with policy now 2023-07-11 18:34:11,203 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:34:11,203 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_144_21da72f5de55ac2c601b.png with policy now 2023-07-11 18:34:11,242 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:34:11,243 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:34:11,243 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:34:11,246 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:34:11,478 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/4on0slco-media/images/validation_144_bf1d4866e27ceb3b9a7c.png 2023-07-11 18:34:11,507 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/bx0qfnof-media/images/validation_144_9571275e76adeef75d31.png 2023-07-11 18:34:11,510 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/s4kmkn81-media/images/validation_144_2a98bd32e667f5d3ffa0.png 2023-07-11 18:34:11,530 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/t0k9weim-media/images/validation_144_21da72f5de55ac2c601b.png 2023-07-11 18:34:11,832 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:34:11,832 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_144_bf1d4866e27ceb3b9a7c.png 2023-07-11 18:34:11,833 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_144_9571275e76adeef75d31.png 2023-07-11 18:34:11,833 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_144_2a98bd32e667f5d3ffa0.png 2023-07-11 18:34:11,834 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_144_21da72f5de55ac2c601b.png 2023-07-11 18:34:11,834 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 18:34:12,158 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/4nly3nbr-media/images/validation_144_bf1d4866e27ceb3b9a7c.png 2023-07-11 18:34:12,190 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/f1l6j37z-media/images/validation_144_2a98bd32e667f5d3ffa0.png 2023-07-11 18:34:12,192 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/2xw5009q-media/images/validation_144_9571275e76adeef75d31.png 2023-07-11 18:34:12,202 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/p6vwwck1-media/images/validation_144_21da72f5de55ac2c601b.png 2023-07-11 18:34:12,993 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:34:12,994 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:34:14,466 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:34:14,467 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:34:14,841 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:34:16,849 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:34:17,805 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:34:17,807 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:34:17,807 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:34:17,809 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:34:17,852 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:34:18,855 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:34:19,740 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:34:20,862 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:34:21,742 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:34:21,744 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:34:21,745 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:34:21,746 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:34:21,866 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:34:22,869 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:34:24,694 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:34:24,695 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:34:24,696 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:34:24,697 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:34:24,877 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:34:24,878 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:34:25,352 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:34:26,883 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:34:28,029 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:34:28,030 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:34:28,157 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:34:28,158 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:34:28,159 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:34:28,160 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:34:28,890 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:34:28,890 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:34:30,630 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:34:30,897 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:34:31,470 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:34:31,472 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:34:31,472 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:34:31,474 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:34:31,900 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:34:32,904 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:34:34,906 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:34:34,908 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:34:34,909 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:34:34,911 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:34:34,913 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:34:35,188 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:34:35,190 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:34:35,190 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:34:35,192 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:34:35,761 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:34:35,917 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:34:36,921 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:34:40,420 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 18:34:41,421 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:34:43,030 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:34:43,031 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:34:47,072 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:34:47,963 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:34:47,963 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_152_630cc9c3ad8164670f80.png with policy now 2023-07-11 18:34:47,977 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:34:47,977 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_152_eb8b309f2e7d0c0900fe.png with policy now 2023-07-11 18:34:47,986 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:34:47,987 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_152_e43b38eafcadeda16c8c.png with policy now 2023-07-11 18:34:47,996 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:34:47,996 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_152_e20fdb183cf6dd559eb8.png with policy now 2023-07-11 18:34:48,030 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:34:48,032 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:34:48,032 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:34:48,034 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:34:48,324 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/rcwp4phi-media/images/validation_152_630cc9c3ad8164670f80.png 2023-07-11 18:34:48,370 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/nu2pkzsy-media/images/validation_152_eb8b309f2e7d0c0900fe.png 2023-07-11 18:34:48,378 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ullz2ljv-media/images/validation_152_e43b38eafcadeda16c8c.png 2023-07-11 18:34:48,386 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/dnl8fc7z-media/images/validation_152_e20fdb183cf6dd559eb8.png 2023-07-11 18:34:48,956 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:34:48,956 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_152_e43b38eafcadeda16c8c.png 2023-07-11 18:34:48,957 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_152_eb8b309f2e7d0c0900fe.png 2023-07-11 18:34:48,957 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_152_630cc9c3ad8164670f80.png 2023-07-11 18:34:48,958 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_152_e20fdb183cf6dd559eb8.png 2023-07-11 18:34:48,958 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 18:34:49,297 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/irh1w4m0-media/images/validation_152_e43b38eafcadeda16c8c.png 2023-07-11 18:34:49,323 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/sb9sisy8-media/images/validation_152_eb8b309f2e7d0c0900fe.png 2023-07-11 18:34:49,332 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/6g8ujxyc-media/images/validation_152_630cc9c3ad8164670f80.png 2023-07-11 18:34:49,381 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/bya03r4l-media/images/validation_152_e20fdb183cf6dd559eb8.png 2023-07-11 18:34:50,962 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:34:51,327 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:34:52,329 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:34:52,970 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:34:54,977 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:34:54,991 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:34:54,993 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:34:54,994 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:34:54,995 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:34:55,980 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:34:56,983 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:34:57,527 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:34:58,270 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:34:58,271 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:34:58,448 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:34:58,450 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:34:58,450 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:34:58,452 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:34:58,991 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:34:58,991 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:35:00,997 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:35:01,499 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:35:01,501 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:35:01,501 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:35:01,502 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:35:02,000 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:35:03,002 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:35:03,174 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:35:05,008 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:35:05,009 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:35:05,010 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:35:05,012 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:35:05,012 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:35:06,015 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:35:07,019 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:35:07,803 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:35:07,805 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:35:07,805 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:35:07,807 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:35:08,022 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:35:08,555 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:35:09,026 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:35:10,423 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 18:35:10,836 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:35:10,837 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:35:10,838 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:35:10,839 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:35:11,033 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:35:11,034 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:35:11,117 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:35:11,119 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:35:11,119 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:35:11,121 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:35:12,036 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:35:13,041 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:35:13,290 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:35:13,291 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:35:14,347 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:35:19,348 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:35:23,703 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:35:23,703 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_160_fd584a28b09e2fa88b66.png with policy now 2023-07-11 18:35:23,745 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:35:23,745 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_160_268e9a732b517d634b0f.png with policy now 2023-07-11 18:35:23,746 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:35:23,746 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_160_f9cc3e65bbd82b475a59.png with policy now 2023-07-11 18:35:23,747 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:35:23,747 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_160_e8dd26e2169153e2c665.png with policy now 2023-07-11 18:35:23,774 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:35:23,775 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:35:23,776 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:35:23,777 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:35:24,001 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/cu1gcubo-media/images/validation_160_fd584a28b09e2fa88b66.png 2023-07-11 18:35:24,061 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/n15u0zo0-media/images/validation_160_e8dd26e2169153e2c665.png 2023-07-11 18:35:24,073 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/lqqs32o7-media/images/validation_160_f9cc3e65bbd82b475a59.png 2023-07-11 18:35:24,076 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:35:24,076 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_160_e8dd26e2169153e2c665.png 2023-07-11 18:35:24,077 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_160_f9cc3e65bbd82b475a59.png 2023-07-11 18:35:24,077 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_160_fd584a28b09e2fa88b66.png 2023-07-11 18:35:24,078 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_160_268e9a732b517d634b0f.png 2023-07-11 18:35:24,078 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 18:35:24,084 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/vslso4eu-media/images/validation_160_268e9a732b517d634b0f.png 2023-07-11 18:35:24,433 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/rc185snh-media/images/validation_160_fd584a28b09e2fa88b66.png 2023-07-11 18:35:24,467 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/3w313w4l-media/images/validation_160_f9cc3e65bbd82b475a59.png 2023-07-11 18:35:24,488 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/nokmnh2l-media/images/validation_160_268e9a732b517d634b0f.png 2023-07-11 18:35:24,495 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/rru3910c-media/images/validation_160_e8dd26e2169153e2c665.png 2023-07-11 18:35:24,778 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:35:27,087 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:35:28,192 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:35:28,290 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:35:28,290 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:35:29,094 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:35:30,421 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:35:31,101 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:35:31,324 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:35:31,326 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:35:31,326 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:35:31,328 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:35:32,104 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:35:33,108 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:35:35,033 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:35:35,035 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:35:35,035 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:35:35,037 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:35:35,116 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:35:35,116 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:35:35,904 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:35:37,123 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:35:37,706 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:35:37,707 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:35:37,708 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:35:37,709 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:35:38,126 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:35:39,130 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:35:40,426 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 18:35:41,136 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:35:41,338 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:35:41,340 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:35:41,340 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:35:41,341 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:35:41,342 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:35:42,140 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:35:43,143 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:35:43,290 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:35:43,291 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:35:45,150 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:35:45,168 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:35:45,169 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:35:45,169 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:35:45,171 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:35:46,154 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:35:46,484 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:35:47,158 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:35:47,868 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:35:47,870 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:35:47,870 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:35:47,872 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:35:48,149 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:35:48,151 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:35:48,152 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:35:48,153 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:35:48,162 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:35:49,165 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:35:51,172 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:35:51,936 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:35:56,937 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:35:58,291 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:35:58,292 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:36:01,015 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:36:01,015 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_168_7a6f1b9b97ae1a405c3d.png with policy now 2023-07-11 18:36:01,057 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:36:01,057 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_168_770dcafceef728f19c85.png with policy now 2023-07-11 18:36:01,058 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:36:01,058 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_168_01bbee240219c25af928.png with policy now 2023-07-11 18:36:01,071 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:36:01,072 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_168_fe40b1e9c3e598d742f2.png with policy now 2023-07-11 18:36:01,106 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:36:01,108 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:36:01,108 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:36:01,110 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:36:01,204 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:36:01,205 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_168_fe40b1e9c3e598d742f2.png 2023-07-11 18:36:01,205 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_168_01bbee240219c25af928.png 2023-07-11 18:36:01,206 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_168_7a6f1b9b97ae1a405c3d.png 2023-07-11 18:36:01,207 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_168_770dcafceef728f19c85.png 2023-07-11 18:36:01,207 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 18:36:01,333 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/sjrc2qwr-media/images/validation_168_7a6f1b9b97ae1a405c3d.png 2023-07-11 18:36:01,378 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/lksbd97d-media/images/validation_168_01bbee240219c25af928.png 2023-07-11 18:36:01,397 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/rd83xqvk-media/images/validation_168_fe40b1e9c3e598d742f2.png 2023-07-11 18:36:01,436 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/lcisycnc-media/images/validation_168_770dcafceef728f19c85.png 2023-07-11 18:36:01,801 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/3glmw66m-media/images/validation_168_770dcafceef728f19c85.png 2023-07-11 18:36:01,855 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ibh2373q-media/images/validation_168_7a6f1b9b97ae1a405c3d.png 2023-07-11 18:36:01,908 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/nfw07gvs-media/images/validation_168_01bbee240219c25af928.png 2023-07-11 18:36:02,111 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:36:02,158 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/cmkiczma-media/images/validation_168_fe40b1e9c3e598d742f2.png 2023-07-11 18:36:04,768 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:36:05,218 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:36:07,225 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:36:07,540 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:36:07,542 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:36:07,542 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:36:07,543 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:36:07,544 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:36:08,228 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:36:09,232 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:36:10,412 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:36:10,414 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:36:10,414 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:36:10,416 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:36:10,428 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 18:36:11,240 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:36:11,241 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:36:13,097 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:36:13,247 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:36:13,308 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:36:13,310 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:36:14,340 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:36:14,342 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:36:14,342 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:36:14,344 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:36:15,254 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:36:15,255 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:36:17,218 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:36:17,220 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:36:17,220 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:36:17,222 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:36:17,262 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:36:17,262 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:36:18,848 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:36:19,271 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:36:20,275 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:36:21,411 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:36:21,413 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:36:21,414 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:36:21,415 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:36:22,282 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:36:22,283 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:36:24,289 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:36:24,509 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:36:25,953 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:36:25,955 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:36:25,956 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:36:25,958 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:36:26,237 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:36:26,240 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:36:26,240 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:36:26,242 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:36:26,299 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:36:26,300 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:36:28,306 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:36:28,308 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:36:28,308 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:36:30,362 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:36:35,363 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:36:38,937 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:36:38,937 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_176_7c3251a3bc6842a14eaf.png with policy now 2023-07-11 18:36:38,952 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:36:38,952 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_176_ddb9188a2ecb3ba1a967.png with policy now 2023-07-11 18:36:38,961 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:36:38,962 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_176_b57338304a381b55774f.png with policy now 2023-07-11 18:36:38,970 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:36:38,971 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_176_77496af6afd4fcfc7802.png with policy now 2023-07-11 18:36:39,009 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:36:39,011 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:36:39,011 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:36:39,017 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:36:39,265 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/cg15yeq7-media/images/validation_176_7c3251a3bc6842a14eaf.png 2023-07-11 18:36:39,269 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/va1ca61l-media/images/validation_176_b57338304a381b55774f.png 2023-07-11 18:36:39,278 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/xddufyj3-media/images/validation_176_ddb9188a2ecb3ba1a967.png 2023-07-11 18:36:39,338 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/wtxcox5a-media/images/validation_176_77496af6afd4fcfc7802.png 2023-07-11 18:36:39,343 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:36:39,344 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_176_7c3251a3bc6842a14eaf.png 2023-07-11 18:36:39,345 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_176_77496af6afd4fcfc7802.png 2023-07-11 18:36:39,345 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_176_b57338304a381b55774f.png 2023-07-11 18:36:39,346 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_176_ddb9188a2ecb3ba1a967.png 2023-07-11 18:36:39,346 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 18:36:39,743 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/jl4rwllv-media/images/validation_176_ddb9188a2ecb3ba1a967.png 2023-07-11 18:36:39,743 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/1m9xg8jk-media/images/validation_176_b57338304a381b55774f.png 2023-07-11 18:36:39,785 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/sxxkmu3h-media/images/validation_176_7c3251a3bc6842a14eaf.png 2023-07-11 18:36:39,805 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/spor8k3r-media/images/validation_176_77496af6afd4fcfc7802.png 2023-07-11 18:36:40,431 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 18:36:40,436 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:36:42,355 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:36:42,664 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:36:43,720 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:36:43,721 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:36:44,362 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:36:46,236 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:36:46,237 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:36:46,238 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:36:46,239 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:36:46,240 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:36:46,369 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:36:46,370 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:36:48,377 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:36:49,800 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:36:49,802 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:36:49,802 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:36:49,804 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:36:50,384 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:36:50,385 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:36:51,569 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:36:52,392 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:36:52,972 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:36:52,974 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:36:52,975 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:36:52,981 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:36:53,395 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:36:54,399 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:36:56,407 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:36:57,211 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:36:57,401 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:36:57,403 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:36:57,404 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:36:57,405 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:36:57,410 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:36:58,413 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:36:58,720 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:36:58,721 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:37:00,421 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:37:00,655 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:37:00,657 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:37:00,657 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:37:00,658 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:37:01,425 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:37:02,350 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:37:02,429 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:37:02,812 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:37:02,815 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:37:02,815 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:37:02,817 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:37:03,093 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:37:03,095 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:37:03,096 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:37:03,098 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:37:03,433 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:37:04,437 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:37:07,704 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:37:10,434 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 18:37:13,436 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:37:13,720 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:37:13,721 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:37:15,772 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:37:15,772 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_184_973f6a3c3ee07eda891b.png with policy now 2023-07-11 18:37:15,813 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:37:15,813 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_184_e6bdd88d9dc7c75d4ca3.png with policy now 2023-07-11 18:37:15,814 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:37:15,814 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_184_b07a08bc9ab86b402e2a.png with policy now 2023-07-11 18:37:15,815 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:37:15,815 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_184_7c9b69b4f13b5c0abece.png with policy now 2023-07-11 18:37:15,847 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:37:15,848 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:37:15,849 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:37:15,850 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:37:16,151 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/rqlflv99-media/images/validation_184_973f6a3c3ee07eda891b.png 2023-07-11 18:37:16,186 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/psx52bjz-media/images/validation_184_e6bdd88d9dc7c75d4ca3.png 2023-07-11 18:37:16,189 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/fgt7hlgr-media/images/validation_184_b07a08bc9ab86b402e2a.png 2023-07-11 18:37:16,232 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/iefm8wah-media/images/validation_184_7c9b69b4f13b5c0abece.png 2023-07-11 18:37:16,477 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:37:16,477 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_184_973f6a3c3ee07eda891b.png 2023-07-11 18:37:16,478 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_184_e6bdd88d9dc7c75d4ca3.png 2023-07-11 18:37:16,479 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_184_b07a08bc9ab86b402e2a.png 2023-07-11 18:37:16,479 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_184_7c9b69b4f13b5c0abece.png 2023-07-11 18:37:16,480 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 18:37:16,925 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/9afnns0j-media/images/validation_184_973f6a3c3ee07eda891b.png 2023-07-11 18:37:16,928 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/32m55q0s-media/images/validation_184_e6bdd88d9dc7c75d4ca3.png 2023-07-11 18:37:16,938 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/3p1xlqaz-media/images/validation_184_b07a08bc9ab86b402e2a.png 2023-07-11 18:37:16,941 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/12qr19b2-media/images/validation_184_7c9b69b4f13b5c0abece.png 2023-07-11 18:37:18,484 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:37:18,663 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:37:19,523 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:37:20,493 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:37:22,500 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:37:23,179 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:37:23,181 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:37:23,181 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:37:23,183 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:37:23,504 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:37:24,508 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:37:24,552 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:37:26,516 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:37:26,672 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:37:26,674 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:37:26,674 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:37:26,676 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:37:27,519 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:37:28,523 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:37:28,720 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:37:28,721 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:37:29,420 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:37:29,421 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:37:29,422 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:37:29,423 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:37:29,527 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:37:30,105 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:37:30,531 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:37:32,164 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:37:32,166 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:37:32,167 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:37:32,168 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:37:32,539 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:37:32,539 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:37:34,546 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:37:35,494 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:37:35,496 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:37:35,496 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:37:35,497 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:37:35,499 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:37:35,550 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:37:36,554 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:37:38,562 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:37:39,284 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:37:39,286 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:37:39,287 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:37:39,288 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:37:39,566 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:37:39,567 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:37:39,569 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:37:39,569 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:37:39,571 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:37:40,444 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 18:37:40,571 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:37:40,571 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:37:41,446 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:37:43,721 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:37:43,721 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:37:46,771 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:37:51,772 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:37:52,350 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:37:52,350 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_192_d1e1f9c8aa7873e56eed.png with policy now 2023-07-11 18:37:52,393 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:37:52,393 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_192_63bb647b973f8f813f9e.png with policy now 2023-07-11 18:37:52,394 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:37:52,394 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_192_9846167229bd54634f68.png with policy now 2023-07-11 18:37:52,395 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:37:52,395 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_192_5fc6b92733f1b72454a9.png with policy now 2023-07-11 18:37:52,422 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:37:52,424 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:37:52,424 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:37:52,426 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:37:52,616 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:37:52,616 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_192_9846167229bd54634f68.png 2023-07-11 18:37:52,617 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_192_d1e1f9c8aa7873e56eed.png 2023-07-11 18:37:52,617 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_192_5fc6b92733f1b72454a9.png 2023-07-11 18:37:52,618 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_192_63bb647b973f8f813f9e.png 2023-07-11 18:37:52,618 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 18:37:52,738 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/3hsfk8a4-media/images/validation_192_9846167229bd54634f68.png 2023-07-11 18:37:52,739 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/2cufzb9u-media/images/validation_192_d1e1f9c8aa7873e56eed.png 2023-07-11 18:37:53,300 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/hamcvjea-media/images/validation_192_d1e1f9c8aa7873e56eed.png 2023-07-11 18:37:53,473 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/qlwxaw6t-media/images/validation_192_63bb647b973f8f813f9e.png 2023-07-11 18:37:53,564 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/fha5ionb-media/images/validation_192_5fc6b92733f1b72454a9.png 2023-07-11 18:37:53,577 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/2ou77v8c-media/images/validation_192_9846167229bd54634f68.png 2023-07-11 18:37:53,947 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/tgrwt6bh-media/images/validation_192_5fc6b92733f1b72454a9.png 2023-07-11 18:37:53,969 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/u01gw1th-media/images/validation_192_63bb647b973f8f813f9e.png 2023-07-11 18:37:55,624 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:37:55,781 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:37:57,188 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:37:57,628 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:37:59,030 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:37:59,031 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:37:59,200 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:37:59,202 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:37:59,202 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:37:59,204 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:37:59,632 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:37:59,633 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:38:01,637 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:38:02,695 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:38:02,697 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:38:02,697 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:38:02,697 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:38:02,699 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:38:03,640 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:38:03,641 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:38:05,644 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:38:06,610 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:38:06,612 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:38:06,612 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:38:06,614 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:38:07,648 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:38:07,649 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:38:08,623 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:38:09,653 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:38:10,443 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 18:38:10,574 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:38:10,576 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:38:10,576 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:38:10,578 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:38:11,656 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:38:11,657 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:38:13,113 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:38:13,115 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:38:13,115 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:38:13,117 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:38:13,661 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:38:13,661 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:38:14,035 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:38:14,036 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:38:14,037 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:38:15,665 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:38:16,917 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:38:16,918 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:38:16,919 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:38:16,921 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:38:17,199 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:38:17,201 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:38:17,201 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:38:17,203 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:38:17,671 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:38:17,671 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:38:19,674 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:38:19,803 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:38:24,804 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:38:29,036 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:38:29,036 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:38:29,822 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:38:29,822 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_200_b47039a85898acafffb2.png with policy now 2023-07-11 18:38:29,823 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:38:29,865 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:38:29,865 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_200_3dc0cb79df01ab7c84bc.png with policy now 2023-07-11 18:38:29,866 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:38:29,866 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_200_7b675596c6407752ba34.png with policy now 2023-07-11 18:38:29,867 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:38:29,867 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_200_96aa6aebfb42c1649c43.png with policy now 2023-07-11 18:38:29,893 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:38:29,894 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:38:29,895 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:38:29,897 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:38:30,149 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ytgyr71r-media/images/validation_200_b47039a85898acafffb2.png 2023-07-11 18:38:30,176 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/o13ww51p-media/images/validation_200_96aa6aebfb42c1649c43.png 2023-07-11 18:38:30,185 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/mzreg38q-media/images/validation_200_3dc0cb79df01ab7c84bc.png 2023-07-11 18:38:30,191 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/o3d6pxr2-media/images/validation_200_7b675596c6407752ba34.png 2023-07-11 18:38:31,696 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:38:31,697 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_200_b47039a85898acafffb2.png 2023-07-11 18:38:31,697 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_200_3dc0cb79df01ab7c84bc.png 2023-07-11 18:38:31,697 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_200_96aa6aebfb42c1649c43.png 2023-07-11 18:38:31,698 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_200_7b675596c6407752ba34.png 2023-07-11 18:38:31,698 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 18:38:32,071 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/126l81pc-media/images/validation_200_7b675596c6407752ba34.png 2023-07-11 18:38:32,071 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/5su9k724-media/images/validation_200_96aa6aebfb42c1649c43.png 2023-07-11 18:38:32,090 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/8hbvw9zq-media/images/validation_200_b47039a85898acafffb2.png 2023-07-11 18:38:32,098 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/wuas93ci-media/images/validation_200_3dc0cb79df01ab7c84bc.png 2023-07-11 18:38:33,701 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:38:33,919 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:38:35,062 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:38:35,706 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:38:36,612 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:38:36,613 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:38:36,614 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:38:36,615 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:38:37,710 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:38:37,710 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:38:39,714 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:38:40,126 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:38:40,248 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:38:40,250 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:38:40,250 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:38:40,252 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:38:40,447 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 18:38:41,719 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:38:41,720 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:38:42,955 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:38:42,957 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:38:42,957 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:38:42,959 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:38:43,723 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:38:43,724 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:38:44,036 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:38:44,037 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:38:45,727 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:38:45,882 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:38:46,013 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:38:46,014 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:38:46,015 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:38:46,016 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:38:47,731 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:38:47,732 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:38:49,415 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:38:49,417 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:38:49,417 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:38:49,419 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:38:49,736 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:38:49,736 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:38:51,202 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:38:51,740 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:38:53,034 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:38:53,036 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:38:53,036 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:38:53,038 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:38:53,318 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:38:53,320 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:38:53,320 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:38:53,322 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:38:53,744 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:38:53,744 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:38:55,749 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:38:56,863 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:38:59,037 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:38:59,038 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:39:02,078 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:39:05,932 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:39:05,933 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_208_b6b72350a5e90be236b8.png with policy now 2023-07-11 18:39:05,973 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:39:05,973 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_208_b0925536eb78c8a0506f.png with policy now 2023-07-11 18:39:05,974 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:39:05,974 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_208_8f8d4515856693522ab8.png with policy now 2023-07-11 18:39:05,975 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:39:05,975 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_208_75c3a361aae943fc98c0.png with policy now 2023-07-11 18:39:06,005 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:39:06,007 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:39:06,008 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:39:06,009 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:39:06,238 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/i09b5kqe-media/images/validation_208_b6b72350a5e90be236b8.png 2023-07-11 18:39:06,281 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/1eyvhq6y-media/images/validation_208_8f8d4515856693522ab8.png 2023-07-11 18:39:06,300 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/enecksyd-media/images/validation_208_75c3a361aae943fc98c0.png 2023-07-11 18:39:06,312 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/kk2rcvaz-media/images/validation_208_b0925536eb78c8a0506f.png 2023-07-11 18:39:07,364 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:39:07,771 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:39:07,771 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_208_b6b72350a5e90be236b8.png 2023-07-11 18:39:07,772 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_208_8f8d4515856693522ab8.png 2023-07-11 18:39:07,772 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_208_75c3a361aae943fc98c0.png 2023-07-11 18:39:07,773 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_208_b0925536eb78c8a0506f.png 2023-07-11 18:39:07,773 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 18:39:08,101 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/wq1jyb0d-media/images/validation_208_b6b72350a5e90be236b8.png 2023-07-11 18:39:08,125 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/7d8a4us1-media/images/validation_208_75c3a361aae943fc98c0.png 2023-07-11 18:39:08,153 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/xvgovsk4-media/images/validation_208_b0925536eb78c8a0506f.png 2023-07-11 18:39:08,155 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/dxg04uq2-media/images/validation_208_8f8d4515856693522ab8.png 2023-07-11 18:39:09,232 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:39:09,775 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:39:10,449 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 18:39:11,779 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:39:12,385 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:39:12,387 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:39:12,387 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:39:12,388 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:39:12,389 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:39:13,783 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:39:13,784 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:39:14,037 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:39:14,037 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:39:15,787 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:39:15,937 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:39:15,939 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:39:15,939 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:39:15,941 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:39:17,792 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:39:17,792 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:39:17,930 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:39:18,931 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:39:18,933 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:39:18,933 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:39:18,935 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:39:19,796 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:39:19,796 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:39:21,800 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:39:23,529 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:39:23,804 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:39:23,865 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:39:23,867 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:39:23,867 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:39:23,868 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:39:25,808 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:39:25,809 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:39:27,075 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:39:27,077 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:39:27,077 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:39:27,079 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:39:27,812 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:39:27,813 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:39:28,992 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:39:29,037 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:39:29,038 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:39:29,817 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:39:31,120 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:39:31,122 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:39:31,122 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:39:31,124 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:39:31,402 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:39:31,403 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:39:31,404 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:39:31,405 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:39:31,822 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:39:31,822 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:39:33,826 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:39:34,939 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:39:39,940 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:39:40,451 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 18:39:43,971 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:39:43,971 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_216_4a85055a3c02c45c760a.png with policy now 2023-07-11 18:39:43,985 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:39:43,986 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_216_4d5eab540def9d6b25f3.png with policy now 2023-07-11 18:39:44,000 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:39:44,000 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_216_cfc28146873310f734c3.png with policy now 2023-07-11 18:39:44,009 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:39:44,010 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_216_7a3a532a14ba9a1b34c2.png with policy now 2023-07-11 18:39:44,037 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:39:44,037 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:39:44,052 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:39:44,225 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:39:44,225 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:39:44,227 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:39:44,357 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/lg505x3i-media/images/validation_216_4a85055a3c02c45c760a.png 2023-07-11 18:39:44,390 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ekldwoy2-media/images/validation_216_cfc28146873310f734c3.png 2023-07-11 18:39:44,395 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/mofc3u2u-media/images/validation_216_4d5eab540def9d6b25f3.png 2023-07-11 18:39:44,503 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/auw662wq-media/images/validation_216_7a3a532a14ba9a1b34c2.png 2023-07-11 18:39:45,228 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:39:45,848 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:39:45,848 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_216_7a3a532a14ba9a1b34c2.png 2023-07-11 18:39:45,849 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_216_cfc28146873310f734c3.png 2023-07-11 18:39:45,849 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_216_4d5eab540def9d6b25f3.png 2023-07-11 18:39:45,850 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_216_4a85055a3c02c45c760a.png 2023-07-11 18:39:45,850 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 18:39:46,201 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/guwrqiox-media/images/validation_216_7a3a532a14ba9a1b34c2.png 2023-07-11 18:39:46,212 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/e7k9xcaq-media/images/validation_216_4a85055a3c02c45c760a.png 2023-07-11 18:39:46,226 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/j6hh5vn4-media/images/validation_216_4d5eab540def9d6b25f3.png 2023-07-11 18:39:46,234 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/q3nbjwo3-media/images/validation_216_cfc28146873310f734c3.png 2023-07-11 18:39:47,852 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:39:48,066 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:39:49,856 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:39:51,123 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:39:51,861 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:39:51,967 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:39:51,969 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:39:51,969 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:39:51,971 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:39:53,865 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:39:53,865 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:39:55,527 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:39:55,529 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:39:55,529 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:39:55,531 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:39:55,870 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:39:55,870 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:39:56,532 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:39:57,874 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:39:58,893 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:39:58,895 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:39:58,895 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:39:58,896 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:39:59,037 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:39:59,038 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:39:59,879 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:39:59,879 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:40:01,538 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:40:01,539 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:40:01,540 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:40:01,540 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:40:01,542 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:40:01,883 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:40:01,884 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:40:03,887 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:40:05,719 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:40:05,721 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:40:05,721 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:40:05,723 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:40:05,892 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:40:05,892 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:40:07,434 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:40:07,897 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:40:09,449 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:40:09,451 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:40:09,452 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:40:09,453 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:40:09,733 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:40:09,735 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:40:09,735 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:40:09,736 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:40:09,902 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:40:09,902 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:40:10,454 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 18:40:11,906 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:40:12,456 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:40:13,910 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:40:14,037 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:40:14,038 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:40:18,104 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:40:22,285 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:40:22,285 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_224_693eb3a2354f1b571e7c.png with policy now 2023-07-11 18:40:22,329 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:40:22,329 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_224_301d4cf0e8bf493be61e.png with policy now 2023-07-11 18:40:22,330 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:40:22,330 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_224_c41cb95d77436452e8b0.png with policy now 2023-07-11 18:40:22,331 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:40:22,331 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_224_e2e0e2a8e4633ed2e014.png with policy now 2023-07-11 18:40:22,368 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:40:22,369 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:40:22,369 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:40:22,371 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:40:22,601 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/d9gajqe4-media/images/validation_224_693eb3a2354f1b571e7c.png 2023-07-11 18:40:22,644 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/odyu92km-media/images/validation_224_301d4cf0e8bf493be61e.png 2023-07-11 18:40:22,658 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/4ov0vtci-media/images/validation_224_c41cb95d77436452e8b0.png 2023-07-11 18:40:22,674 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/dhnvi0c5-media/images/validation_224_e2e0e2a8e4633ed2e014.png 2023-07-11 18:40:23,342 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:40:23,930 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:40:23,930 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_224_693eb3a2354f1b571e7c.png 2023-07-11 18:40:23,931 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_224_e2e0e2a8e4633ed2e014.png 2023-07-11 18:40:23,931 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_224_301d4cf0e8bf493be61e.png 2023-07-11 18:40:23,932 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_224_c41cb95d77436452e8b0.png 2023-07-11 18:40:23,932 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 18:40:24,263 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/afie115n-media/images/validation_224_693eb3a2354f1b571e7c.png 2023-07-11 18:40:24,298 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/3v70pkyr-media/images/validation_224_e2e0e2a8e4633ed2e014.png 2023-07-11 18:40:24,308 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ur2u0gsp-media/images/validation_224_301d4cf0e8bf493be61e.png 2023-07-11 18:40:24,510 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/5wk3lvaq-media/images/validation_224_c41cb95d77436452e8b0.png 2023-07-11 18:40:25,934 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:40:26,073 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:40:27,939 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:40:28,652 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:40:29,264 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:40:29,265 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:40:29,440 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:40:29,442 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:40:29,442 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:40:29,444 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:40:29,944 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:40:29,944 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:40:31,948 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:40:32,370 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:40:32,372 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:40:32,372 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:40:32,374 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:40:33,953 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:40:33,954 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:40:34,205 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:40:35,958 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:40:36,081 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:40:36,083 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:40:36,083 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:40:36,085 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:40:37,963 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:40:37,963 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:40:39,083 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:40:39,084 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:40:39,085 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:40:39,086 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:40:39,785 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:40:39,967 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:40:39,967 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:40:40,457 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 18:40:41,971 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:40:42,824 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:40:42,825 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:40:42,826 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:40:42,827 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:40:43,976 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:40:43,976 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:40:44,301 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:40:44,302 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:40:45,078 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:40:45,981 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:40:46,013 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:40:46,014 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:40:46,015 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:40:46,016 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:40:46,296 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:40:46,298 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:40:46,298 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:40:46,300 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:40:47,986 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:40:47,987 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:40:49,991 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:40:50,982 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:40:55,983 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:40:59,064 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:40:59,064 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_232_a060116adb492f4b9a59.png with policy now 2023-07-11 18:40:59,078 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:40:59,079 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_232_c7a422f0918a7e63fef3.png with policy now 2023-07-11 18:40:59,088 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:40:59,089 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_232_c57b0c82f494de77f729.png with policy now 2023-07-11 18:40:59,097 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:40:59,097 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_232_713378cc3efc25bf233a.png with policy now 2023-07-11 18:40:59,132 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:40:59,133 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:40:59,134 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:40:59,135 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:40:59,300 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:40:59,301 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:40:59,390 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/3tqf7i6k-media/images/validation_232_a060116adb492f4b9a59.png 2023-07-11 18:40:59,413 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/l9wlnmfi-media/images/validation_232_c57b0c82f494de77f729.png 2023-07-11 18:40:59,421 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ke4zjxlv-media/images/validation_232_713378cc3efc25bf233a.png 2023-07-11 18:40:59,423 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/td307led-media/images/validation_232_c7a422f0918a7e63fef3.png 2023-07-11 18:41:00,011 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:41:00,012 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_232_c7a422f0918a7e63fef3.png 2023-07-11 18:41:00,012 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_232_a060116adb492f4b9a59.png 2023-07-11 18:41:00,013 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_232_713378cc3efc25bf233a.png 2023-07-11 18:41:00,014 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_232_c57b0c82f494de77f729.png 2023-07-11 18:41:00,014 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 18:41:00,353 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/owxo7wz6-media/images/validation_232_c7a422f0918a7e63fef3.png 2023-07-11 18:41:00,353 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/soag5ys4-media/images/validation_232_a060116adb492f4b9a59.png 2023-07-11 18:41:00,361 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/wq451ub5-media/images/validation_232_c57b0c82f494de77f729.png 2023-07-11 18:41:00,368 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/pkxxs3av-media/images/validation_232_713378cc3efc25bf233a.png 2023-07-11 18:41:01,234 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:41:02,833 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:41:04,021 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:41:06,025 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:41:06,099 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:41:06,101 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:41:06,102 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:41:06,103 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:41:06,881 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:41:08,030 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:41:08,030 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:41:09,065 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:41:09,067 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:41:09,067 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:41:09,069 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:41:10,035 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:41:10,035 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:41:10,460 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 18:41:12,040 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:41:12,348 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:41:12,350 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:41:12,350 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:41:12,351 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:41:12,352 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:41:14,044 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:41:14,045 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:41:14,300 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:41:14,301 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:41:15,437 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:41:15,439 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:41:15,439 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:41:15,441 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:41:16,049 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:41:16,049 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:41:17,718 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:41:18,054 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:41:18,649 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:41:18,651 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:41:18,651 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:41:18,653 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:41:20,058 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:41:20,059 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:41:22,063 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:41:22,671 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:41:22,673 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:41:22,674 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:41:22,675 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:41:22,955 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:41:22,956 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:41:22,957 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:41:22,957 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:41:22,959 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:41:24,069 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:41:24,069 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:41:26,073 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:41:28,477 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:41:29,301 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:41:29,301 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:41:34,342 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:41:35,458 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:41:35,458 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_240_cf699d9c0706f5fbf1d1.png with policy now 2023-07-11 18:41:35,501 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:41:35,501 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_240_b17212d3811e99b0c92e.png with policy now 2023-07-11 18:41:35,502 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:41:35,502 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_240_4363a80dfcc1b540ae2c.png with policy now 2023-07-11 18:41:35,503 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:41:35,503 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_240_131f48b2cfb4ebcfb628.png with policy now 2023-07-11 18:41:35,537 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:41:35,538 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:41:35,538 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:41:35,540 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:41:35,777 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/l80qagw1-media/images/validation_240_cf699d9c0706f5fbf1d1.png 2023-07-11 18:41:35,822 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/puj52y9l-media/images/validation_240_4363a80dfcc1b540ae2c.png 2023-07-11 18:41:35,830 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/y2drvg1r-media/images/validation_240_131f48b2cfb4ebcfb628.png 2023-07-11 18:41:36,095 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:41:36,096 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_240_131f48b2cfb4ebcfb628.png 2023-07-11 18:41:36,096 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_240_4363a80dfcc1b540ae2c.png 2023-07-11 18:41:36,097 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_240_cf699d9c0706f5fbf1d1.png 2023-07-11 18:41:36,097 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_240_b17212d3811e99b0c92e.png 2023-07-11 18:41:36,098 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 18:41:36,485 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/g2bzem72-media/images/validation_240_131f48b2cfb4ebcfb628.png 2023-07-11 18:41:36,511 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/q2vaz85q-media/images/validation_240_cf699d9c0706f5fbf1d1.png 2023-07-11 18:41:36,529 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/kjnk13hi-media/images/validation_240_4363a80dfcc1b540ae2c.png 2023-07-11 18:41:36,625 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/znp2ugz8-media/images/validation_240_b17212d3811e99b0c92e.png 2023-07-11 18:41:37,005 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/tdrxgmsw-media/images/validation_240_b17212d3811e99b0c92e.png 2023-07-11 18:41:39,321 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:41:40,104 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:41:40,322 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:41:40,463 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 18:41:42,109 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:41:42,768 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:41:42,769 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:41:42,770 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:41:42,771 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:41:44,114 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:41:44,114 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:41:44,301 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:41:44,301 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:41:45,500 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:41:46,119 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:41:46,467 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:41:46,469 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:41:46,470 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:41:46,471 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:41:48,124 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:41:48,124 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:41:49,841 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:41:49,843 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:41:49,843 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:41:49,845 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:41:50,128 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:41:50,129 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:41:50,846 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:41:52,133 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:41:52,838 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:41:52,840 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:41:52,840 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:41:52,842 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:41:54,137 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:41:54,138 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:41:56,037 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:41:56,039 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:41:56,039 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:41:56,040 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:41:56,041 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:41:56,142 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:41:56,142 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:41:58,146 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:41:59,404 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:41:59,405 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:41:59,531 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:41:59,533 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:41:59,533 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:41:59,535 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:41:59,814 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:41:59,816 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:41:59,817 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:41:59,818 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:42:00,152 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:42:00,152 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:42:01,460 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:42:02,156 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:42:06,461 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:42:10,466 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 18:42:11,468 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:42:12,502 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:42:12,503 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_248_9e9ae0e3b6539cacbcfd.png with policy now 2023-07-11 18:42:12,517 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:42:12,518 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_248_84e81851db303d973735.png with policy now 2023-07-11 18:42:12,527 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:42:12,527 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_248_57792db981d24437505d.png with policy now 2023-07-11 18:42:12,535 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:42:12,535 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_248_981e491cb9b69cf79bc6.png with policy now 2023-07-11 18:42:12,574 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:42:12,576 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:42:12,576 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:42:12,578 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:42:12,866 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/uabsr0vk-media/images/validation_248_981e491cb9b69cf79bc6.png 2023-07-11 18:42:12,866 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/s7sq9d1p-media/images/validation_248_57792db981d24437505d.png 2023-07-11 18:42:12,866 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/zvrjr3ej-media/images/validation_248_9e9ae0e3b6539cacbcfd.png 2023-07-11 18:42:12,878 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/7savf4iz-media/images/validation_248_84e81851db303d973735.png 2023-07-11 18:42:14,181 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:42:14,182 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_248_981e491cb9b69cf79bc6.png 2023-07-11 18:42:14,182 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_248_84e81851db303d973735.png 2023-07-11 18:42:14,182 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_248_9e9ae0e3b6539cacbcfd.png 2023-07-11 18:42:14,183 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_248_57792db981d24437505d.png 2023-07-11 18:42:14,183 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 18:42:14,404 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:42:14,405 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:42:14,554 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/tz8wbyzo-media/images/validation_248_57792db981d24437505d.png 2023-07-11 18:42:14,563 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/5q6rqqzm-media/images/validation_248_9e9ae0e3b6539cacbcfd.png 2023-07-11 18:42:14,567 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/hhwxv7q5-media/images/validation_248_84e81851db303d973735.png 2023-07-11 18:42:14,574 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/3h6xmkqv-media/images/validation_248_981e491cb9b69cf79bc6.png 2023-07-11 18:42:16,186 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:42:16,683 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:42:17,208 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:42:18,191 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:42:20,195 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:42:20,475 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:42:20,477 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:42:20,477 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:42:20,479 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:42:21,793 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:42:22,200 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:42:22,201 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:42:24,042 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:42:24,043 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:42:24,044 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:42:24,045 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:42:24,205 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:42:24,206 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:42:26,210 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:42:27,266 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:42:27,425 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:42:27,426 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:42:27,427 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:42:27,428 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:42:28,214 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:42:28,215 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:42:29,404 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:42:29,405 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:42:30,219 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:42:30,817 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:42:30,819 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:42:30,819 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:42:30,821 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:42:32,223 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:42:32,224 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:42:32,927 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:42:33,748 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:42:33,750 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:42:33,750 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:42:33,752 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:42:34,228 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:42:34,229 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:42:36,234 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:42:37,127 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:42:37,129 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:42:37,129 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:42:37,131 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:42:37,411 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:42:37,413 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:42:37,413 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:42:37,415 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:42:37,955 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:42:38,239 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:42:38,239 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:42:40,244 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:42:40,469 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 18:42:43,472 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:42:44,404 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:42:44,405 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:42:48,478 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:42:49,991 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:42:49,991 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_256_e0fd66e23b03e8fb0137.png with policy now 2023-07-11 18:42:50,005 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:42:50,006 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_256_003e58e75a4b35087fc3.png with policy now 2023-07-11 18:42:50,010 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:42:50,010 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_256_6e977e09b8e28e89d88b.png with policy now 2023-07-11 18:42:50,020 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:42:50,020 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_256_398994785799033e6506.png with policy now 2023-07-11 18:42:50,057 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:42:50,058 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:42:50,059 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:42:50,060 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:42:50,265 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:42:50,266 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_256_398994785799033e6506.png 2023-07-11 18:42:50,266 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_256_6e977e09b8e28e89d88b.png 2023-07-11 18:42:50,267 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_256_e0fd66e23b03e8fb0137.png 2023-07-11 18:42:50,267 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_256_003e58e75a4b35087fc3.png 2023-07-11 18:42:50,267 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 18:42:50,304 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/7yy68npr-media/images/validation_256_e0fd66e23b03e8fb0137.png 2023-07-11 18:42:50,332 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/yryoetqr-media/images/validation_256_003e58e75a4b35087fc3.png 2023-07-11 18:42:50,339 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/it2egb6d-media/images/validation_256_6e977e09b8e28e89d88b.png 2023-07-11 18:42:50,346 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/tj37gj75-media/images/validation_256_398994785799033e6506.png 2023-07-11 18:42:50,662 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/hsuumo6s-media/images/validation_256_e0fd66e23b03e8fb0137.png 2023-07-11 18:42:50,685 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/c862fgtk-media/images/validation_256_003e58e75a4b35087fc3.png 2023-07-11 18:42:51,026 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/06ujeluv-media/images/validation_256_398994785799033e6506.png 2023-07-11 18:42:51,187 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/qiwz0li0-media/images/validation_256_6e977e09b8e28e89d88b.png 2023-07-11 18:42:53,548 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:42:54,274 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:42:54,606 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:42:56,279 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:42:58,284 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:42:58,759 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:42:58,760 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:42:58,761 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:42:58,761 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:42:58,763 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:42:59,762 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:42:59,763 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:43:00,289 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:43:00,290 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:43:02,294 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:43:02,394 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:43:02,396 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:43:02,396 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:43:02,398 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:43:04,114 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:43:04,299 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:43:04,299 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:43:05,782 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:43:05,784 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:43:05,784 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:43:05,786 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:43:06,303 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:43:06,304 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:43:07,969 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:43:07,971 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:43:07,971 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:43:07,973 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:43:08,308 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:43:08,309 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:43:09,445 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:43:10,313 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:43:10,472 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 18:43:12,077 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:43:12,079 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:43:12,079 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:43:12,081 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:43:12,319 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:43:12,319 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:43:14,324 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:43:14,452 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:43:15,466 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:43:15,467 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:43:15,632 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:43:15,633 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:43:15,634 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:43:15,635 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:43:15,879 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:43:15,881 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:43:15,881 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:43:15,883 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:43:16,328 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:43:16,329 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:43:18,333 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:43:19,482 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:43:24,483 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:43:28,485 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:43:28,485 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_264_df9f8e90426ed7bffb77.png with policy now 2023-07-11 18:43:28,500 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:43:28,500 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_264_be323385875893b62402.png with policy now 2023-07-11 18:43:28,510 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:43:28,510 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_264_50b6b68312c4f0c0f919.png with policy now 2023-07-11 18:43:28,513 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:43:28,513 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_264_0ba4a6c56127f845b572.png with policy now 2023-07-11 18:43:28,555 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:43:28,556 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:43:28,556 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:43:28,558 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:43:28,826 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ltgqe2x5-media/images/validation_264_df9f8e90426ed7bffb77.png 2023-07-11 18:43:28,828 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ixq6w3vt-media/images/validation_264_50b6b68312c4f0c0f919.png 2023-07-11 18:43:28,844 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/b3hz759d-media/images/validation_264_0ba4a6c56127f845b572.png 2023-07-11 18:43:28,860 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/7rfvvydm-media/images/validation_264_be323385875893b62402.png 2023-07-11 18:43:29,559 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:43:30,360 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:43:30,360 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_264_50b6b68312c4f0c0f919.png 2023-07-11 18:43:30,361 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_264_df9f8e90426ed7bffb77.png 2023-07-11 18:43:30,361 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_264_0ba4a6c56127f845b572.png 2023-07-11 18:43:30,362 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_264_be323385875893b62402.png 2023-07-11 18:43:30,362 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 18:43:30,533 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:43:30,533 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:43:30,710 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/am8thh8j-media/images/validation_264_50b6b68312c4f0c0f919.png 2023-07-11 18:43:30,717 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/wrp6rw75-media/images/validation_264_0ba4a6c56127f845b572.png 2023-07-11 18:43:30,719 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/hynyg0ow-media/images/validation_264_df9f8e90426ed7bffb77.png 2023-07-11 18:43:30,736 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/gx2mjrdp-media/images/validation_264_be323385875893b62402.png 2023-07-11 18:43:32,365 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:43:32,936 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:43:34,370 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:43:34,568 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:43:36,376 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:43:36,865 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:43:36,866 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:43:36,867 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:43:36,868 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:43:38,380 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:43:38,381 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:43:39,881 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:43:40,386 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:43:40,475 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 18:43:40,938 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:43:40,940 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:43:40,941 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:43:40,942 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:43:42,390 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:43:42,391 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:43:44,395 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:43:44,578 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:43:44,579 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:43:44,580 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:43:44,581 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:43:45,511 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:43:45,533 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:43:45,534 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:43:46,400 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:43:46,401 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:43:47,968 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:43:47,970 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:43:47,971 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:43:47,972 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:43:48,405 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:43:48,406 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:43:50,410 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:43:50,864 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:43:50,865 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:43:50,866 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:43:50,867 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:43:50,868 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:43:52,414 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:43:52,414 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:43:54,208 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:43:54,209 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:43:54,210 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:43:54,212 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:43:54,419 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:43:54,420 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:43:54,490 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:43:54,491 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:43:54,492 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:43:54,493 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:43:56,077 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:43:56,479 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:43:56,480 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:43:58,484 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:44:00,534 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:44:00,535 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:44:01,577 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:44:06,578 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:44:07,133 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:44:07,133 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_272_c2159fc2ce0f2913430b.png with policy now 2023-07-11 18:44:07,144 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:44:07,145 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_272_cd804946c37522db1906.png with policy now 2023-07-11 18:44:07,156 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:44:07,156 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_272_5f6299d67a2c504b2dcc.png with policy now 2023-07-11 18:44:07,168 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:44:07,168 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_272_2787282524c47ad5cb18.png with policy now 2023-07-11 18:44:07,202 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:44:07,204 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:44:07,204 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:44:07,211 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:44:07,460 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/wjfl6qzm-media/images/validation_272_c2159fc2ce0f2913430b.png 2023-07-11 18:44:07,484 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/cxryq9r3-media/images/validation_272_cd804946c37522db1906.png 2023-07-11 18:44:07,493 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/jw9vjbv4-media/images/validation_272_5f6299d67a2c504b2dcc.png 2023-07-11 18:44:07,502 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/0778tnmj-media/images/validation_272_2787282524c47ad5cb18.png 2023-07-11 18:44:08,509 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:44:08,510 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_272_cd804946c37522db1906.png 2023-07-11 18:44:08,510 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_272_5f6299d67a2c504b2dcc.png 2023-07-11 18:44:08,511 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_272_2787282524c47ad5cb18.png 2023-07-11 18:44:08,511 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_272_c2159fc2ce0f2913430b.png 2023-07-11 18:44:08,512 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 18:44:08,883 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/254gcnms-media/images/validation_272_5f6299d67a2c504b2dcc.png 2023-07-11 18:44:08,890 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/q9f3l7nb-media/images/validation_272_cd804946c37522db1906.png 2023-07-11 18:44:08,902 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/1jdqb3ih-media/images/validation_272_2787282524c47ad5cb18.png 2023-07-11 18:44:08,904 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/bjgojdnd-media/images/validation_272_c2159fc2ce0f2913430b.png 2023-07-11 18:44:10,478 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 18:44:10,514 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:44:10,745 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:44:11,747 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:44:12,520 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:44:14,525 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:44:14,715 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:44:14,716 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:44:14,717 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:44:14,718 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:44:15,559 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:44:15,560 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:44:16,529 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:44:16,530 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:44:16,901 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:44:17,568 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:44:17,569 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:44:17,570 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:44:17,571 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:44:18,535 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:44:18,535 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:44:20,540 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:44:21,042 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:44:21,044 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:44:21,045 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:44:21,046 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:44:21,997 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:44:22,545 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:44:22,545 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:44:24,473 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:44:24,475 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:44:24,475 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:44:24,477 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:44:24,550 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:44:24,550 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:44:26,555 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:44:27,037 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:44:27,039 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:44:27,040 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:44:27,040 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:44:27,042 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:44:28,560 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:44:28,560 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:44:30,473 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:44:30,475 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:44:30,475 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:44:30,477 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:44:30,559 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:44:30,560 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:44:30,569 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:44:30,570 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:44:30,800 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:44:30,802 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:44:30,802 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:44:30,804 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:44:32,308 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:44:32,575 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:44:32,576 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:44:34,580 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:44:37,309 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:44:40,482 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 18:44:42,484 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:44:43,336 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:44:43,337 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_280_a60cb3ddac08d073d30c.png with policy now 2023-07-11 18:44:43,347 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:44:43,347 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_280_4af7c3c5f9585a30429c.png with policy now 2023-07-11 18:44:43,356 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:44:43,356 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_280_558ec026a5f3b0c54aab.png with policy now 2023-07-11 18:44:43,364 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:44:43,365 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_280_c9a2c308a53c060e2f9e.png with policy now 2023-07-11 18:44:43,406 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:44:43,407 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:44:43,407 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:44:43,409 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:44:43,666 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/qg8c9r5e-media/images/validation_280_a60cb3ddac08d073d30c.png 2023-07-11 18:44:43,692 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/nqz93pby-media/images/validation_280_c9a2c308a53c060e2f9e.png 2023-07-11 18:44:43,710 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/y2al1r3v-media/images/validation_280_4af7c3c5f9585a30429c.png 2023-07-11 18:44:43,795 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/2syrhx9i-media/images/validation_280_558ec026a5f3b0c54aab.png 2023-07-11 18:44:44,604 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:44:44,605 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_280_a60cb3ddac08d073d30c.png 2023-07-11 18:44:44,605 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_280_c9a2c308a53c060e2f9e.png 2023-07-11 18:44:44,606 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_280_558ec026a5f3b0c54aab.png 2023-07-11 18:44:44,607 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_280_4af7c3c5f9585a30429c.png 2023-07-11 18:44:44,607 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 18:44:44,958 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/jpf1rjcb-media/images/validation_280_c9a2c308a53c060e2f9e.png 2023-07-11 18:44:44,988 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/d4x6e3gf-media/images/validation_280_558ec026a5f3b0c54aab.png 2023-07-11 18:44:44,990 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/adljwzfr-media/images/validation_280_a60cb3ddac08d073d30c.png 2023-07-11 18:44:45,000 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/cysdvc2o-media/images/validation_280_4af7c3c5f9585a30429c.png 2023-07-11 18:44:45,560 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:44:45,560 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:44:46,609 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:44:47,233 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:44:48,235 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:44:48,615 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:44:50,619 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:44:50,647 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:44:50,649 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:44:50,649 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:44:50,651 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:44:52,624 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:44:52,625 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:44:53,327 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:44:54,538 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:44:54,539 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:44:54,540 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:44:54,541 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:44:54,630 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:44:54,631 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:44:56,635 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:44:58,005 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:44:58,007 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:44:58,008 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:44:58,010 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:44:58,640 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:44:58,641 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:44:58,785 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:45:00,560 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:45:00,561 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:45:00,645 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:45:01,296 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:45:01,298 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:45:01,298 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:45:01,300 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:45:02,650 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:45:02,651 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:45:03,942 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:45:03,943 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:45:03,944 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:45:03,944 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:45:03,946 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:45:04,655 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:45:04,656 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:45:06,661 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:45:06,687 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:45:06,689 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:45:06,689 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:45:06,691 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:45:06,971 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:45:06,973 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:45:06,973 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:45:06,975 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:45:08,667 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:45:08,667 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:45:09,526 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:45:10,483 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 18:45:10,672 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:45:15,486 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:45:15,560 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:45:15,560 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:45:19,534 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:45:19,535 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_288_d322f4dde2e9d2d4c2d0.png with policy now 2023-07-11 18:45:19,577 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:45:19,577 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_288_e8cc62c8cd0593a12258.png with policy now 2023-07-11 18:45:19,578 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:45:19,578 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_288_2a46f6447f317761e631.png with policy now 2023-07-11 18:45:19,579 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:45:19,579 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_288_7092f5d1e02e7ebba8fb.png with policy now 2023-07-11 18:45:19,604 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:45:19,605 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:45:19,605 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:45:19,607 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:45:19,893 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/qg18nak6-media/images/validation_288_d322f4dde2e9d2d4c2d0.png 2023-07-11 18:45:19,901 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/vzgdd4zj-media/images/validation_288_7092f5d1e02e7ebba8fb.png 2023-07-11 18:45:19,914 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/v9jguvyj-media/images/validation_288_e8cc62c8cd0593a12258.png 2023-07-11 18:45:19,915 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/94ro8x21-media/images/validation_288_2a46f6447f317761e631.png 2023-07-11 18:45:20,502 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:45:20,696 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:45:20,697 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_288_e8cc62c8cd0593a12258.png 2023-07-11 18:45:20,697 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_288_2a46f6447f317761e631.png 2023-07-11 18:45:20,698 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_288_d322f4dde2e9d2d4c2d0.png 2023-07-11 18:45:20,698 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_288_7092f5d1e02e7ebba8fb.png 2023-07-11 18:45:20,699 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 18:45:21,101 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/zbs7l7bb-media/images/validation_288_d322f4dde2e9d2d4c2d0.png 2023-07-11 18:45:21,102 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/zqsp1885-media/images/validation_288_e8cc62c8cd0593a12258.png 2023-07-11 18:45:21,133 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/mm346j0b-media/images/validation_288_7092f5d1e02e7ebba8fb.png 2023-07-11 18:45:21,133 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/gmkam6cq-media/images/validation_288_2a46f6447f317761e631.png 2023-07-11 18:45:22,701 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:45:23,489 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:45:24,707 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:45:26,287 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:45:26,712 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:45:27,430 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:45:27,432 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:45:27,433 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:45:27,434 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:45:28,717 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:45:28,718 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:45:30,723 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:45:30,760 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:45:30,761 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:45:30,932 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:45:30,934 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:45:30,934 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:45:30,936 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:45:31,486 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:45:32,728 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:45:32,729 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:45:34,185 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:45:34,187 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:45:34,187 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:45:34,189 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:45:34,733 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:45:34,734 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:45:36,739 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:45:36,807 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:45:37,274 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:45:37,276 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:45:37,276 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:45:37,278 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:45:38,742 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:45:38,742 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:45:40,486 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 18:45:40,682 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:45:40,683 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:45:40,684 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:45:40,685 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:45:40,745 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:45:40,745 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:45:42,363 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:45:42,751 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:45:44,145 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:45:44,146 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:45:44,147 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:45:44,148 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:45:44,426 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:45:44,428 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:45:44,428 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:45:44,430 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:45:44,754 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:45:44,755 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:45:45,760 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:45:45,761 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:45:46,757 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:45:47,853 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:45:48,761 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:45:52,854 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:45:57,239 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:45:57,239 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_296_9e30465455fb3e14420c.png with policy now 2023-07-11 18:45:57,249 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:45:57,249 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_296_ce669110321002a765f4.png with policy now 2023-07-11 18:45:57,258 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:45:57,258 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_296_1ed73a63f24513e30257.png with policy now 2023-07-11 18:45:57,268 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:45:57,268 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_296_88dc791f343412ad2af1.png with policy now 2023-07-11 18:45:57,313 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:45:57,314 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:45:57,315 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:45:57,316 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:45:57,557 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/7kvf6gpd-media/images/validation_296_9e30465455fb3e14420c.png 2023-07-11 18:45:57,565 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/yq0pwxb6-media/images/validation_296_ce669110321002a765f4.png 2023-07-11 18:45:57,588 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/u8gmvl5g-media/images/validation_296_1ed73a63f24513e30257.png 2023-07-11 18:45:57,604 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/k4fuylmu-media/images/validation_296_88dc791f343412ad2af1.png 2023-07-11 18:45:58,317 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:45:58,778 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:45:58,779 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_296_ce669110321002a765f4.png 2023-07-11 18:45:58,779 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_296_88dc791f343412ad2af1.png 2023-07-11 18:45:58,780 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_296_1ed73a63f24513e30257.png 2023-07-11 18:45:58,780 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_296_9e30465455fb3e14420c.png 2023-07-11 18:45:58,781 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 18:45:59,104 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/yrp0don7-media/images/validation_296_ce669110321002a765f4.png 2023-07-11 18:45:59,120 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/o3m1zdg0-media/images/validation_296_1ed73a63f24513e30257.png 2023-07-11 18:45:59,126 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/6kivjwzk-media/images/validation_296_9e30465455fb3e14420c.png 2023-07-11 18:45:59,143 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/92ad5fye-media/images/validation_296_88dc791f343412ad2af1.png 2023-07-11 18:46:00,760 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:46:00,761 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:46:00,781 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:46:01,724 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:46:02,785 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:46:03,823 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:46:04,511 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:46:04,513 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:46:04,514 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:46:04,515 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:46:04,788 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:46:04,789 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:46:06,795 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:46:08,375 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:46:08,377 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:46:08,378 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:46:08,379 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:46:08,801 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:46:08,802 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:46:09,381 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:46:10,489 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 18:46:10,806 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:46:12,110 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:46:12,112 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:46:12,112 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:46:12,114 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:46:12,812 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:46:12,812 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:46:14,817 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:46:15,055 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:46:15,058 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:46:15,058 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:46:15,058 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:46:15,060 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:46:16,151 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:46:16,152 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:46:16,822 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:46:16,823 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:46:18,828 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:46:18,862 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:46:18,863 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:46:18,863 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:46:18,865 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:46:20,305 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:46:20,833 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:46:20,834 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:46:21,156 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:46:21,158 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:46:21,158 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:46:21,160 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:46:21,438 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:46:21,440 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:46:21,440 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:46:21,442 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:46:22,839 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:46:22,840 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:46:24,844 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:46:25,988 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:46:30,989 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:46:31,188 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:46:31,189 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:46:34,000 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:46:34,000 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_304_ce2d8ab42d2884a6a507.png with policy now 2023-07-11 18:46:34,015 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:46:34,016 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_304_acbea64c9e5d9aa9013b.png with policy now 2023-07-11 18:46:34,019 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:46:34,019 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_304_408941d3a9aad78e3825.png with policy now 2023-07-11 18:46:34,033 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:46:34,033 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_304_8cb43723394c9698f98b.png with policy now 2023-07-11 18:46:34,070 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:46:34,072 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:46:34,073 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:46:34,074 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:46:34,306 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/qne6p083-media/images/validation_304_ce2d8ab42d2884a6a507.png 2023-07-11 18:46:34,322 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/stom8rar-media/images/validation_304_acbea64c9e5d9aa9013b.png 2023-07-11 18:46:34,330 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ssgqjqpa-media/images/validation_304_8cb43723394c9698f98b.png 2023-07-11 18:46:34,332 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/umoq0jgc-media/images/validation_304_408941d3a9aad78e3825.png 2023-07-11 18:46:34,869 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:46:34,870 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_304_408941d3a9aad78e3825.png 2023-07-11 18:46:34,870 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_304_ce2d8ab42d2884a6a507.png 2023-07-11 18:46:34,871 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_304_acbea64c9e5d9aa9013b.png 2023-07-11 18:46:34,871 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_304_8cb43723394c9698f98b.png 2023-07-11 18:46:34,872 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 18:46:35,205 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/iwtjm8cb-media/images/validation_304_408941d3a9aad78e3825.png 2023-07-11 18:46:35,227 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/08s7f9e9-media/images/validation_304_ce2d8ab42d2884a6a507.png 2023-07-11 18:46:35,237 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/iwqqdfh1-media/images/validation_304_8cb43723394c9698f98b.png 2023-07-11 18:46:35,469 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/sndx8f2s-media/images/validation_304_acbea64c9e5d9aa9013b.png 2023-07-11 18:46:36,621 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:46:37,340 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:46:38,880 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:46:40,234 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:46:40,236 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:46:40,236 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:46:40,238 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:46:40,492 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 18:46:40,885 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:46:40,886 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:46:42,548 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:46:42,890 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:46:43,321 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:46:43,323 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:46:43,324 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:46:43,325 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:46:44,896 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:46:44,897 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:46:46,235 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:46:46,236 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:46:46,902 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:46:46,981 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:46:46,983 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:46:46,983 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:46:46,984 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:46:47,678 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:46:48,908 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:46:48,909 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:46:50,284 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:46:50,286 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:46:50,286 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:46:50,288 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:46:50,914 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:46:50,914 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:46:52,919 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:46:52,933 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:46:54,034 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:46:54,036 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:46:54,036 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:46:54,038 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:46:54,925 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:46:54,925 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:46:56,931 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:46:57,836 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:46:57,838 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:46:57,838 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:46:57,840 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:46:58,118 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:46:58,119 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:46:58,122 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:46:58,122 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:46:58,124 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:46:58,937 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:46:58,938 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:47:00,942 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:47:01,235 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:47:01,235 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:47:03,284 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:47:08,285 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:47:10,495 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 18:47:10,736 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:47:10,736 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_312_d97fb722dbbfe3980d7a.png with policy now 2023-07-11 18:47:10,777 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:47:10,777 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_312_ae0fbaa0b9e169ccfaf7.png with policy now 2023-07-11 18:47:10,778 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:47:10,778 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_312_3f17f5fe28a713544173.png with policy now 2023-07-11 18:47:10,779 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:47:10,779 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_312_8d287303a5c0f6c9b182.png with policy now 2023-07-11 18:47:10,808 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:47:10,809 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:47:10,810 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:47:10,811 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:47:10,968 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:47:10,969 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_312_3f17f5fe28a713544173.png 2023-07-11 18:47:10,969 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_312_8d287303a5c0f6c9b182.png 2023-07-11 18:47:10,970 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_312_ae0fbaa0b9e169ccfaf7.png 2023-07-11 18:47:10,970 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_312_d97fb722dbbfe3980d7a.png 2023-07-11 18:47:10,971 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 18:47:11,084 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/0sqm5y8h-media/images/validation_312_3f17f5fe28a713544173.png 2023-07-11 18:47:11,092 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/k5bu0hxt-media/images/validation_312_d97fb722dbbfe3980d7a.png 2023-07-11 18:47:11,093 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/1wxv78fm-media/images/validation_312_ae0fbaa0b9e169ccfaf7.png 2023-07-11 18:47:11,108 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/vk5ar6rt-media/images/validation_312_8d287303a5c0f6c9b182.png 2023-07-11 18:47:11,454 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/pbumhxq5-media/images/validation_312_3f17f5fe28a713544173.png 2023-07-11 18:47:11,499 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/6iwywsfj-media/images/validation_312_ae0fbaa0b9e169ccfaf7.png 2023-07-11 18:47:11,499 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/kjuxbjok-media/images/validation_312_d97fb722dbbfe3980d7a.png 2023-07-11 18:47:11,812 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/jwx5k6jl-media/images/validation_312_8d287303a5c0f6c9b182.png 2023-07-11 18:47:13,719 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:47:14,979 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:47:15,028 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:47:16,235 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:47:16,235 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:47:16,985 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:47:18,113 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:47:18,115 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:47:18,115 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:47:18,116 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:47:18,772 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:47:18,990 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:47:18,991 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:47:20,996 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:47:21,002 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:47:21,004 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:47:21,004 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:47:21,006 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:47:23,000 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:47:23,000 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:47:24,282 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:47:25,005 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:47:25,041 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:47:25,042 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:47:25,042 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:47:25,043 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:47:27,011 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:47:27,011 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:47:28,020 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:47:28,022 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:47:28,022 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:47:28,024 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:47:29,016 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:47:29,017 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:47:29,975 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:47:31,022 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:47:31,235 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:47:31,236 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:47:31,360 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:47:31,362 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:47:31,362 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:47:31,364 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:47:33,028 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:47:33,028 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:47:33,722 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:47:33,723 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:47:33,724 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:47:33,725 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:47:34,005 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:47:34,007 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:47:34,007 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:47:34,009 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:47:35,034 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:47:35,034 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:47:35,544 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:47:37,040 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:47:40,498 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 18:47:41,499 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:47:46,236 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:47:46,237 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:47:46,628 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:47:46,628 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_320_ca6b8df2ff2f328c13ab.png with policy now 2023-07-11 18:47:46,629 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:47:46,673 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:47:46,673 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_320_787669ccfd6f2189c2e9.png with policy now 2023-07-11 18:47:46,674 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:47:46,674 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_320_6fac76e86a9c94c7429f.png with policy now 2023-07-11 18:47:46,675 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:47:46,675 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_320_59eb061cba639d911944.png with policy now 2023-07-11 18:47:46,703 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:47:46,705 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:47:46,705 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:47:46,706 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:47:46,943 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/mo9lnk83-media/images/validation_320_ca6b8df2ff2f328c13ab.png 2023-07-11 18:47:46,983 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/es5vklwz-media/images/validation_320_787669ccfd6f2189c2e9.png 2023-07-11 18:47:47,016 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/p9az94cx-media/images/validation_320_6fac76e86a9c94c7429f.png 2023-07-11 18:47:47,023 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/dy1rewak-media/images/validation_320_59eb061cba639d911944.png 2023-07-11 18:47:47,065 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:47:47,066 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_320_6fac76e86a9c94c7429f.png 2023-07-11 18:47:47,066 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_320_ca6b8df2ff2f328c13ab.png 2023-07-11 18:47:47,067 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_320_59eb061cba639d911944.png 2023-07-11 18:47:47,067 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_320_787669ccfd6f2189c2e9.png 2023-07-11 18:47:47,068 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 18:47:47,431 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/jptnxye4-media/images/validation_320_6fac76e86a9c94c7429f.png 2023-07-11 18:47:47,439 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/s7lyquv6-media/images/validation_320_ca6b8df2ff2f328c13ab.png 2023-07-11 18:47:47,464 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/sc1r7ql0-media/images/validation_320_787669ccfd6f2189c2e9.png 2023-07-11 18:47:47,677 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/7s9qaz3n-media/images/validation_320_59eb061cba639d911944.png 2023-07-11 18:47:50,397 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:47:51,076 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:47:51,724 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:47:53,082 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:47:53,450 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:47:53,452 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:47:53,452 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:47:53,454 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:47:55,087 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:47:55,088 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:47:56,742 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:47:56,786 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:47:56,788 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:47:56,788 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:47:56,790 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:47:57,093 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:47:57,094 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:47:59,099 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:47:59,905 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:47:59,907 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:47:59,907 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:47:59,909 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:48:01,104 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:48:01,105 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:48:01,236 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:48:01,236 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:48:01,998 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:48:03,094 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:48:03,096 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:48:03,096 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:48:03,097 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:48:03,109 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:48:03,109 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:48:05,115 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:48:06,973 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:48:06,975 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:48:06,976 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:48:06,977 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:48:07,120 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:48:07,121 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:48:07,833 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:48:09,128 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:48:10,502 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 18:48:11,134 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:48:11,266 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:48:11,267 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:48:11,267 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:48:11,269 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:48:11,550 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:48:11,552 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:48:11,553 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:48:11,554 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:48:13,112 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:48:13,140 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:48:13,141 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:48:15,146 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:48:16,246 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:48:16,247 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:48:18,287 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:48:23,288 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:48:24,125 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:48:24,125 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_328_9a95e0071e58dbf2382e.png with policy now 2023-07-11 18:48:24,169 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:48:24,169 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_328_a295737d90d34f9422e2.png with policy now 2023-07-11 18:48:24,170 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:48:24,170 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_328_d261d0f406cd6d609cb4.png with policy now 2023-07-11 18:48:24,171 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:48:24,171 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_328_5ee531180049cce5568d.png with policy now 2023-07-11 18:48:24,196 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:48:24,197 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:48:24,199 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:48:24,202 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:48:24,443 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/hj92w6v5-media/images/validation_328_9a95e0071e58dbf2382e.png 2023-07-11 18:48:24,500 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/xixsoasb-media/images/validation_328_d261d0f406cd6d609cb4.png 2023-07-11 18:48:24,511 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/r84hbccj-media/images/validation_328_a295737d90d34f9422e2.png 2023-07-11 18:48:24,528 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/nmy6uvhx-media/images/validation_328_5ee531180049cce5568d.png 2023-07-11 18:48:25,172 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:48:25,173 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_328_d261d0f406cd6d609cb4.png 2023-07-11 18:48:25,173 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_328_a295737d90d34f9422e2.png 2023-07-11 18:48:25,174 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_328_9a95e0071e58dbf2382e.png 2023-07-11 18:48:25,175 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_328_5ee531180049cce5568d.png 2023-07-11 18:48:25,175 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 18:48:25,523 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/9hwcbqfv-media/images/validation_328_a295737d90d34f9422e2.png 2023-07-11 18:48:25,524 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/0ah1lxrl-media/images/validation_328_d261d0f406cd6d609cb4.png 2023-07-11 18:48:25,527 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/cqj04u11-media/images/validation_328_9a95e0071e58dbf2382e.png 2023-07-11 18:48:25,588 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/0vf22g7d-media/images/validation_328_5ee531180049cce5568d.png 2023-07-11 18:48:28,416 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:48:28,417 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:48:29,183 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:48:31,189 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:48:31,246 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:48:31,247 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:48:32,008 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:48:32,010 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:48:32,010 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:48:32,018 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:48:33,195 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:48:33,195 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:48:33,590 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:48:35,117 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:48:35,119 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:48:35,120 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:48:35,121 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:48:35,201 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:48:35,201 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:48:37,207 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:48:37,684 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:48:37,686 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:48:37,686 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:48:37,687 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:48:38,602 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:48:39,213 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:48:39,214 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:48:40,503 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 18:48:41,218 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:48:41,864 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:48:41,865 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:48:41,866 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:48:41,867 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:48:43,224 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:48:43,225 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:48:43,971 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:48:45,230 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:48:45,293 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:48:45,295 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:48:45,295 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:48:45,297 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:48:46,336 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:48:46,337 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:48:47,236 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:48:47,237 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:48:48,360 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:48:48,362 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:48:48,362 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:48:48,364 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:48:48,649 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:48:48,651 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:48:48,652 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:48:48,653 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:48:49,040 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:48:49,243 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:48:49,243 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:48:51,248 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:48:54,179 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:48:59,180 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:49:01,239 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:49:01,240 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_336_0ffc6176fb6d7e1c0839.png with policy now 2023-07-11 18:49:01,250 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:49:01,250 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_336_d3ae5109418d504e15a5.png with policy now 2023-07-11 18:49:01,263 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:49:01,263 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_336_a39fd3fa08d829b242c5.png with policy now 2023-07-11 18:49:01,272 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:49:01,273 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_336_db973e73f7779e6592b8.png with policy now 2023-07-11 18:49:01,284 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_336_db973e73f7779e6592b8.png 2023-07-11 18:49:01,285 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_336_d3ae5109418d504e15a5.png 2023-07-11 18:49:01,285 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_336_0ffc6176fb6d7e1c0839.png 2023-07-11 18:49:01,285 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_336_a39fd3fa08d829b242c5.png 2023-07-11 18:49:01,286 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 18:49:01,308 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:49:01,309 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:49:01,309 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:49:01,312 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:49:01,345 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:49:01,346 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:49:01,561 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/qdj6d7s8-media/images/validation_336_0ffc6176fb6d7e1c0839.png 2023-07-11 18:49:01,596 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/xhiaicvc-media/images/validation_336_d3ae5109418d504e15a5.png 2023-07-11 18:49:01,603 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/gqf6exb1-media/images/validation_336_a39fd3fa08d829b242c5.png 2023-07-11 18:49:01,611 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/awg8yzs6-media/images/validation_336_db973e73f7779e6592b8.png 2023-07-11 18:49:01,944 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/rn3o1m7z-media/images/validation_336_a39fd3fa08d829b242c5.png 2023-07-11 18:49:01,944 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ekxymqa2-media/images/validation_336_0ffc6176fb6d7e1c0839.png 2023-07-11 18:49:02,207 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/56m8jg1e-media/images/validation_336_d3ae5109418d504e15a5.png 2023-07-11 18:49:02,289 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/5oklkwh8-media/images/validation_336_db973e73f7779e6592b8.png 2023-07-11 18:49:03,289 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:49:04,268 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:49:05,141 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:49:05,295 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:49:07,301 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:49:08,917 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:49:08,919 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:49:08,919 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:49:08,920 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:49:09,307 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:49:09,307 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:49:09,661 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:49:10,506 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 18:49:11,313 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:49:11,761 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:49:11,763 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:49:11,763 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:49:11,765 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:49:13,319 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:49:13,319 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:49:14,811 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:49:14,813 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:49:14,813 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:49:14,814 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:49:14,815 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:49:15,325 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:49:15,325 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:49:16,345 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:49:16,345 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:49:17,330 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:49:17,710 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:49:17,713 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:49:17,713 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:49:17,715 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:49:19,336 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:49:19,336 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:49:20,248 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:49:20,995 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:49:20,996 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:49:20,997 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:49:20,998 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:49:21,341 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:49:21,342 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:49:23,347 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:49:25,354 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:49:25,373 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:49:25,413 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:49:25,415 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:49:25,416 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:49:25,417 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:49:25,700 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:49:25,702 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:49:25,702 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:49:25,704 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:49:27,360 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:49:27,361 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:49:29,366 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:49:31,246 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:49:31,352 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:49:31,353 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:49:36,418 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:49:38,275 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:49:38,275 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_344_f5043cd63cbfcc5b803e.png with policy now 2023-07-11 18:49:38,317 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:49:38,317 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_344_f73ffb4d2e046ab554c4.png with policy now 2023-07-11 18:49:38,318 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:49:38,318 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_344_cea16b4401e675df5e86.png with policy now 2023-07-11 18:49:38,319 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:49:38,319 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_344_6aa625d41ec15aca550a.png with policy now 2023-07-11 18:49:38,349 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:49:38,351 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:49:38,351 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:49:38,353 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:49:38,609 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/t5pxhh7w-media/images/validation_344_f5043cd63cbfcc5b803e.png 2023-07-11 18:49:38,644 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/zjtmqze7-media/images/validation_344_6aa625d41ec15aca550a.png 2023-07-11 18:49:38,646 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/vvxm38ha-media/images/validation_344_f73ffb4d2e046ab554c4.png 2023-07-11 18:49:38,654 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/lnk6j0zx-media/images/validation_344_cea16b4401e675df5e86.png 2023-07-11 18:49:39,392 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:49:39,393 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_344_6aa625d41ec15aca550a.png 2023-07-11 18:49:39,394 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_344_f5043cd63cbfcc5b803e.png 2023-07-11 18:49:39,394 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_344_cea16b4401e675df5e86.png 2023-07-11 18:49:39,395 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_344_f73ffb4d2e046ab554c4.png 2023-07-11 18:49:39,395 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 18:49:39,744 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/wxwf3q5v-media/images/validation_344_f5043cd63cbfcc5b803e.png 2023-07-11 18:49:39,744 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/t8p5imhl-media/images/validation_344_6aa625d41ec15aca550a.png 2023-07-11 18:49:39,770 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/3ubadmi7-media/images/validation_344_f73ffb4d2e046ab554c4.png 2023-07-11 18:49:39,790 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/h6douro2-media/images/validation_344_cea16b4401e675df5e86.png 2023-07-11 18:49:40,508 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 18:49:41,398 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:49:42,058 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:49:42,202 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:49:43,405 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:49:45,053 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:49:45,054 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:49:45,055 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:49:45,056 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:49:45,410 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:49:45,411 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:49:46,352 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:49:46,353 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:49:47,416 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:49:47,747 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:49:48,908 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:49:48,909 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:49:48,910 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:49:48,911 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:49:49,422 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:49:49,423 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:49:51,428 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:49:51,602 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:49:51,603 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:49:51,604 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:49:51,605 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:49:52,833 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:49:53,434 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:49:53,435 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:49:55,440 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:49:55,771 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:49:55,773 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:49:55,773 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:49:55,775 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:49:57,446 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:49:57,446 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:49:57,982 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:49:58,911 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:49:58,913 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:49:58,913 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:49:58,915 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:49:59,451 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:49:59,452 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:50:01,458 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:50:01,619 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:50:01,619 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:50:01,745 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:50:01,746 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:50:01,747 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:50:01,748 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:50:02,026 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:50:02,028 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:50:02,028 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:50:02,030 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:50:03,464 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:50:03,464 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:50:03,692 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:50:05,470 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:50:08,693 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:50:10,511 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 18:50:14,514 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:50:14,741 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:50:14,741 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_352_816057c5787d207de5ff.png with policy now 2023-07-11 18:50:14,755 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:50:14,756 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_352_6788b6099bc68d956e71.png with policy now 2023-07-11 18:50:14,765 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:50:14,765 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_352_8c3b42ad97f04b9342e2.png with policy now 2023-07-11 18:50:14,775 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:50:14,775 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_352_3a044ff8368fa9c583c0.png with policy now 2023-07-11 18:50:14,813 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:50:14,814 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:50:14,815 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:50:14,817 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:50:15,046 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/jeyc019q-media/images/validation_352_816057c5787d207de5ff.png 2023-07-11 18:50:15,088 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/8u744at4-media/images/validation_352_8c3b42ad97f04b9342e2.png 2023-07-11 18:50:15,101 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/z8kbya1b-media/images/validation_352_6788b6099bc68d956e71.png 2023-07-11 18:50:15,165 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/uedesb20-media/images/validation_352_3a044ff8368fa9c583c0.png 2023-07-11 18:50:15,499 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:50:15,499 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_352_3a044ff8368fa9c583c0.png 2023-07-11 18:50:15,500 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_352_6788b6099bc68d956e71.png 2023-07-11 18:50:15,500 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_352_8c3b42ad97f04b9342e2.png 2023-07-11 18:50:15,501 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_352_816057c5787d207de5ff.png 2023-07-11 18:50:15,501 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 18:50:15,846 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/y1cn6ghz-media/images/validation_352_3a044ff8368fa9c583c0.png 2023-07-11 18:50:15,853 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/xsmuzac2-media/images/validation_352_8c3b42ad97f04b9342e2.png 2023-07-11 18:50:15,862 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/6ad6jvrb-media/images/validation_352_816057c5787d207de5ff.png 2023-07-11 18:50:15,874 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/1l659tc5-media/images/validation_352_6788b6099bc68d956e71.png 2023-07-11 18:50:16,618 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:50:16,619 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:50:18,826 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:50:19,511 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:50:19,550 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:50:21,386 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:50:21,388 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:50:21,388 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:50:21,390 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:50:21,517 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:50:21,517 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:50:23,523 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:50:24,559 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:50:24,653 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:50:24,654 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:50:24,655 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:50:24,656 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:50:25,529 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:50:25,530 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:50:27,535 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:50:28,292 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:50:28,294 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:50:28,294 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:50:28,296 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:50:29,541 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:50:29,541 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:50:29,872 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:50:31,547 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:50:31,724 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:50:31,725 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:50:31,849 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:50:31,850 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:50:31,851 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:50:31,852 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:50:33,553 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:50:33,553 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:50:34,471 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:50:34,473 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:50:34,473 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:50:34,475 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:50:35,476 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:50:35,558 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:50:35,559 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:50:37,565 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:50:38,163 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:50:38,165 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:50:38,165 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:50:38,167 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:50:38,445 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:50:38,446 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:50:38,447 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:50:38,448 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:50:39,571 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:50:39,572 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:50:40,514 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 18:50:40,519 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:50:41,577 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:50:45,517 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:50:46,724 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:50:46,725 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:50:50,772 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:50:51,073 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:50:51,074 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_360_c20dcd4a948eaf4fa197.png with policy now 2023-07-11 18:50:51,085 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:50:51,085 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_360_9a31898c9077cdd9a41d.png with policy now 2023-07-11 18:50:51,098 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:50:51,099 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_360_e2e4e89dbfed8848cd88.png with policy now 2023-07-11 18:50:51,109 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:50:51,109 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_360_c8519aaa69a9bd44cd79.png with policy now 2023-07-11 18:50:51,144 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:50:51,146 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:50:51,146 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:50:51,148 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:50:51,401 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/9pse80h1-media/images/validation_360_c20dcd4a948eaf4fa197.png 2023-07-11 18:50:51,412 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/9t8x6pwh-media/images/validation_360_e2e4e89dbfed8848cd88.png 2023-07-11 18:50:51,426 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/hgg2a7tp-media/images/validation_360_9a31898c9077cdd9a41d.png 2023-07-11 18:50:51,439 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/fgb7tyg6-media/images/validation_360_c8519aaa69a9bd44cd79.png 2023-07-11 18:50:51,605 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:50:51,605 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_360_9a31898c9077cdd9a41d.png 2023-07-11 18:50:51,606 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_360_c8519aaa69a9bd44cd79.png 2023-07-11 18:50:51,606 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_360_e2e4e89dbfed8848cd88.png 2023-07-11 18:50:51,607 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_360_c20dcd4a948eaf4fa197.png 2023-07-11 18:50:51,607 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 18:50:51,961 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/d07wimod-media/images/validation_360_e2e4e89dbfed8848cd88.png 2023-07-11 18:50:51,970 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/35a20kd3-media/images/validation_360_9a31898c9077cdd9a41d.png 2023-07-11 18:50:51,986 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/jmh37llx-media/images/validation_360_c8519aaa69a9bd44cd79.png 2023-07-11 18:50:52,126 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/w64j1htg-media/images/validation_360_c20dcd4a948eaf4fa197.png 2023-07-11 18:50:55,616 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:50:56,164 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:50:56,408 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:50:57,622 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:50:59,272 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:50:59,274 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:50:59,274 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:50:59,276 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:50:59,628 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:50:59,629 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:51:01,635 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:51:01,841 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:51:01,841 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:51:01,842 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:51:02,593 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:51:02,594 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:51:02,594 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:51:02,596 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:51:03,640 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:51:03,641 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:51:05,646 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:51:06,103 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:51:06,104 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:51:06,105 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:51:06,106 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:51:06,907 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:51:07,652 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:51:07,653 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:51:09,567 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:51:09,569 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:51:09,569 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:51:09,571 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:51:09,658 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:51:09,659 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:51:10,517 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 18:51:11,664 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:51:12,037 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:51:12,961 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:51:12,963 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:51:12,963 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:51:12,965 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:51:13,671 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:51:13,671 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:51:15,677 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:51:16,496 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:51:16,497 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:51:16,498 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:51:16,499 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:51:16,779 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:51:16,781 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:51:16,781 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:51:16,783 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:51:16,841 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:51:16,842 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:51:17,226 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:51:17,684 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:51:17,684 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:51:19,690 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:51:22,366 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:51:27,368 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:51:29,383 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:51:29,383 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_368_e3360ff6b5d76e8dac77.png with policy now 2023-07-11 18:51:29,397 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:51:29,398 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_368_f4da615e9afd2c2cd8b4.png with policy now 2023-07-11 18:51:29,411 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:51:29,411 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_368_718865898338ad2310bd.png with policy now 2023-07-11 18:51:29,420 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:51:29,420 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_368_72cbe5ebdd0210e3d6fe.png with policy now 2023-07-11 18:51:29,461 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:51:29,463 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:51:29,463 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:51:29,471 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:51:29,716 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/j8y74t82-media/images/validation_368_e3360ff6b5d76e8dac77.png 2023-07-11 18:51:29,719 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/h22d3wms-media/images/validation_368_f4da615e9afd2c2cd8b4.png 2023-07-11 18:51:29,721 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:51:29,722 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_368_718865898338ad2310bd.png 2023-07-11 18:51:29,722 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_368_f4da615e9afd2c2cd8b4.png 2023-07-11 18:51:29,723 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_368_e3360ff6b5d76e8dac77.png 2023-07-11 18:51:29,723 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_368_72cbe5ebdd0210e3d6fe.png 2023-07-11 18:51:29,724 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 18:51:29,734 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/gk1z37jg-media/images/validation_368_718865898338ad2310bd.png 2023-07-11 18:51:29,741 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/atkek9yl-media/images/validation_368_72cbe5ebdd0210e3d6fe.png 2023-07-11 18:51:30,080 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/pf8c7zzc-media/images/validation_368_718865898338ad2310bd.png 2023-07-11 18:51:30,100 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/clshp3c3-media/images/validation_368_f4da615e9afd2c2cd8b4.png 2023-07-11 18:51:30,109 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ot87wucp-media/images/validation_368_72cbe5ebdd0210e3d6fe.png 2023-07-11 18:51:30,205 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/4hyjsqsh-media/images/validation_368_e3360ff6b5d76e8dac77.png 2023-07-11 18:51:31,841 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:51:31,842 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:51:32,802 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:51:32,803 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:51:33,733 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:51:35,739 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:51:36,869 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:51:36,870 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:51:36,871 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:51:36,872 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:51:37,745 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:51:37,746 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:51:37,873 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:51:39,752 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:51:40,283 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:51:40,285 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:51:40,285 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:51:40,287 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:51:40,521 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 18:51:41,758 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:51:41,758 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:51:43,539 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:51:43,544 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:51:43,546 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:51:43,546 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:51:43,548 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:51:43,764 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:51:43,764 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:51:45,770 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:51:46,841 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:51:46,842 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:51:47,243 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:51:47,245 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:51:47,246 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:51:47,247 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:51:47,776 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:51:47,776 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:51:49,385 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:51:49,782 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:51:51,247 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:51:51,248 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:51:51,249 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:51:51,250 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:51:51,788 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:51:51,789 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:51:53,795 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:51:54,812 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:51:54,814 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:51:54,814 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:51:54,815 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:51:54,816 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:51:55,095 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:51:55,097 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:51:55,098 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:51:55,099 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:51:55,801 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:51:55,802 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:51:57,807 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:52:00,697 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:52:01,852 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:52:01,853 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:52:05,889 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:52:07,732 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:52:07,733 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_376_c945a66b43ba7bfe4fc4.png with policy now 2023-07-11 18:52:07,773 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:52:07,773 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_376_1c6e0b1c9c4a02e2487e.png with policy now 2023-07-11 18:52:07,775 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:52:07,775 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_376_e09462e3bcde1058d20b.png with policy now 2023-07-11 18:52:07,791 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:52:07,792 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_376_05d73dd8573520eeccd6.png with policy now 2023-07-11 18:52:07,831 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_376_c945a66b43ba7bfe4fc4.png 2023-07-11 18:52:07,832 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_376_05d73dd8573520eeccd6.png 2023-07-11 18:52:07,833 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_376_e09462e3bcde1058d20b.png 2023-07-11 18:52:07,833 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_376_1c6e0b1c9c4a02e2487e.png 2023-07-11 18:52:07,834 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 18:52:07,834 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:52:07,835 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:52:07,836 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:52:07,837 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:52:08,058 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ximcvyru-media/images/validation_376_c945a66b43ba7bfe4fc4.png 2023-07-11 18:52:08,087 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/9dgzt086-media/images/validation_376_1c6e0b1c9c4a02e2487e.png 2023-07-11 18:52:08,093 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/64vj4g8g-media/images/validation_376_05d73dd8573520eeccd6.png 2023-07-11 18:52:08,105 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/iucs4u0l-media/images/validation_376_e09462e3bcde1058d20b.png 2023-07-11 18:52:08,382 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/kx0yex4u-media/images/validation_376_c945a66b43ba7bfe4fc4.png 2023-07-11 18:52:08,452 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/3ksylb5o-media/images/validation_376_1c6e0b1c9c4a02e2487e.png 2023-07-11 18:52:08,729 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/mttk1o29-media/images/validation_376_05d73dd8573520eeccd6.png 2023-07-11 18:52:08,799 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/8q9d1542-media/images/validation_376_e09462e3bcde1058d20b.png 2023-07-11 18:52:09,835 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:52:10,523 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 18:52:11,243 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:52:11,840 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:52:11,948 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:52:13,844 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:52:15,219 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:52:15,221 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:52:15,221 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:52:15,223 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:52:15,848 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:52:15,849 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:52:16,851 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:52:16,852 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:52:16,915 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:52:17,855 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:52:18,423 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:52:18,425 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:52:18,425 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:52:18,427 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:52:19,862 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:52:19,862 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:52:21,643 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:52:21,645 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:52:21,645 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:52:21,647 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:52:21,868 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:52:21,868 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:52:22,648 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:52:23,874 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:52:24,675 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:52:24,676 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:52:24,676 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:52:24,678 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:52:25,880 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:52:25,880 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:52:27,886 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:52:27,968 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:52:28,092 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:52:28,094 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:52:28,094 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:52:28,096 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:52:29,893 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:52:29,893 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:52:31,187 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:52:31,189 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:52:31,189 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:52:31,192 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:52:31,470 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:52:31,472 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:52:31,473 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:52:31,474 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:52:31,851 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:52:31,852 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:52:31,901 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:52:31,902 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:52:33,076 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:52:33,907 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:52:38,078 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:52:40,526 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 18:52:43,528 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:52:44,156 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:52:44,157 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_384_155d4ae5c7fbde37d2d4.png with policy now 2023-07-11 18:52:44,174 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:52:44,174 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_384_11cb9d1bc6f6a97d0955.png with policy now 2023-07-11 18:52:44,188 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:52:44,188 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_384_7a702633095b41268383.png with policy now 2023-07-11 18:52:44,201 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:52:44,202 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_384_905222ac5248c3db9c7f.png with policy now 2023-07-11 18:52:44,244 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:52:44,246 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:52:44,246 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:52:44,247 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:52:44,453 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/y0belok2-media/images/validation_384_155d4ae5c7fbde37d2d4.png 2023-07-11 18:52:44,491 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/f4t0o67n-media/images/validation_384_11cb9d1bc6f6a97d0955.png 2023-07-11 18:52:44,508 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/hr6l4c46-media/images/validation_384_7a702633095b41268383.png 2023-07-11 18:52:44,518 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/6iu3iagn-media/images/validation_384_905222ac5248c3db9c7f.png 2023-07-11 18:52:45,943 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:52:45,943 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_384_905222ac5248c3db9c7f.png 2023-07-11 18:52:45,943 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_384_11cb9d1bc6f6a97d0955.png 2023-07-11 18:52:45,944 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_384_7a702633095b41268383.png 2023-07-11 18:52:45,944 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_384_155d4ae5c7fbde37d2d4.png 2023-07-11 18:52:45,944 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 18:52:46,272 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/6kfqfahc-media/images/validation_384_905222ac5248c3db9c7f.png 2023-07-11 18:52:46,287 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/174ikzxe-media/images/validation_384_11cb9d1bc6f6a97d0955.png 2023-07-11 18:52:46,291 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/d0n2w210-media/images/validation_384_7a702633095b41268383.png 2023-07-11 18:52:46,299 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/jcaauqgr-media/images/validation_384_155d4ae5c7fbde37d2d4.png 2023-07-11 18:52:46,852 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:52:46,853 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:52:47,949 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:52:48,138 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:52:49,140 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:52:49,956 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:52:51,962 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:52:52,127 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:52:52,129 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:52:52,129 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:52:52,131 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:52:53,968 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:52:53,968 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:52:54,809 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:52:55,579 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:52:55,581 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:52:55,581 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:52:55,583 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:52:55,975 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:52:55,975 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:52:57,981 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:52:59,078 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:52:59,080 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:52:59,080 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:52:59,082 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:52:59,889 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:52:59,987 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:52:59,988 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:53:01,852 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:53:01,853 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:53:01,993 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:53:02,128 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:53:02,130 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:53:02,130 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:53:02,132 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:53:03,999 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:53:04,000 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:53:05,284 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:53:06,005 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:53:06,183 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:53:06,185 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:53:06,185 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:53:06,187 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:53:08,012 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:53:08,012 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:53:09,489 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:53:09,491 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:53:09,492 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:53:09,493 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:53:09,770 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:53:09,772 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:53:09,773 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:53:09,774 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:53:10,018 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:53:10,019 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:53:10,310 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:53:10,529 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 18:53:12,024 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:53:14,030 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:53:15,531 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:53:16,852 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:53:16,853 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:53:20,902 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:53:22,332 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:53:22,332 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_392_86c67c37eda7ead98c90.png with policy now 2023-07-11 18:53:22,366 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:53:22,366 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_392_6c8f25f05cf96b671ec4.png with policy now 2023-07-11 18:53:22,380 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:53:22,380 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_392_7d995d67c424df992085.png with policy now 2023-07-11 18:53:22,394 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:53:22,394 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_392_cba110247f071a63aa27.png with policy now 2023-07-11 18:53:22,437 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:53:22,438 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:53:22,439 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:53:22,440 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:53:22,679 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/mecawlmt-media/images/validation_392_6c8f25f05cf96b671ec4.png 2023-07-11 18:53:22,704 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/qaktoeto-media/images/validation_392_86c67c37eda7ead98c90.png 2023-07-11 18:53:22,709 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/13w36mig-media/images/validation_392_7d995d67c424df992085.png 2023-07-11 18:53:22,933 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/bcmn5c6j-media/images/validation_392_cba110247f071a63aa27.png 2023-07-11 18:53:24,060 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:53:24,061 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_392_7d995d67c424df992085.png 2023-07-11 18:53:24,061 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_392_6c8f25f05cf96b671ec4.png 2023-07-11 18:53:24,062 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_392_cba110247f071a63aa27.png 2023-07-11 18:53:24,062 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_392_86c67c37eda7ead98c90.png 2023-07-11 18:53:24,063 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 18:53:24,403 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/kz2m5xj5-media/images/validation_392_cba110247f071a63aa27.png 2023-07-11 18:53:24,414 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/e73jzjuw-media/images/validation_392_7d995d67c424df992085.png 2023-07-11 18:53:24,464 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/8z2k9rle-media/images/validation_392_86c67c37eda7ead98c90.png 2023-07-11 18:53:24,541 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/vps9axuz-media/images/validation_392_6c8f25f05cf96b671ec4.png 2023-07-11 18:53:25,242 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:53:26,066 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:53:26,589 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:53:28,524 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:53:28,526 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:53:28,526 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:53:28,528 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:53:29,071 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:53:29,072 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:53:32,078 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:53:32,080 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:53:32,449 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:53:32,449 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:53:32,620 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:53:32,622 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:53:32,622 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:53:32,624 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:53:35,085 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:53:35,085 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:53:36,509 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:53:36,511 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:53:36,511 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:53:36,513 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:53:37,321 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:53:38,092 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:53:38,092 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:53:39,824 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:53:39,826 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:53:39,826 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:53:39,828 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:53:40,532 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 18:53:41,098 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:53:41,098 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:53:43,026 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:53:43,888 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:53:43,891 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:53:43,891 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:53:43,893 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:53:44,104 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:53:44,105 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:53:47,076 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:53:47,078 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:53:47,078 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:53:47,080 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:53:47,111 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:53:47,112 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:53:47,358 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:53:47,360 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:53:47,361 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:53:47,362 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:53:47,448 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:53:47,449 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:53:49,003 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:53:50,118 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:53:50,119 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:53:53,124 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:53:54,004 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:53:59,005 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:54:00,072 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:54:00,072 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_400_ed68c4b1954c2b559c72.png with policy now 2023-07-11 18:54:00,091 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:54:00,091 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_400_b541dd5a4efd2e7f476e.png with policy now 2023-07-11 18:54:00,104 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:54:00,104 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_400_1a34069e82abe9147618.png with policy now 2023-07-11 18:54:00,116 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:54:00,117 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_400_d81856892dae7f37a9ef.png with policy now 2023-07-11 18:54:00,159 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:54:00,161 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:54:00,161 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:54:00,162 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:54:00,419 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/bks3bd6c-media/images/validation_400_ed68c4b1954c2b559c72.png 2023-07-11 18:54:00,459 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/9zhrkaze-media/images/validation_400_d81856892dae7f37a9ef.png 2023-07-11 18:54:00,463 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/2cm28rib-media/images/validation_400_b541dd5a4efd2e7f476e.png 2023-07-11 18:54:00,466 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/aa35lixy-media/images/validation_400_1a34069e82abe9147618.png 2023-07-11 18:54:02,143 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:54:02,144 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_400_d81856892dae7f37a9ef.png 2023-07-11 18:54:02,144 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_400_ed68c4b1954c2b559c72.png 2023-07-11 18:54:02,145 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_400_1a34069e82abe9147618.png 2023-07-11 18:54:02,146 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_400_b541dd5a4efd2e7f476e.png 2023-07-11 18:54:02,146 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 18:54:02,448 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:54:02,449 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:54:02,493 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/c2ufhjtt-media/images/validation_400_d81856892dae7f37a9ef.png 2023-07-11 18:54:02,507 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ytilq8a8-media/images/validation_400_ed68c4b1954c2b559c72.png 2023-07-11 18:54:02,510 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/tnpuj6zc-media/images/validation_400_b541dd5a4efd2e7f476e.png 2023-07-11 18:54:02,523 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/2jv5xrbf-media/images/validation_400_1a34069e82abe9147618.png 2023-07-11 18:54:04,290 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:54:04,291 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:54:05,190 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:54:07,378 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:54:07,379 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:54:07,380 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:54:07,381 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:54:08,196 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:54:08,197 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:54:09,921 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:54:10,535 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 18:54:10,965 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:54:10,967 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:54:10,967 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:54:10,968 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:54:11,203 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:54:11,204 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:54:13,763 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:54:13,765 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:54:13,766 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:54:13,767 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:54:14,210 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:54:14,210 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:54:15,535 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:54:17,190 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:54:17,192 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:54:17,193 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:54:17,194 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:54:17,216 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:54:17,217 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:54:17,782 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:54:17,783 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:54:20,223 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:54:20,484 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:54:20,486 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:54:20,487 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:54:20,488 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:54:21,489 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:54:23,230 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:54:23,230 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:54:25,056 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:54:25,058 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:54:25,059 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:54:25,060 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:54:25,338 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:54:25,339 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:54:25,340 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:54:25,341 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:54:26,237 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:54:26,238 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:54:27,003 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:54:29,244 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:54:32,004 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:54:32,812 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:54:32,812 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:54:37,854 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:54:38,009 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:54:38,009 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_408_dea2f3fa8cc40610700c.png with policy now 2023-07-11 18:54:38,024 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:54:38,024 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_408_8049b1e57a48ac722fb3.png with policy now 2023-07-11 18:54:38,037 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:54:38,037 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_408_db3da578241ccb488fb6.png with policy now 2023-07-11 18:54:38,051 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:54:38,051 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_408_5100cba967fc4cc1cdf6.png with policy now 2023-07-11 18:54:38,095 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:54:38,098 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:54:38,098 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:54:38,100 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:54:38,263 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:54:38,263 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_408_8049b1e57a48ac722fb3.png 2023-07-11 18:54:38,264 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_408_db3da578241ccb488fb6.png 2023-07-11 18:54:38,265 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_408_5100cba967fc4cc1cdf6.png 2023-07-11 18:54:38,265 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_408_dea2f3fa8cc40610700c.png 2023-07-11 18:54:38,265 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 18:54:38,319 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/8x0xqpdr-media/images/validation_408_dea2f3fa8cc40610700c.png 2023-07-11 18:54:38,333 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/zirgpjyx-media/images/validation_408_8049b1e57a48ac722fb3.png 2023-07-11 18:54:38,354 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/kqsu2vs2-media/images/validation_408_5100cba967fc4cc1cdf6.png 2023-07-11 18:54:38,362 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/m8bxvwbe-media/images/validation_408_db3da578241ccb488fb6.png 2023-07-11 18:54:38,819 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/mzwys88q-media/images/validation_408_5100cba967fc4cc1cdf6.png 2023-07-11 18:54:38,833 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/fgchzzzs-media/images/validation_408_dea2f3fa8cc40610700c.png 2023-07-11 18:54:39,171 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/jbsu2x7r-media/images/validation_408_db3da578241ccb488fb6.png 2023-07-11 18:54:39,264 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/7ynn99xm-media/images/validation_408_8049b1e57a48ac722fb3.png 2023-07-11 18:54:40,539 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 18:54:42,981 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:54:43,184 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:54:44,277 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:54:45,466 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:54:45,468 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:54:45,469 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:54:45,470 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:54:47,282 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:54:47,282 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:54:47,849 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:54:47,850 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:54:48,971 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:54:49,073 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:54:49,075 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:54:49,075 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:54:49,077 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:54:50,288 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:54:50,288 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:54:52,984 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:54:52,986 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:54:52,986 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:54:52,988 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:54:53,294 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:54:53,295 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:54:54,337 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:54:55,409 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:54:55,411 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:54:55,411 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:54:55,413 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:54:56,301 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:54:56,301 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:54:58,681 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:54:58,682 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:54:58,683 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:54:58,684 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:54:59,307 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:54:59,308 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:55:00,172 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:55:01,797 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:55:01,798 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:55:01,799 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:55:01,800 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:55:02,078 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:55:02,080 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:55:02,080 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:55:02,082 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:55:02,315 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:55:02,315 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:55:02,850 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:55:02,850 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:55:05,321 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:55:05,887 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:55:10,541 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 18:55:11,543 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:55:14,744 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:55:14,744 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_416_d31ae94219028126481b.png with policy now 2023-07-11 18:55:14,785 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:55:14,785 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_416_0fb15379db5c9ff2e0fc.png with policy now 2023-07-11 18:55:14,786 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:55:14,787 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_416_cb07e3b163172a97c565.png with policy now 2023-07-11 18:55:14,798 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:55:14,798 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_416_33b6f1678b8b6a41d991.png with policy now 2023-07-11 18:55:14,839 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:55:14,840 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:55:14,840 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:55:14,842 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:55:15,060 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ufm18ri3-media/images/validation_416_d31ae94219028126481b.png 2023-07-11 18:55:15,119 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/soud16v4-media/images/validation_416_0fb15379db5c9ff2e0fc.png 2023-07-11 18:55:15,135 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/wfmszrto-media/images/validation_416_cb07e3b163172a97c565.png 2023-07-11 18:55:15,137 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/p9cyb665-media/images/validation_416_33b6f1678b8b6a41d991.png 2023-07-11 18:55:16,797 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:55:17,349 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:55:17,349 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:55:17,349 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_416_0fb15379db5c9ff2e0fc.png 2023-07-11 18:55:17,350 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_416_cb07e3b163172a97c565.png 2023-07-11 18:55:17,351 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_416_d31ae94219028126481b.png 2023-07-11 18:55:17,351 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_416_33b6f1678b8b6a41d991.png 2023-07-11 18:55:17,351 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 18:55:17,725 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/m6otffwm-media/images/validation_416_cb07e3b163172a97c565.png 2023-07-11 18:55:17,726 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/zey54b9r-media/images/validation_416_0fb15379db5c9ff2e0fc.png 2023-07-11 18:55:17,729 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ustzrsr0-media/images/validation_416_d31ae94219028126481b.png 2023-07-11 18:55:17,735 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/xf5f5fqg-media/images/validation_416_33b6f1678b8b6a41d991.png 2023-07-11 18:55:17,849 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:55:17,850 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:55:18,620 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:55:20,356 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:55:22,527 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:55:22,579 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:55:22,581 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:55:22,581 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:55:22,583 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:55:23,363 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:55:23,363 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:55:26,278 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:55:26,280 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:55:26,280 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:55:26,282 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:55:26,369 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:55:26,370 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:55:28,083 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:55:29,376 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:55:29,840 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:55:29,842 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:55:29,842 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:55:29,844 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:55:32,383 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:55:32,383 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:55:32,880 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:55:32,881 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:55:34,000 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:55:34,068 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:55:34,070 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:55:34,070 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:55:34,072 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:55:35,389 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:55:35,390 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:55:37,363 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:55:37,365 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:55:37,366 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:55:37,367 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:55:38,396 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:55:38,397 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:55:39,372 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:55:40,070 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:55:40,072 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:55:40,073 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:55:40,075 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:55:40,353 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:55:40,354 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:55:40,355 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:55:40,356 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:55:40,544 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 18:55:41,405 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:55:41,406 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:55:44,412 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:55:45,018 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:55:47,883 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:55:47,884 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:55:50,932 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:55:53,080 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:55:53,081 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_424_33240f6708272a52eea8.png with policy now 2023-07-11 18:55:53,096 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:55:53,096 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_424_3e2d62ab8a779b41aad1.png with policy now 2023-07-11 18:55:53,111 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:55:53,111 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_424_85fe6cca04ba628c90f1.png with policy now 2023-07-11 18:55:53,125 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:55:53,126 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_424_ff3c9534a6bae344d14f.png with policy now 2023-07-11 18:55:53,170 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:55:53,171 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:55:53,172 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:55:53,173 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:55:53,411 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/f3jzzquy-media/images/validation_424_33240f6708272a52eea8.png 2023-07-11 18:55:53,433 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:55:53,433 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/k16jn45s-media/images/validation_424_85fe6cca04ba628c90f1.png 2023-07-11 18:55:53,434 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_424_85fe6cca04ba628c90f1.png 2023-07-11 18:55:53,434 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_424_3e2d62ab8a779b41aad1.png 2023-07-11 18:55:53,435 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_424_ff3c9534a6bae344d14f.png 2023-07-11 18:55:53,436 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_424_33240f6708272a52eea8.png 2023-07-11 18:55:53,436 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 18:55:53,445 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/qgwfijul-media/images/validation_424_3e2d62ab8a779b41aad1.png 2023-07-11 18:55:53,446 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/8wljk73u-media/images/validation_424_ff3c9534a6bae344d14f.png 2023-07-11 18:55:53,778 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/8fubsjbf-media/images/validation_424_85fe6cca04ba628c90f1.png 2023-07-11 18:55:53,790 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/9jehgj6l-media/images/validation_424_ff3c9534a6bae344d14f.png 2023-07-11 18:55:53,794 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/jq9fxzu8-media/images/validation_424_3e2d62ab8a779b41aad1.png 2023-07-11 18:55:53,805 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/v6ujtsug-media/images/validation_424_33240f6708272a52eea8.png 2023-07-11 18:55:56,610 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:55:57,672 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:55:59,446 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:56:00,683 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:56:00,685 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:56:00,685 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:56:00,687 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:56:02,140 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:56:02,453 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:56:02,453 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:56:02,883 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:56:02,884 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:56:03,905 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:56:03,906 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:56:03,907 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:56:03,908 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:56:05,460 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:56:05,460 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:56:06,159 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:56:06,161 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:56:06,161 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:56:06,163 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:56:07,526 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:56:08,467 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:56:08,467 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:56:09,341 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:56:09,343 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:56:09,343 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:56:09,345 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:56:10,547 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 18:56:11,473 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:56:11,474 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:56:12,686 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:56:12,826 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:56:12,827 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:56:12,828 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:56:12,829 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:56:14,480 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:56:14,480 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:56:16,165 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:56:16,167 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:56:16,167 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:56:16,169 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:56:16,447 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:56:16,449 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:56:16,450 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:56:16,451 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:56:17,487 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:56:17,488 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:56:17,883 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:56:17,884 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:56:17,884 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:56:20,494 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:56:22,920 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:56:27,922 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:56:29,001 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:56:29,001 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_432_18313983b36cde50dd3b.png with policy now 2023-07-11 18:56:29,045 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:56:29,045 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_432_9aaaaf8c3f6ad3ebcd99.png with policy now 2023-07-11 18:56:29,046 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:56:29,046 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_432_c7dc0e4c561f71eb5b3e.png with policy now 2023-07-11 18:56:29,047 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:56:29,047 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_432_0dfe0505f8d9adcc6ad1.png with policy now 2023-07-11 18:56:29,091 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:56:29,093 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:56:29,093 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:56:29,095 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:56:29,340 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/zon4z2eg-media/images/validation_432_18313983b36cde50dd3b.png 2023-07-11 18:56:29,385 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/nxqiiio0-media/images/validation_432_9aaaaf8c3f6ad3ebcd99.png 2023-07-11 18:56:29,385 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/bs26uklo-media/images/validation_432_c7dc0e4c561f71eb5b3e.png 2023-07-11 18:56:29,399 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/m5g8c56d-media/images/validation_432_0dfe0505f8d9adcc6ad1.png 2023-07-11 18:56:29,514 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:56:29,515 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_432_18313983b36cde50dd3b.png 2023-07-11 18:56:29,516 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_432_9aaaaf8c3f6ad3ebcd99.png 2023-07-11 18:56:29,516 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_432_0dfe0505f8d9adcc6ad1.png 2023-07-11 18:56:29,517 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_432_c7dc0e4c561f71eb5b3e.png 2023-07-11 18:56:29,517 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 18:56:29,902 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/aj0g8vkm-media/images/validation_432_18313983b36cde50dd3b.png 2023-07-11 18:56:29,904 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/5skgo8ce-media/images/validation_432_c7dc0e4c561f71eb5b3e.png 2023-07-11 18:56:29,923 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/utk36nc9-media/images/validation_432_0dfe0505f8d9adcc6ad1.png 2023-07-11 18:56:29,957 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/geg01vy9-media/images/validation_432_9aaaaf8c3f6ad3ebcd99.png 2023-07-11 18:56:33,113 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:56:33,114 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:56:33,181 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:56:33,241 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:56:35,529 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:56:37,050 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:56:37,052 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:56:37,052 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:56:37,054 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:56:38,536 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:56:38,537 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:56:38,708 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:56:40,013 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:56:40,015 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:56:40,015 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:56:40,017 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:56:40,549 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 18:56:41,544 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:56:41,544 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:56:42,921 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:56:42,923 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:56:42,923 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:56:42,925 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:56:43,892 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:56:44,551 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:56:44,551 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:56:46,925 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:56:46,926 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:56:46,927 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:56:46,928 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:56:47,558 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:56:47,558 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:56:48,113 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:56:48,114 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:56:49,185 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:56:49,849 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:56:49,851 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:56:49,851 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:56:49,853 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:56:50,565 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:56:50,565 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:56:53,573 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:56:53,794 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:56:53,796 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:56:53,797 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:56:53,798 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:56:54,075 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:56:54,077 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:56:54,078 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:56:54,079 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:56:54,477 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:56:56,580 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:56:56,580 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:56:59,586 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:56:59,616 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:57:03,148 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:57:03,149 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:57:05,218 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:57:06,650 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:57:06,650 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_440_bf3806856d1001a26a5b.png with policy now 2023-07-11 18:57:06,693 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:57:06,693 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_440_12692f8ccbda27de07b9.png with policy now 2023-07-11 18:57:06,694 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:57:06,694 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_440_d5960ae197569bb66752.png with policy now 2023-07-11 18:57:06,695 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:57:06,695 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_440_b23e879f7839905b9cab.png with policy now 2023-07-11 18:57:06,740 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:57:06,741 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:57:06,742 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:57:06,743 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:57:06,970 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/1c17w2oh-media/images/validation_440_bf3806856d1001a26a5b.png 2023-07-11 18:57:07,011 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/l72rc19a-media/images/validation_440_12692f8ccbda27de07b9.png 2023-07-11 18:57:07,011 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/odzmnaiy-media/images/validation_440_d5960ae197569bb66752.png 2023-07-11 18:57:07,016 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/amylspsf-media/images/validation_440_b23e879f7839905b9cab.png 2023-07-11 18:57:08,606 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:57:08,607 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_440_b23e879f7839905b9cab.png 2023-07-11 18:57:08,608 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_440_d5960ae197569bb66752.png 2023-07-11 18:57:08,608 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_440_12692f8ccbda27de07b9.png 2023-07-11 18:57:08,609 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_440_bf3806856d1001a26a5b.png 2023-07-11 18:57:08,609 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 18:57:08,955 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/yl1eg8c9-media/images/validation_440_d5960ae197569bb66752.png 2023-07-11 18:57:08,975 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/reeqvnrq-media/images/validation_440_b23e879f7839905b9cab.png 2023-07-11 18:57:08,985 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/jb82pt4k-media/images/validation_440_bf3806856d1001a26a5b.png 2023-07-11 18:57:08,989 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/rhqzx7q9-media/images/validation_440_12692f8ccbda27de07b9.png 2023-07-11 18:57:10,552 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 18:57:10,557 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:57:10,748 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:57:11,614 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:57:13,722 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:57:13,724 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:57:13,724 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:57:13,726 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:57:14,621 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:57:14,621 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:57:15,740 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:57:16,262 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:57:16,264 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:57:16,264 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:57:16,266 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:57:17,628 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:57:17,629 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:57:18,148 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:57:18,149 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:57:19,517 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:57:19,519 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:57:19,520 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:57:19,521 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:57:20,636 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:57:20,636 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:57:21,074 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:57:23,643 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:57:23,650 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:57:23,652 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:57:23,652 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:57:23,654 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:57:26,131 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:57:26,650 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:57:26,650 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:57:27,178 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:57:27,180 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:57:27,181 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:57:27,182 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:57:29,657 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:57:29,658 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:57:31,244 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:57:31,357 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:57:31,359 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:57:31,359 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:57:31,361 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:57:31,640 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:57:31,643 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:57:31,643 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:57:31,645 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:57:32,665 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:57:32,666 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:57:33,148 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:57:33,149 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:57:35,672 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:57:37,230 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:57:40,555 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 18:57:42,558 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:57:44,254 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:57:44,254 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_448_9e5ba8c3ea8c4ba6489b.png with policy now 2023-07-11 18:57:44,270 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:57:44,271 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_448_a781cbefdd5dbd10be81.png with policy now 2023-07-11 18:57:44,284 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:57:44,284 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_448_9d3f7736aabd23e4c942.png with policy now 2023-07-11 18:57:44,297 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:57:44,297 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_448_b685a88a96b2c1a8a665.png with policy now 2023-07-11 18:57:44,342 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:57:44,346 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:57:44,347 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:57:44,349 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:57:44,579 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ky0ydkjj-media/images/validation_448_9e5ba8c3ea8c4ba6489b.png 2023-07-11 18:57:44,597 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/vqt51kj9-media/images/validation_448_a781cbefdd5dbd10be81.png 2023-07-11 18:57:44,597 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/57c2jske-media/images/validation_448_9d3f7736aabd23e4c942.png 2023-07-11 18:57:44,676 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/0wcfhdmj-media/images/validation_448_b685a88a96b2c1a8a665.png 2023-07-11 18:57:44,693 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:57:44,694 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_448_b685a88a96b2c1a8a665.png 2023-07-11 18:57:44,694 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_448_9d3f7736aabd23e4c942.png 2023-07-11 18:57:44,695 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_448_9e5ba8c3ea8c4ba6489b.png 2023-07-11 18:57:44,695 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_448_a781cbefdd5dbd10be81.png 2023-07-11 18:57:44,696 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 18:57:45,052 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/5j09aphz-media/images/validation_448_b685a88a96b2c1a8a665.png 2023-07-11 18:57:45,061 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/bezv5kvl-media/images/validation_448_9e5ba8c3ea8c4ba6489b.png 2023-07-11 18:57:45,061 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/hjcfc48h-media/images/validation_448_a781cbefdd5dbd10be81.png 2023-07-11 18:57:45,153 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/rr00twvj-media/images/validation_448_9d3f7736aabd23e4c942.png 2023-07-11 18:57:47,700 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:57:48,170 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:57:48,171 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:57:48,207 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:57:48,340 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:57:50,708 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:57:52,416 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:57:52,418 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:57:52,418 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:57:52,420 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:57:53,218 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:57:53,715 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:57:53,715 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:57:55,063 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:57:55,065 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:57:55,065 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:57:55,067 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:57:56,722 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:57:56,723 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:57:58,297 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:57:59,228 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:57:59,230 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:57:59,230 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:57:59,232 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:57:59,729 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:57:59,730 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:58:02,000 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:58:02,002 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:58:02,002 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:58:02,004 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:58:02,737 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:58:02,737 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:58:03,170 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:58:03,171 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:58:03,741 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:58:05,463 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:58:05,464 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:58:05,465 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:58:05,466 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:58:05,744 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:58:05,744 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:58:08,751 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:58:09,011 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:58:09,013 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:58:09,013 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:58:09,013 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:58:09,015 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:58:09,295 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:58:09,297 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:58:09,297 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:58:09,299 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:58:10,558 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 18:58:11,758 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:58:11,759 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:58:14,561 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:58:18,170 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:58:18,171 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:58:20,213 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:58:21,893 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:58:21,893 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_456_7274f3777b238c5f0628.png with policy now 2023-07-11 18:58:21,911 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:58:21,912 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_456_3c82fec3d730872e6a5e.png with policy now 2023-07-11 18:58:21,926 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:58:21,926 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_456_8e44c66b80c9ca0412ba.png with policy now 2023-07-11 18:58:21,940 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:58:21,940 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_456_ed31315a0d1d059dae78.png with policy now 2023-07-11 18:58:21,981 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:58:21,982 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:58:21,983 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:58:21,984 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:58:22,185 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/55l7brsj-media/images/validation_456_7274f3777b238c5f0628.png 2023-07-11 18:58:22,221 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/y389aowq-media/images/validation_456_3c82fec3d730872e6a5e.png 2023-07-11 18:58:22,264 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/imatgj8b-media/images/validation_456_ed31315a0d1d059dae78.png 2023-07-11 18:58:22,284 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/7upx4dqf-media/images/validation_456_8e44c66b80c9ca0412ba.png 2023-07-11 18:58:23,788 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:58:23,789 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_456_7274f3777b238c5f0628.png 2023-07-11 18:58:23,789 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_456_ed31315a0d1d059dae78.png 2023-07-11 18:58:23,790 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_456_8e44c66b80c9ca0412ba.png 2023-07-11 18:58:23,791 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_456_3c82fec3d730872e6a5e.png 2023-07-11 18:58:23,791 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 18:58:24,127 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/37omvaja-media/images/validation_456_7274f3777b238c5f0628.png 2023-07-11 18:58:24,127 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/d8snoba8-media/images/validation_456_ed31315a0d1d059dae78.png 2023-07-11 18:58:24,147 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ygj6bx8c-media/images/validation_456_3c82fec3d730872e6a5e.png 2023-07-11 18:58:24,161 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ivk2f61z-media/images/validation_456_8e44c66b80c9ca0412ba.png 2023-07-11 18:58:25,974 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:58:26,796 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:58:26,998 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:58:29,803 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:58:30,338 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:58:30,340 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:58:30,340 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:58:30,342 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:58:31,705 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:58:32,811 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:58:32,811 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:58:33,170 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:58:33,171 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:58:33,486 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:58:33,488 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:58:33,489 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:58:33,490 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:58:35,818 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:58:35,819 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:58:36,028 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:58:36,030 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:58:36,031 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:58:36,032 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:58:36,758 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:58:38,826 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:58:38,826 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:58:38,979 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:58:38,981 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:58:38,981 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:58:38,983 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:58:40,562 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 18:58:41,833 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:58:41,833 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:58:41,909 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:58:41,911 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:58:41,911 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:58:41,911 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:58:41,913 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:58:44,840 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:58:44,841 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:58:45,784 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:58:45,786 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:58:45,786 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:58:45,788 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:58:46,066 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:58:46,068 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:58:46,068 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:58:46,070 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:58:47,849 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:58:47,849 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:58:47,891 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:58:48,171 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:58:48,172 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:58:50,856 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:58:53,220 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:58:58,221 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:58:59,008 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:58:59,008 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_464_b52266a063208eac90d1.png with policy now 2023-07-11 18:58:59,025 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:58:59,025 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_464_b0bdbc989e686d54eacb.png with policy now 2023-07-11 18:58:59,040 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:58:59,040 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_464_66cbaba9a39ca8bfc815.png with policy now 2023-07-11 18:58:59,056 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:58:59,056 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_464_97d923b01964cb8ebb37.png with policy now 2023-07-11 18:58:59,099 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:58:59,100 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:58:59,100 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:58:59,102 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:58:59,320 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/e0dau61r-media/images/validation_464_b52266a063208eac90d1.png 2023-07-11 18:58:59,356 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/0u9plg6o-media/images/validation_464_b0bdbc989e686d54eacb.png 2023-07-11 18:58:59,362 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/hz6o4tny-media/images/validation_464_66cbaba9a39ca8bfc815.png 2023-07-11 18:58:59,381 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ztujjfgs-media/images/validation_464_97d923b01964cb8ebb37.png 2023-07-11 18:58:59,877 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:58:59,878 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_464_b52266a063208eac90d1.png 2023-07-11 18:58:59,878 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_464_66cbaba9a39ca8bfc815.png 2023-07-11 18:58:59,879 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_464_b0bdbc989e686d54eacb.png 2023-07-11 18:58:59,879 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_464_97d923b01964cb8ebb37.png 2023-07-11 18:58:59,880 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 18:59:00,240 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/l6mzpa2c-media/images/validation_464_b0bdbc989e686d54eacb.png 2023-07-11 18:59:00,242 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ik5f5y6f-media/images/validation_464_b52266a063208eac90d1.png 2023-07-11 18:59:00,260 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/tfb7e2sh-media/images/validation_464_66cbaba9a39ca8bfc815.png 2023-07-11 18:59:00,268 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/f1a2wi2c-media/images/validation_464_97d923b01964cb8ebb37.png 2023-07-11 18:59:02,885 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:59:03,171 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:59:03,172 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:59:03,224 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:59:03,264 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:59:05,892 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:59:06,119 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:59:06,121 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:59:06,121 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:59:06,123 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:59:08,805 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:59:08,899 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:59:08,900 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:59:09,483 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:59:09,485 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:59:09,485 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:59:09,487 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:59:10,564 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 18:59:11,907 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:59:11,907 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:59:12,243 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:59:12,245 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:59:12,245 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:59:12,247 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:59:14,686 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:59:14,914 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:59:14,914 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:59:15,372 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:59:15,374 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:59:15,374 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:59:15,376 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:59:17,921 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:59:17,922 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:59:19,015 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:59:19,016 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:59:19,188 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:59:19,190 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:59:19,190 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:59:19,192 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:59:20,468 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:59:20,929 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:59:20,929 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:59:22,572 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:59:22,574 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:59:22,574 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:59:22,576 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:59:22,853 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:59:22,855 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:59:22,856 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:59:22,857 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:59:23,937 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:59:23,938 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:59:26,451 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:59:26,944 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:59:31,452 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:59:34,015 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:59:34,016 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:59:35,456 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:59:35,456 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_472_cfb31d01ff9d8fcc9935.png with policy now 2023-07-11 18:59:35,497 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:59:35,497 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_472_820adba40c6ce80d2f4a.png with policy now 2023-07-11 18:59:35,498 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:59:35,498 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_472_059ba63ee59da583f8b0.png with policy now 2023-07-11 18:59:35,499 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 18:59:35,499 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_472_ce05e5603e88c594f7a0.png with policy now 2023-07-11 18:59:35,541 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:59:35,543 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:59:35,543 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:59:35,545 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:59:35,804 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/aoa2zaz8-media/images/validation_472_cfb31d01ff9d8fcc9935.png 2023-07-11 18:59:35,830 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/o9yeqfr9-media/images/validation_472_820adba40c6ce80d2f4a.png 2023-07-11 18:59:35,830 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/35blgbyj-media/images/validation_472_ce05e5603e88c594f7a0.png 2023-07-11 18:59:35,861 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/mu0k4e95-media/images/validation_472_059ba63ee59da583f8b0.png 2023-07-11 18:59:35,966 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:59:35,966 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_472_820adba40c6ce80d2f4a.png 2023-07-11 18:59:35,967 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_472_ce05e5603e88c594f7a0.png 2023-07-11 18:59:35,968 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_472_cfb31d01ff9d8fcc9935.png 2023-07-11 18:59:35,968 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_472_059ba63ee59da583f8b0.png 2023-07-11 18:59:35,969 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 18:59:36,310 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/q49fv8so-media/images/validation_472_ce05e5603e88c594f7a0.png 2023-07-11 18:59:36,313 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/sny624nq-media/images/validation_472_cfb31d01ff9d8fcc9935.png 2023-07-11 18:59:36,330 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/m7c4cq1e-media/images/validation_472_059ba63ee59da583f8b0.png 2023-07-11 18:59:36,333 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/qj8mt3fa-media/images/validation_472_820adba40c6ce80d2f4a.png 2023-07-11 18:59:36,546 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:59:39,019 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:59:39,220 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:59:40,567 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 18:59:42,026 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:59:42,283 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:59:43,088 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:59:43,090 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:59:43,090 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:59:43,092 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:59:45,034 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:59:45,034 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:59:46,442 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:59:46,444 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:59:46,445 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:59:46,446 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:59:47,447 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:59:48,041 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:59:48,042 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:59:49,016 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 18:59:49,017 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 18:59:50,273 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:59:50,275 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:59:50,275 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:59:50,277 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:59:51,048 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:59:51,049 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:59:53,386 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:59:53,388 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:59:53,388 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:59:53,389 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:59:53,390 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:59:54,055 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:59:54,056 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:59:56,615 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:59:56,617 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:59:56,618 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:59:56,619 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:59:57,063 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 18:59:57,063 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 18:59:58,875 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 18:59:59,466 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:59:59,468 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:59:59,469 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:59:59,470 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 18:59:59,750 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 18:59:59,752 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 18:59:59,752 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 18:59:59,754 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:00:00,070 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:00:00,071 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:00:03,076 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:00:04,015 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:00:04,016 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:00:04,016 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:00:09,076 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:00:10,570 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:00:12,337 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:00:12,337 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_480_cccdf5a911d086c92c1b.png with policy now 2023-07-11 19:00:12,381 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:00:12,381 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_480_ccca929b7019e723af09.png with policy now 2023-07-11 19:00:12,382 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:00:12,382 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_480_be6c1fc77e931ea5eccf.png with policy now 2023-07-11 19:00:12,383 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:00:12,383 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_480_a98a50da4795b7ed390e.png with policy now 2023-07-11 19:00:12,423 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:00:12,425 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:00:12,425 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:00:12,426 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:00:12,657 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/zdwhk7ne-media/images/validation_480_cccdf5a911d086c92c1b.png 2023-07-11 19:00:12,693 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/uswjcd1q-media/images/validation_480_ccca929b7019e723af09.png 2023-07-11 19:00:12,696 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/hd6wgk4g-media/images/validation_480_be6c1fc77e931ea5eccf.png 2023-07-11 19:00:12,710 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/lgluwwrr-media/images/validation_480_a98a50da4795b7ed390e.png 2023-07-11 19:00:14,348 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:00:15,104 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:00:15,105 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_480_cccdf5a911d086c92c1b.png 2023-07-11 19:00:15,105 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_480_a98a50da4795b7ed390e.png 2023-07-11 19:00:15,106 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_480_ccca929b7019e723af09.png 2023-07-11 19:00:15,106 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_480_be6c1fc77e931ea5eccf.png 2023-07-11 19:00:15,107 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 19:00:15,463 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/pxwtgll1-media/images/validation_480_cccdf5a911d086c92c1b.png 2023-07-11 19:00:15,480 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/3jkz6epz-media/images/validation_480_ccca929b7019e723af09.png 2023-07-11 19:00:15,484 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ol89kmld-media/images/validation_480_a98a50da4795b7ed390e.png 2023-07-11 19:00:15,508 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/q016urc1-media/images/validation_480_be6c1fc77e931ea5eccf.png 2023-07-11 19:00:16,382 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:00:18,112 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:00:19,015 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:00:19,016 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:00:19,799 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:00:19,840 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:00:19,842 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:00:19,842 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:00:19,844 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:00:21,120 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:00:21,120 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:00:23,234 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:00:23,236 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:00:23,236 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:00:23,238 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:00:24,127 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:00:24,128 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:00:25,273 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:00:27,013 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:00:27,015 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:00:27,015 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:00:27,017 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:00:27,135 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:00:27,135 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:00:30,075 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:00:30,077 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:00:30,077 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:00:30,078 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:00:30,142 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:00:30,143 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:00:30,703 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:00:32,910 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:00:32,912 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:00:32,912 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:00:32,913 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:00:33,149 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:00:33,150 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:00:34,016 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:00:34,017 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:00:36,051 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:00:36,139 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:00:36,140 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:00:36,141 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:00:36,142 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:00:36,157 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:00:36,157 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:00:36,420 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:00:36,422 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:00:36,423 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:00:36,424 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:00:39,165 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:00:39,165 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:00:40,573 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:00:41,575 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:00:46,576 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:00:49,016 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:00:49,018 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:00:49,091 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:00:49,091 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_488_edd688b52163221b6553.png with policy now 2023-07-11 19:00:49,092 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:00:49,092 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_488_446494043ddef0089999.png with policy now 2023-07-11 19:00:49,093 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:00:49,093 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_488_f79a2569df6f42330b22.png with policy now 2023-07-11 19:00:49,094 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:00:49,094 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_488_1f7f3d27cf4d0d695023.png with policy now 2023-07-11 19:00:49,110 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:00:49,112 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:00:49,112 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:00:49,119 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:00:49,403 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/vtxzu4dc-media/images/validation_488_edd688b52163221b6553.png 2023-07-11 19:00:49,424 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/xdqw5u3s-media/images/validation_488_f79a2569df6f42330b22.png 2023-07-11 19:00:49,426 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/av0ku390-media/images/validation_488_446494043ddef0089999.png 2023-07-11 19:00:49,427 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/l7dekfqk-media/images/validation_488_1f7f3d27cf4d0d695023.png 2023-07-11 19:00:51,193 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:00:51,193 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_488_446494043ddef0089999.png 2023-07-11 19:00:51,193 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_488_f79a2569df6f42330b22.png 2023-07-11 19:00:51,194 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_488_edd688b52163221b6553.png 2023-07-11 19:00:51,194 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_488_1f7f3d27cf4d0d695023.png 2023-07-11 19:00:51,195 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 19:00:51,529 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/rm0c0e9o-media/images/validation_488_f79a2569df6f42330b22.png 2023-07-11 19:00:51,547 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/369e7ihs-media/images/validation_488_1f7f3d27cf4d0d695023.png 2023-07-11 19:00:51,550 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/85voudvb-media/images/validation_488_edd688b52163221b6553.png 2023-07-11 19:00:51,558 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/3ofhd9a5-media/images/validation_488_446494043ddef0089999.png 2023-07-11 19:00:51,984 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:00:52,864 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:00:54,200 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:00:56,077 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:00:56,080 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:00:56,080 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:00:56,082 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:00:57,208 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:00:57,208 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:00:57,486 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:00:59,300 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:00:59,302 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:00:59,303 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:00:59,304 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:01:00,215 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:01:00,216 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:01:02,448 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:01:02,450 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:01:02,450 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:01:02,452 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:01:03,202 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:01:03,223 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:01:03,224 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:01:04,163 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:01:04,164 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:01:06,214 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:01:06,216 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:01:06,217 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:01:06,218 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:01:06,230 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:01:06,230 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:01:08,910 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:01:09,237 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:01:09,806 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:01:09,807 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:01:09,808 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:01:09,810 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:01:10,576 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:01:12,245 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:01:12,245 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:01:13,890 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:01:13,892 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:01:13,893 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:01:13,894 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:01:14,174 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:01:14,176 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:01:14,177 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:01:14,177 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:01:14,179 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:01:15,253 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:01:15,254 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:01:18,261 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:01:19,192 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:01:19,193 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:01:19,258 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:01:24,259 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:01:26,782 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:01:26,783 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_496_68e06148c0b530b292ef.png with policy now 2023-07-11 19:01:26,799 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:01:26,799 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_496_2a1d91ee7dad3aca3377.png with policy now 2023-07-11 19:01:26,813 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:01:26,813 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_496_37d891b4d6c71e28eca9.png with policy now 2023-07-11 19:01:26,827 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:01:26,827 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_496_ab5da33b4fab6d97badb.png with policy now 2023-07-11 19:01:26,877 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:01:26,879 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:01:26,880 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:01:26,885 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:01:27,128 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/p5oemud3-media/images/validation_496_2a1d91ee7dad3aca3377.png 2023-07-11 19:01:27,129 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/8qor8ci7-media/images/validation_496_37d891b4d6c71e28eca9.png 2023-07-11 19:01:27,132 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/jvjubtyk-media/images/validation_496_68e06148c0b530b292ef.png 2023-07-11 19:01:27,140 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/z0czlla8-media/images/validation_496_ab5da33b4fab6d97badb.png 2023-07-11 19:01:27,283 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:01:27,283 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_496_68e06148c0b530b292ef.png 2023-07-11 19:01:27,284 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_496_37d891b4d6c71e28eca9.png 2023-07-11 19:01:27,284 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_496_ab5da33b4fab6d97badb.png 2023-07-11 19:01:27,285 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_496_2a1d91ee7dad3aca3377.png 2023-07-11 19:01:27,285 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 19:01:27,665 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/txct5wqk-media/images/validation_496_37d891b4d6c71e28eca9.png 2023-07-11 19:01:27,667 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/17d1h5s0-media/images/validation_496_68e06148c0b530b292ef.png 2023-07-11 19:01:27,682 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/yg1v4p5x-media/images/validation_496_ab5da33b4fab6d97badb.png 2023-07-11 19:01:27,701 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/763hvfo8-media/images/validation_496_2a1d91ee7dad3aca3377.png 2023-07-11 19:01:29,663 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:01:30,290 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:01:30,726 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:01:33,298 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:01:33,952 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:01:33,954 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:01:33,955 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:01:33,956 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:01:34,685 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:01:34,685 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:01:34,686 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:01:36,306 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:01:36,306 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:01:37,034 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:01:37,035 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:01:37,035 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:01:37,037 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:01:39,314 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:01:39,314 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:01:40,402 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:01:40,403 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:01:40,404 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:01:40,404 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:01:40,406 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:01:40,579 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:01:42,322 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:01:42,322 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:01:43,962 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:01:43,964 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:01:43,964 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:01:43,966 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:01:45,328 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:01:45,329 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:01:45,483 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:01:46,922 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:01:46,924 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:01:46,925 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:01:46,926 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:01:48,336 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:01:48,337 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:01:49,685 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:01:49,686 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:01:50,388 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:01:50,390 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:01:50,390 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:01:50,392 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:01:50,625 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:01:50,627 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:01:50,627 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:01:50,627 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:01:50,629 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:01:51,345 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:01:51,345 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:01:54,352 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:01:56,226 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:02:01,227 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:02:03,405 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_504_f98f85886287d586fb24.png 2023-07-11 19:02:03,406 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:02:03,406 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_504_f98f85886287d586fb24.png with policy now 2023-07-11 19:02:03,407 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 19:02:03,423 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:02:03,423 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_504_aab7b24d7b2a7c51475f.png with policy now 2023-07-11 19:02:03,434 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:02:03,434 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_504_8d503921a8e36638c133.png with policy now 2023-07-11 19:02:03,446 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:02:03,446 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_504_4cd9dbf50122f56d08c9.png with policy now 2023-07-11 19:02:03,486 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:02:03,487 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:02:03,489 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:02:03,495 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:02:03,723 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/okpcw86h-media/images/validation_504_f98f85886287d586fb24.png 2023-07-11 19:02:03,744 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/kujwplth-media/images/validation_504_8d503921a8e36638c133.png 2023-07-11 19:02:03,750 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/a7kpz5tc-media/images/validation_504_4cd9dbf50122f56d08c9.png 2023-07-11 19:02:03,772 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/iltrjvtr-media/images/validation_504_aab7b24d7b2a7c51475f.png 2023-07-11 19:02:04,050 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/9m9gv57d-media/images/validation_504_f98f85886287d586fb24.png 2023-07-11 19:02:04,685 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:02:04,686 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:02:06,414 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:02:06,415 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_504_f98f85886287d586fb24.png 2023-07-11 19:02:06,416 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:02:06,416 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_504_4cd9dbf50122f56d08c9.png 2023-07-11 19:02:06,417 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_504_aab7b24d7b2a7c51475f.png 2023-07-11 19:02:06,417 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_504_8d503921a8e36638c133.png 2023-07-11 19:02:06,417 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 19:02:06,522 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:02:06,791 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/c8pzivsb-media/images/validation_504_4cd9dbf50122f56d08c9.png 2023-07-11 19:02:06,813 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/74sdqk7q-media/images/validation_504_f98f85886287d586fb24.png 2023-07-11 19:02:06,856 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/hwwjrbho-media/images/validation_504_aab7b24d7b2a7c51475f.png 2023-07-11 19:02:07,061 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/yjniob2h-media/images/validation_504_8d503921a8e36638c133.png 2023-07-11 19:02:07,484 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:02:09,426 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:02:10,229 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:02:10,231 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:02:10,232 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:02:10,233 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:02:10,582 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:02:12,382 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:02:12,434 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:02:12,434 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:02:13,322 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:02:13,324 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:02:13,324 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:02:13,326 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:02:15,444 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:02:15,444 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:02:17,376 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:02:17,377 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:02:17,378 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:02:17,379 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:02:18,380 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:02:18,452 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:02:18,452 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:02:19,685 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:02:19,686 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:02:21,069 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:02:21,071 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:02:21,071 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:02:21,073 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:02:21,461 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:02:21,462 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:02:23,741 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:02:24,469 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:02:24,595 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:02:24,597 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:02:24,597 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:02:24,598 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:02:27,479 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:02:27,480 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:02:28,096 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:02:28,098 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:02:28,098 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:02:28,100 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:02:28,377 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:02:28,379 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:02:28,379 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:02:28,381 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:02:28,798 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:02:30,487 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:02:30,488 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:02:33,938 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:02:34,706 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:02:34,707 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:02:39,770 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:02:40,585 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:02:41,020 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:02:41,020 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_512_95c88d4ee00e6dceedec.png with policy now 2023-07-11 19:02:41,061 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:02:41,061 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_512_c7af5aea69ad1b787bdd.png with policy now 2023-07-11 19:02:41,062 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:02:41,062 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_512_b6b8325d8fe282e8ee86.png with policy now 2023-07-11 19:02:41,076 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:02:41,076 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_512_87b03e3c08874339a548.png with policy now 2023-07-11 19:02:41,116 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:02:41,118 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:02:41,119 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:02:41,120 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:02:41,340 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/wsuo2dg0-media/images/validation_512_95c88d4ee00e6dceedec.png 2023-07-11 19:02:41,382 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/uivybw87-media/images/validation_512_c7af5aea69ad1b787bdd.png 2023-07-11 19:02:41,389 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/wb5bjl9l-media/images/validation_512_b6b8325d8fe282e8ee86.png 2023-07-11 19:02:41,426 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/42o8u9uv-media/images/validation_512_87b03e3c08874339a548.png 2023-07-11 19:02:42,518 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:02:42,518 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_512_c7af5aea69ad1b787bdd.png 2023-07-11 19:02:42,519 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_512_87b03e3c08874339a548.png 2023-07-11 19:02:42,520 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_512_95c88d4ee00e6dceedec.png 2023-07-11 19:02:42,520 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_512_b6b8325d8fe282e8ee86.png 2023-07-11 19:02:42,521 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 19:02:42,880 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/daku452a-media/images/validation_512_87b03e3c08874339a548.png 2023-07-11 19:02:42,895 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/gbc583th-media/images/validation_512_b6b8325d8fe282e8ee86.png 2023-07-11 19:02:42,907 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/qyjwbu5x-media/images/validation_512_95c88d4ee00e6dceedec.png 2023-07-11 19:02:42,908 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/wma7hj5r-media/images/validation_512_c7af5aea69ad1b787bdd.png 2023-07-11 19:02:44,014 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:02:45,015 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:02:45,526 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:02:47,721 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:02:47,723 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:02:47,723 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:02:47,725 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:02:48,534 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:02:48,535 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:02:49,706 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:02:49,707 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:02:50,355 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:02:51,372 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:02:51,374 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:02:51,375 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:02:51,376 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:02:51,542 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:02:51,543 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:02:54,550 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:02:54,633 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:02:54,635 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:02:54,635 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:02:54,637 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:02:55,638 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:02:57,558 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:02:57,559 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:02:58,354 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:02:58,355 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:02:58,355 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:02:58,357 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:03:00,566 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:03:00,566 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:03:00,776 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:03:01,631 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:03:01,633 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:03:01,633 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:03:01,635 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:03:03,574 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:03:03,574 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:03:04,706 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:03:04,707 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:03:04,948 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:03:04,950 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:03:04,950 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:03:04,952 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:03:05,190 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:03:05,192 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:03:05,193 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:03:05,194 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:03:06,583 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:03:06,583 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:03:06,737 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:03:09,590 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:03:10,589 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:03:12,591 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:03:17,592 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:03:17,765 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:03:17,765 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_520_2111edf7d29b2a9e9ffa.png with policy now 2023-07-11 19:03:17,782 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:03:17,782 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_520_b5b4ceac10276cd5b030.png with policy now 2023-07-11 19:03:17,796 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:03:17,796 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_520_3a4cf4d788de75533c67.png with policy now 2023-07-11 19:03:17,812 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:03:17,812 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_520_78d0488f99cba790913e.png with policy now 2023-07-11 19:03:17,855 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:03:17,857 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:03:17,857 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:03:17,862 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:03:18,107 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/s6iwxc9v-media/images/validation_520_2111edf7d29b2a9e9ffa.png 2023-07-11 19:03:18,137 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/du3hbb7m-media/images/validation_520_b5b4ceac10276cd5b030.png 2023-07-11 19:03:18,144 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/7pmt6686-media/images/validation_520_3a4cf4d788de75533c67.png 2023-07-11 19:03:18,260 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/w7e08p9g-media/images/validation_520_78d0488f99cba790913e.png 2023-07-11 19:03:18,614 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:03:18,615 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_520_2111edf7d29b2a9e9ffa.png 2023-07-11 19:03:18,615 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_520_b5b4ceac10276cd5b030.png 2023-07-11 19:03:18,616 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_520_78d0488f99cba790913e.png 2023-07-11 19:03:18,616 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_520_3a4cf4d788de75533c67.png 2023-07-11 19:03:18,617 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 19:03:18,949 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ga5mhhna-media/images/validation_520_2111edf7d29b2a9e9ffa.png 2023-07-11 19:03:18,965 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/zzbdokz8-media/images/validation_520_78d0488f99cba790913e.png 2023-07-11 19:03:18,967 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/fh1xj9o9-media/images/validation_520_3a4cf4d788de75533c67.png 2023-07-11 19:03:18,991 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ncox328y-media/images/validation_520_b5b4ceac10276cd5b030.png 2023-07-11 19:03:19,706 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:03:19,707 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:03:21,622 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:03:22,395 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:03:22,976 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:03:24,631 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:03:25,314 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:03:25,316 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:03:25,316 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:03:25,318 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:03:27,639 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:03:27,640 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:03:28,423 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:03:28,535 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:03:28,537 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:03:28,538 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:03:28,539 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:03:30,647 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:03:30,648 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:03:32,243 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:03:32,245 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:03:32,245 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:03:32,247 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:03:33,652 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:03:33,652 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:03:34,063 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:03:34,706 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:03:34,707 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:03:35,608 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:03:35,610 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:03:35,610 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:03:35,612 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:03:36,660 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:03:36,660 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:03:38,586 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:03:38,587 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:03:38,588 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:03:38,589 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:03:39,266 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:03:39,668 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:03:39,669 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:03:40,591 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:03:42,070 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:03:42,072 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:03:42,072 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:03:42,074 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:03:42,352 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:03:42,354 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:03:42,354 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:03:42,356 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:03:42,677 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:03:42,678 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:03:45,090 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:03:45,685 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:03:49,713 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:03:49,714 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:03:50,749 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:03:55,143 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:03:55,143 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_528_875885eb3355d2fc1a7a.png with policy now 2023-07-11 19:03:55,185 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:03:55,185 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_528_b7baa092f64c64dba260.png with policy now 2023-07-11 19:03:55,186 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:03:55,186 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_528_5348c071dbdd44e6d366.png with policy now 2023-07-11 19:03:55,198 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:03:55,198 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_528_767c54c5dd821b8fd78d.png with policy now 2023-07-11 19:03:55,236 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:03:55,237 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:03:55,237 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:03:55,239 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:03:55,479 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/linp1yjj-media/images/validation_528_875885eb3355d2fc1a7a.png 2023-07-11 19:03:55,542 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/wzd8vi5q-media/images/validation_528_5348c071dbdd44e6d366.png 2023-07-11 19:03:55,543 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/fbe0f8nh-media/images/validation_528_767c54c5dd821b8fd78d.png 2023-07-11 19:03:55,546 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/xytwqb77-media/images/validation_528_b7baa092f64c64dba260.png 2023-07-11 19:03:56,240 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:03:57,716 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:03:57,717 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:03:57,717 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_528_b7baa092f64c64dba260.png 2023-07-11 19:03:57,717 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_528_875885eb3355d2fc1a7a.png 2023-07-11 19:03:57,718 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_528_767c54c5dd821b8fd78d.png 2023-07-11 19:03:57,719 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_528_5348c071dbdd44e6d366.png 2023-07-11 19:03:57,719 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 19:03:58,049 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/fuze3sl1-media/images/validation_528_875885eb3355d2fc1a7a.png 2023-07-11 19:03:58,073 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/fqegvcuc-media/images/validation_528_b7baa092f64c64dba260.png 2023-07-11 19:03:58,086 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/4m1c3aor-media/images/validation_528_5348c071dbdd44e6d366.png 2023-07-11 19:03:58,089 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/4mpmcfql-media/images/validation_528_767c54c5dd821b8fd78d.png 2023-07-11 19:03:59,016 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:04:00,724 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:04:02,004 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:04:02,006 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:04:02,006 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:04:02,006 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:04:02,008 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:04:03,729 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:04:03,730 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:04:04,713 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:04:04,714 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:04:06,323 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:04:06,325 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:04:06,325 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:04:06,327 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:04:06,737 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:04:06,738 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:04:07,328 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:04:09,745 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:04:10,222 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:04:10,224 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:04:10,225 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:04:10,226 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:04:10,594 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:04:12,432 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:04:12,753 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:04:12,753 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:04:13,138 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:04:13,140 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:04:13,141 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:04:13,142 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:04:15,761 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:04:15,762 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:04:15,844 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:04:15,846 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:04:15,846 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:04:15,848 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:04:17,455 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:04:18,769 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:04:18,770 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:04:18,902 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:04:18,903 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:04:18,904 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:04:18,905 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:04:19,186 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:04:19,188 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:04:19,188 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:04:19,190 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:04:19,713 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:04:19,715 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:04:21,778 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:04:21,779 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:04:22,761 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:04:27,763 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:04:31,726 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:04:31,726 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_536_a8e5af09990ec6f7b046.png with policy now 2023-07-11 19:04:31,769 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:04:31,769 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_536_08ad374c5434931379b1.png with policy now 2023-07-11 19:04:31,770 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:04:31,770 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_536_e6b2848e4a4a5a1fa3b2.png with policy now 2023-07-11 19:04:31,780 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:04:31,781 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_536_915b0997964674e19d30.png with policy now 2023-07-11 19:04:31,821 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:04:31,823 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:04:31,823 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:04:31,825 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:04:32,041 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/efhzelx2-media/images/validation_536_a8e5af09990ec6f7b046.png 2023-07-11 19:04:32,080 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/osge0ya5-media/images/validation_536_08ad374c5434931379b1.png 2023-07-11 19:04:32,101 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/eg0gjeq0-media/images/validation_536_e6b2848e4a4a5a1fa3b2.png 2023-07-11 19:04:32,122 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/h2gblscl-media/images/validation_536_915b0997964674e19d30.png 2023-07-11 19:04:32,826 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:04:33,813 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:04:33,814 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:04:33,814 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_536_915b0997964674e19d30.png 2023-07-11 19:04:33,815 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_536_e6b2848e4a4a5a1fa3b2.png 2023-07-11 19:04:33,818 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_536_08ad374c5434931379b1.png 2023-07-11 19:04:33,818 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_536_a8e5af09990ec6f7b046.png 2023-07-11 19:04:33,819 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 19:04:34,287 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/x5kyz5v3-media/images/validation_536_915b0997964674e19d30.png 2023-07-11 19:04:34,315 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/han6b9n8-media/images/validation_536_08ad374c5434931379b1.png 2023-07-11 19:04:34,347 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/54pp5094-media/images/validation_536_a8e5af09990ec6f7b046.png 2023-07-11 19:04:34,474 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/337hlv6v-media/images/validation_536_e6b2848e4a4a5a1fa3b2.png 2023-07-11 19:04:34,748 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:04:34,750 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:04:35,880 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:04:36,821 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:04:38,451 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:04:39,668 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:04:39,670 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:04:39,670 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:04:39,672 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:04:39,832 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:04:39,832 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:04:40,597 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:04:42,778 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:04:42,780 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:04:42,781 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:04:42,782 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:04:42,839 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:04:42,840 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:04:43,705 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:04:45,848 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:04:45,925 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:04:45,927 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:04:45,927 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:04:45,929 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:04:48,855 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:04:48,856 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:04:48,965 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:04:49,127 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:04:49,129 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:04:49,129 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:04:49,131 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:04:49,749 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:04:49,750 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:04:51,863 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:04:51,863 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:04:52,283 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:04:52,285 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:04:52,286 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:04:52,287 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:04:54,138 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:04:54,871 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:04:54,872 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:04:56,127 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:04:56,129 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:04:56,129 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:04:56,131 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:04:56,409 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:04:56,411 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:04:56,411 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:04:56,413 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:04:57,880 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:04:57,880 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:04:59,970 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:05:04,790 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:05:04,791 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:05:05,845 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:05:09,006 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:05:09,006 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_544_d527e1f00fb003991478.png with policy now 2023-07-11 19:05:09,049 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:05:09,049 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_544_b99da3c0fcb7ededf7b0.png with policy now 2023-07-11 19:05:09,049 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:05:09,050 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_544_f145c478050d42496e6e.png with policy now 2023-07-11 19:05:09,062 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:05:09,063 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_544_ec40722860c852345c23.png with policy now 2023-07-11 19:05:09,100 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:05:09,102 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:05:09,102 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:05:09,104 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:05:09,313 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/bk58kcmp-media/images/validation_544_d527e1f00fb003991478.png 2023-07-11 19:05:09,372 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/0snjvvbb-media/images/validation_544_b99da3c0fcb7ededf7b0.png 2023-07-11 19:05:09,376 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/hx8a31iy-media/images/validation_544_f145c478050d42496e6e.png 2023-07-11 19:05:09,390 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/zmcy0iwh-media/images/validation_544_ec40722860c852345c23.png 2023-07-11 19:05:09,912 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:05:09,912 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_544_f145c478050d42496e6e.png 2023-07-11 19:05:09,913 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_544_b99da3c0fcb7ededf7b0.png 2023-07-11 19:05:09,913 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_544_d527e1f00fb003991478.png 2023-07-11 19:05:09,914 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_544_ec40722860c852345c23.png 2023-07-11 19:05:09,914 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 19:05:10,297 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/egxmf1ze-media/images/validation_544_b99da3c0fcb7ededf7b0.png 2023-07-11 19:05:10,305 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ra7n8wsv-media/images/validation_544_f145c478050d42496e6e.png 2023-07-11 19:05:10,319 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/9qd9z62r-media/images/validation_544_ec40722860c852345c23.png 2023-07-11 19:05:10,322 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/synbk4i1-media/images/validation_544_d527e1f00fb003991478.png 2023-07-11 19:05:10,600 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:05:11,014 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:05:12,195 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:05:12,920 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:05:15,928 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:05:16,395 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:05:16,565 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:05:16,567 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:05:16,567 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:05:16,569 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:05:18,936 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:05:18,937 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:05:19,368 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:05:19,369 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:05:19,370 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:05:19,371 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:05:20,273 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:05:20,274 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:05:21,907 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:05:21,945 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:05:21,945 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:05:22,816 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:05:22,818 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:05:22,818 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:05:22,820 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:05:24,953 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:05:24,953 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:05:26,277 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:05:26,279 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:05:26,280 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:05:26,281 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:05:26,977 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:05:27,961 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:05:27,961 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:05:29,426 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:05:29,427 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:05:29,428 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:05:29,429 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:05:30,969 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:05:30,969 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:05:32,630 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:05:32,632 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:05:32,632 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:05:32,633 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:05:32,634 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:05:32,913 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:05:32,915 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:05:32,916 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:05:32,917 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:05:33,978 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:05:33,979 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:05:35,273 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:05:35,274 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:05:38,357 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:05:40,603 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:05:43,605 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:05:45,570 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:05:45,571 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_552_8c78fd5084c9eef35e50.png with policy now 2023-07-11 19:05:45,613 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:05:45,613 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_552_41181c5e1fbbaa3688f6.png with policy now 2023-07-11 19:05:45,614 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:05:45,614 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_552_257d3c0b8f2426466221.png with policy now 2023-07-11 19:05:45,616 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:05:45,617 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_552_8812f45a50bdc79a8517.png with policy now 2023-07-11 19:05:45,661 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:05:45,662 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:05:45,663 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:05:45,664 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:05:45,891 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/a0nkkrbl-media/images/validation_552_8c78fd5084c9eef35e50.png 2023-07-11 19:05:45,934 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/0qf6vvge-media/images/validation_552_8812f45a50bdc79a8517.png 2023-07-11 19:05:45,955 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/pj0cpvof-media/images/validation_552_257d3c0b8f2426466221.png 2023-07-11 19:05:45,962 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/lblwgztc-media/images/validation_552_41181c5e1fbbaa3688f6.png 2023-07-11 19:05:46,011 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:05:46,011 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_552_257d3c0b8f2426466221.png 2023-07-11 19:05:46,012 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_552_8c78fd5084c9eef35e50.png 2023-07-11 19:05:46,013 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_552_8812f45a50bdc79a8517.png 2023-07-11 19:05:46,013 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_552_41181c5e1fbbaa3688f6.png 2023-07-11 19:05:46,013 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 19:05:46,387 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/gp8is2zr-media/images/validation_552_8c78fd5084c9eef35e50.png 2023-07-11 19:05:46,388 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/0jouqylg-media/images/validation_552_257d3c0b8f2426466221.png 2023-07-11 19:05:46,392 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/3akriroy-media/images/validation_552_41181c5e1fbbaa3688f6.png 2023-07-11 19:05:46,413 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/40xi4wpb-media/images/validation_552_8812f45a50bdc79a8517.png 2023-07-11 19:05:49,019 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:05:49,164 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:05:49,165 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:05:50,812 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:05:50,813 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:05:52,026 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:05:52,502 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:05:52,504 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:05:52,505 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:05:52,506 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:05:54,452 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:05:55,034 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:05:55,034 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:05:56,113 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:05:56,115 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:05:56,115 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:05:56,117 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:05:58,042 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:05:58,043 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:05:59,003 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:05:59,005 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:05:59,006 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:05:59,007 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:06:00,009 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:06:01,051 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:06:01,051 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:06:03,201 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:06:03,203 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:06:03,203 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:06:03,205 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:06:04,059 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:06:04,059 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:06:05,215 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:06:06,059 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:06:06,060 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:06:06,232 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:06:06,234 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:06:06,234 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:06:06,235 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:06:07,067 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:06:07,067 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:06:08,978 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:06:08,980 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:06:08,980 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:06:08,982 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:06:09,259 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:06:09,261 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:06:09,262 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:06:09,263 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:06:10,076 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:06:10,076 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:06:10,606 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:06:10,610 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:06:15,608 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:06:20,610 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:06:21,065 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:06:21,066 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:06:21,863 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:06:21,864 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_560_f76cf11600d692be2502.png with policy now 2023-07-11 19:06:21,905 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:06:21,905 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_560_bbef43cf934160e04cfd.png with policy now 2023-07-11 19:06:21,906 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:06:21,906 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_560_8a39c93dbe4043d18b4f.png with policy now 2023-07-11 19:06:21,915 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:06:21,916 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_560_b956f5c40cffd0dcba98.png with policy now 2023-07-11 19:06:21,953 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:06:21,954 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:06:21,955 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:06:21,956 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:06:22,108 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:06:22,109 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_560_8a39c93dbe4043d18b4f.png 2023-07-11 19:06:22,110 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_560_bbef43cf934160e04cfd.png 2023-07-11 19:06:22,110 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_560_f76cf11600d692be2502.png 2023-07-11 19:06:22,111 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_560_b956f5c40cffd0dcba98.png 2023-07-11 19:06:22,111 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 19:06:22,173 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/7e3g9thi-media/images/validation_560_f76cf11600d692be2502.png 2023-07-11 19:06:22,221 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/g1i64zvw-media/images/validation_560_bbef43cf934160e04cfd.png 2023-07-11 19:06:22,228 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/s6pncgu6-media/images/validation_560_8a39c93dbe4043d18b4f.png 2023-07-11 19:06:22,235 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/0tqe8sts-media/images/validation_560_b956f5c40cffd0dcba98.png 2023-07-11 19:06:22,549 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/bpgynat1-media/images/validation_560_bbef43cf934160e04cfd.png 2023-07-11 19:06:22,563 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/safaion5-media/images/validation_560_f76cf11600d692be2502.png 2023-07-11 19:06:22,563 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/xeg1kswm-media/images/validation_560_b956f5c40cffd0dcba98.png 2023-07-11 19:06:22,864 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ifldr9yb-media/images/validation_560_8a39c93dbe4043d18b4f.png 2023-07-11 19:06:25,119 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:06:25,415 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:06:26,060 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:06:28,128 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:06:28,894 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:06:28,896 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:06:28,896 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:06:28,898 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:06:31,136 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:06:31,137 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:06:31,418 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:06:32,726 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:06:32,728 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:06:32,729 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:06:32,730 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:06:34,144 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:06:34,145 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:06:35,842 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:06:35,844 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:06:35,844 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:06:35,846 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:06:36,065 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:06:36,066 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:06:36,577 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:06:37,153 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:06:37,153 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:06:38,804 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:06:38,806 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:06:38,807 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:06:38,808 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:06:40,161 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:06:40,162 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:06:40,608 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:06:42,025 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:06:42,027 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:06:42,027 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:06:42,027 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:06:42,029 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:06:43,170 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:06:43,170 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:06:45,306 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:06:45,307 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:06:45,308 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:06:45,309 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:06:45,588 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:06:45,590 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:06:45,590 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:06:45,592 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:06:46,178 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:06:46,179 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:06:47,157 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:06:49,186 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:06:51,065 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:06:51,065 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:06:53,108 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:06:58,110 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:06:58,214 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:06:58,215 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_568_dba4f198a349b1343cdd.png with policy now 2023-07-11 19:06:58,227 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_568_dba4f198a349b1343cdd.png 2023-07-11 19:06:58,231 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:06:58,231 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_568_04d62c7653c5d162a2e3.png with policy now 2023-07-11 19:06:58,246 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:06:58,246 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_568_e297ab1a0ce4b9855bb5.png with policy now 2023-07-11 19:06:58,258 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:06:58,258 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_568_68596aff73764ec36f54.png with policy now 2023-07-11 19:06:58,298 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:06:58,300 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:06:58,300 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:06:58,302 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:06:58,561 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/5bizf9ht-media/images/validation_568_04d62c7653c5d162a2e3.png 2023-07-11 19:06:58,563 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/quxosgzg-media/images/validation_568_dba4f198a349b1343cdd.png 2023-07-11 19:06:58,570 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/0r9gsczg-media/images/validation_568_e297ab1a0ce4b9855bb5.png 2023-07-11 19:06:58,578 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/08gpqw5o-media/images/validation_568_68596aff73764ec36f54.png 2023-07-11 19:06:59,183 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/h6trbm19-media/images/validation_568_dba4f198a349b1343cdd.png 2023-07-11 19:07:01,237 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:07:01,238 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:07:01,239 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_568_68596aff73764ec36f54.png 2023-07-11 19:07:01,239 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_568_04d62c7653c5d162a2e3.png 2023-07-11 19:07:01,240 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_568_e297ab1a0ce4b9855bb5.png 2023-07-11 19:07:01,240 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 19:07:01,578 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/46kvjw2v-media/images/validation_568_68596aff73764ec36f54.png 2023-07-11 19:07:01,586 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/esiut0a6-media/images/validation_568_e297ab1a0ce4b9855bb5.png 2023-07-11 19:07:01,593 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/yfarbho0-media/images/validation_568_04d62c7653c5d162a2e3.png 2023-07-11 19:07:01,716 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:07:03,376 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:07:04,246 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:07:04,618 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:07:04,620 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:07:04,621 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:07:04,622 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:07:06,535 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:07:06,536 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:07:07,254 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:07:07,255 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:07:08,485 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:07:08,487 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:07:08,487 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:07:08,488 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:07:08,490 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:07:10,263 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:07:10,264 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:07:10,612 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:07:11,959 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:07:11,961 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:07:11,961 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:07:11,963 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:07:13,272 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:07:13,272 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:07:13,789 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:07:15,461 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:07:15,463 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:07:15,463 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:07:15,465 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:07:16,280 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:07:16,280 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:07:18,680 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:07:18,682 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:07:18,683 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:07:18,684 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:07:19,288 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:07:19,289 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:07:19,460 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:07:21,535 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:07:21,536 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:07:21,992 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:07:21,994 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:07:21,994 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:07:21,996 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:07:22,231 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:07:22,233 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:07:22,234 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:07:22,235 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:07:22,292 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:07:22,293 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:07:24,805 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:07:25,302 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:07:29,806 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:07:34,783 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:07:34,783 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_576_3b7a05a66d1f89e67bb8.png with policy now 2023-07-11 19:07:34,799 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:07:34,799 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_576_b1cd8c64caeaede72534.png with policy now 2023-07-11 19:07:34,814 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:07:34,814 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_576_0fa5d5fa63f3526989d2.png with policy now 2023-07-11 19:07:34,815 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:07:34,828 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:07:34,828 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_576_b8f228173e9910efda0e.png with policy now 2023-07-11 19:07:34,871 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:07:34,873 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:07:34,873 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:07:34,875 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:07:35,133 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/bycfsrfk-media/images/validation_576_3b7a05a66d1f89e67bb8.png 2023-07-11 19:07:35,140 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/o18bgdxu-media/images/validation_576_0fa5d5fa63f3526989d2.png 2023-07-11 19:07:35,150 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/qlagyrui-media/images/validation_576_b1cd8c64caeaede72534.png 2023-07-11 19:07:35,167 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/nyryw63q-media/images/validation_576_b8f228173e9910efda0e.png 2023-07-11 19:07:36,551 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:07:36,552 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:07:37,335 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:07:37,335 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_576_3b7a05a66d1f89e67bb8.png 2023-07-11 19:07:37,336 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_576_b1cd8c64caeaede72534.png 2023-07-11 19:07:37,336 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_576_b8f228173e9910efda0e.png 2023-07-11 19:07:37,337 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_576_0fa5d5fa63f3526989d2.png 2023-07-11 19:07:37,337 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 19:07:37,672 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/qp3lm6v5-media/images/validation_576_3b7a05a66d1f89e67bb8.png 2023-07-11 19:07:37,687 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/tztrsdbz-media/images/validation_576_0fa5d5fa63f3526989d2.png 2023-07-11 19:07:37,710 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/frha2ojs-media/images/validation_576_b8f228173e9910efda0e.png 2023-07-11 19:07:37,851 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/6hwjd2wk-media/images/validation_576_b1cd8c64caeaede72534.png 2023-07-11 19:07:38,947 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:07:39,949 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:07:40,339 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:07:40,614 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:07:42,711 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:07:42,712 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:07:42,713 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:07:42,714 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:07:43,343 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:07:43,344 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:07:45,114 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:07:45,923 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:07:45,925 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:07:45,925 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:07:45,927 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:07:46,347 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:07:46,348 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:07:49,168 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:07:49,170 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:07:49,171 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:07:49,172 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:07:49,356 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:07:49,357 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:07:50,611 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:07:51,549 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:07:51,550 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:07:52,343 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:07:52,344 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:07:52,345 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:07:52,346 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:07:52,360 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:07:52,360 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:07:55,364 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:07:56,418 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:07:56,420 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:07:56,420 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:07:56,421 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:07:56,423 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:07:58,372 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:07:58,373 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:07:59,150 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:07:59,151 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:07:59,152 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:07:59,153 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:07:59,432 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:07:59,434 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:07:59,434 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:07:59,436 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:08:01,382 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:08:01,382 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:08:02,026 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:08:06,568 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:08:06,569 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:08:07,611 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:08:10,617 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:08:12,074 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:08:12,074 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_584_fea9e895a6bad4c69a30.png with policy now 2023-07-11 19:08:12,092 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:08:12,092 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_584_b49048e90ace4b32dca7.png with policy now 2023-07-11 19:08:12,122 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:08:12,122 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_584_434c1624a967a3f3a962.png with policy now 2023-07-11 19:08:12,144 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:08:12,144 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_584_29997c31305c3912cd04.png with policy now 2023-07-11 19:08:12,208 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:08:12,210 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:08:12,210 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:08:12,212 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:08:12,400 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/y3zzrthm-media/images/validation_584_fea9e895a6bad4c69a30.png 2023-07-11 19:08:12,402 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/qutqfwlo-media/images/validation_584_b49048e90ace4b32dca7.png 2023-07-11 19:08:12,427 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ryqkot01-media/images/validation_584_434c1624a967a3f3a962.png 2023-07-11 19:08:12,481 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/0y7dr4nt-media/images/validation_584_29997c31305c3912cd04.png 2023-07-11 19:08:13,213 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:08:13,412 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:08:13,413 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_584_29997c31305c3912cd04.png 2023-07-11 19:08:13,413 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_584_434c1624a967a3f3a962.png 2023-07-11 19:08:13,414 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_584_fea9e895a6bad4c69a30.png 2023-07-11 19:08:13,414 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_584_b49048e90ace4b32dca7.png 2023-07-11 19:08:13,415 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 19:08:13,757 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/2mq0gkkm-media/images/validation_584_29997c31305c3912cd04.png 2023-07-11 19:08:13,763 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/xyyxoac3-media/images/validation_584_fea9e895a6bad4c69a30.png 2023-07-11 19:08:13,770 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/4bqj5btj-media/images/validation_584_434c1624a967a3f3a962.png 2023-07-11 19:08:13,796 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ea89fkqr-media/images/validation_584_b49048e90ace4b32dca7.png 2023-07-11 19:08:16,201 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:08:16,421 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:08:18,731 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:08:19,431 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:08:19,893 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:08:19,895 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:08:19,895 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:08:19,897 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:08:21,567 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:08:21,568 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:08:22,439 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:08:22,440 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:08:23,608 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:08:23,610 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:08:23,611 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:08:23,612 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:08:24,312 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:08:25,448 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:08:25,449 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:08:26,454 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:08:26,456 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:08:26,456 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:08:26,458 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:08:28,457 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:08:28,457 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:08:29,829 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:08:29,831 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:08:29,831 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:08:29,832 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:08:29,834 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:08:31,465 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:08:31,466 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:08:33,246 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:08:33,248 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:08:33,248 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:08:33,249 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:08:34,474 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:08:34,474 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:08:34,831 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:08:36,694 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:08:36,695 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:08:36,821 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:08:37,103 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:08:37,484 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:08:37,514 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:08:37,515 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:08:37,517 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:08:37,517 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:08:37,518 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:08:37,519 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:08:40,492 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:08:40,493 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:08:40,620 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:08:40,625 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:08:45,623 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:08:49,773 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:08:49,773 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_592_3924670710a9e29d06d3.png with policy now 2023-07-11 19:08:49,792 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:08:49,793 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_592_0e50e9bfb15ce39c112d.png with policy now 2023-07-11 19:08:49,808 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:08:49,808 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_592_9a03f64388706e1447b6.png with policy now 2023-07-11 19:08:49,824 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:08:49,824 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_592_cd080a9de17f2873d239.png with policy now 2023-07-11 19:08:49,868 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:08:49,869 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:08:49,869 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:08:49,871 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:08:50,073 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/pazcq2ve-media/images/validation_592_3924670710a9e29d06d3.png 2023-07-11 19:08:50,126 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/97fci7y3-media/images/validation_592_0e50e9bfb15ce39c112d.png 2023-07-11 19:08:50,127 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/anwauq2r-media/images/validation_592_9a03f64388706e1447b6.png 2023-07-11 19:08:50,175 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/e3wbotq6-media/images/validation_592_cd080a9de17f2873d239.png 2023-07-11 19:08:50,861 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:08:51,694 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:08:51,695 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:08:52,525 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:08:52,526 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:08:52,526 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_592_9a03f64388706e1447b6.png 2023-07-11 19:08:52,527 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_592_cd080a9de17f2873d239.png 2023-07-11 19:08:52,527 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_592_3924670710a9e29d06d3.png 2023-07-11 19:08:52,528 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_592_0e50e9bfb15ce39c112d.png 2023-07-11 19:08:52,528 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 19:08:52,861 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/2zcti1h5-media/images/validation_592_9a03f64388706e1447b6.png 2023-07-11 19:08:52,866 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/d110r1d0-media/images/validation_592_3924670710a9e29d06d3.png 2023-07-11 19:08:52,868 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/s2sje9cb-media/images/validation_592_cd080a9de17f2873d239.png 2023-07-11 19:08:52,892 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/j0ty0ms1-media/images/validation_592_0e50e9bfb15ce39c112d.png 2023-07-11 19:08:53,384 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:08:55,534 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:08:56,212 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:08:57,470 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:08:57,472 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:08:57,472 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:08:57,474 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:08:59,543 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:08:59,544 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:09:00,138 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:09:00,140 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:09:00,140 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:09:00,142 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:09:01,942 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:09:03,241 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:09:03,243 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:09:03,243 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:09:03,245 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:09:03,552 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:09:03,552 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:09:06,185 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:09:06,187 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:09:06,188 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:09:06,189 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:09:07,190 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:09:07,396 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:09:07,397 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:09:07,561 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:09:07,562 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:09:10,151 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:09:10,153 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:09:10,153 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:09:10,155 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:09:10,622 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:09:11,570 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:09:11,570 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:09:12,205 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:09:13,393 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:09:13,395 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:09:13,396 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:09:13,397 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:09:13,675 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:09:13,677 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:09:13,677 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:09:13,679 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:09:15,578 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:09:15,579 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:09:17,314 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:09:22,315 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:09:22,406 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:09:22,407 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:09:26,303 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:09:26,304 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_600_8300fd15af7363a95498.png with policy now 2023-07-11 19:09:26,345 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:09:26,345 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_600_4dddb5352b02ea5893dc.png with policy now 2023-07-11 19:09:26,346 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:09:26,346 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_600_1d97a6cab0fe06041ca1.png with policy now 2023-07-11 19:09:26,347 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:09:26,347 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_600_917bd69bfc0758359482.png with policy now 2023-07-11 19:09:26,389 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:09:26,390 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:09:26,391 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:09:26,392 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:09:26,698 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/sflwoahk-media/images/validation_600_8300fd15af7363a95498.png 2023-07-11 19:09:26,754 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/49bu905h-media/images/validation_600_4dddb5352b02ea5893dc.png 2023-07-11 19:09:26,754 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/h7y5egyx-media/images/validation_600_917bd69bfc0758359482.png 2023-07-11 19:09:26,766 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/i74aoamp-media/images/validation_600_1d97a6cab0fe06041ca1.png 2023-07-11 19:09:27,393 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:09:27,605 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:09:27,605 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_600_8300fd15af7363a95498.png 2023-07-11 19:09:27,606 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_600_917bd69bfc0758359482.png 2023-07-11 19:09:27,607 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_600_1d97a6cab0fe06041ca1.png 2023-07-11 19:09:27,607 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_600_4dddb5352b02ea5893dc.png 2023-07-11 19:09:27,608 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 19:09:27,931 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/jjmgwddb-media/images/validation_600_8300fd15af7363a95498.png 2023-07-11 19:09:27,949 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/rvmzcabw-media/images/validation_600_917bd69bfc0758359482.png 2023-07-11 19:09:27,985 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/q4xs02te-media/images/validation_600_1d97a6cab0fe06041ca1.png 2023-07-11 19:09:27,988 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/wlo30gvc-media/images/validation_600_4dddb5352b02ea5893dc.png 2023-07-11 19:09:30,244 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:09:31,615 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:09:32,555 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:09:33,235 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:09:33,237 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:09:33,237 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:09:33,239 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:09:35,623 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:09:35,624 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:09:37,406 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:09:37,407 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:09:37,587 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:09:37,628 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:09:37,630 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:09:37,630 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:09:37,632 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:09:39,633 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:09:39,633 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:09:40,479 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:09:40,481 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:09:40,482 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:09:40,483 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:09:40,626 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:09:43,181 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:09:43,562 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:09:43,564 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:09:43,565 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:09:43,566 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:09:43,642 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:09:43,643 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:09:47,651 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:09:47,893 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:09:47,895 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:09:47,895 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:09:47,897 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:09:48,573 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:09:50,955 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:09:50,957 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:09:50,958 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:09:50,959 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:09:51,237 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:09:51,239 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:09:51,239 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:09:51,241 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:09:51,661 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:09:51,661 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:09:52,406 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:09:52,407 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:09:54,443 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:09:55,669 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:09:59,444 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:10:03,825 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:10:03,826 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_608_2104758725886dd01ccd.png with policy now 2023-07-11 19:10:03,869 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:10:03,869 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_608_e93612b41299320c0daf.png with policy now 2023-07-11 19:10:03,870 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:10:03,870 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_608_6863da4ca0ed68a9855f.png with policy now 2023-07-11 19:10:03,872 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:10:03,872 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_608_0f027a7a2f974076b6e9.png with policy now 2023-07-11 19:10:03,914 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:10:03,915 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:10:03,916 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:10:03,917 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:10:04,116 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/f6a0fl9g-media/images/validation_608_2104758725886dd01ccd.png 2023-07-11 19:10:04,184 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/f75x3x8d-media/images/validation_608_6863da4ca0ed68a9855f.png 2023-07-11 19:10:04,190 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/srapc9kc-media/images/validation_608_e93612b41299320c0daf.png 2023-07-11 19:10:04,243 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ubi4tma5-media/images/validation_608_0f027a7a2f974076b6e9.png 2023-07-11 19:10:04,918 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:10:07,524 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:10:07,525 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:10:07,697 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:10:07,697 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:10:07,698 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:10:07,699 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_608_e93612b41299320c0daf.png 2023-07-11 19:10:07,699 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_608_0f027a7a2f974076b6e9.png 2023-07-11 19:10:07,700 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_608_2104758725886dd01ccd.png 2023-07-11 19:10:07,700 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_608_6863da4ca0ed68a9855f.png 2023-07-11 19:10:07,701 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 19:10:08,049 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/e9aji08x-media/images/validation_608_0f027a7a2f974076b6e9.png 2023-07-11 19:10:08,055 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/j35w4ibn-media/images/validation_608_2104758725886dd01ccd.png 2023-07-11 19:10:08,056 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ilq2vnhz-media/images/validation_608_e93612b41299320c0daf.png 2023-07-11 19:10:08,078 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ad32jcdu-media/images/validation_608_6863da4ca0ed68a9855f.png 2023-07-11 19:10:09,991 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:10:10,628 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:10:11,208 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:10:11,209 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:10:11,210 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:10:11,211 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:10:11,706 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:10:11,706 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:10:14,374 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:10:14,376 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:10:14,376 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:10:14,378 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:10:15,379 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:10:15,715 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:10:15,715 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:10:17,764 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:10:17,766 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:10:17,766 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:10:17,768 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:10:19,724 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:10:19,724 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:10:21,023 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:10:21,034 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:10:21,036 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:10:21,036 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:10:21,038 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:10:22,524 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:10:22,525 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:10:23,733 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:10:23,733 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:10:24,091 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:10:24,093 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:10:24,093 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:10:24,095 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:10:26,238 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:10:27,043 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:10:27,044 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:10:27,045 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:10:27,047 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:10:27,326 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:10:27,328 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:10:27,329 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:10:27,330 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:10:27,743 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:10:27,744 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:10:31,752 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:10:31,879 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:10:36,880 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:10:37,524 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:10:37,525 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:10:39,913 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:10:39,913 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_616_a89cec9bd984af45d04e.png with policy now 2023-07-11 19:10:39,957 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:10:39,957 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_616_3cf079583c2c3172a1af.png with policy now 2023-07-11 19:10:39,958 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:10:39,958 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_616_7dcb623099bcf2fb9044.png with policy now 2023-07-11 19:10:39,959 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:10:39,959 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_616_b8d74a65b6fc0e8293ef.png with policy now 2023-07-11 19:10:40,002 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:10:40,004 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:10:40,004 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:10:40,006 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:10:40,235 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/yaxpmqop-media/images/validation_616_a89cec9bd984af45d04e.png 2023-07-11 19:10:40,284 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/myjax818-media/images/validation_616_7dcb623099bcf2fb9044.png 2023-07-11 19:10:40,302 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/nfww1w4x-media/images/validation_616_3cf079583c2c3172a1af.png 2023-07-11 19:10:40,343 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/7s7vtrcv-media/images/validation_616_b8d74a65b6fc0e8293ef.png 2023-07-11 19:10:40,631 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:10:42,249 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:10:43,781 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:10:43,781 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:10:43,781 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_616_3cf079583c2c3172a1af.png 2023-07-11 19:10:43,782 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_616_7dcb623099bcf2fb9044.png 2023-07-11 19:10:43,782 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_616_a89cec9bd984af45d04e.png 2023-07-11 19:10:43,783 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_616_b8d74a65b6fc0e8293ef.png 2023-07-11 19:10:43,783 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 19:10:44,132 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/18pomf9e-media/images/validation_616_3cf079583c2c3172a1af.png 2023-07-11 19:10:44,153 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/k4eo5pw2-media/images/validation_616_7dcb623099bcf2fb9044.png 2023-07-11 19:10:44,156 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/yiep4el8-media/images/validation_616_b8d74a65b6fc0e8293ef.png 2023-07-11 19:10:44,180 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/5lyncauq-media/images/validation_616_a89cec9bd984af45d04e.png 2023-07-11 19:10:44,447 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:10:47,492 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:10:47,494 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:10:47,494 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:10:47,495 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:10:47,496 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:10:47,790 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:10:47,790 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:10:50,153 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:10:50,155 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:10:50,156 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:10:50,157 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:10:51,800 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:10:51,800 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:10:52,665 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:10:52,800 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:10:52,801 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:10:53,664 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:10:53,667 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:10:53,667 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:10:53,669 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:10:55,809 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:10:55,809 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:10:56,500 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:10:56,502 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:10:56,503 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:10:56,506 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:10:57,989 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:10:59,818 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:10:59,818 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:11:00,233 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:11:00,235 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:11:00,235 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:11:00,237 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:11:03,074 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:11:03,828 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:11:03,828 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:11:04,517 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:11:04,520 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:11:04,520 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:11:04,522 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:11:04,800 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:11:04,802 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:11:04,802 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:11:04,804 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:11:07,801 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:11:07,802 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:11:07,837 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:11:07,837 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:11:08,861 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:11:10,634 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:11:14,637 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:11:17,469 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:11:17,470 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_624_870de5f70c0e3c35f00a.png with policy now 2023-07-11 19:11:17,513 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:11:17,513 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_624_5194e21fa06d24dfa3a2.png with policy now 2023-07-11 19:11:17,514 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:11:17,514 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_624_7988a620d724c156773b.png with policy now 2023-07-11 19:11:17,516 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:11:17,516 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_624_9cd574bf60106cbeb7f7.png with policy now 2023-07-11 19:11:17,559 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:11:17,561 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:11:17,561 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:11:17,563 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:11:17,810 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/8mgsr87v-media/images/validation_624_870de5f70c0e3c35f00a.png 2023-07-11 19:11:17,832 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/7gfd59ue-media/images/validation_624_7988a620d724c156773b.png 2023-07-11 19:11:17,836 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/7gxvd4d7-media/images/validation_624_9cd574bf60106cbeb7f7.png 2023-07-11 19:11:17,841 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/2387gx8l-media/images/validation_624_5194e21fa06d24dfa3a2.png 2023-07-11 19:11:19,865 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:11:19,865 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_624_9cd574bf60106cbeb7f7.png 2023-07-11 19:11:19,866 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_624_5194e21fa06d24dfa3a2.png 2023-07-11 19:11:19,867 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_624_870de5f70c0e3c35f00a.png 2023-07-11 19:11:19,867 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_624_7988a620d724c156773b.png 2023-07-11 19:11:19,868 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 19:11:19,911 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:11:20,219 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/k0ewrwk2-media/images/validation_624_5194e21fa06d24dfa3a2.png 2023-07-11 19:11:20,221 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/5y1xv9f0-media/images/validation_624_9cd574bf60106cbeb7f7.png 2023-07-11 19:11:20,239 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/iszqmv2r-media/images/validation_624_7988a620d724c156773b.png 2023-07-11 19:11:20,252 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/r8aupd9i-media/images/validation_624_870de5f70c0e3c35f00a.png 2023-07-11 19:11:21,369 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:11:22,807 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:11:22,809 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:11:23,874 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:11:24,575 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:11:24,576 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:11:24,576 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:11:24,578 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:11:25,379 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:11:27,883 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:11:27,884 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:11:27,907 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:11:27,909 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:11:27,909 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:11:27,911 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:11:31,125 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:11:31,893 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:11:31,893 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:11:32,527 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:11:32,529 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:11:32,530 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:11:32,531 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:11:35,610 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:11:35,611 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:11:35,612 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:11:35,613 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:11:35,904 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:11:35,905 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:11:36,390 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:11:37,808 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:11:37,809 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:11:39,692 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:11:39,694 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:11:39,694 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:11:39,695 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:11:39,914 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:11:39,915 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:11:40,637 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:11:41,756 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:11:42,698 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:11:42,700 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:11:42,700 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:11:42,702 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:11:42,985 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:11:42,987 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:11:42,987 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:11:42,988 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:11:43,925 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:11:43,925 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:11:47,510 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:11:47,933 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:11:52,511 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:11:52,827 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:11:52,828 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:11:55,560 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:11:55,561 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_632_9db58ceffde2ac6026b2.png with policy now 2023-07-11 19:11:55,601 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:11:55,601 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_632_90d4a1e76489230b3d0e.png with policy now 2023-07-11 19:11:55,602 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:11:55,602 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_632_cbe1621c20b110b72424.png with policy now 2023-07-11 19:11:55,603 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:11:55,603 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_632_e3edcf43076ac8fbe703.png with policy now 2023-07-11 19:11:55,646 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:11:55,647 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:11:55,647 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:11:55,649 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:11:55,869 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/c5o6sw30-media/images/validation_632_9db58ceffde2ac6026b2.png 2023-07-11 19:11:55,919 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/r19mmner-media/images/validation_632_cbe1621c20b110b72424.png 2023-07-11 19:11:55,945 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/7hz0tl4o-media/images/validation_632_90d4a1e76489230b3d0e.png 2023-07-11 19:11:55,950 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/6egqimbz-media/images/validation_632_e3edcf43076ac8fbe703.png 2023-07-11 19:11:55,953 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:11:55,953 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_632_cbe1621c20b110b72424.png 2023-07-11 19:11:55,954 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_632_90d4a1e76489230b3d0e.png 2023-07-11 19:11:55,955 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_632_e3edcf43076ac8fbe703.png 2023-07-11 19:11:55,955 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_632_9db58ceffde2ac6026b2.png 2023-07-11 19:11:55,955 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 19:11:56,296 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/7xczlp23-media/images/validation_632_cbe1621c20b110b72424.png 2023-07-11 19:11:56,310 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/55j1qlzf-media/images/validation_632_90d4a1e76489230b3d0e.png 2023-07-11 19:11:56,313 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/zpz7nyqc-media/images/validation_632_e3edcf43076ac8fbe703.png 2023-07-11 19:11:56,329 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/vy3iprrr-media/images/validation_632_9db58ceffde2ac6026b2.png 2023-07-11 19:11:58,219 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:11:59,642 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:11:59,959 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:12:02,501 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:12:02,503 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:12:02,504 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:12:02,505 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:12:03,456 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:12:03,968 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:12:03,968 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:12:06,709 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:12:06,711 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:12:06,711 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:12:06,713 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:12:07,827 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:12:07,828 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:12:07,977 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:12:07,977 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:12:08,961 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:12:09,809 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:12:09,810 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:12:09,811 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:12:09,812 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:12:10,642 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:12:11,981 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:12:11,982 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:12:12,883 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:12:12,885 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:12:12,885 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:12:12,887 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:12:14,090 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:12:15,985 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:12:15,986 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:12:17,192 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:12:17,194 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:12:17,194 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:12:17,196 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:12:19,125 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:12:19,991 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:12:19,991 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:12:20,207 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:12:20,209 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:12:20,209 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:12:20,211 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:12:20,490 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:12:20,492 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:12:20,492 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:12:20,494 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:12:22,859 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:12:22,860 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:12:23,994 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:12:23,995 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:12:24,913 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:12:29,914 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:12:33,043 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:12:33,043 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_640_b0840e49c29d03b310d6.png with policy now 2023-07-11 19:12:33,085 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:12:33,085 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_640_926cd9184da41f18f64c.png with policy now 2023-07-11 19:12:33,086 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:12:33,086 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_640_988f0fde8c0b0bda7e16.png with policy now 2023-07-11 19:12:33,096 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:12:33,097 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_640_cf8925a87c42c9b29e53.png with policy now 2023-07-11 19:12:33,134 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:12:33,136 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:12:33,136 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:12:33,138 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:12:34,925 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:12:36,007 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:12:36,008 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_640_b0840e49c29d03b310d6.png 2023-07-11 19:12:36,008 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_640_988f0fde8c0b0bda7e16.png 2023-07-11 19:12:36,009 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_640_cf8925a87c42c9b29e53.png 2023-07-11 19:12:36,009 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_640_926cd9184da41f18f64c.png 2023-07-11 19:12:36,010 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 19:12:36,925 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/b9x6oxhp-media/images/validation_640_b0840e49c29d03b310d6.png 2023-07-11 19:12:37,105 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/o1ha19r5-media/images/validation_640_988f0fde8c0b0bda7e16.png 2023-07-11 19:12:37,859 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:12:37,860 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:12:37,925 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/fpoiw32c-media/images/validation_640_926cd9184da41f18f64c.png 2023-07-11 19:12:37,926 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:12:38,055 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/vx2rd80v-media/images/validation_640_cf8925a87c42c9b29e53.png 2023-07-11 19:12:38,297 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/2my82bmb-media/images/validation_640_b0840e49c29d03b310d6.png 2023-07-11 19:12:38,415 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/msc4gy57-media/images/validation_640_988f0fde8c0b0bda7e16.png 2023-07-11 19:12:38,444 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/wq6r2v0m-media/images/validation_640_926cd9184da41f18f64c.png 2023-07-11 19:12:38,721 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/9kn9df0l-media/images/validation_640_cf8925a87c42c9b29e53.png 2023-07-11 19:12:40,017 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:12:40,461 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:12:40,643 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:12:41,367 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:12:41,369 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:12:41,370 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:12:41,371 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:12:43,852 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:12:43,854 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:12:43,854 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:12:43,855 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:12:44,026 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:12:44,027 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:12:45,593 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:12:47,028 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:12:47,030 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:12:47,030 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:12:47,031 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:12:48,036 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:12:48,037 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:12:49,885 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:12:49,887 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:12:49,887 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:12:49,888 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:12:51,285 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:12:52,045 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:12:52,046 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:12:52,727 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:12:52,728 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:12:52,728 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:12:52,730 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:12:52,859 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:12:52,860 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:12:56,055 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:12:56,055 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:12:56,624 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:12:56,625 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:12:56,626 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:12:56,626 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:12:56,627 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:12:56,907 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:12:56,909 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:12:56,909 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:12:56,911 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:13:00,065 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:13:00,065 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:13:02,447 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:13:07,448 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:13:07,860 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:13:07,861 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:13:09,464 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:13:09,464 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_648_728dd91334abb53b86d0.png with policy now 2023-07-11 19:13:09,505 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:13:09,505 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_648_55287d9a3ca04b88ba22.png with policy now 2023-07-11 19:13:09,506 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:13:09,506 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_648_7c2b88efee87b1a3d3bc.png with policy now 2023-07-11 19:13:09,515 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:13:09,515 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_648_5b836be338a558b667f9.png with policy now 2023-07-11 19:13:09,556 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:13:09,558 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:13:09,558 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:13:09,560 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:13:09,778 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/j60srj0x-media/images/validation_648_728dd91334abb53b86d0.png 2023-07-11 19:13:09,834 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ltx2qakt-media/images/validation_648_7c2b88efee87b1a3d3bc.png 2023-07-11 19:13:09,837 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/rvbfde1p-media/images/validation_648_5b836be338a558b667f9.png 2023-07-11 19:13:09,848 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/5q6tfs4q-media/images/validation_648_55287d9a3ca04b88ba22.png 2023-07-11 19:13:10,646 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:13:12,093 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:13:12,094 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_648_728dd91334abb53b86d0.png 2023-07-11 19:13:12,095 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_648_7c2b88efee87b1a3d3bc.png 2023-07-11 19:13:12,095 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_648_5b836be338a558b667f9.png 2023-07-11 19:13:12,096 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_648_55287d9a3ca04b88ba22.png 2023-07-11 19:13:12,096 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 19:13:12,436 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/02p1p3zw-media/images/validation_648_728dd91334abb53b86d0.png 2023-07-11 19:13:12,452 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/hwxwdq45-media/images/validation_648_7c2b88efee87b1a3d3bc.png 2023-07-11 19:13:12,473 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/dpp8u71m-media/images/validation_648_55287d9a3ca04b88ba22.png 2023-07-11 19:13:12,509 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/7d6y6tsw-media/images/validation_648_5b836be338a558b667f9.png 2023-07-11 19:13:13,182 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:13:13,199 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:13:16,104 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:13:17,245 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:13:17,247 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:13:17,247 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:13:17,249 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:13:18,884 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:13:20,113 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:13:20,113 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:13:21,468 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:13:21,470 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:13:21,470 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:13:21,472 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:13:22,859 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:13:22,860 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:13:24,123 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:13:24,123 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:13:24,322 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:13:24,324 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:13:24,324 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:13:24,325 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:13:24,326 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:13:26,911 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:13:26,913 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:13:26,913 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:13:26,915 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:13:28,132 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:13:28,133 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:13:29,456 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:13:30,827 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:13:30,829 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:13:30,829 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:13:30,831 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:13:32,142 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:13:32,142 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:13:34,155 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:13:34,157 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:13:34,157 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:13:34,159 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:13:34,439 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:13:34,441 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:13:34,441 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:13:34,443 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:13:34,872 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:13:36,152 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:13:36,153 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:13:37,873 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:13:37,874 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:13:39,933 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:13:40,162 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:13:40,649 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:13:45,651 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:13:47,078 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:13:47,079 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_656_44ad3c8fcf80a92c74c0.png with policy now 2023-07-11 19:13:47,121 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:13:47,121 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_656_c5399b2a25bca627fffe.png with policy now 2023-07-11 19:13:47,122 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:13:47,122 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_656_09858d6b1f00f78cae5c.png with policy now 2023-07-11 19:13:47,122 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:13:47,122 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_656_c7768aa4e07faefe0067.png with policy now 2023-07-11 19:13:47,163 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:13:47,164 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:13:47,165 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:13:47,166 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:13:47,450 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/atoey0bg-media/images/validation_656_c5399b2a25bca627fffe.png 2023-07-11 19:13:47,460 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ld8ap8sr-media/images/validation_656_09858d6b1f00f78cae5c.png 2023-07-11 19:13:47,464 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/fwo4eoao-media/images/validation_656_c7768aa4e07faefe0067.png 2023-07-11 19:13:47,466 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ebm2ok6g-media/images/validation_656_44ad3c8fcf80a92c74c0.png 2023-07-11 19:13:48,182 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:13:48,182 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_656_c5399b2a25bca627fffe.png 2023-07-11 19:13:48,183 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_656_c7768aa4e07faefe0067.png 2023-07-11 19:13:48,183 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_656_09858d6b1f00f78cae5c.png 2023-07-11 19:13:48,184 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_656_44ad3c8fcf80a92c74c0.png 2023-07-11 19:13:48,184 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 19:13:48,545 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/x1lw1pfj-media/images/validation_656_c7768aa4e07faefe0067.png 2023-07-11 19:13:48,546 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/5zaz551c-media/images/validation_656_c5399b2a25bca627fffe.png 2023-07-11 19:13:48,666 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/nup0s4t9-media/images/validation_656_44ad3c8fcf80a92c74c0.png 2023-07-11 19:13:48,690 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/g73icbsc-media/images/validation_656_09858d6b1f00f78cae5c.png 2023-07-11 19:13:50,993 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:13:52,197 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:13:52,204 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:13:52,875 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:13:52,876 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:13:54,995 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:13:54,997 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:13:54,998 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:13:54,999 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:13:56,206 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:13:56,207 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:13:56,275 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:13:57,827 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:13:57,829 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:13:57,829 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:13:57,830 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:14:00,216 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:14:00,217 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:14:01,221 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:14:01,223 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:14:01,223 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:14:01,225 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:14:01,798 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:14:04,226 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:14:04,226 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:14:04,560 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:14:04,561 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:14:04,561 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:14:04,563 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:14:07,425 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:14:07,427 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:14:07,427 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:14:07,428 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:14:07,430 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:14:07,875 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:14:07,876 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:14:08,236 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:14:08,236 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:14:10,651 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:14:11,220 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:14:11,221 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:14:11,222 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:14:11,223 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:14:11,503 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:14:11,505 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:14:11,505 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:14:11,507 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:14:12,245 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:14:12,246 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:14:13,074 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:14:16,255 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:14:18,075 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:14:22,875 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:14:22,876 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:14:23,926 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:14:24,057 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:14:24,058 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_664_158f555f9864c3964f75.png with policy now 2023-07-11 19:14:24,072 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:14:24,072 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_664_95c67446531a911a320b.png with policy now 2023-07-11 19:14:24,086 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:14:24,086 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_664_092cf810659ff0a3ad9b.png with policy now 2023-07-11 19:14:24,099 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:14:24,099 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_664_6d9e47d176e82143fca1.png with policy now 2023-07-11 19:14:24,141 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:14:24,144 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:14:24,145 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:14:24,147 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:14:24,274 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:14:24,274 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_664_092cf810659ff0a3ad9b.png 2023-07-11 19:14:24,275 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_664_158f555f9864c3964f75.png 2023-07-11 19:14:24,275 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_664_95c67446531a911a320b.png 2023-07-11 19:14:24,275 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_664_6d9e47d176e82143fca1.png 2023-07-11 19:14:24,275 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 19:14:24,359 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/0fwuiqqv-media/images/validation_664_158f555f9864c3964f75.png 2023-07-11 19:14:24,393 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/5m72ez2s-media/images/validation_664_95c67446531a911a320b.png 2023-07-11 19:14:24,404 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/c6uys8co-media/images/validation_664_092cf810659ff0a3ad9b.png 2023-07-11 19:14:24,432 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/xsp7f6yj-media/images/validation_664_6d9e47d176e82143fca1.png 2023-07-11 19:14:24,750 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/3zcir8y2-media/images/validation_664_158f555f9864c3964f75.png 2023-07-11 19:14:24,794 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/yevjegd6-media/images/validation_664_95c67446531a911a320b.png 2023-07-11 19:14:24,976 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/1hf8ly7q-media/images/validation_664_6d9e47d176e82143fca1.png 2023-07-11 19:14:25,109 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/vic771vp-media/images/validation_664_092cf810659ff0a3ad9b.png 2023-07-11 19:14:27,833 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:14:28,284 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:14:29,308 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:14:31,355 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:14:31,356 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:14:31,357 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:14:31,358 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:14:32,293 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:14:32,294 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:14:35,175 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:14:35,177 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:14:35,177 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:14:35,177 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:14:35,179 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:14:36,303 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:14:36,303 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:14:37,879 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:14:37,880 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:14:38,614 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:14:38,615 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:14:38,616 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:14:38,618 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:14:40,313 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:14:40,313 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:14:40,486 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:14:40,655 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:14:41,380 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:14:41,382 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:14:41,382 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:14:41,384 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:14:44,323 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:14:44,323 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:14:45,556 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:14:45,557 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:14:45,558 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:14:45,558 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:14:45,559 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:14:48,333 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:14:48,334 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:14:48,592 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:14:48,594 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:14:48,595 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:14:48,596 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:14:48,874 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:14:48,876 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:14:48,876 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:14:48,878 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:14:50,561 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:14:52,343 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:14:52,343 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:14:52,885 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:14:52,886 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:14:55,943 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:15:00,944 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:15:01,612 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:15:01,613 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_672_e6e648fc0801971ed27f.png with policy now 2023-07-11 19:15:01,653 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:15:01,653 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_672_640323be00d22421969d.png with policy now 2023-07-11 19:15:01,653 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:15:01,653 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_672_e7cccd2e24c981238fe9.png with policy now 2023-07-11 19:15:01,663 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:15:01,663 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_672_0a35aec0325a8b5931bd.png with policy now 2023-07-11 19:15:01,726 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:15:01,728 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:15:01,728 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:15:01,734 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:15:01,932 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/m1rjlvyq-media/images/validation_672_e6e648fc0801971ed27f.png 2023-07-11 19:15:01,969 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/oua40wqp-media/images/validation_672_e7cccd2e24c981238fe9.png 2023-07-11 19:15:01,975 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/j7pi2w4a-media/images/validation_672_640323be00d22421969d.png 2023-07-11 19:15:01,987 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/y1ythefs-media/images/validation_672_0a35aec0325a8b5931bd.png 2023-07-11 19:15:04,373 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:15:04,373 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:15:04,374 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_672_0a35aec0325a8b5931bd.png 2023-07-11 19:15:04,374 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_672_e6e648fc0801971ed27f.png 2023-07-11 19:15:04,375 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_672_640323be00d22421969d.png 2023-07-11 19:15:04,375 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_672_e7cccd2e24c981238fe9.png 2023-07-11 19:15:04,376 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 19:15:04,711 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/8bmozrj5-media/images/validation_672_0a35aec0325a8b5931bd.png 2023-07-11 19:15:04,717 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/9lpx4fuc-media/images/validation_672_e6e648fc0801971ed27f.png 2023-07-11 19:15:04,719 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/1tvn6gte-media/images/validation_672_640323be00d22421969d.png 2023-07-11 19:15:04,735 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/whzwkbw0-media/images/validation_672_e7cccd2e24c981238fe9.png 2023-07-11 19:15:05,260 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:15:06,261 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:15:07,899 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:15:07,900 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:15:07,948 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:15:08,024 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:15:08,024 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:15:08,026 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:15:08,383 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:15:08,383 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:15:10,658 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:15:11,660 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:15:11,749 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:15:11,750 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:15:11,750 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:15:11,752 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:15:12,393 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:15:12,393 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:15:15,378 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:15:15,379 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:15:15,380 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:15:15,381 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:15:16,403 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:15:16,403 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:15:17,251 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:15:18,125 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:15:18,126 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:15:18,127 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:15:18,128 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:15:20,408 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:15:20,409 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:15:21,559 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:15:21,560 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:15:21,561 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:15:21,562 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:15:22,364 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:15:22,913 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:15:22,914 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:15:24,385 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:15:24,387 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:15:24,387 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:15:24,389 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:15:24,418 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:15:24,418 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:15:24,666 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:15:24,668 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:15:24,669 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:15:24,670 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:15:28,216 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:15:28,428 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:15:28,429 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:15:33,217 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:15:37,280 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:15:37,280 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_680_aa194dccdb4cde13bf28.png with policy now 2023-07-11 19:15:37,297 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:15:37,298 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_680_5296d50b33663e449a0e.png with policy now 2023-07-11 19:15:37,312 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:15:37,312 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_680_72ef33de170d84e10fda.png with policy now 2023-07-11 19:15:37,330 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:15:37,330 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_680_590049620b94f7f699ff.png with policy now 2023-07-11 19:15:37,378 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:15:37,379 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:15:37,379 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:15:37,386 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:15:37,611 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/iu7u8cyf-media/images/validation_680_aa194dccdb4cde13bf28.png 2023-07-11 19:15:37,636 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/7nojgqrm-media/images/validation_680_590049620b94f7f699ff.png 2023-07-11 19:15:37,663 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/5itg5fwa-media/images/validation_680_72ef33de170d84e10fda.png 2023-07-11 19:15:37,913 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:15:37,914 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:15:38,321 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/jo4piuzt-media/images/validation_680_5296d50b33663e449a0e.png 2023-07-11 19:15:38,432 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:15:40,460 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:15:40,460 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:15:40,461 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_680_5296d50b33663e449a0e.png 2023-07-11 19:15:40,461 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_680_aa194dccdb4cde13bf28.png 2023-07-11 19:15:40,462 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_680_590049620b94f7f699ff.png 2023-07-11 19:15:40,462 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_680_72ef33de170d84e10fda.png 2023-07-11 19:15:40,463 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 19:15:40,660 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:15:40,841 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ivd1c7yk-media/images/validation_680_5296d50b33663e449a0e.png 2023-07-11 19:15:40,844 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/e9oc6ttn-media/images/validation_680_590049620b94f7f699ff.png 2023-07-11 19:15:40,844 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/l1venovd-media/images/validation_680_aa194dccdb4cde13bf28.png 2023-07-11 19:15:40,857 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/88wrv275-media/images/validation_680_72ef33de170d84e10fda.png 2023-07-11 19:15:41,387 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:15:44,131 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:15:44,468 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:15:44,823 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:15:44,825 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:15:44,825 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:15:44,827 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:15:48,257 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:15:48,258 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:15:48,259 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:15:48,260 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:15:48,479 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:15:48,479 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:15:49,963 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:15:51,311 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:15:51,313 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:15:51,313 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:15:51,315 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:15:52,489 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:15:52,489 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:15:53,163 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:15:53,164 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:15:54,569 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:15:54,570 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:15:54,571 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:15:54,572 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:15:55,343 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:15:56,499 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:15:56,499 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:15:57,661 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:15:57,663 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:15:57,663 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:15:57,665 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:16:00,509 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:16:00,509 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:16:00,784 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:16:00,786 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:16:00,786 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:16:00,787 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:16:00,789 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:16:01,066 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:16:01,068 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:16:01,069 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:16:01,070 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:16:04,519 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:16:04,519 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:16:06,711 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:16:08,163 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:16:08,163 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:16:10,663 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:16:12,666 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:16:13,725 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:16:13,726 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_688_dc7a2400adb3ddfda5f1.png with policy now 2023-07-11 19:16:13,741 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:16:13,742 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_688_9f4c23b3d6799b0996b4.png with policy now 2023-07-11 19:16:13,757 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:16:13,757 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_688_1987fb0f39545442f7d1.png with policy now 2023-07-11 19:16:13,773 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:16:13,773 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_688_59f2ed6f733f8149780a.png with policy now 2023-07-11 19:16:13,816 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:16:13,818 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:16:13,818 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:16:13,819 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:16:14,022 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/f421mrct-media/images/validation_688_dc7a2400adb3ddfda5f1.png 2023-07-11 19:16:14,055 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/8i64gt3h-media/images/validation_688_9f4c23b3d6799b0996b4.png 2023-07-11 19:16:14,059 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/se9b2ukl-media/images/validation_688_1987fb0f39545442f7d1.png 2023-07-11 19:16:14,072 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/b9h1qwzj-media/images/validation_688_59f2ed6f733f8149780a.png 2023-07-11 19:16:16,535 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:16:16,536 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_688_dc7a2400adb3ddfda5f1.png 2023-07-11 19:16:16,536 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_688_59f2ed6f733f8149780a.png 2023-07-11 19:16:16,537 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_688_1987fb0f39545442f7d1.png 2023-07-11 19:16:16,537 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_688_9f4c23b3d6799b0996b4.png 2023-07-11 19:16:16,538 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 19:16:16,931 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ez7h1149-media/images/validation_688_59f2ed6f733f8149780a.png 2023-07-11 19:16:16,937 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ixv5rodl-media/images/validation_688_dc7a2400adb3ddfda5f1.png 2023-07-11 19:16:16,956 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/c7ng19u2-media/images/validation_688_1987fb0f39545442f7d1.png 2023-07-11 19:16:16,972 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/jtjlc0f2-media/images/validation_688_9f4c23b3d6799b0996b4.png 2023-07-11 19:16:18,233 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:16:18,234 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:16:20,546 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:16:20,694 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:16:20,696 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:16:20,697 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:16:20,698 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:16:23,332 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:16:23,332 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:16:23,333 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:16:24,184 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:16:24,186 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:16:24,187 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:16:24,188 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:16:24,558 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:16:24,558 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:16:27,542 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:16:27,544 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:16:27,545 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:16:27,546 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:16:28,570 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:16:28,571 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:16:28,965 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:16:30,713 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:16:30,714 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:16:30,715 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:16:30,716 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:16:32,579 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:16:32,579 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:16:33,334 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:16:33,335 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:16:33,336 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:16:33,337 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:16:34,338 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:16:36,589 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:16:36,589 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:16:36,662 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:16:36,664 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:16:36,664 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:16:36,666 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:16:36,945 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:16:36,947 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:16:36,947 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:16:36,949 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:16:38,372 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:16:38,373 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:16:39,477 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:16:40,599 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:16:40,600 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:16:40,666 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:16:44,668 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:16:49,509 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:16:49,509 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_696_bf57326caaf8c0b6ad64.png with policy now 2023-07-11 19:16:49,525 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:16:49,525 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_696_b61dd12befa2c9cd68d2.png with policy now 2023-07-11 19:16:49,540 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:16:49,540 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_696_72197a206eb0dae2a9b9.png with policy now 2023-07-11 19:16:49,553 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:16:49,553 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_696_597ae9e605560c042f4a.png with policy now 2023-07-11 19:16:49,597 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:16:49,600 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:16:49,600 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:16:49,602 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:16:49,840 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/7a7092ff-media/images/validation_696_bf57326caaf8c0b6ad64.png 2023-07-11 19:16:49,869 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ntcy8voa-media/images/validation_696_b61dd12befa2c9cd68d2.png 2023-07-11 19:16:49,881 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/qdctbiqu-media/images/validation_696_72197a206eb0dae2a9b9.png 2023-07-11 19:16:49,884 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/vj9fu1y2-media/images/validation_696_597ae9e605560c042f4a.png 2023-07-11 19:16:50,603 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:16:52,631 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:16:52,631 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:16:52,632 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_696_bf57326caaf8c0b6ad64.png 2023-07-11 19:16:52,632 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_696_72197a206eb0dae2a9b9.png 2023-07-11 19:16:52,633 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_696_b61dd12befa2c9cd68d2.png 2023-07-11 19:16:52,633 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_696_597ae9e605560c042f4a.png 2023-07-11 19:16:52,634 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 19:16:52,731 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:16:52,975 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/wqbi3dti-media/images/validation_696_bf57326caaf8c0b6ad64.png 2023-07-11 19:16:52,977 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/kzhsmekt-media/images/validation_696_72197a206eb0dae2a9b9.png 2023-07-11 19:16:52,988 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/6o12bjsa-media/images/validation_696_597ae9e605560c042f4a.png 2023-07-11 19:16:52,999 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/k0kvrvat-media/images/validation_696_b61dd12befa2c9cd68d2.png 2023-07-11 19:16:53,530 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:16:53,531 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:16:55,306 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:16:55,308 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:16:55,309 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:16:55,310 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:16:56,064 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:16:56,640 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:16:56,640 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:16:58,112 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:16:58,114 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:16:58,115 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:16:58,116 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:17:00,649 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:17:00,650 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:17:01,269 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:17:01,270 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:17:01,271 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:17:01,271 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:17:01,273 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:17:04,659 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:17:04,660 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:17:04,805 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:17:04,807 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:17:04,807 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:17:04,809 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:17:06,686 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:17:08,474 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:17:08,476 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:17:08,477 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:17:08,478 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:17:08,529 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:17:08,530 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:17:08,669 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:17:08,669 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:17:10,668 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:17:12,017 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:17:12,019 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:17:12,019 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:17:12,020 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:17:12,021 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:17:12,303 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:17:12,305 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:17:12,306 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:17:12,307 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:17:12,679 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:17:12,680 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:17:16,689 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:17:17,884 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:17:22,885 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:17:23,530 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:17:23,530 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:17:24,889 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:17:24,889 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_704_c190c2a29844123428a8.png with policy now 2023-07-11 19:17:24,904 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:17:24,904 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_704_ff8ed0b57b72882d609c.png with policy now 2023-07-11 19:17:24,938 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:17:24,938 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_704_072c318a44d7c7157563.png with policy now 2023-07-11 19:17:24,954 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:17:24,954 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_704_81266f0e45c6c5305caa.png with policy now 2023-07-11 19:17:24,992 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:17:24,993 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:17:24,994 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:17:24,995 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:17:25,188 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/t29w27hg-media/images/validation_704_c190c2a29844123428a8.png 2023-07-11 19:17:25,229 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/7naimfoa-media/images/validation_704_ff8ed0b57b72882d609c.png 2023-07-11 19:17:25,250 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/8tzmf6dx-media/images/validation_704_072c318a44d7c7157563.png 2023-07-11 19:17:25,254 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/79yexbdw-media/images/validation_704_81266f0e45c6c5305caa.png 2023-07-11 19:17:27,967 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:17:28,721 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:17:28,721 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:17:28,722 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_704_81266f0e45c6c5305caa.png 2023-07-11 19:17:28,722 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_704_072c318a44d7c7157563.png 2023-07-11 19:17:28,723 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_704_ff8ed0b57b72882d609c.png 2023-07-11 19:17:28,723 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_704_c190c2a29844123428a8.png 2023-07-11 19:17:28,724 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 19:17:28,751 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:17:29,080 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/16nj8tad-media/images/validation_704_81266f0e45c6c5305caa.png 2023-07-11 19:17:29,102 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/p10pvz2j-media/images/validation_704_072c318a44d7c7157563.png 2023-07-11 19:17:29,128 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/nacnevp7-media/images/validation_704_c190c2a29844123428a8.png 2023-07-11 19:17:29,150 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/znea3ba4-media/images/validation_704_ff8ed0b57b72882d609c.png 2023-07-11 19:17:31,759 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:17:31,760 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:17:31,761 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:17:31,762 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:17:32,731 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:17:32,731 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:17:33,127 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:17:34,486 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:17:34,488 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:17:34,488 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:17:34,490 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:17:36,741 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:17:36,742 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:17:37,613 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:17:37,614 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:17:37,615 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:17:37,616 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:17:38,530 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:17:38,531 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:17:38,639 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:17:40,672 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:17:40,751 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:17:40,751 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:17:41,033 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:17:41,035 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:17:41,035 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:17:41,036 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:17:44,066 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:17:44,068 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:17:44,068 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:17:44,069 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:17:44,070 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:17:44,761 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:17:44,762 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:17:47,831 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:17:47,833 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:17:47,834 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:17:47,835 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:17:48,114 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:17:48,116 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:17:48,116 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:17:48,118 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:17:48,772 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:17:48,772 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:17:49,732 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:17:52,781 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:17:53,558 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:17:53,559 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:17:55,599 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:18:00,600 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:18:00,826 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:18:00,826 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_712_6ba33f9b1438e729f3e3.png with policy now 2023-07-11 19:18:00,869 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:18:00,869 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_712_7fff33d2cf3b6bfe2855.png with policy now 2023-07-11 19:18:00,870 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:18:00,870 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_712_a41798bd5ba67711765f.png with policy now 2023-07-11 19:18:00,870 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:18:00,871 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_712_e66ad705dd270fae09fa.png with policy now 2023-07-11 19:18:00,899 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:18:00,900 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:18:00,900 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:18:00,901 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:18:01,130 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/0eo2vrn5-media/images/validation_712_6ba33f9b1438e729f3e3.png 2023-07-11 19:18:01,180 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/14ou5d47-media/images/validation_712_a41798bd5ba67711765f.png 2023-07-11 19:18:01,197 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/4n4mahvp-media/images/validation_712_7fff33d2cf3b6bfe2855.png 2023-07-11 19:18:01,206 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/9s3ncj21-media/images/validation_712_e66ad705dd270fae09fa.png 2023-07-11 19:18:04,773 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:18:04,813 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:18:04,813 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:18:04,814 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_712_6ba33f9b1438e729f3e3.png 2023-07-11 19:18:04,814 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_712_e66ad705dd270fae09fa.png 2023-07-11 19:18:04,815 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_712_a41798bd5ba67711765f.png 2023-07-11 19:18:04,816 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_712_7fff33d2cf3b6bfe2855.png 2023-07-11 19:18:04,816 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 19:18:05,167 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/e3js9pwr-media/images/validation_712_6ba33f9b1438e729f3e3.png 2023-07-11 19:18:05,175 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/4xg3xx6u-media/images/validation_712_a41798bd5ba67711765f.png 2023-07-11 19:18:05,184 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/bgpqxfw1-media/images/validation_712_e66ad705dd270fae09fa.png 2023-07-11 19:18:05,188 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/4bib2m65-media/images/validation_712_7fff33d2cf3b6bfe2855.png 2023-07-11 19:18:05,774 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:18:08,041 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:18:08,043 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:18:08,043 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:18:08,158 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:18:08,785 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:18:08,786 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:18:08,824 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:18:08,824 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:18:10,675 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:18:11,467 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:18:11,469 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:18:11,469 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:18:11,470 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:18:11,471 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:18:12,834 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:18:12,834 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:18:15,453 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:18:15,455 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:18:15,455 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:18:15,457 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:18:16,844 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:18:16,844 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:18:17,057 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:18:18,254 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:18:18,255 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:18:18,256 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:18:18,258 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:18:20,854 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:18:20,854 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:18:21,343 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:18:21,345 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:18:21,345 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:18:21,347 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:18:22,955 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:18:23,784 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:18:23,785 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:18:24,864 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:18:24,865 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:18:25,092 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:18:25,094 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:18:25,094 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:18:25,096 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:18:25,377 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:18:25,379 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:18:25,379 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:18:25,380 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:18:28,038 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:18:28,875 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:18:28,875 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:18:33,039 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:18:38,040 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:18:38,066 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:18:38,066 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_720_a76739d06259d72923e3.png with policy now 2023-07-11 19:18:38,081 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:18:38,081 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_720_3ac906f6178bc7cc5030.png with policy now 2023-07-11 19:18:38,089 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:18:38,090 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_720_18dae1edaa995fefc02c.png with policy now 2023-07-11 19:18:38,099 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:18:38,099 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_720_f473f15d1fa6dae04183.png with policy now 2023-07-11 19:18:38,136 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:18:38,137 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:18:38,137 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:18:38,139 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:18:38,414 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/7thisoua-media/images/validation_720_18dae1edaa995fefc02c.png 2023-07-11 19:18:38,420 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ozh5lehx-media/images/validation_720_a76739d06259d72923e3.png 2023-07-11 19:18:38,427 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/5jpqq70h-media/images/validation_720_f473f15d1fa6dae04183.png 2023-07-11 19:18:38,433 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/l5ncd0dk-media/images/validation_720_3ac906f6178bc7cc5030.png 2023-07-11 19:18:38,784 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:18:38,786 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:18:40,677 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:18:40,906 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:18:40,906 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_720_a76739d06259d72923e3.png 2023-07-11 19:18:40,907 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_720_f473f15d1fa6dae04183.png 2023-07-11 19:18:40,907 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_720_18dae1edaa995fefc02c.png 2023-07-11 19:18:40,908 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_720_3ac906f6178bc7cc5030.png 2023-07-11 19:18:40,908 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 19:18:41,295 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/5xq4efm9-media/images/validation_720_f473f15d1fa6dae04183.png 2023-07-11 19:18:41,300 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/qqg67wfi-media/images/validation_720_a76739d06259d72923e3.png 2023-07-11 19:18:41,317 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/fim1xq5d-media/images/validation_720_18dae1edaa995fefc02c.png 2023-07-11 19:18:41,324 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/2h1uy2y4-media/images/validation_720_3ac906f6178bc7cc5030.png 2023-07-11 19:18:42,269 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:18:43,593 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:18:44,917 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:18:45,475 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:18:45,477 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:18:45,477 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:18:45,479 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:18:48,626 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:18:48,627 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:18:48,629 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:18:48,629 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:18:48,631 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:18:48,927 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:18:48,927 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:18:52,093 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:18:52,095 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:18:52,095 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:18:52,097 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:18:52,937 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:18:52,938 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:18:53,785 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:18:53,785 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:18:53,786 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:18:55,659 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:18:55,661 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:18:55,661 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:18:55,663 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:18:56,946 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:18:56,946 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:18:58,497 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:18:58,499 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:18:58,499 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:18:58,501 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:18:59,014 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:19:00,953 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:19:00,954 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:19:01,130 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:19:01,132 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:19:01,132 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:19:01,134 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:19:01,415 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:19:01,417 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:19:01,418 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:19:01,419 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:19:04,965 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:19:04,965 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:19:04,973 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:19:08,803 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:19:08,804 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:19:10,680 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:19:10,685 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:19:14,065 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:19:14,065 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_728_85f42fd5b9d94ea9077f.png with policy now 2023-07-11 19:19:14,109 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:19:14,109 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_728_6d3391d06d6901eb149d.png with policy now 2023-07-11 19:19:14,110 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:19:14,110 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_728_b82c35a5387d49e858c0.png with policy now 2023-07-11 19:19:14,110 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:19:14,110 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_728_7db45a659309d31efe67.png with policy now 2023-07-11 19:19:14,136 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:19:14,138 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:19:14,138 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:19:14,139 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:19:14,387 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/nlck81d1-media/images/validation_728_85f42fd5b9d94ea9077f.png 2023-07-11 19:19:14,425 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/qmir8prm-media/images/validation_728_7db45a659309d31efe67.png 2023-07-11 19:19:14,440 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/fluryrfs-media/images/validation_728_b82c35a5387d49e858c0.png 2023-07-11 19:19:14,445 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/lkwroqu8-media/images/validation_728_6d3391d06d6901eb149d.png 2023-07-11 19:19:16,380 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:19:16,994 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:19:16,995 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_728_b82c35a5387d49e858c0.png 2023-07-11 19:19:16,995 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_728_85f42fd5b9d94ea9077f.png 2023-07-11 19:19:16,996 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_728_7db45a659309d31efe67.png 2023-07-11 19:19:16,996 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_728_6d3391d06d6901eb149d.png 2023-07-11 19:19:16,997 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 19:19:17,337 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ulwy54y4-media/images/validation_728_7db45a659309d31efe67.png 2023-07-11 19:19:17,341 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/7eplmf77-media/images/validation_728_85f42fd5b9d94ea9077f.png 2023-07-11 19:19:17,344 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/qfvp2yp9-media/images/validation_728_b82c35a5387d49e858c0.png 2023-07-11 19:19:17,504 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/rbcyxqhm-media/images/validation_728_6d3391d06d6901eb149d.png 2023-07-11 19:19:18,866 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:19:21,005 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:19:21,858 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:19:21,860 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:19:21,860 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:19:21,861 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:19:21,863 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:19:23,803 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:19:23,804 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:19:25,016 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:19:25,016 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:19:25,445 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:19:25,447 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:19:25,447 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:19:25,449 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:19:27,673 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:19:29,026 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:19:29,026 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:19:29,614 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:19:29,616 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:19:29,617 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:19:29,618 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:19:32,847 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:19:32,849 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:19:32,850 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:19:32,850 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:19:32,852 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:19:33,035 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:19:33,036 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:19:36,030 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:19:36,031 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:19:36,032 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:19:36,033 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:19:37,043 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:19:37,044 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:19:38,253 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:19:38,830 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:19:38,830 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:19:39,000 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:19:39,002 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:19:39,002 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:19:39,003 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:19:39,239 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:19:39,240 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:19:39,240 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:19:39,242 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:19:40,683 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:19:41,057 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:19:41,057 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:19:43,684 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:19:45,066 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:19:48,686 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:19:51,963 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:19:51,963 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_736_ac4e29b11ce1d305f530.png with policy now 2023-07-11 19:19:51,979 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:19:51,979 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_736_fae683dc9806394cb2e9.png with policy now 2023-07-11 19:19:51,981 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:19:51,982 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_736_f0caac32194909cb1981.png with policy now 2023-07-11 19:19:51,995 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:19:51,995 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_736_d6436e676d16768a25aa.png with policy now 2023-07-11 19:19:52,032 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:19:52,034 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:19:52,034 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:19:52,037 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:19:52,301 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/dh5w0g5x-media/images/validation_736_fae683dc9806394cb2e9.png 2023-07-11 19:19:52,315 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/48psztbj-media/images/validation_736_d6436e676d16768a25aa.png 2023-07-11 19:19:52,354 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/li0byose-media/images/validation_736_f0caac32194909cb1981.png 2023-07-11 19:19:52,475 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/chvwn98v-media/images/validation_736_ac4e29b11ce1d305f530.png 2023-07-11 19:19:53,086 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:19:53,086 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_736_fae683dc9806394cb2e9.png 2023-07-11 19:19:53,087 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_736_d6436e676d16768a25aa.png 2023-07-11 19:19:53,087 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_736_f0caac32194909cb1981.png 2023-07-11 19:19:53,088 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_736_ac4e29b11ce1d305f530.png 2023-07-11 19:19:53,088 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 19:19:53,439 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/3rpbr5wb-media/images/validation_736_f0caac32194909cb1981.png 2023-07-11 19:19:53,445 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/xqsx8j5s-media/images/validation_736_d6436e676d16768a25aa.png 2023-07-11 19:19:53,473 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/gwthjc4u-media/images/validation_736_ac4e29b11ce1d305f530.png 2023-07-11 19:19:53,630 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/su6dvlpd-media/images/validation_736_fae683dc9806394cb2e9.png 2023-07-11 19:19:54,383 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:19:54,384 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:19:54,432 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:19:56,536 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:19:57,095 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:19:59,399 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:19:59,400 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:19:59,401 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:19:59,402 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:20:00,403 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:20:01,106 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:20:01,106 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:20:02,133 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:20:02,135 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:20:02,135 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:20:02,137 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:20:05,117 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:20:05,117 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:20:05,251 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:20:05,253 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:20:05,253 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:20:05,255 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:20:05,932 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:20:08,299 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:20:08,301 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:20:08,302 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:20:08,303 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:20:09,127 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:20:09,128 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:20:09,383 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:20:09,384 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:20:10,686 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:20:11,705 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:20:11,792 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:20:11,793 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:20:11,794 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:20:11,795 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:20:13,136 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:20:13,137 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:20:15,831 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:20:15,832 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:20:15,833 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:20:15,835 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:20:16,112 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:20:16,114 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:20:16,115 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:20:16,116 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:20:16,765 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:20:17,146 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:20:17,147 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:20:21,229 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:20:21,766 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:20:24,411 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:20:24,412 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:20:27,493 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:20:28,797 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:20:28,797 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_744_88c24683fdcc6de68876.png with policy now 2023-07-11 19:20:28,841 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:20:28,841 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_744_835fc7ba76fe5bdd463b.png with policy now 2023-07-11 19:20:28,842 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:20:28,842 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_744_54915f91d21e93f22f28.png with policy now 2023-07-11 19:20:28,843 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:20:28,843 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_744_24c539112b53ecad6996.png with policy now 2023-07-11 19:20:28,871 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:20:28,872 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:20:28,873 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:20:28,874 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:20:29,162 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/hud3noln-media/images/validation_744_24c539112b53ecad6996.png 2023-07-11 19:20:29,166 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/rlhznli3-media/images/validation_744_54915f91d21e93f22f28.png 2023-07-11 19:20:29,169 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/b9opsrc9-media/images/validation_744_835fc7ba76fe5bdd463b.png 2023-07-11 19:20:29,204 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/zae45ahl-media/images/validation_744_88c24683fdcc6de68876.png 2023-07-11 19:20:29,250 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:20:29,251 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_744_54915f91d21e93f22f28.png 2023-07-11 19:20:29,251 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_744_835fc7ba76fe5bdd463b.png 2023-07-11 19:20:29,252 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_744_88c24683fdcc6de68876.png 2023-07-11 19:20:29,252 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_744_24c539112b53ecad6996.png 2023-07-11 19:20:29,253 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 19:20:29,586 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/cjyvk60h-media/images/validation_744_88c24683fdcc6de68876.png 2023-07-11 19:20:29,590 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/9kpr7vlj-media/images/validation_744_835fc7ba76fe5bdd463b.png 2023-07-11 19:20:29,599 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/mqp5y78n-media/images/validation_744_24c539112b53ecad6996.png 2023-07-11 19:20:29,701 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/lylxcr41-media/images/validation_744_54915f91d21e93f22f28.png 2023-07-11 19:20:32,594 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:20:32,595 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:20:33,261 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:20:35,689 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:20:35,691 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:20:35,691 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:20:35,693 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:20:37,271 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:20:37,272 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:20:38,099 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:20:38,792 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:20:38,794 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:20:38,795 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:20:38,796 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:20:39,458 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:20:39,459 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:20:40,689 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:20:41,281 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:20:41,282 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:20:42,334 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:20:42,335 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:20:42,336 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:20:42,337 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:20:43,198 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:20:45,292 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:20:45,293 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:20:46,052 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:20:46,054 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:20:46,055 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:20:46,056 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:20:48,616 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:20:49,303 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:20:49,304 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:20:49,311 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:20:49,313 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:20:49,314 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:20:49,315 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:20:52,739 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:20:52,740 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:20:52,741 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:20:52,743 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:20:53,020 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:20:53,022 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:20:53,022 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:20:53,024 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:20:53,313 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:20:53,313 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:20:54,468 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:20:54,469 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:20:54,520 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:20:57,323 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:20:59,520 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:21:04,521 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:21:05,603 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:21:05,603 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_752_aa82701d58c5e4be3efd.png with policy now 2023-07-11 19:21:05,617 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:21:05,617 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_752_e8178d5464133df3775d.png with policy now 2023-07-11 19:21:05,628 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:21:05,628 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_752_95d256cb61eaa36920d3.png with policy now 2023-07-11 19:21:05,637 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:21:05,637 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_752_ec4a84736de32ae6ae3c.png with policy now 2023-07-11 19:21:05,673 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:21:05,676 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:21:05,676 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:21:05,678 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:21:05,945 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/hgu4u9wa-media/images/validation_752_95d256cb61eaa36920d3.png 2023-07-11 19:21:05,962 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/s181a66d-media/images/validation_752_e8178d5464133df3775d.png 2023-07-11 19:21:05,964 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/r9zwgnfc-media/images/validation_752_ec4a84736de32ae6ae3c.png 2023-07-11 19:21:06,051 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/nhvpgacd-media/images/validation_752_aa82701d58c5e4be3efd.png 2023-07-11 19:21:09,355 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:21:09,356 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:21:09,356 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_752_ec4a84736de32ae6ae3c.png 2023-07-11 19:21:09,357 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_752_95d256cb61eaa36920d3.png 2023-07-11 19:21:09,357 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_752_aa82701d58c5e4be3efd.png 2023-07-11 19:21:09,358 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_752_e8178d5464133df3775d.png 2023-07-11 19:21:09,358 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 19:21:09,467 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:21:09,468 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:21:09,692 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/r3nv7zzu-media/images/validation_752_95d256cb61eaa36920d3.png 2023-07-11 19:21:09,695 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/kprgmq82-media/images/validation_752_aa82701d58c5e4be3efd.png 2023-07-11 19:21:09,721 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/0ump0jq4-media/images/validation_752_e8178d5464133df3775d.png 2023-07-11 19:21:09,813 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/8317vakz-media/images/validation_752_ec4a84736de32ae6ae3c.png 2023-07-11 19:21:10,518 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:21:10,673 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:21:10,692 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:21:13,366 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:21:13,721 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:21:13,722 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:21:13,723 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:21:13,724 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:21:15,859 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:21:16,503 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:21:16,505 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:21:16,505 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:21:16,507 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:21:17,377 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:21:17,377 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:21:19,867 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:21:19,868 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:21:19,869 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:21:19,870 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:21:21,292 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:21:21,387 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:21:21,388 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:21:23,097 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:21:23,098 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:21:23,099 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:21:23,101 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:21:24,667 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:21:24,668 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:21:25,397 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:21:25,398 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:21:25,701 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:21:25,703 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:21:25,703 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:21:25,705 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:21:26,406 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:21:28,977 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:21:28,979 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:21:28,980 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:21:28,981 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:21:29,258 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:21:29,260 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:21:29,260 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:21:29,262 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:21:29,408 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:21:29,409 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:21:31,806 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:21:33,420 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:21:36,808 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:21:39,667 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:21:39,668 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:21:40,695 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:21:41,897 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:21:41,897 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_760_8c7815ce3a6257b58e00.png with policy now 2023-07-11 19:21:41,898 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:21:41,907 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:21:41,908 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_760_cf617b6b91a851ec265b.png with policy now 2023-07-11 19:21:41,919 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:21:41,919 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_760_a1a1c2ba06a8ef07f019.png with policy now 2023-07-11 19:21:41,930 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:21:41,930 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_760_fb0a52e92cd7ee579cca.png with policy now 2023-07-11 19:21:41,967 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:21:41,969 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:21:41,969 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:21:41,971 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:21:42,207 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ucirat18-media/images/validation_760_8c7815ce3a6257b58e00.png 2023-07-11 19:21:42,228 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/n8siz8d1-media/images/validation_760_cf617b6b91a851ec265b.png 2023-07-11 19:21:42,229 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/m51pef9c-media/images/validation_760_a1a1c2ba06a8ef07f019.png 2023-07-11 19:21:42,244 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/gcw3dq99-media/images/validation_760_fb0a52e92cd7ee579cca.png 2023-07-11 19:21:45,451 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:21:45,451 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:21:45,451 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_760_8c7815ce3a6257b58e00.png 2023-07-11 19:21:45,452 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_760_fb0a52e92cd7ee579cca.png 2023-07-11 19:21:45,452 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_760_a1a1c2ba06a8ef07f019.png 2023-07-11 19:21:45,453 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_760_cf617b6b91a851ec265b.png 2023-07-11 19:21:45,453 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 19:21:45,817 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/v3qw4atq-media/images/validation_760_8c7815ce3a6257b58e00.png 2023-07-11 19:21:45,819 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/kalekcz6-media/images/validation_760_a1a1c2ba06a8ef07f019.png 2023-07-11 19:21:45,841 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/30omdoic-media/images/validation_760_cf617b6b91a851ec265b.png 2023-07-11 19:21:45,848 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/lxdwfuel-media/images/validation_760_fb0a52e92cd7ee579cca.png 2023-07-11 19:21:46,220 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:21:47,075 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:21:49,173 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:21:49,175 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:21:49,175 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:21:49,177 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:21:49,461 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:21:49,462 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:21:52,589 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:21:52,591 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:21:52,591 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:21:52,591 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:21:52,593 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:21:53,471 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:21:53,471 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:21:54,667 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:21:54,668 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:21:55,304 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:21:55,306 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:21:55,307 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:21:55,308 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:21:57,481 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:21:57,482 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:21:58,304 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:21:58,628 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:21:58,630 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:21:58,630 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:21:58,632 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:22:01,492 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:22:01,493 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:22:02,171 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:22:02,173 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:22:02,173 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:22:02,175 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:22:03,655 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:22:04,972 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:22:04,974 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:22:04,974 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:22:04,976 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:22:05,252 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:22:05,254 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:22:05,255 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:22:05,256 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:22:05,504 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:22:05,505 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:22:08,849 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:22:09,515 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:22:09,667 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:22:09,668 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:22:10,698 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:22:14,700 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:22:17,850 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:22:17,851 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_768_693e8b6430b9351126e2.png with policy now 2023-07-11 19:22:17,865 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:22:17,865 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_768_917890c98c8478bc01dd.png with policy now 2023-07-11 19:22:17,875 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:22:17,875 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_768_373eddb1acba3661e232.png with policy now 2023-07-11 19:22:17,883 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:22:17,883 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_768_282437d359e52d34f4f8.png with policy now 2023-07-11 19:22:17,920 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:22:17,921 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:22:17,922 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:22:17,924 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:22:18,148 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/24fqpail-media/images/validation_768_693e8b6430b9351126e2.png 2023-07-11 19:22:18,186 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/shaoujzw-media/images/validation_768_373eddb1acba3661e232.png 2023-07-11 19:22:18,201 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/cifcvcg4-media/images/validation_768_282437d359e52d34f4f8.png 2023-07-11 19:22:18,203 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/zim26jm3-media/images/validation_768_917890c98c8478bc01dd.png 2023-07-11 19:22:20,136 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:22:21,547 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:22:21,547 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:22:21,547 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_768_282437d359e52d34f4f8.png 2023-07-11 19:22:21,548 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_768_373eddb1acba3661e232.png 2023-07-11 19:22:21,549 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_768_693e8b6430b9351126e2.png 2023-07-11 19:22:21,549 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_768_917890c98c8478bc01dd.png 2023-07-11 19:22:21,550 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 19:22:21,883 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/6crkmggk-media/images/validation_768_373eddb1acba3661e232.png 2023-07-11 19:22:21,890 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/hliyhjd8-media/images/validation_768_282437d359e52d34f4f8.png 2023-07-11 19:22:21,918 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/0iwrxcob-media/images/validation_768_693e8b6430b9351126e2.png 2023-07-11 19:22:21,925 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/kfnhbjoe-media/images/validation_768_917890c98c8478bc01dd.png 2023-07-11 19:22:22,073 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:22:24,667 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:22:24,668 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:22:25,272 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:22:25,274 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:22:25,274 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:22:25,275 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:22:25,277 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:22:25,557 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:22:25,558 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:22:28,604 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:22:28,606 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:22:28,607 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:22:28,608 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:22:29,569 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:22:29,569 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:22:30,445 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:22:32,523 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:22:32,524 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:22:32,524 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:22:32,526 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:22:33,580 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:22:33,580 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:22:35,977 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:22:36,158 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:22:36,160 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:22:36,160 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:22:36,162 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:22:37,589 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:22:37,590 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:22:39,426 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:22:39,428 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:22:39,428 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:22:39,430 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:22:39,668 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:22:39,669 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:22:40,701 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:22:41,329 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:22:41,601 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:22:41,601 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:22:41,964 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:22:41,966 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:22:41,966 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:22:41,968 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:22:42,247 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:22:42,249 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:22:42,249 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:22:42,251 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:22:45,612 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:22:45,613 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:22:46,923 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:22:51,924 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:22:54,668 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:22:54,669 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:22:54,967 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:22:54,967 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_776_2aac3dd33ac8c1b799cb.png with policy now 2023-07-11 19:22:54,977 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:22:54,977 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_776_eb54d867993cb09e2b53.png with policy now 2023-07-11 19:22:54,986 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:22:54,986 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_776_06b4410d2195a28ee2e4.png with policy now 2023-07-11 19:22:54,994 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:22:54,994 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_776_938b1a9310105c2c30a8.png with policy now 2023-07-11 19:22:55,035 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:22:55,036 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:22:55,036 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:22:55,039 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:22:55,303 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ozhb0c2a-media/images/validation_776_2aac3dd33ac8c1b799cb.png 2023-07-11 19:22:55,305 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ey9xujdr-media/images/validation_776_06b4410d2195a28ee2e4.png 2023-07-11 19:22:55,311 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/mhjbsl3x-media/images/validation_776_eb54d867993cb09e2b53.png 2023-07-11 19:22:55,342 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/thrc3d7n-media/images/validation_776_938b1a9310105c2c30a8.png 2023-07-11 19:22:57,633 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:22:57,646 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:22:57,646 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:22:57,647 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_776_eb54d867993cb09e2b53.png 2023-07-11 19:22:57,647 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_776_06b4410d2195a28ee2e4.png 2023-07-11 19:22:57,648 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_776_938b1a9310105c2c30a8.png 2023-07-11 19:22:57,648 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_776_2aac3dd33ac8c1b799cb.png 2023-07-11 19:22:57,649 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 19:22:58,007 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/b05rstuo-media/images/validation_776_2aac3dd33ac8c1b799cb.png 2023-07-11 19:22:58,035 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/9m2cdc9h-media/images/validation_776_06b4410d2195a28ee2e4.png 2023-07-11 19:22:58,092 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/g5uwtk5g-media/images/validation_776_938b1a9310105c2c30a8.png 2023-07-11 19:22:58,208 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/5psks3oz-media/images/validation_776_eb54d867993cb09e2b53.png 2023-07-11 19:22:58,535 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:23:01,657 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:23:01,836 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:23:01,838 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:23:01,838 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:23:01,839 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:23:03,137 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:23:05,130 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:23:05,132 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:23:05,132 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:23:05,134 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:23:05,668 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:23:05,668 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:23:07,967 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:23:07,969 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:23:07,969 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:23:07,971 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:23:08,972 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:23:09,668 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:23:09,669 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:23:09,682 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:23:09,683 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:23:10,705 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:23:11,389 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:23:11,390 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:23:11,391 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:23:11,392 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:23:13,693 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:23:13,693 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:23:13,984 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:23:14,589 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:23:14,591 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:23:14,591 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:23:14,593 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:23:17,141 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:23:17,142 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:23:17,142 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:23:17,144 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:23:17,422 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:23:17,424 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:23:17,425 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:23:17,426 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:23:17,704 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:23:17,704 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:23:19,051 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:23:21,716 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:23:24,052 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:23:24,685 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:23:24,686 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:23:29,731 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:23:30,033 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:23:30,033 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_784_731753b83553f4858c73.png with policy now 2023-07-11 19:23:30,043 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:23:30,043 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_784_07f7d7557663601508eb.png with policy now 2023-07-11 19:23:30,052 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:23:30,052 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_784_489eed56c83ee92b8e9c.png with policy now 2023-07-11 19:23:30,061 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:23:30,061 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_784_0fa68e4c56abe4cd2e0d.png with policy now 2023-07-11 19:23:30,104 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:23:30,105 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:23:30,106 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:23:30,107 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:23:30,379 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/gqso0o9s-media/images/validation_784_731753b83553f4858c73.png 2023-07-11 19:23:30,390 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/3d5wkm11-media/images/validation_784_0fa68e4c56abe4cd2e0d.png 2023-07-11 19:23:30,407 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/nfx0lfv0-media/images/validation_784_07f7d7557663601508eb.png 2023-07-11 19:23:30,409 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/0qweyx5n-media/images/validation_784_489eed56c83ee92b8e9c.png 2023-07-11 19:23:33,749 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:23:33,750 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:23:33,750 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_784_489eed56c83ee92b8e9c.png 2023-07-11 19:23:33,750 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_784_731753b83553f4858c73.png 2023-07-11 19:23:33,751 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_784_07f7d7557663601508eb.png 2023-07-11 19:23:33,751 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_784_0fa68e4c56abe4cd2e0d.png 2023-07-11 19:23:33,752 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 19:23:34,151 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/xta7ufpb-media/images/validation_784_731753b83553f4858c73.png 2023-07-11 19:23:34,170 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/jyhamaeh-media/images/validation_784_07f7d7557663601508eb.png 2023-07-11 19:23:34,170 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/hfigt46k-media/images/validation_784_0fa68e4c56abe4cd2e0d.png 2023-07-11 19:23:34,257 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/xk26xwu9-media/images/validation_784_489eed56c83ee92b8e9c.png 2023-07-11 19:23:34,540 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:23:35,368 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:23:37,623 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:23:37,624 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:23:37,625 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:23:37,626 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:23:37,759 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:23:37,759 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:23:39,686 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:23:39,687 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:23:40,706 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:23:40,708 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:23:40,918 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:23:40,920 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:23:40,921 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:23:40,922 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:23:41,766 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:23:41,767 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:23:43,984 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:23:43,986 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:23:43,987 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:23:43,988 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:23:45,776 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:23:45,777 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:23:46,525 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:23:47,686 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:23:47,688 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:23:47,688 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:23:47,690 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:23:49,788 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:23:49,789 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:23:51,267 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:23:51,269 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:23:51,270 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:23:51,271 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:23:52,241 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:23:53,797 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:23:53,798 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:23:54,147 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:23:54,149 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:23:54,149 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:23:54,151 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:23:54,428 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:23:54,430 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:23:54,430 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:23:54,432 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:23:54,685 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:23:54,686 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:23:57,809 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:23:57,809 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:23:59,687 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:24:04,688 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:24:05,927 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:24:09,792 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:24:09,793 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:24:09,793 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:24:10,709 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:24:11,711 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:24:13,849 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:24:16,712 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:24:21,290 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:24:21,290 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_792_e061fd9e41fdfb8c934a.png with policy now 2023-07-11 19:24:21,305 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:24:21,305 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_792_2db0a6b3dbfebedf2d8e.png with policy now 2023-07-11 19:24:21,313 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:24:21,313 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_792_3f75331b78a95b5b40b7.png with policy now 2023-07-11 19:24:21,322 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:24:21,322 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_792_eac5499ee475cd33206f.png with policy now 2023-07-11 19:24:21,359 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:24:21,360 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:24:21,360 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:24:21,362 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:24:21,655 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/2b6dxfwg-media/images/validation_792_eac5499ee475cd33206f.png 2023-07-11 19:24:21,660 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/xkkr6814-media/images/validation_792_3f75331b78a95b5b40b7.png 2023-07-11 19:24:21,666 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/reybpaiw-media/images/validation_792_2db0a6b3dbfebedf2d8e.png 2023-07-11 19:24:21,706 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/0dch89vx-media/images/validation_792_e061fd9e41fdfb8c934a.png 2023-07-11 19:24:21,871 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:24:21,872 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_792_eac5499ee475cd33206f.png 2023-07-11 19:24:21,872 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_792_3f75331b78a95b5b40b7.png 2023-07-11 19:24:21,873 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_792_2db0a6b3dbfebedf2d8e.png 2023-07-11 19:24:21,874 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_792_e061fd9e41fdfb8c934a.png 2023-07-11 19:24:21,874 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 19:24:22,289 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/mbe3mgnu-media/images/validation_792_eac5499ee475cd33206f.png 2023-07-11 19:24:22,363 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:24:22,366 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ypm1f9hw-media/images/validation_792_2db0a6b3dbfebedf2d8e.png 2023-07-11 19:24:22,372 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ozz2yycc-media/images/validation_792_3f75331b78a95b5b40b7.png 2023-07-11 19:24:22,470 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/501n9dpa-media/images/validation_792_e061fd9e41fdfb8c934a.png 2023-07-11 19:24:24,793 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:24:24,793 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:24:24,794 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:24:25,099 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:24:25,883 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:24:27,800 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:24:28,412 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:24:28,415 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:24:28,416 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:24:28,417 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:24:30,893 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:24:30,894 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:24:31,622 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:24:31,624 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:24:31,625 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:24:31,627 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:24:32,872 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:24:34,388 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:24:34,390 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:24:34,391 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:24:34,392 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:24:35,905 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:24:35,905 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:24:37,644 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:24:37,646 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:24:37,646 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:24:37,648 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:24:38,355 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:24:39,793 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:24:39,793 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:24:39,794 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:24:40,712 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:24:40,915 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:24:40,916 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:24:41,274 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:24:41,276 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:24:41,276 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:24:41,278 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:24:43,454 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:24:44,123 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:24:44,125 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:24:44,126 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:24:44,127 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:24:44,407 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:24:44,409 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:24:44,409 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:24:44,411 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:24:45,926 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:24:45,926 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:24:49,019 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:24:50,939 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:24:54,020 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:24:54,829 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:24:54,829 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:24:54,830 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:24:57,080 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:24:57,080 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_800_31cd8fae17813d7d0e65.png with policy now 2023-07-11 19:24:57,094 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:24:57,095 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_800_d905700551ce75840705.png with policy now 2023-07-11 19:24:57,104 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:24:57,104 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_800_3d6397867b321dddf262.png with policy now 2023-07-11 19:24:57,111 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:24:57,112 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_800_2ed775ff9a30d99a0a43.png with policy now 2023-07-11 19:24:57,149 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:24:57,150 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:24:57,150 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:24:57,152 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:24:57,476 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/uf8sc0qu-media/images/validation_800_31cd8fae17813d7d0e65.png 2023-07-11 19:24:57,500 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/c0rjj0pd-media/images/validation_800_3d6397867b321dddf262.png 2023-07-11 19:24:57,503 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/g7xq46oa-media/images/validation_800_2ed775ff9a30d99a0a43.png 2023-07-11 19:24:57,621 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/gbcfvujm-media/images/validation_800_d905700551ce75840705.png 2023-07-11 19:24:59,163 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:25:00,713 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:25:00,962 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:25:00,962 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:25:00,963 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_800_31cd8fae17813d7d0e65.png 2023-07-11 19:25:00,963 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_800_d905700551ce75840705.png 2023-07-11 19:25:00,964 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_800_2ed775ff9a30d99a0a43.png 2023-07-11 19:25:00,964 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_800_3d6397867b321dddf262.png 2023-07-11 19:25:00,965 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 19:25:01,360 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/rijrc93n-media/images/validation_800_2ed775ff9a30d99a0a43.png 2023-07-11 19:25:01,366 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/7fnf9pdo-media/images/validation_800_31cd8fae17813d7d0e65.png 2023-07-11 19:25:01,379 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/5w07i6z2-media/images/validation_800_d905700551ce75840705.png 2023-07-11 19:25:01,384 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/kwsu7kqd-media/images/validation_800_3d6397867b321dddf262.png 2023-07-11 19:25:04,059 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:25:04,061 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:25:04,061 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:25:04,063 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:25:05,064 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:25:05,973 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:25:05,973 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:25:07,753 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:25:07,755 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:25:07,755 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:25:07,757 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:25:09,828 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:25:09,829 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:25:09,829 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:25:10,410 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:25:10,715 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:25:10,985 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:25:10,985 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:25:11,561 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:25:11,563 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:25:11,563 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:25:11,565 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:25:14,627 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:25:14,629 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:25:14,629 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:25:14,631 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:25:15,613 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:25:15,996 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:25:15,997 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:25:18,344 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:25:18,346 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:25:18,347 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:25:18,348 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:25:21,007 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:25:21,008 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:25:21,046 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:25:21,584 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:25:21,586 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:25:21,587 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:25:21,589 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:25:21,867 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:25:21,869 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:25:21,869 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:25:21,871 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:25:24,829 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:25:24,830 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:25:24,830 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:25:26,018 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:25:26,019 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:25:26,884 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:25:31,885 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:25:34,503 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:25:34,504 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_808_198cfae2d5879abcba46.png with policy now 2023-07-11 19:25:34,518 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:25:34,519 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_808_7aaebfffee54696e50f8.png with policy now 2023-07-11 19:25:34,528 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:25:34,529 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_808_f9c3bd83a1bb11eb4cff.png with policy now 2023-07-11 19:25:34,537 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:25:34,538 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_808_b053e35f6e9aac4bea20.png with policy now 2023-07-11 19:25:34,574 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:25:34,575 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:25:34,576 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:25:34,577 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:25:34,873 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/gt7gefm0-media/images/validation_808_198cfae2d5879abcba46.png 2023-07-11 19:25:34,887 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ikk95ql8-media/images/validation_808_b053e35f6e9aac4bea20.png 2023-07-11 19:25:34,899 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/3ih61s1y-media/images/validation_808_f9c3bd83a1bb11eb4cff.png 2023-07-11 19:25:35,004 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/2c9efk02-media/images/validation_808_7aaebfffee54696e50f8.png 2023-07-11 19:25:36,041 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:25:36,042 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_808_f9c3bd83a1bb11eb4cff.png 2023-07-11 19:25:36,042 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_808_198cfae2d5879abcba46.png 2023-07-11 19:25:36,043 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_808_b053e35f6e9aac4bea20.png 2023-07-11 19:25:36,044 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_808_7aaebfffee54696e50f8.png 2023-07-11 19:25:36,044 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 19:25:36,394 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/r757t3at-media/images/validation_808_b053e35f6e9aac4bea20.png 2023-07-11 19:25:36,395 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/pdk63f9b-media/images/validation_808_198cfae2d5879abcba46.png 2023-07-11 19:25:36,395 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/el0ikyxx-media/images/validation_808_f9c3bd83a1bb11eb4cff.png 2023-07-11 19:25:36,423 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/7u5uw3cm-media/images/validation_808_7aaebfffee54696e50f8.png 2023-07-11 19:25:37,131 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:25:39,063 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:25:40,493 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:25:40,494 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:25:40,494 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:25:40,718 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:25:41,052 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:25:42,469 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:25:42,515 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:25:42,517 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:25:42,517 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:25:42,519 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:25:45,236 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:25:45,238 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:25:45,238 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:25:45,240 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:25:46,062 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:25:46,062 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:25:47,963 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:25:47,965 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:25:47,965 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:25:47,966 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:25:47,968 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:25:51,018 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:25:51,019 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:25:51,020 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:25:51,312 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:25:51,318 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:25:51,319 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:25:53,702 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:25:54,383 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:25:54,385 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:25:54,386 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:25:54,387 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:25:55,565 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:25:55,565 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:25:55,566 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:25:56,329 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:25:56,330 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:25:57,895 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:25:57,897 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:25:57,898 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:25:57,899 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:25:58,177 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:25:58,179 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:25:58,180 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:25:58,181 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:25:58,716 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:26:01,339 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:26:01,339 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:26:03,854 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:26:08,855 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:26:10,566 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:26:10,566 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:26:10,567 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:26:10,721 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:26:10,938 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:26:10,939 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_816_061281f0958189cb5d31.png with policy now 2023-07-11 19:26:10,950 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:26:10,950 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_816_e9e3f32b220e23c697cf.png with policy now 2023-07-11 19:26:10,962 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:26:10,962 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_816_5b6b74a7d7fd49b3636b.png with policy now 2023-07-11 19:26:10,972 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:26:10,972 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_816_69329d4e2a0a472e0d04.png with policy now 2023-07-11 19:26:11,032 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:26:11,034 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:26:11,034 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:26:11,036 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:26:11,255 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/9pnhozfa-media/images/validation_816_061281f0958189cb5d31.png 2023-07-11 19:26:11,257 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/r5z6uzzr-media/images/validation_816_e9e3f32b220e23c697cf.png 2023-07-11 19:26:11,302 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/s7x28y9j-media/images/validation_816_5b6b74a7d7fd49b3636b.png 2023-07-11 19:26:11,308 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/07dswkq6-media/images/validation_816_69329d4e2a0a472e0d04.png 2023-07-11 19:26:11,362 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:26:11,362 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_816_69329d4e2a0a472e0d04.png 2023-07-11 19:26:11,362 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_816_e9e3f32b220e23c697cf.png 2023-07-11 19:26:11,363 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_816_061281f0958189cb5d31.png 2023-07-11 19:26:11,363 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_816_5b6b74a7d7fd49b3636b.png 2023-07-11 19:26:11,364 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 19:26:11,712 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/9nowzo1u-media/images/validation_816_69329d4e2a0a472e0d04.png 2023-07-11 19:26:11,742 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/awoysuo1-media/images/validation_816_e9e3f32b220e23c697cf.png 2023-07-11 19:26:11,747 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/sfhklirj-media/images/validation_816_5b6b74a7d7fd49b3636b.png 2023-07-11 19:26:11,840 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/qj2j8o9z-media/images/validation_816_061281f0958189cb5d31.png 2023-07-11 19:26:14,411 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:26:14,412 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:26:16,372 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:26:16,889 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:26:16,891 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:26:16,891 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:26:16,893 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:26:20,031 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:26:20,033 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:26:20,033 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:26:20,034 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:26:20,035 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:26:21,384 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:26:21,384 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:26:23,655 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:26:23,657 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:26:23,659 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:26:23,661 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:26:25,198 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:26:25,638 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:26:25,638 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:26:25,639 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:26:26,394 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:26:26,395 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:26:26,832 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:26:26,834 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:26:26,834 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:26:26,836 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:26:30,581 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:26:30,583 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:26:30,583 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:26:30,583 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:26:30,585 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:26:31,405 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:26:31,406 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:26:33,637 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:26:33,639 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:26:33,639 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:26:33,641 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:26:33,918 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:26:33,921 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:26:33,921 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:26:33,922 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:26:36,417 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:26:36,417 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:26:36,487 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:26:40,638 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:26:40,638 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:26:40,639 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:26:40,724 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:26:41,726 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:26:46,472 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:26:46,472 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_824_27f999fd2d618c705cb1.png with policy now 2023-07-11 19:26:46,482 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:26:46,482 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_824_9c2a756ba8f9a949c00e.png with policy now 2023-07-11 19:26:46,491 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:26:46,491 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_824_e40da72508d9965ce259.png with policy now 2023-07-11 19:26:46,500 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:26:46,500 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_824_40304dd8f904f4d4b176.png with policy now 2023-07-11 19:26:46,542 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:26:46,543 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:26:46,544 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:26:46,545 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:26:46,802 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/9450ry73-media/images/validation_824_e40da72508d9965ce259.png 2023-07-11 19:26:46,812 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/wxh1cu9e-media/images/validation_824_40304dd8f904f4d4b176.png 2023-07-11 19:26:46,821 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/4ozf1lgp-media/images/validation_824_9c2a756ba8f9a949c00e.png 2023-07-11 19:26:46,913 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ly7x9yjq-media/images/validation_824_27f999fd2d618c705cb1.png 2023-07-11 19:26:47,546 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:26:50,906 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:26:51,451 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:26:51,452 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:26:51,452 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_824_9c2a756ba8f9a949c00e.png 2023-07-11 19:26:51,453 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_824_e40da72508d9965ce259.png 2023-07-11 19:26:51,453 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_824_27f999fd2d618c705cb1.png 2023-07-11 19:26:51,453 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_824_40304dd8f904f4d4b176.png 2023-07-11 19:26:51,454 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 19:26:51,835 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/s08trr6e-media/images/validation_824_27f999fd2d618c705cb1.png 2023-07-11 19:26:51,842 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/k3xzgu5n-media/images/validation_824_e40da72508d9965ce259.png 2023-07-11 19:26:51,847 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/fg3ucsf7-media/images/validation_824_9c2a756ba8f9a949c00e.png 2023-07-11 19:26:51,848 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/rwohef3f-media/images/validation_824_40304dd8f904f4d4b176.png 2023-07-11 19:26:52,953 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:26:55,073 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:26:55,075 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:26:55,075 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:26:55,076 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:26:55,639 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:26:55,639 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:26:55,640 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:26:56,462 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:26:56,462 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:26:57,769 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:26:57,771 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:26:57,771 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:26:57,773 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:26:58,326 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:27:00,490 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:27:00,491 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:27:00,492 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:27:00,493 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:27:01,472 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:27:01,473 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:27:03,914 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:27:04,063 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:27:04,065 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:27:04,066 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:27:04,067 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:27:06,483 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:27:06,484 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:27:08,171 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:27:08,173 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:27:08,174 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:27:08,175 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:27:09,769 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:27:10,639 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:27:10,639 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:27:10,640 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:27:10,727 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:27:11,368 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:27:11,369 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:27:11,369 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:27:11,371 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:27:11,494 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:27:11,494 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:27:11,650 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:27:11,651 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:27:11,652 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:27:11,653 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:27:15,233 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:27:16,505 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:27:16,506 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:27:20,234 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:27:24,266 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:27:24,266 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_832_7645690b8d91851e7d9d.png with policy now 2023-07-11 19:27:24,277 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:27:24,278 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_832_430e5d6128d45a25cb6f.png with policy now 2023-07-11 19:27:24,287 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:27:24,287 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_832_306e72d0667b956b2311.png with policy now 2023-07-11 19:27:24,307 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:27:24,307 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_832_262acb7fdbd40f4b9740.png with policy now 2023-07-11 19:27:24,350 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:27:24,352 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:27:24,352 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:27:24,354 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:27:24,594 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/l6wyhbzd-media/images/validation_832_7645690b8d91851e7d9d.png 2023-07-11 19:27:24,617 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/c2c5bb3o-media/images/validation_832_430e5d6128d45a25cb6f.png 2023-07-11 19:27:24,628 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/3y6u3u2n-media/images/validation_832_306e72d0667b956b2311.png 2023-07-11 19:27:24,643 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/7xnyszt3-media/images/validation_832_262acb7fdbd40f4b9740.png 2023-07-11 19:27:25,865 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:27:25,866 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:27:25,868 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:27:25,927 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:27:26,530 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:27:26,530 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_832_430e5d6128d45a25cb6f.png 2023-07-11 19:27:26,530 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_832_262acb7fdbd40f4b9740.png 2023-07-11 19:27:26,531 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_832_7645690b8d91851e7d9d.png 2023-07-11 19:27:26,532 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_832_306e72d0667b956b2311.png 2023-07-11 19:27:26,532 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 19:27:26,876 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/b54vohbv-media/images/validation_832_430e5d6128d45a25cb6f.png 2023-07-11 19:27:26,888 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/thdovtos-media/images/validation_832_7645690b8d91851e7d9d.png 2023-07-11 19:27:26,889 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/0tygs4ma-media/images/validation_832_306e72d0667b956b2311.png 2023-07-11 19:27:26,906 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/eyst2sh7-media/images/validation_832_262acb7fdbd40f4b9740.png 2023-07-11 19:27:27,553 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:27:30,684 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:27:30,686 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:27:30,686 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:27:30,688 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:27:31,338 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:27:31,541 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:27:31,542 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:27:33,184 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:27:33,185 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:27:33,186 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:27:33,187 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:27:36,552 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:27:36,553 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:27:36,594 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:27:36,596 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:27:36,597 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:27:36,597 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:27:36,599 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:27:40,384 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:27:40,386 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:27:40,386 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:27:40,388 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:27:40,731 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:27:40,955 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:27:40,955 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:27:40,956 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:27:41,563 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:27:41,564 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:27:42,127 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:27:44,193 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:27:44,195 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:27:44,196 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:27:44,197 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:27:46,574 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:27:46,574 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:27:47,630 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:27:47,632 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:27:47,633 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:27:47,633 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:27:47,635 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:27:47,912 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:27:47,913 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:27:47,914 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:27:47,915 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:27:51,585 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:27:51,586 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:27:53,454 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:27:55,972 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:27:55,972 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:27:55,973 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:27:59,037 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:28:00,511 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:28:00,511 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_840_2be7449ad230441219d7.png with policy now 2023-07-11 19:28:00,521 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:28:00,521 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_840_b38629e31548571aa3be.png with policy now 2023-07-11 19:28:00,530 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:28:00,531 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_840_e7fcf8abc3886e356bb8.png with policy now 2023-07-11 19:28:00,539 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:28:00,539 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_840_ffd650d33cbd150f9d5a.png with policy now 2023-07-11 19:28:00,588 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:28:00,591 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:28:00,591 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:28:00,594 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:28:00,818 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/1sk59iyx-media/images/validation_840_2be7449ad230441219d7.png 2023-07-11 19:28:00,842 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/impdztw5-media/images/validation_840_b38629e31548571aa3be.png 2023-07-11 19:28:00,857 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/mmtxxbiz-media/images/validation_840_ffd650d33cbd150f9d5a.png 2023-07-11 19:28:00,869 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/96qlghl3-media/images/validation_840_e7fcf8abc3886e356bb8.png 2023-07-11 19:28:01,610 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:28:01,610 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_840_ffd650d33cbd150f9d5a.png 2023-07-11 19:28:01,611 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_840_2be7449ad230441219d7.png 2023-07-11 19:28:01,612 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_840_b38629e31548571aa3be.png 2023-07-11 19:28:01,612 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_840_e7fcf8abc3886e356bb8.png 2023-07-11 19:28:01,613 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 19:28:01,959 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/hbqzo0pq-media/images/validation_840_ffd650d33cbd150f9d5a.png 2023-07-11 19:28:01,966 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/1yyrdx2a-media/images/validation_840_2be7449ad230441219d7.png 2023-07-11 19:28:01,979 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/cybubabm-media/images/validation_840_b38629e31548571aa3be.png 2023-07-11 19:28:01,982 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/iqqxk7q0-media/images/validation_840_e7fcf8abc3886e356bb8.png 2023-07-11 19:28:04,702 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:28:04,704 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:28:06,621 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:28:08,065 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:28:08,067 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:28:08,068 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:28:08,069 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:28:09,917 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:28:10,732 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:28:11,343 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:28:11,343 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:28:11,344 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:28:11,469 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:28:11,471 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:28:11,471 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:28:11,472 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:28:11,632 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:28:11,633 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:28:14,924 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:28:14,926 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:28:14,926 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:28:14,926 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:28:14,928 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:28:16,644 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:28:16,645 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:28:17,865 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:28:17,867 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:28:17,867 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:28:17,869 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:28:20,656 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:28:21,311 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:28:21,313 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:28:21,313 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:28:21,315 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:28:21,654 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:28:21,655 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:28:24,896 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:28:24,898 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:28:24,898 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:28:24,900 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:28:25,179 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:28:25,181 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:28:25,181 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:28:25,183 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:28:25,724 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:28:26,343 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:28:26,343 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:28:26,344 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:28:26,666 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:28:26,666 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:28:31,396 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:28:31,677 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:28:36,397 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:28:37,750 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:28:37,750 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_848_9fd46f5e0289db110e5a.png with policy now 2023-07-11 19:28:37,764 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:28:37,765 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_848_0da7b9c733de297e7940.png with policy now 2023-07-11 19:28:37,773 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:28:37,774 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_848_d3851e1b7aee0085f878.png with policy now 2023-07-11 19:28:37,784 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:28:37,785 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_848_7e99547529310b4b5dbe.png with policy now 2023-07-11 19:28:37,821 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:28:37,824 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:28:37,824 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:28:37,826 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:28:38,084 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/weez4yl6-media/images/validation_848_d3851e1b7aee0085f878.png 2023-07-11 19:28:38,092 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/572gm2cg-media/images/validation_848_0da7b9c733de297e7940.png 2023-07-11 19:28:38,100 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/djcj0mzz-media/images/validation_848_9fd46f5e0289db110e5a.png 2023-07-11 19:28:38,231 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/y01mdk0x-media/images/validation_848_7e99547529310b4b5dbe.png 2023-07-11 19:28:40,735 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:28:41,518 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:28:41,518 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:28:41,519 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:28:41,556 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:28:41,643 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:28:41,702 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:28:41,702 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:28:41,703 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_848_9fd46f5e0289db110e5a.png 2023-07-11 19:28:41,703 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_848_d3851e1b7aee0085f878.png 2023-07-11 19:28:41,704 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_848_7e99547529310b4b5dbe.png 2023-07-11 19:28:41,704 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_848_0da7b9c733de297e7940.png 2023-07-11 19:28:41,705 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 19:28:42,043 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/f3tozl47-media/images/validation_848_9fd46f5e0289db110e5a.png 2023-07-11 19:28:42,053 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/2udlxl29-media/images/validation_848_d3851e1b7aee0085f878.png 2023-07-11 19:28:42,087 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/8my3mrcn-media/images/validation_848_0da7b9c733de297e7940.png 2023-07-11 19:28:42,471 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ao1sacok-media/images/validation_848_7e99547529310b4b5dbe.png 2023-07-11 19:28:44,447 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:28:44,450 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:28:44,450 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:28:44,452 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:28:46,713 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:28:46,714 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:28:47,204 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:28:48,601 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:28:48,603 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:28:48,603 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:28:48,605 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:28:51,724 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:28:51,724 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:28:52,251 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:28:52,253 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:28:52,253 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:28:52,254 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:28:52,256 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:28:56,192 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:28:56,194 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:28:56,195 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:28:56,196 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:28:56,519 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:28:56,519 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:28:56,520 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:28:56,734 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:28:56,734 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:28:57,396 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:28:58,519 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:28:58,521 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:28:58,521 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:28:58,523 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:29:01,745 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:29:01,746 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:29:01,943 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:29:01,945 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:29:01,945 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:29:01,947 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:29:02,224 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:29:02,226 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:29:02,226 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:29:02,228 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:29:02,601 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:29:06,757 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:29:06,758 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:29:07,740 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:29:10,738 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:29:11,548 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:29:11,548 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:29:11,549 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:29:13,604 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:29:14,798 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:29:14,798 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_856_b93f1ea746e64027d9cf.png with policy now 2023-07-11 19:29:14,812 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:29:14,813 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_856_a9c51bde36c7b91e7157.png with policy now 2023-07-11 19:29:14,822 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:29:14,823 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_856_34456aced927b6fdcddf.png with policy now 2023-07-11 19:29:14,831 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:29:14,831 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_856_738cae12b93e3de8b6ad.png with policy now 2023-07-11 19:29:14,866 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:29:14,867 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:29:14,868 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:29:14,870 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:29:15,097 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/bo7skoff-media/images/validation_856_b93f1ea746e64027d9cf.png 2023-07-11 19:29:15,125 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/9juipd7e-media/images/validation_856_34456aced927b6fdcddf.png 2023-07-11 19:29:15,161 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/o7biq7a8-media/images/validation_856_a9c51bde36c7b91e7157.png 2023-07-11 19:29:15,166 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/iayxixhh-media/images/validation_856_738cae12b93e3de8b6ad.png 2023-07-11 19:29:16,782 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:29:16,782 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_856_a9c51bde36c7b91e7157.png 2023-07-11 19:29:16,783 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_856_34456aced927b6fdcddf.png 2023-07-11 19:29:16,783 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_856_b93f1ea746e64027d9cf.png 2023-07-11 19:29:16,784 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_856_738cae12b93e3de8b6ad.png 2023-07-11 19:29:16,784 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 19:29:17,146 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/rdn4qfoq-media/images/validation_856_34456aced927b6fdcddf.png 2023-07-11 19:29:17,146 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/qm5sny6z-media/images/validation_856_b93f1ea746e64027d9cf.png 2023-07-11 19:29:17,159 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/eivnabpr-media/images/validation_856_a9c51bde36c7b91e7157.png 2023-07-11 19:29:17,166 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/99r4d4qg-media/images/validation_856_738cae12b93e3de8b6ad.png 2023-07-11 19:29:18,922 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:29:18,923 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:29:21,793 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:29:22,213 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:29:22,215 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:29:22,215 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:29:22,217 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:29:24,201 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:29:24,901 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:29:24,902 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:29:24,903 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:29:24,904 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:29:26,548 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:29:26,548 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:29:26,549 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:29:26,805 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:29:26,805 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:29:28,483 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:29:28,485 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:29:28,486 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:29:28,487 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:29:29,616 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:29:31,483 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:29:31,485 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:29:31,485 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:29:31,486 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:29:31,815 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:29:31,816 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:29:35,175 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:29:35,215 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:29:35,217 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:29:35,217 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:29:35,219 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:29:36,826 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:29:36,826 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:29:39,038 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:29:39,040 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:29:39,041 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:29:39,042 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:29:39,321 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:29:39,323 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:29:39,323 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:29:39,325 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:29:40,741 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:29:40,746 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:29:41,549 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:29:41,549 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:29:41,550 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:29:41,837 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:29:41,837 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:29:46,587 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:29:51,588 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:29:51,871 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:29:51,871 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_864_950acf7e39a1d4e12f7d.png with policy now 2023-07-11 19:29:51,913 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:29:51,913 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_864_8ecbb274e51eb741dfb5.png with policy now 2023-07-11 19:29:51,914 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:29:51,914 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_864_8aa45d21612bc1191b24.png with policy now 2023-07-11 19:29:51,916 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:29:51,916 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_864_5a6f78f0d182d46a8beb.png with policy now 2023-07-11 19:29:51,943 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:29:51,945 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:29:51,946 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:29:51,947 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:29:52,167 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/pi9rs1d7-media/images/validation_864_950acf7e39a1d4e12f7d.png 2023-07-11 19:29:52,244 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/plndphaj-media/images/validation_864_5a6f78f0d182d46a8beb.png 2023-07-11 19:29:52,248 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/tv4iwgwi-media/images/validation_864_8aa45d21612bc1191b24.png 2023-07-11 19:29:52,255 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/rni120ug-media/images/validation_864_8ecbb274e51eb741dfb5.png 2023-07-11 19:29:55,576 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:29:56,551 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:29:56,551 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:29:56,552 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:29:56,610 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:29:56,875 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:29:56,875 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:29:56,876 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_864_950acf7e39a1d4e12f7d.png 2023-07-11 19:29:56,876 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_864_8aa45d21612bc1191b24.png 2023-07-11 19:29:56,877 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_864_8ecbb274e51eb741dfb5.png 2023-07-11 19:29:56,878 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_864_5a6f78f0d182d46a8beb.png 2023-07-11 19:29:56,878 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 19:29:57,197 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/voxxo8ja-media/images/validation_864_950acf7e39a1d4e12f7d.png 2023-07-11 19:29:57,248 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/41ldh6ay-media/images/validation_864_5a6f78f0d182d46a8beb.png 2023-07-11 19:29:57,255 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/rgngy4yr-media/images/validation_864_8aa45d21612bc1191b24.png 2023-07-11 19:29:57,255 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/nglyrx92-media/images/validation_864_8ecbb274e51eb741dfb5.png 2023-07-11 19:29:58,914 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:29:58,916 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:29:58,917 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:29:58,918 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:30:01,681 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:30:01,885 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:30:01,886 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:30:02,422 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:30:02,423 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:30:02,423 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:30:02,425 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:30:06,253 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:30:06,255 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:30:06,255 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:30:06,257 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:30:06,897 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:30:06,898 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:30:07,031 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:30:08,897 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:30:08,899 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:30:08,899 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:30:08,901 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:30:10,744 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:30:11,549 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:30:11,551 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:30:11,551 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:30:11,909 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:30:11,910 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:30:12,828 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:30:12,858 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:30:12,860 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:30:12,860 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:30:12,862 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:30:15,381 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:30:15,383 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:30:15,383 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:30:15,385 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:30:15,662 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:30:15,664 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:30:15,665 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:30:15,666 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:30:16,921 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:30:16,921 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:30:18,293 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:30:21,933 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:30:23,294 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:30:26,554 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:30:26,554 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:30:26,555 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:30:28,317 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:30:28,317 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_872_ca57733755532b74591b.png with policy now 2023-07-11 19:30:28,318 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:30:28,332 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:30:28,332 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_872_92a3a20861d1d9a07c22.png with policy now 2023-07-11 19:30:28,341 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:30:28,341 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_872_72610369def04cbca766.png with policy now 2023-07-11 19:30:28,350 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:30:28,351 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_872_a69cb09077cee2f55c62.png with policy now 2023-07-11 19:30:28,391 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:30:28,392 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:30:28,392 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:30:28,394 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:30:28,640 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/2f027433-media/images/validation_872_ca57733755532b74591b.png 2023-07-11 19:30:28,665 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/bblf24ex-media/images/validation_872_92a3a20861d1d9a07c22.png 2023-07-11 19:30:28,670 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/mep5d1jq-media/images/validation_872_72610369def04cbca766.png 2023-07-11 19:30:28,676 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/f9y1lov1-media/images/validation_872_a69cb09077cee2f55c62.png 2023-07-11 19:30:31,958 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:30:31,959 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:30:31,959 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_872_72610369def04cbca766.png 2023-07-11 19:30:31,959 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_872_a69cb09077cee2f55c62.png 2023-07-11 19:30:31,960 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_872_ca57733755532b74591b.png 2023-07-11 19:30:31,961 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_872_92a3a20861d1d9a07c22.png 2023-07-11 19:30:31,961 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 19:30:32,294 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/02ry823u-media/images/validation_872_72610369def04cbca766.png 2023-07-11 19:30:32,317 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/3lifa3wi-media/images/validation_872_ca57733755532b74591b.png 2023-07-11 19:30:32,322 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/8t0ikcv4-media/images/validation_872_a69cb09077cee2f55c62.png 2023-07-11 19:30:32,335 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/59wk0rhp-media/images/validation_872_92a3a20861d1d9a07c22.png 2023-07-11 19:30:32,740 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:30:33,685 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:30:36,694 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:30:36,696 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:30:36,697 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:30:36,698 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:30:36,969 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:30:36,970 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:30:39,559 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:30:40,722 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:30:40,724 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:30:40,724 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:30:40,726 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:30:40,747 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:30:41,554 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:30:41,554 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:30:41,555 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:30:41,981 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:30:41,982 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:30:43,595 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:30:43,596 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:30:43,597 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:30:43,599 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:30:44,600 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:30:46,527 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:30:46,529 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:30:46,529 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:30:46,531 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:30:46,993 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:30:46,994 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:30:49,602 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:30:49,604 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:30:49,604 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:30:49,605 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:30:49,606 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:30:52,005 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:30:52,005 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:30:53,222 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:30:53,224 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:30:53,225 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:30:53,226 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:30:53,507 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:30:53,508 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:30:53,509 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:30:53,510 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:30:55,073 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:30:56,555 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:30:56,555 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:30:56,555 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:30:57,137 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:30:57,138 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:31:00,613 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:31:05,614 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:31:06,126 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:31:06,126 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_880_2ab57feb81125c56c260.png with policy now 2023-07-11 19:31:06,140 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:31:06,141 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_880_18fe769afdeec594e0f0.png with policy now 2023-07-11 19:31:06,151 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:31:06,151 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_880_3bf88939232b65d3898d.png with policy now 2023-07-11 19:31:06,159 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:31:06,159 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_880_4ca27ee28456bc10a80b.png with policy now 2023-07-11 19:31:06,194 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:31:06,195 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:31:06,196 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:31:06,197 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:31:06,469 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/322s9eqi-media/images/validation_880_2ab57feb81125c56c260.png 2023-07-11 19:31:06,485 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/upz712aa-media/images/validation_880_18fe769afdeec594e0f0.png 2023-07-11 19:31:06,494 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/16v5vu2v-media/images/validation_880_4ca27ee28456bc10a80b.png 2023-07-11 19:31:06,526 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/js88qh7p-media/images/validation_880_3bf88939232b65d3898d.png 2023-07-11 19:31:07,162 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:31:07,162 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_880_3bf88939232b65d3898d.png 2023-07-11 19:31:07,163 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_880_4ca27ee28456bc10a80b.png 2023-07-11 19:31:07,164 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_880_18fe769afdeec594e0f0.png 2023-07-11 19:31:07,164 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_880_2ab57feb81125c56c260.png 2023-07-11 19:31:07,165 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 19:31:07,512 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/149ymb9j-media/images/validation_880_4ca27ee28456bc10a80b.png 2023-07-11 19:31:07,520 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/sytp70lv-media/images/validation_880_18fe769afdeec594e0f0.png 2023-07-11 19:31:07,524 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ze2nfsm2-media/images/validation_880_2ab57feb81125c56c260.png 2023-07-11 19:31:07,621 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/paawocez-media/images/validation_880_3bf88939232b65d3898d.png 2023-07-11 19:31:09,716 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:31:10,718 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:31:10,750 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:31:11,554 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:31:11,555 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:31:12,173 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:31:13,204 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:31:13,206 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:31:13,206 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:31:13,208 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:31:16,473 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:31:16,475 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:31:16,475 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:31:16,476 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:31:16,478 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:31:17,185 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:31:17,185 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:31:19,323 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:31:19,324 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:31:19,325 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:31:19,327 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:31:22,004 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:31:22,198 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:31:22,199 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:31:22,677 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:31:22,679 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:31:22,679 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:31:22,681 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:31:25,571 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:31:25,572 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:31:25,573 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:31:25,574 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:31:26,554 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:31:26,555 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:31:27,210 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:31:27,210 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:31:27,992 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:31:28,857 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:31:28,858 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:31:28,859 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:31:28,860 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:31:29,139 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:31:29,140 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:31:29,140 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:31:29,142 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:31:32,221 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:31:32,222 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:31:33,765 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:31:38,766 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:31:40,753 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:31:41,555 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:31:41,555 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:31:41,556 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:31:41,781 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:31:41,782 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_888_a7440f71b9ffcf04c59e.png with policy now 2023-07-11 19:31:41,793 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:31:41,793 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_888_8729ef5a4b0fe41acc66.png with policy now 2023-07-11 19:31:41,806 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:31:41,806 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_888_19b900daab11785f4793.png with policy now 2023-07-11 19:31:41,816 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:31:41,816 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_888_aec0fa4930afae7a7541.png with policy now 2023-07-11 19:31:41,853 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:31:41,854 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:31:41,855 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:31:41,860 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:31:42,104 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/08jcdbtu-media/images/validation_888_a7440f71b9ffcf04c59e.png 2023-07-11 19:31:42,107 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/1vbt07bl-media/images/validation_888_8729ef5a4b0fe41acc66.png 2023-07-11 19:31:42,109 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ri4dknyu-media/images/validation_888_19b900daab11785f4793.png 2023-07-11 19:31:42,136 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/6w2i43wk-media/images/validation_888_aec0fa4930afae7a7541.png 2023-07-11 19:31:42,246 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:31:42,247 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_888_19b900daab11785f4793.png 2023-07-11 19:31:42,247 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_888_aec0fa4930afae7a7541.png 2023-07-11 19:31:42,248 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_888_a7440f71b9ffcf04c59e.png 2023-07-11 19:31:42,249 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_888_8729ef5a4b0fe41acc66.png 2023-07-11 19:31:42,249 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 19:31:42,617 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/yhf12iba-media/images/validation_888_a7440f71b9ffcf04c59e.png 2023-07-11 19:31:42,645 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/dhbrt8j6-media/images/validation_888_8729ef5a4b0fe41acc66.png 2023-07-11 19:31:42,645 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/40dd945y-media/images/validation_888_aec0fa4930afae7a7541.png 2023-07-11 19:31:42,827 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/3ipddyi1-media/images/validation_888_19b900daab11785f4793.png 2023-07-11 19:31:44,695 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:31:45,870 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:31:47,258 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:31:49,109 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:31:49,111 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:31:49,111 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:31:49,113 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:31:50,114 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:31:52,235 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:31:52,236 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:31:52,237 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:31:52,238 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:31:52,266 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:31:52,267 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:31:55,862 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:31:55,864 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:31:55,864 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:31:55,865 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:31:55,866 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:31:57,233 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:31:57,234 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:31:57,234 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:31:57,278 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:31:57,278 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:32:00,373 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:32:00,375 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:32:00,375 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:32:00,377 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:32:01,059 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:32:02,290 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:32:02,290 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:32:04,042 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:32:04,044 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:32:04,045 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:32:04,046 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:32:06,405 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:32:07,139 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:32:07,141 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:32:07,141 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:32:07,143 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:32:07,300 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:32:07,301 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:32:07,421 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:32:07,422 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:32:07,423 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:32:07,424 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:32:10,755 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:32:11,757 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:32:12,247 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:32:12,247 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:32:12,248 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:32:12,313 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:32:12,313 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:32:17,304 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:32:20,035 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:32:20,035 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_896_1195075f26c9c85c17be.png with policy now 2023-07-11 19:32:20,050 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:32:20,050 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_896_8b01143d063fb3d088dd.png with policy now 2023-07-11 19:32:20,060 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:32:20,060 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_896_a33d92e49bb3819c9398.png with policy now 2023-07-11 19:32:20,070 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:32:20,070 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_896_2d49f7aa4f79dc70f56e.png with policy now 2023-07-11 19:32:20,104 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:32:20,105 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:32:20,106 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:32:20,107 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:32:20,372 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ol4gozi6-media/images/validation_896_1195075f26c9c85c17be.png 2023-07-11 19:32:20,381 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/pbv6wnyg-media/images/validation_896_8b01143d063fb3d088dd.png 2023-07-11 19:32:20,386 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/xzqz0xfi-media/images/validation_896_a33d92e49bb3819c9398.png 2023-07-11 19:32:20,389 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/rb1zqbyq-media/images/validation_896_2d49f7aa4f79dc70f56e.png 2023-07-11 19:32:22,338 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:32:22,339 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:32:22,339 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_896_a33d92e49bb3819c9398.png 2023-07-11 19:32:22,339 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_896_2d49f7aa4f79dc70f56e.png 2023-07-11 19:32:22,339 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_896_8b01143d063fb3d088dd.png 2023-07-11 19:32:22,340 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_896_1195075f26c9c85c17be.png 2023-07-11 19:32:22,340 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 19:32:22,673 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/atk664lw-media/images/validation_896_a33d92e49bb3819c9398.png 2023-07-11 19:32:22,686 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/9ua92f5x-media/images/validation_896_2d49f7aa4f79dc70f56e.png 2023-07-11 19:32:22,692 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/eo4qp2nn-media/images/validation_896_1195075f26c9c85c17be.png 2023-07-11 19:32:22,723 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/u54tfuev-media/images/validation_896_8b01143d063fb3d088dd.png 2023-07-11 19:32:23,017 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:32:23,706 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:32:27,198 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:32:27,200 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:32:27,200 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:32:27,202 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:32:27,248 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:32:27,248 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:32:27,248 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:32:27,349 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:32:27,350 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:32:28,289 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:32:30,514 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:32:30,515 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:32:30,516 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:32:30,518 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:32:32,361 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:32:32,361 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:32:33,414 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:32:34,378 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:32:34,380 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:32:34,381 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:32:34,382 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:32:37,372 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:32:37,373 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:32:37,757 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:32:37,758 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:32:37,759 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:32:37,760 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:32:38,761 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:32:40,758 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:32:41,206 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:32:41,208 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:32:41,208 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:32:41,210 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:32:42,248 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:32:42,248 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:32:42,248 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:32:42,385 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:32:42,385 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:32:44,113 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:32:44,254 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:32:44,256 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:32:44,257 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:32:44,258 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:32:44,536 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:32:44,537 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:32:44,538 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:32:44,539 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:32:47,397 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:32:47,398 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:32:49,135 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:32:54,136 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:32:57,191 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:32:57,192 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_904_d08a528486fd9d24317e.png with policy now 2023-07-11 19:32:57,206 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:32:57,206 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_904_74a58ccedcd0aba02d9d.png with policy now 2023-07-11 19:32:57,215 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:32:57,216 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_904_01099aeefecc508a7281.png with policy now 2023-07-11 19:32:57,224 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:32:57,224 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_904_d9f4aaa16030ef084b93.png with policy now 2023-07-11 19:32:57,248 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:32:57,248 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:32:57,249 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:32:57,266 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:32:57,394 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:32:57,395 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:32:57,397 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:32:57,422 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:32:57,422 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_904_74a58ccedcd0aba02d9d.png 2023-07-11 19:32:57,423 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_904_d08a528486fd9d24317e.png 2023-07-11 19:32:57,423 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_904_d9f4aaa16030ef084b93.png 2023-07-11 19:32:57,424 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_904_01099aeefecc508a7281.png 2023-07-11 19:32:57,424 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 19:32:57,506 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/8q8wddpf-media/images/validation_904_d08a528486fd9d24317e.png 2023-07-11 19:32:57,533 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/49jnvbdc-media/images/validation_904_74a58ccedcd0aba02d9d.png 2023-07-11 19:32:57,541 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/rpjr0yka-media/images/validation_904_d9f4aaa16030ef084b93.png 2023-07-11 19:32:57,547 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/sh9oyhts-media/images/validation_904_01099aeefecc508a7281.png 2023-07-11 19:32:57,891 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/quthozvm-media/images/validation_904_d08a528486fd9d24317e.png 2023-07-11 19:32:57,916 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/d0p0g9n1-media/images/validation_904_d9f4aaa16030ef084b93.png 2023-07-11 19:32:57,917 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/lkcrqh1j-media/images/validation_904_01099aeefecc508a7281.png 2023-07-11 19:32:58,244 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/7sm4pgwr-media/images/validation_904_74a58ccedcd0aba02d9d.png 2023-07-11 19:32:59,589 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:33:01,265 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:33:02,433 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:33:04,799 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:33:04,800 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:33:04,801 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:33:04,801 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:33:04,803 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:33:07,422 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:33:07,423 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:33:07,424 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:33:07,426 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:33:07,444 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:33:07,444 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:33:10,698 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:33:10,700 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:33:10,700 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:33:10,701 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:33:10,703 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:33:10,761 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:33:12,382 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:33:12,382 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:33:12,383 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:33:12,455 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:33:12,455 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:33:14,254 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:33:14,256 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:33:14,256 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:33:14,258 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:33:15,990 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:33:17,466 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:33:17,466 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:33:17,775 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:33:17,777 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:33:17,778 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:33:17,779 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:33:21,533 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:33:21,535 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:33:21,535 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:33:21,535 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:33:21,537 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:33:21,815 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:33:21,821 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:33:21,821 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:33:21,823 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:33:22,477 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:33:22,478 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:33:27,371 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:33:27,383 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:33:27,383 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:33:27,383 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:33:27,489 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:33:32,447 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:33:34,374 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:33:34,374 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_912_7c4eaf576d54dc0892c4.png with policy now 2023-07-11 19:33:34,385 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:33:34,385 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_912_4a51dc0e598ef75ade74.png with policy now 2023-07-11 19:33:34,394 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:33:34,394 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_912_87ef29d0d2134d35550f.png with policy now 2023-07-11 19:33:34,403 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:33:34,403 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_912_ac2d97b761361e2c7f95.png with policy now 2023-07-11 19:33:34,444 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:33:34,445 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:33:34,445 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:33:34,447 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:33:34,788 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/9d008pz5-media/images/validation_912_4a51dc0e598ef75ade74.png 2023-07-11 19:33:34,793 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/n1u03plo-media/images/validation_912_ac2d97b761361e2c7f95.png 2023-07-11 19:33:34,801 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/9poti4yu-media/images/validation_912_87ef29d0d2134d35550f.png 2023-07-11 19:33:34,932 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/sl1314dd-media/images/validation_912_7c4eaf576d54dc0892c4.png 2023-07-11 19:33:37,515 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:33:37,516 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_912_7c4eaf576d54dc0892c4.png 2023-07-11 19:33:37,516 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_912_ac2d97b761361e2c7f95.png 2023-07-11 19:33:37,517 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_912_87ef29d0d2134d35550f.png 2023-07-11 19:33:37,517 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_912_4a51dc0e598ef75ade74.png 2023-07-11 19:33:37,517 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 19:33:37,578 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:33:37,861 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/sgb8s9fn-media/images/validation_912_ac2d97b761361e2c7f95.png 2023-07-11 19:33:37,885 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/oi84yd0i-media/images/validation_912_4a51dc0e598ef75ade74.png 2023-07-11 19:33:37,888 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/m4hlx86q-media/images/validation_912_87ef29d0d2134d35550f.png 2023-07-11 19:33:37,948 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/3g795f87-media/images/validation_912_7c4eaf576d54dc0892c4.png 2023-07-11 19:33:39,045 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:33:40,764 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:33:41,674 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:33:41,676 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:33:41,676 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:33:41,678 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:33:42,516 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:33:42,517 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:33:42,517 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:33:42,637 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:33:42,688 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:33:42,688 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:33:45,584 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:33:45,586 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:33:45,586 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:33:45,588 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:33:47,699 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:33:47,699 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:33:47,899 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:33:48,457 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:33:48,458 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:33:48,459 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:33:48,492 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:33:51,904 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:33:51,906 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:33:51,906 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:33:51,919 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:33:52,710 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:33:52,710 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:33:53,297 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:33:54,905 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:33:54,907 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:33:54,907 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:33:54,909 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:33:57,517 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:33:57,517 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:33:57,518 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:33:57,722 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:33:57,722 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:33:58,340 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:33:58,341 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:33:58,342 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:33:58,342 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:33:58,344 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:33:58,622 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:33:58,624 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:33:58,624 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:33:58,626 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:34:02,733 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:34:02,734 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:34:03,456 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:34:08,457 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:34:10,768 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:34:11,513 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:34:11,513 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_920_dd362beabef9f0f07f1c.png with policy now 2023-07-11 19:34:11,524 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:34:11,524 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_920_ceb5dd8574ce0588ad3d.png with policy now 2023-07-11 19:34:11,537 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:34:11,537 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_920_cb802cc8bb9421190135.png with policy now 2023-07-11 19:34:11,546 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:34:11,546 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_920_5b7b240cdae3212a828d.png with policy now 2023-07-11 19:34:11,584 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:34:11,585 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:34:11,586 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:34:11,587 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:34:11,815 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/lo4u0937-media/images/validation_920_dd362beabef9f0f07f1c.png 2023-07-11 19:34:11,849 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/x0mrhfmi-media/images/validation_920_5b7b240cdae3212a828d.png 2023-07-11 19:34:11,857 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/nhoawxaj-media/images/validation_920_ceb5dd8574ce0588ad3d.png 2023-07-11 19:34:11,885 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/iygs1asy-media/images/validation_920_cb802cc8bb9421190135.png 2023-07-11 19:34:12,517 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:34:12,517 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:34:12,518 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:34:12,758 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:34:12,758 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_920_ceb5dd8574ce0588ad3d.png 2023-07-11 19:34:12,759 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_920_5b7b240cdae3212a828d.png 2023-07-11 19:34:12,759 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_920_cb802cc8bb9421190135.png 2023-07-11 19:34:12,759 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_920_dd362beabef9f0f07f1c.png 2023-07-11 19:34:12,760 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 19:34:13,118 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/gseb731a-media/images/validation_920_cb802cc8bb9421190135.png 2023-07-11 19:34:13,120 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/zgucnaar-media/images/validation_920_5b7b240cdae3212a828d.png 2023-07-11 19:34:13,124 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/022u9wsw-media/images/validation_920_dd362beabef9f0f07f1c.png 2023-07-11 19:34:13,135 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/m3c6ed1o-media/images/validation_920_ceb5dd8574ce0588ad3d.png 2023-07-11 19:34:14,011 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:34:15,148 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:34:17,775 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:34:18,044 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:34:18,046 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:34:18,046 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:34:18,083 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:34:19,738 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:34:21,409 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:34:21,411 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:34:21,412 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:34:21,426 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:34:22,786 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:34:22,787 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:34:24,583 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:34:24,585 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:34:24,585 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:34:24,617 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:34:25,296 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:34:27,405 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:34:27,407 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:34:27,408 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:34:27,410 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:34:27,517 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:34:27,517 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:34:27,518 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:34:27,797 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:34:27,798 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:34:30,499 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:34:30,501 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:34:30,501 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:34:30,501 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:34:30,503 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:34:32,809 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:34:32,809 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:34:33,679 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:34:33,681 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:34:33,682 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:34:33,683 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:34:33,962 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:34:33,964 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:34:33,964 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:34:33,977 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:34:35,540 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:34:37,821 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:34:37,821 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:34:40,541 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:34:40,770 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:34:42,517 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:34:42,518 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:34:42,518 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:34:45,559 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:34:46,653 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:34:46,654 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_928_35f2e6d732e429b27cef.png with policy now 2023-07-11 19:34:46,710 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:34:46,710 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_928_c7d072dca160d6de598d.png with policy now 2023-07-11 19:34:46,776 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:34:46,777 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_928_6c25bd7611e598bbfbe5.png with policy now 2023-07-11 19:34:46,821 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:34:46,821 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_928_0ccf3153354added95d1.png with policy now 2023-07-11 19:34:46,870 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:34:46,871 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:34:46,871 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:34:46,884 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:34:46,977 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/8g2zrrf6-media/images/validation_928_35f2e6d732e429b27cef.png 2023-07-11 19:34:47,054 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/qvg7n67c-media/images/validation_928_c7d072dca160d6de598d.png 2023-07-11 19:34:47,084 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/rwbi7kbs-media/images/validation_928_6c25bd7611e598bbfbe5.png 2023-07-11 19:34:47,154 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/dcaepygp-media/images/validation_928_0ccf3153354added95d1.png 2023-07-11 19:34:47,844 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:34:47,844 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_928_35f2e6d732e429b27cef.png 2023-07-11 19:34:47,844 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_928_0ccf3153354added95d1.png 2023-07-11 19:34:47,845 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_928_6c25bd7611e598bbfbe5.png 2023-07-11 19:34:47,845 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_928_c7d072dca160d6de598d.png 2023-07-11 19:34:47,845 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 19:34:48,186 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/iexigjwi-media/images/validation_928_35f2e6d732e429b27cef.png 2023-07-11 19:34:48,212 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ntsmwitr-media/images/validation_928_6c25bd7611e598bbfbe5.png 2023-07-11 19:34:48,215 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/nr3l153r-media/images/validation_928_0ccf3153354added95d1.png 2023-07-11 19:34:48,237 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/1c1a23wk-media/images/validation_928_c7d072dca160d6de598d.png 2023-07-11 19:34:50,779 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:34:50,780 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:34:52,855 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:34:53,518 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:34:53,520 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:34:53,520 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:34:53,521 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:34:56,043 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:34:57,282 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:34:57,283 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:34:57,283 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:34:57,285 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:34:57,656 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:34:57,657 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:34:57,657 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:34:57,866 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:34:57,867 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:35:00,288 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:35:00,289 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:35:00,290 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:35:00,291 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:35:01,904 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:35:02,881 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:35:02,881 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:35:04,167 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:35:04,169 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:35:04,169 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:35:04,171 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:35:06,756 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:35:06,758 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:35:06,758 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:35:06,760 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:35:07,604 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:35:07,893 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:35:07,893 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:35:10,519 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:35:10,521 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:35:10,521 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:35:10,523 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:35:10,774 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:35:10,801 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:35:10,803 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:35:10,804 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:35:10,805 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:35:12,656 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:35:12,656 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:35:12,657 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:35:12,905 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:35:12,906 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:35:12,980 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:35:17,981 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:35:22,982 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:35:23,436 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:35:23,436 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_936_14f68a769ca53b61b6be.png with policy now 2023-07-11 19:35:23,494 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:35:23,495 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_936_fbfe70b4fd3580e4922a.png with policy now 2023-07-11 19:35:23,554 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:35:23,554 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_936_190b7cee3f4ce931d581.png with policy now 2023-07-11 19:35:23,620 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:35:23,620 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_936_427cd756a6fbf28d7994.png with policy now 2023-07-11 19:35:23,668 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:35:23,669 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:35:23,669 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:35:23,680 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:35:23,762 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ivtsv3nc-media/images/validation_936_14f68a769ca53b61b6be.png 2023-07-11 19:35:23,832 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/66v7563b-media/images/validation_936_fbfe70b4fd3580e4922a.png 2023-07-11 19:35:23,873 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/j1hg6m1p-media/images/validation_936_190b7cee3f4ce931d581.png 2023-07-11 19:35:23,937 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/p1xx053z-media/images/validation_936_427cd756a6fbf28d7994.png 2023-07-11 19:35:26,859 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:35:27,656 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:35:27,657 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:35:27,657 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:35:27,943 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:35:27,944 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:35:27,944 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_936_14f68a769ca53b61b6be.png 2023-07-11 19:35:27,944 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_936_fbfe70b4fd3580e4922a.png 2023-07-11 19:35:27,945 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_936_427cd756a6fbf28d7994.png 2023-07-11 19:35:27,946 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_936_190b7cee3f4ce931d581.png 2023-07-11 19:35:27,946 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 19:35:28,358 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/xygli45o-media/images/validation_936_14f68a769ca53b61b6be.png 2023-07-11 19:35:28,363 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/6cejjath-media/images/validation_936_427cd756a6fbf28d7994.png 2023-07-11 19:35:28,382 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/e1xpidn1-media/images/validation_936_fbfe70b4fd3580e4922a.png 2023-07-11 19:35:28,397 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/wjo33vxg-media/images/validation_936_190b7cee3f4ce931d581.png 2023-07-11 19:35:28,759 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:35:30,375 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:35:30,377 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:35:30,377 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:35:30,378 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:35:32,954 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:35:32,955 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:35:33,893 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:35:33,895 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:35:33,895 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:35:33,895 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:35:33,897 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:35:37,815 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:35:37,817 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:35:37,817 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:35:37,818 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:35:37,966 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:35:37,966 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:35:39,099 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:35:40,354 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:35:40,356 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:35:40,356 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:35:40,357 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:35:40,776 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:35:42,824 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:35:42,824 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:35:42,825 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:35:42,978 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:35:42,978 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:35:44,075 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:35:44,077 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:35:44,078 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:35:44,079 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:35:45,080 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:35:47,632 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:35:47,633 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:35:47,634 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:35:47,671 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:35:47,916 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:35:47,917 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:35:47,918 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:35:47,920 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:35:47,990 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:35:47,990 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:35:50,488 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:35:53,006 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:35:55,489 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:35:57,861 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:35:57,862 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:35:57,862 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:36:00,523 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:36:00,523 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_944_131e7280660d61568c15.png with policy now 2023-07-11 19:36:00,524 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:36:00,590 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:36:00,590 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_944_41a5dce9cfa5cf744e71.png with policy now 2023-07-11 19:36:00,663 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:36:00,663 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_944_d7a08791c37a0d7e716d.png with policy now 2023-07-11 19:36:00,707 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:36:00,707 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_944_8d33167a8809bc5e2e10.png with policy now 2023-07-11 19:36:00,755 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:36:00,756 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:36:00,756 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:36:00,770 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:36:00,849 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/o1d2vu2q-media/images/validation_944_131e7280660d61568c15.png 2023-07-11 19:36:00,902 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/3eboyitv-media/images/validation_944_41a5dce9cfa5cf744e71.png 2023-07-11 19:36:00,977 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/7bprv60j-media/images/validation_944_d7a08791c37a0d7e716d.png 2023-07-11 19:36:01,006 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/xk69tfof-media/images/validation_944_8d33167a8809bc5e2e10.png 2023-07-11 19:36:03,031 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:36:03,031 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_944_d7a08791c37a0d7e716d.png 2023-07-11 19:36:03,032 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_944_8d33167a8809bc5e2e10.png 2023-07-11 19:36:03,032 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_944_41a5dce9cfa5cf744e71.png 2023-07-11 19:36:03,033 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_944_131e7280660d61568c15.png 2023-07-11 19:36:03,033 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 19:36:03,371 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/18ijyicw-media/images/validation_944_8d33167a8809bc5e2e10.png 2023-07-11 19:36:03,378 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/lzkptwjc-media/images/validation_944_d7a08791c37a0d7e716d.png 2023-07-11 19:36:03,394 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/39mjwa27-media/images/validation_944_41a5dce9cfa5cf744e71.png 2023-07-11 19:36:03,403 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/eaxuztyr-media/images/validation_944_131e7280660d61568c15.png 2023-07-11 19:36:04,069 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:36:06,060 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:36:06,687 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:36:06,689 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:36:06,690 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:36:06,706 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:36:08,043 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:36:08,044 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:36:10,146 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:36:10,147 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:36:10,147 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:36:10,149 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:36:10,779 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:36:11,611 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:36:12,861 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:36:12,862 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:36:12,862 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:36:13,054 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:36:13,055 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:36:14,006 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:36:14,009 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:36:14,009 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:36:14,011 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:36:16,918 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:36:16,920 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:36:16,920 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:36:16,921 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:36:16,937 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:36:18,066 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:36:18,067 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:36:20,350 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:36:20,352 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:36:20,352 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:36:20,354 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:36:21,957 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:36:23,078 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:36:23,078 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:36:24,280 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:36:24,282 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:36:24,282 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:36:24,298 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:36:24,564 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:36:24,566 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:36:24,566 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:36:24,585 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:36:27,122 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:36:27,861 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:36:27,862 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:36:27,862 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:36:28,090 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:36:28,091 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:36:32,913 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:36:37,215 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:36:37,216 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_952_fc214fd3c543e1062168.png with policy now 2023-07-11 19:36:37,307 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:36:37,307 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_952_49c73fa7e966791f5390.png with policy now 2023-07-11 19:36:37,372 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:36:37,372 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_952_c88dcfd62ec0e16b152a.png with policy now 2023-07-11 19:36:37,433 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:36:37,433 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_952_e9c245f08754c65b88da.png with policy now 2023-07-11 19:36:37,478 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:36:37,479 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:36:37,479 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:36:37,483 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:36:37,546 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/gi7iqdde-media/images/validation_952_fc214fd3c543e1062168.png 2023-07-11 19:36:37,618 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/6vv7cq2s-media/images/validation_952_49c73fa7e966791f5390.png 2023-07-11 19:36:37,688 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/lzwk67vp-media/images/validation_952_c88dcfd62ec0e16b152a.png 2023-07-11 19:36:37,758 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/vvug7jfb-media/images/validation_952_e9c245f08754c65b88da.png 2023-07-11 19:36:38,115 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:36:38,115 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_952_fc214fd3c543e1062168.png 2023-07-11 19:36:38,116 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_952_49c73fa7e966791f5390.png 2023-07-11 19:36:38,117 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_952_c88dcfd62ec0e16b152a.png 2023-07-11 19:36:38,117 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_952_e9c245f08754c65b88da.png 2023-07-11 19:36:38,117 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 19:36:38,484 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:36:38,493 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/3hajqimu-media/images/validation_952_fc214fd3c543e1062168.png 2023-07-11 19:36:38,509 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ytfhqbwj-media/images/validation_952_49c73fa7e966791f5390.png 2023-07-11 19:36:38,520 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ws790l59-media/images/validation_952_c88dcfd62ec0e16b152a.png 2023-07-11 19:36:38,529 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/efj0i41a-media/images/validation_952_e9c245f08754c65b88da.png 2023-07-11 19:36:40,781 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:36:41,369 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:36:42,861 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:36:42,862 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:36:42,862 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:36:43,127 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:36:44,266 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:36:44,356 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:36:44,358 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:36:44,358 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:36:44,759 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:36:48,139 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:36:48,139 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:36:48,216 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:36:48,218 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:36:48,219 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:36:48,242 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:36:49,304 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:36:50,969 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:36:50,971 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:36:50,971 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:36:50,976 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:36:53,151 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:36:53,151 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:36:54,077 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:36:54,079 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:36:54,079 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:36:54,104 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:36:55,042 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:36:57,423 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:36:57,424 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:36:57,425 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:36:57,450 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:36:58,163 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:36:58,164 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:36:58,207 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:36:58,208 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:36:58,208 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:37:00,172 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:37:01,023 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:37:01,025 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:37:01,025 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:37:01,049 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:37:01,307 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:37:01,309 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:37:01,309 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:37:01,324 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:37:03,194 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:37:03,194 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:37:06,070 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:37:08,222 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:37:10,784 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:37:11,786 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:37:13,246 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:37:13,247 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:37:13,247 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:37:14,172 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:37:14,172 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_960_bbcff34d61dc915cfffa.png with policy now 2023-07-11 19:37:14,217 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:37:14,217 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_960_d943310ca37254daf167.png with policy now 2023-07-11 19:37:14,276 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:37:14,276 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_960_1f1bb4f5e11489979814.png with policy now 2023-07-11 19:37:14,375 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:37:14,375 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_960_dab10b88bdf06be619c2.png with policy now 2023-07-11 19:37:14,422 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:37:14,423 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:37:14,423 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:37:14,443 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:37:14,492 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ulerpov4-media/images/validation_960_bbcff34d61dc915cfffa.png 2023-07-11 19:37:14,540 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/9crhjdia-media/images/validation_960_d943310ca37254daf167.png 2023-07-11 19:37:14,602 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/g9a4id4a-media/images/validation_960_1f1bb4f5e11489979814.png 2023-07-11 19:37:14,690 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/14u2a6v3-media/images/validation_960_dab10b88bdf06be619c2.png 2023-07-11 19:37:16,984 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:37:17,953 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:37:18,261 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:37:18,262 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:37:18,262 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_960_dab10b88bdf06be619c2.png 2023-07-11 19:37:18,263 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_960_bbcff34d61dc915cfffa.png 2023-07-11 19:37:18,263 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_960_d943310ca37254daf167.png 2023-07-11 19:37:18,264 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_960_1f1bb4f5e11489979814.png 2023-07-11 19:37:18,264 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 19:37:18,603 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/f66jgr2v-media/images/validation_960_dab10b88bdf06be619c2.png 2023-07-11 19:37:18,610 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/0rfthdn1-media/images/validation_960_bbcff34d61dc915cfffa.png 2023-07-11 19:37:18,628 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/8ddav4rw-media/images/validation_960_d943310ca37254daf167.png 2023-07-11 19:37:18,653 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/82w6kfuh-media/images/validation_960_1f1bb4f5e11489979814.png 2023-07-11 19:37:21,349 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:37:21,392 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:37:21,393 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:37:21,434 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:37:22,111 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:37:23,276 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:37:23,276 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:37:24,484 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:37:24,486 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:37:24,487 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:37:24,525 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:37:27,176 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:37:28,246 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:37:28,247 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:37:28,247 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:37:28,291 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:37:28,292 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:37:28,300 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:37:28,303 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:37:28,304 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:37:28,326 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:37:31,468 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:37:31,470 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:37:31,471 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:37:31,477 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:37:32,839 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:37:33,312 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:37:33,313 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:37:34,250 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:37:34,252 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:37:34,252 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:37:34,264 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:37:37,059 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:37:37,061 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:37:37,061 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:37:37,074 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:37:37,340 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:37:37,342 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:37:37,343 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:37:37,350 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:37:37,957 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:37:38,327 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:37:38,327 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:37:40,787 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:37:43,268 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:37:43,269 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:37:43,269 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:37:43,343 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:37:43,361 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:37:48,362 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:37:50,018 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:37:50,018 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_968_d57c92e936cc34db3b66.png with policy now 2023-07-11 19:37:50,235 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:37:50,235 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_968_37b53ef8f7963631c78a.png with policy now 2023-07-11 19:37:50,303 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:37:50,303 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_968_d874c01a95067c21bdbb.png with policy now 2023-07-11 19:37:50,345 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:37:50,345 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_968_73e3db3d1e8039707146.png with policy now 2023-07-11 19:37:50,362 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/2592os36-media/images/validation_968_d57c92e936cc34db3b66.png 2023-07-11 19:37:50,393 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:37:50,395 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:37:50,395 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:37:50,425 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:37:50,560 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/f28r2hgx-media/images/validation_968_37b53ef8f7963631c78a.png 2023-07-11 19:37:50,628 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/q66877b7-media/images/validation_968_d874c01a95067c21bdbb.png 2023-07-11 19:37:50,660 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/xcfsebkg-media/images/validation_968_73e3db3d1e8039707146.png 2023-07-11 19:37:53,367 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:37:53,368 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_968_37b53ef8f7963631c78a.png 2023-07-11 19:37:53,368 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_968_73e3db3d1e8039707146.png 2023-07-11 19:37:53,369 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_968_d57c92e936cc34db3b66.png 2023-07-11 19:37:53,370 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_968_d874c01a95067c21bdbb.png 2023-07-11 19:37:53,370 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 19:37:53,553 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:37:53,711 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/xvv67r30-media/images/validation_968_d57c92e936cc34db3b66.png 2023-07-11 19:37:53,719 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/qt1y1f7h-media/images/validation_968_73e3db3d1e8039707146.png 2023-07-11 19:37:53,724 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/556771gq-media/images/validation_968_37b53ef8f7963631c78a.png 2023-07-11 19:37:53,740 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/62trrrrv-media/images/validation_968_d874c01a95067c21bdbb.png 2023-07-11 19:37:54,265 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:37:57,604 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:37:57,606 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:37:57,606 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:37:57,624 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:37:58,299 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:37:58,300 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:37:58,301 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:37:58,382 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:37:58,383 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:37:59,268 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:38:01,537 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:38:01,539 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:38:01,539 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:38:01,553 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:38:03,410 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:38:03,410 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:38:05,023 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:38:05,025 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:38:05,026 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:38:05,026 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:38:05,081 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:38:08,046 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:38:08,048 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:38:08,048 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:38:08,049 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:38:08,422 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:38:08,423 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:38:10,577 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:38:10,790 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:38:11,382 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:38:11,383 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:38:11,384 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:38:11,399 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:38:13,299 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:38:13,300 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:38:13,300 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:38:13,434 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:38:13,435 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:38:14,519 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:38:14,521 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:38:14,521 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:38:14,535 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:38:14,803 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:38:14,805 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:38:14,805 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:38:14,836 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:38:15,697 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:38:18,446 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:38:18,447 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:38:20,698 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:38:25,699 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:38:27,796 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:38:27,796 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_976_1c6ecb09cf45729ec02d.png with policy now 2023-07-11 19:38:27,854 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:38:27,854 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_976_96467eb797af222d5127.png with policy now 2023-07-11 19:38:27,937 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:38:27,937 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_976_b1d4d79c192e807ea9ea.png with policy now 2023-07-11 19:38:28,002 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:38:28,002 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_976_24834f9350949b5f8e1c.png with policy now 2023-07-11 19:38:28,049 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:38:28,050 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:38:28,050 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:38:28,079 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:38:28,111 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/wbapnmcb-media/images/validation_976_1c6ecb09cf45729ec02d.png 2023-07-11 19:38:28,166 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/5r5anmro-media/images/validation_976_96467eb797af222d5127.png 2023-07-11 19:38:28,247 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/lr7eqqw6-media/images/validation_976_b1d4d79c192e807ea9ea.png 2023-07-11 19:38:28,299 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:38:28,300 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:38:28,300 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:38:28,313 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/kornjjvz-media/images/validation_976_24834f9350949b5f8e1c.png 2023-07-11 19:38:28,471 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:38:28,472 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_976_b1d4d79c192e807ea9ea.png 2023-07-11 19:38:28,472 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_976_96467eb797af222d5127.png 2023-07-11 19:38:28,473 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_976_1c6ecb09cf45729ec02d.png 2023-07-11 19:38:28,473 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_976_24834f9350949b5f8e1c.png 2023-07-11 19:38:28,474 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 19:38:28,820 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/k2ttuqg4-media/images/validation_976_1c6ecb09cf45729ec02d.png 2023-07-11 19:38:28,822 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/6xkqcgsd-media/images/validation_976_b1d4d79c192e807ea9ea.png 2023-07-11 19:38:28,829 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/mejnri3z-media/images/validation_976_96467eb797af222d5127.png 2023-07-11 19:38:28,833 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/iwv7t6pb-media/images/validation_976_24834f9350949b5f8e1c.png 2023-07-11 19:38:30,880 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:38:31,768 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:38:33,483 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:38:34,821 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:38:34,823 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:38:34,823 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:38:34,837 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:38:36,317 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:38:38,072 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:38:38,073 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:38:38,074 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:38:38,103 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:38:38,495 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:38:38,495 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:38:40,793 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:38:41,277 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:38:41,279 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:38:41,279 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:38:41,292 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:38:41,831 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:38:43,299 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:38:43,300 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:38:43,301 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:38:43,505 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:38:43,506 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:38:45,068 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:38:45,070 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:38:45,070 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:38:45,086 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:38:47,496 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:38:48,518 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:38:48,518 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:38:49,131 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:38:49,133 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:38:49,133 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:38:49,135 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:38:52,615 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:38:52,697 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:38:52,699 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:38:52,699 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:38:52,701 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:38:52,984 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:38:52,985 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:38:52,986 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:38:52,987 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:38:53,530 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:38:53,530 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:38:58,300 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:38:58,300 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:38:58,301 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:38:58,353 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:38:58,542 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:39:03,355 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:39:05,710 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:39:05,710 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_984_8ce2b6b9e57abff6242d.png with policy now 2023-07-11 19:39:05,771 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:39:05,771 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_984_6babdd3a11961e590892.png with policy now 2023-07-11 19:39:05,846 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:39:05,846 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_984_8655855a41d6a76b3f50.png with policy now 2023-07-11 19:39:05,903 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:39:05,903 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_984_858c44737753f571d83a.png with policy now 2023-07-11 19:39:05,973 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:39:05,974 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:39:05,975 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:39:06,014 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:39:06,958 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/wgap6hdx-media/images/validation_984_6babdd3a11961e590892.png 2023-07-11 19:39:07,038 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/p4whq2wq-media/images/validation_984_8655855a41d6a76b3f50.png 2023-07-11 19:39:07,113 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ax7ogiea-media/images/validation_984_858c44737753f571d83a.png 2023-07-11 19:39:07,200 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/05uyxeau-media/images/validation_984_8ce2b6b9e57abff6242d.png 2023-07-11 19:39:08,567 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:39:08,567 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_984_8655855a41d6a76b3f50.png 2023-07-11 19:39:08,568 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_984_858c44737753f571d83a.png 2023-07-11 19:39:08,568 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_984_6babdd3a11961e590892.png 2023-07-11 19:39:08,569 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_984_8ce2b6b9e57abff6242d.png 2023-07-11 19:39:08,569 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 19:39:08,886 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:39:08,960 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/49569wwz-media/images/validation_984_8ce2b6b9e57abff6242d.png 2023-07-11 19:39:08,972 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/qrct0e61-media/images/validation_984_858c44737753f571d83a.png 2023-07-11 19:39:08,973 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/fhndwqzf-media/images/validation_984_8655855a41d6a76b3f50.png 2023-07-11 19:39:09,591 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:39:09,639 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/mlkdchnx-media/images/validation_984_6babdd3a11961e590892.png 2023-07-11 19:39:10,796 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:39:12,888 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:39:12,890 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:39:12,890 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:39:12,901 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:39:13,300 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:39:13,300 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:39:13,301 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:39:13,579 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:39:13,579 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:39:14,709 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:39:16,533 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:39:16,534 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:39:16,535 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:39:16,536 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:39:18,590 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:39:18,591 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:39:20,105 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:39:20,107 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:39:20,107 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:39:20,108 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:39:20,110 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:39:23,551 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:39:23,553 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:39:23,553 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:39:23,555 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:39:23,603 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:39:23,603 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:39:25,273 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:39:27,430 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:39:27,432 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:39:27,433 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:39:27,434 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:39:28,405 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:39:28,406 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:39:28,406 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:39:28,615 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:39:28,615 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:39:30,766 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:39:30,768 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:39:30,769 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:39:30,769 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:39:30,771 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:39:31,051 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:39:31,052 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:39:31,053 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:39:31,054 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:39:33,627 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:39:33,627 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:39:36,612 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:39:40,799 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:39:41,801 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:39:43,406 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:39:43,406 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:39:43,407 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:39:43,629 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:39:43,630 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_992_81d31deea5f321a879ea.png with policy now 2023-07-11 19:39:43,644 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:39:43,644 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_992_d3be1fc1429f13f91ba7.png with policy now 2023-07-11 19:39:43,654 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:39:43,654 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_992_b832ae939713ff831840.png with policy now 2023-07-11 19:39:43,657 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:39:43,657 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_992_297cfc9d19c6bf373211.png with policy now 2023-07-11 19:39:43,674 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_992_b832ae939713ff831840.png 2023-07-11 19:39:43,674 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_992_81d31deea5f321a879ea.png 2023-07-11 19:39:43,675 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_992_d3be1fc1429f13f91ba7.png 2023-07-11 19:39:43,676 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 19:39:43,699 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:39:43,700 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:39:43,700 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:39:43,702 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:39:43,986 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/owevqxqs-media/images/validation_992_81d31deea5f321a879ea.png 2023-07-11 19:39:44,018 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/mx7hs6jo-media/images/validation_992_d3be1fc1429f13f91ba7.png 2023-07-11 19:39:44,026 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/fxwazsbx-media/images/validation_992_b832ae939713ff831840.png 2023-07-11 19:39:44,032 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/fg4bvyty-media/images/validation_992_297cfc9d19c6bf373211.png 2023-07-11 19:39:44,488 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/gxf0g1eq-media/images/validation_992_81d31deea5f321a879ea.png 2023-07-11 19:39:44,540 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/d8vuqnbi-media/images/validation_992_d3be1fc1429f13f91ba7.png 2023-07-11 19:39:44,608 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/hohj7e7p-media/images/validation_992_b832ae939713ff831840.png 2023-07-11 19:39:46,830 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:39:47,835 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:39:48,688 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:39:48,688 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:39:48,688 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_992_297cfc9d19c6bf373211.png 2023-07-11 19:39:48,689 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 19:39:49,073 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/zbeyivp4-media/images/validation_992_297cfc9d19c6bf373211.png 2023-07-11 19:39:51,499 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:39:51,501 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:39:51,501 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:39:51,503 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:39:52,504 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:39:53,699 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:39:53,700 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:39:55,282 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:39:55,284 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:39:55,285 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:39:55,286 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:39:57,688 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:39:58,406 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:39:58,406 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:39:58,407 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:39:58,505 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:39:58,507 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:39:58,507 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:39:58,509 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:39:59,711 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:39:59,712 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:40:02,086 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:40:02,088 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:40:02,088 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:40:02,090 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:40:03,091 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:40:05,603 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:40:05,605 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:40:05,605 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:40:05,607 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:40:05,722 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:40:05,723 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:40:08,496 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:40:08,974 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:40:08,976 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:40:08,977 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:40:08,978 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:40:09,270 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:40:09,271 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:40:09,272 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:40:09,274 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:40:10,802 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:40:11,734 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:40:11,735 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:40:13,406 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:40:13,406 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:40:13,407 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:40:14,458 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:40:17,746 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:40:19,459 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:40:22,422 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:40:22,422 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1000_ffc1a9f27bcd6f2a61ac.png with policy now 2023-07-11 19:40:22,436 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:40:22,437 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1000_3bac780bdad763dfe748.png with policy now 2023-07-11 19:40:22,447 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:40:22,447 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1000_3df8df281127a97b5d1a.png with policy now 2023-07-11 19:40:22,449 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:40:22,450 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1000_fe76546257030d738699.png with policy now 2023-07-11 19:40:22,490 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:40:22,491 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:40:22,491 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:40:22,493 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:40:22,757 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/9i010des-media/images/validation_1000_ffc1a9f27bcd6f2a61ac.png 2023-07-11 19:40:22,769 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/413j6g6g-media/images/validation_1000_3bac780bdad763dfe748.png 2023-07-11 19:40:22,771 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/iyq5vfg5-media/images/validation_1000_3df8df281127a97b5d1a.png 2023-07-11 19:40:22,785 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/y5xtvcqt-media/images/validation_1000_fe76546257030d738699.png 2023-07-11 19:40:23,760 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:40:23,760 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1000_ffc1a9f27bcd6f2a61ac.png 2023-07-11 19:40:23,761 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1000_3bac780bdad763dfe748.png 2023-07-11 19:40:23,761 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1000_3df8df281127a97b5d1a.png 2023-07-11 19:40:23,762 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1000_fe76546257030d738699.png 2023-07-11 19:40:23,762 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 19:40:24,100 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/oy37bxc4-media/images/validation_1000_ffc1a9f27bcd6f2a61ac.png 2023-07-11 19:40:24,121 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/bzj7tdk8-media/images/validation_1000_fe76546257030d738699.png 2023-07-11 19:40:24,125 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/6i3bv9xx-media/images/validation_1000_3bac780bdad763dfe748.png 2023-07-11 19:40:24,129 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/1fh2gjp2-media/images/validation_1000_3df8df281127a97b5d1a.png 2023-07-11 19:40:24,749 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:40:25,840 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:40:28,406 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:40:28,407 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:40:28,407 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:40:29,529 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:40:29,530 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:40:29,531 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:40:29,533 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:40:29,772 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:40:29,772 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:40:30,169 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:40:32,654 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:40:32,656 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:40:32,657 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:40:32,658 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:40:35,319 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:40:35,783 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:40:35,784 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:40:36,102 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:40:36,103 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:40:36,103 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:40:36,105 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:40:38,962 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:40:38,963 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:40:38,964 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:40:38,965 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:40:40,545 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:40:40,805 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:40:41,795 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:40:41,795 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:40:42,241 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:40:42,243 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:40:42,243 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:40:42,245 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:40:43,406 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:40:43,407 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:40:43,407 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:40:45,747 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:40:45,916 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:40:45,917 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:40:45,918 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:40:45,920 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:40:46,198 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:40:46,200 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:40:46,201 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:40:46,202 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:40:47,807 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:40:47,808 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:40:51,731 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:40:53,819 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:40:56,732 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:40:58,407 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:40:58,407 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:40:58,408 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:40:58,769 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:40:58,769 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1008_386a62fe7e44ee995bbb.png with policy now 2023-07-11 19:40:58,779 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:40:58,779 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1008_8bde6a294d23990b588f.png with policy now 2023-07-11 19:40:58,789 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:40:58,789 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1008_808168a3292399e3b0a3.png with policy now 2023-07-11 19:40:58,797 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:40:58,797 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1008_2e83a2626ff883d16244.png with policy now 2023-07-11 19:40:58,840 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:40:58,841 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:40:58,842 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:40:58,847 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:40:59,082 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/k0580yj6-media/images/validation_1008_386a62fe7e44ee995bbb.png 2023-07-11 19:40:59,105 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/7t3w1ldj-media/images/validation_1008_2e83a2626ff883d16244.png 2023-07-11 19:40:59,106 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ae76oe0z-media/images/validation_1008_8bde6a294d23990b588f.png 2023-07-11 19:40:59,107 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/802edw9f-media/images/validation_1008_808168a3292399e3b0a3.png 2023-07-11 19:40:59,832 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:40:59,832 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1008_8bde6a294d23990b588f.png 2023-07-11 19:40:59,833 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1008_808168a3292399e3b0a3.png 2023-07-11 19:40:59,833 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1008_2e83a2626ff883d16244.png 2023-07-11 19:40:59,834 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1008_386a62fe7e44ee995bbb.png 2023-07-11 19:40:59,834 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 19:41:00,170 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/qp9d0i5i-media/images/validation_1008_8bde6a294d23990b588f.png 2023-07-11 19:41:00,183 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/v778f6yy-media/images/validation_1008_808168a3292399e3b0a3.png 2023-07-11 19:41:00,184 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/z3dovmcp-media/images/validation_1008_2e83a2626ff883d16244.png 2023-07-11 19:41:00,186 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/wru17iw9-media/images/validation_1008_386a62fe7e44ee995bbb.png 2023-07-11 19:41:01,826 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:41:02,351 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:41:05,812 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:41:05,814 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:41:05,814 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:41:05,887 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:41:05,912 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:41:05,912 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:41:07,368 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:41:09,105 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:41:09,107 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:41:09,108 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:41:09,110 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:41:10,807 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:41:11,923 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:41:11,924 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:41:12,774 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:41:12,776 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:41:12,776 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:41:12,777 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:41:12,779 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:41:13,407 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:41:13,407 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:41:13,408 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:41:16,579 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:41:16,581 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:41:16,581 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:41:16,583 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:41:17,936 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:41:17,936 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:41:18,044 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:41:19,430 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:41:19,431 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:41:19,432 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:41:19,433 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:41:22,852 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:41:22,854 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:41:22,854 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:41:22,856 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:41:23,133 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:41:23,135 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:41:23,136 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:41:23,136 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:41:23,138 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:41:23,948 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:41:23,948 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:41:28,407 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:41:28,408 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:41:28,408 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:41:28,457 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:41:29,959 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:41:33,458 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:41:35,772 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:41:35,773 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1016_49858d5caa724841d560.png with policy now 2023-07-11 19:41:35,801 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:41:35,801 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1016_b2e26ce825c90767327d.png with policy now 2023-07-11 19:41:35,813 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:41:35,813 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1016_30d94ad44f6b1b049026.png with policy now 2023-07-11 19:41:35,822 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:41:35,822 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1016_219d5bff72b5592df3b9.png with policy now 2023-07-11 19:41:35,862 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:41:35,863 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:41:35,864 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:41:35,865 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:41:35,972 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:41:35,973 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1016_49858d5caa724841d560.png 2023-07-11 19:41:35,973 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1016_b2e26ce825c90767327d.png 2023-07-11 19:41:35,974 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1016_219d5bff72b5592df3b9.png 2023-07-11 19:41:35,975 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1016_30d94ad44f6b1b049026.png 2023-07-11 19:41:35,975 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 19:41:36,115 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ycrt7iv7-media/images/validation_1016_49858d5caa724841d560.png 2023-07-11 19:41:36,158 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/vzsrh1sh-media/images/validation_1016_219d5bff72b5592df3b9.png 2023-07-11 19:41:36,164 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/y9iw03xc-media/images/validation_1016_30d94ad44f6b1b049026.png 2023-07-11 19:41:36,356 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/len9nsgm-media/images/validation_1016_b2e26ce825c90767327d.png 2023-07-11 19:41:36,514 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ntfogzes-media/images/validation_1016_49858d5caa724841d560.png 2023-07-11 19:41:36,556 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ldxrhcge-media/images/validation_1016_219d5bff72b5592df3b9.png 2023-07-11 19:41:36,935 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/gskhp25k-media/images/validation_1016_b2e26ce825c90767327d.png 2023-07-11 19:41:37,002 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/wv1cfpmc-media/images/validation_1016_30d94ad44f6b1b049026.png 2023-07-11 19:41:38,468 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:41:40,436 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:41:40,810 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:41:41,985 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:41:43,410 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:41:43,410 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:41:43,411 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:41:43,538 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:41:43,540 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:41:43,540 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:41:43,541 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:41:43,542 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:41:46,764 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:41:46,766 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:41:46,766 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:41:46,768 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:41:47,997 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:41:47,997 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:41:49,253 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:41:50,078 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:41:50,080 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:41:50,080 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:41:50,082 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:41:53,067 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:41:53,069 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:41:53,069 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:41:53,071 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:41:54,010 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:41:54,010 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:41:54,613 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:41:56,427 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:41:56,429 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:41:56,430 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:41:56,431 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:41:58,410 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:41:58,410 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:41:58,411 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:42:00,023 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:42:00,023 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:42:00,083 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:42:00,161 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:42:00,162 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:42:00,162 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:42:00,164 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:42:00,442 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:42:00,444 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:42:00,444 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:42:00,446 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:42:05,101 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:42:06,354 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:42:06,354 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:42:10,103 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:42:10,812 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:42:13,094 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:42:13,094 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1024_06f67fa746d45752b1c0.png with policy now 2023-07-11 19:42:13,105 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:42:13,105 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1024_c8826549b51bc824f1a7.png with policy now 2023-07-11 19:42:13,114 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:42:13,114 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1024_de6a26e8e064b3d4c07b.png with policy now 2023-07-11 19:42:13,122 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:42:13,123 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1024_78be3e36643b4f4e5aeb.png with policy now 2023-07-11 19:42:13,162 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:42:13,164 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:42:13,164 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:42:13,166 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:42:13,410 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:42:13,410 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:42:13,411 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:42:13,434 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/j2ag8tzg-media/images/validation_1024_06f67fa746d45752b1c0.png 2023-07-11 19:42:13,450 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/k3g9w5ka-media/images/validation_1024_de6a26e8e064b3d4c07b.png 2023-07-11 19:42:13,463 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/xy3l0bqy-media/images/validation_1024_c8826549b51bc824f1a7.png 2023-07-11 19:42:13,466 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/nqpz3x72-media/images/validation_1024_78be3e36643b4f4e5aeb.png 2023-07-11 19:42:15,549 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:42:17,211 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:42:18,380 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:42:18,381 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:42:18,381 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1024_78be3e36643b4f4e5aeb.png 2023-07-11 19:42:18,381 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1024_c8826549b51bc824f1a7.png 2023-07-11 19:42:18,382 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1024_06f67fa746d45752b1c0.png 2023-07-11 19:42:18,383 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1024_de6a26e8e064b3d4c07b.png 2023-07-11 19:42:18,383 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 19:42:18,736 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/p1no7bve-media/images/validation_1024_c8826549b51bc824f1a7.png 2023-07-11 19:42:18,751 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/lvz0ebvd-media/images/validation_1024_de6a26e8e064b3d4c07b.png 2023-07-11 19:42:18,753 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ur9q7j3k-media/images/validation_1024_06f67fa746d45752b1c0.png 2023-07-11 19:42:18,770 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/7v7cnlk0-media/images/validation_1024_78be3e36643b4f4e5aeb.png 2023-07-11 19:42:20,278 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:42:20,280 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:42:20,281 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:42:20,282 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:42:21,283 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:42:23,509 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:42:23,511 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:42:23,511 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:42:23,513 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:42:24,392 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:42:24,392 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:42:26,866 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:42:27,588 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:42:27,590 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:42:27,590 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:42:27,592 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:42:28,515 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:42:28,516 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:42:28,516 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:42:30,404 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:42:30,404 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:42:30,889 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:42:30,891 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:42:30,892 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:42:30,893 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:42:32,565 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:42:34,984 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:42:34,986 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:42:34,986 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:42:34,988 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:42:36,416 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:42:36,416 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:42:37,782 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:42:37,783 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:42:37,783 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:42:37,784 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:42:37,786 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:42:38,062 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:42:38,064 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:42:38,064 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:42:38,066 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:42:40,815 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:42:42,428 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:42:42,428 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:42:42,817 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:42:43,515 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:42:43,516 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:42:43,516 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:42:48,560 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:42:50,781 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:42:50,782 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1032_4440c94c8c5b21619372.png with policy now 2023-07-11 19:42:50,796 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:42:50,796 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1032_e0b5d6b0134b15f1acc9.png with policy now 2023-07-11 19:42:50,806 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:42:50,806 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1032_fa3287622e2f185b3f97.png with policy now 2023-07-11 19:42:50,815 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:42:50,816 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1032_f343983400b249dec4cb.png with policy now 2023-07-11 19:42:50,850 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:42:50,851 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:42:50,851 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:42:50,853 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:42:51,102 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/0xx9k5px-media/images/validation_1032_4440c94c8c5b21619372.png 2023-07-11 19:42:51,141 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ohq5r6qq-media/images/validation_1032_f343983400b249dec4cb.png 2023-07-11 19:42:51,142 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/elp1agrh-media/images/validation_1032_fa3287622e2f185b3f97.png 2023-07-11 19:42:51,149 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/8ze30if3-media/images/validation_1032_e0b5d6b0134b15f1acc9.png 2023-07-11 19:42:53,786 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:42:53,788 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:42:54,453 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:42:54,453 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:42:54,453 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1032_f343983400b249dec4cb.png 2023-07-11 19:42:54,454 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1032_e0b5d6b0134b15f1acc9.png 2023-07-11 19:42:54,454 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1032_4440c94c8c5b21619372.png 2023-07-11 19:42:54,455 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1032_fa3287622e2f185b3f97.png 2023-07-11 19:42:54,455 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 19:42:54,785 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/uhbk4zyn-media/images/validation_1032_f343983400b249dec4cb.png 2023-07-11 19:42:54,804 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/flgy555x-media/images/validation_1032_e0b5d6b0134b15f1acc9.png 2023-07-11 19:42:54,817 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/t6bm37cb-media/images/validation_1032_4440c94c8c5b21619372.png 2023-07-11 19:42:54,833 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/9pno9uxp-media/images/validation_1032_fa3287622e2f185b3f97.png 2023-07-11 19:42:57,266 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:42:57,268 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:42:57,268 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:42:57,269 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:42:58,806 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:42:58,806 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:42:58,807 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:42:58,850 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:43:00,466 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:43:00,466 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:43:00,712 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:43:00,713 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:43:00,714 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:43:00,715 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:43:04,189 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:43:04,191 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:43:04,192 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:43:04,192 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:43:04,194 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:43:06,479 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:43:06,479 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:43:07,278 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:43:07,279 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:43:07,280 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:43:07,281 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:43:09,494 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:43:10,652 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:43:10,653 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:43:10,653 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:43:10,655 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:43:10,818 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:43:12,491 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:43:12,492 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:43:13,806 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:43:13,806 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:43:13,807 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:43:13,924 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:43:13,926 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:43:13,926 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:43:13,928 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:43:14,206 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:43:14,207 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:43:14,208 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:43:14,209 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:43:14,596 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:43:18,504 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:43:18,504 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:43:19,733 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:43:24,734 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:43:26,729 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:43:26,729 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1040_a3486a6a586a730d4368.png with policy now 2023-07-11 19:43:26,744 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:43:26,744 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1040_0a890667226ffaf5b704.png with policy now 2023-07-11 19:43:26,754 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:43:26,754 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1040_6fd15f77048c66e45ab2.png with policy now 2023-07-11 19:43:26,763 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:43:26,763 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1040_d91b47aa5350d4e8bc34.png with policy now 2023-07-11 19:43:26,799 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:43:26,800 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:43:26,801 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:43:26,802 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:43:27,058 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/t4r4yx2y-media/images/validation_1040_a3486a6a586a730d4368.png 2023-07-11 19:43:27,074 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/yhshjs1z-media/images/validation_1040_0a890667226ffaf5b704.png 2023-07-11 19:43:27,077 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/lsor2oiv-media/images/validation_1040_d91b47aa5350d4e8bc34.png 2023-07-11 19:43:27,079 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/bvip9cd6-media/images/validation_1040_6fd15f77048c66e45ab2.png 2023-07-11 19:43:28,806 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:43:28,807 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:43:28,807 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:43:29,820 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:43:29,821 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:43:30,529 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:43:30,529 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:43:30,530 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1040_0a890667226ffaf5b704.png 2023-07-11 19:43:30,530 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1040_a3486a6a586a730d4368.png 2023-07-11 19:43:30,531 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1040_d91b47aa5350d4e8bc34.png 2023-07-11 19:43:30,531 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1040_6fd15f77048c66e45ab2.png 2023-07-11 19:43:30,531 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 19:43:30,970 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/18hj5och-media/images/validation_1040_6fd15f77048c66e45ab2.png 2023-07-11 19:43:30,979 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/6h6m7ug9-media/images/validation_1040_a3486a6a586a730d4368.png 2023-07-11 19:43:30,983 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ky4uheyd-media/images/validation_1040_0a890667226ffaf5b704.png 2023-07-11 19:43:30,998 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/xrx58eoa-media/images/validation_1040_d91b47aa5350d4e8bc34.png 2023-07-11 19:43:33,643 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:43:33,645 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:43:33,645 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:43:33,647 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:43:35,328 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:43:36,542 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:43:36,542 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:43:36,737 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:43:36,739 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:43:36,739 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:43:36,741 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:43:39,716 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:43:39,718 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:43:39,718 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:43:39,720 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:43:40,348 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:43:40,822 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:43:42,557 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:43:42,558 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:43:42,849 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:43:42,851 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:43:42,851 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:43:42,853 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:43:43,806 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:43:43,807 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:43:43,807 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:43:45,483 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:43:46,826 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:43:46,827 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:43:46,828 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:43:46,829 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:43:48,570 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:43:48,571 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:43:50,332 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:43:50,334 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:43:50,334 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:43:50,336 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:43:50,613 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:43:50,614 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:43:50,615 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:43:50,615 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:43:50,617 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:43:54,583 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:43:54,584 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:43:56,154 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:43:58,816 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:43:58,816 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:43:58,817 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:44:01,855 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:44:03,181 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:44:03,181 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1048_f14ff48f5c6ee9389577.png with policy now 2023-07-11 19:44:03,225 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:44:03,225 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1048_04bf65c9392f974c75b8.png with policy now 2023-07-11 19:44:03,226 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:44:03,226 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1048_bd23a5c1898f8da42d93.png with policy now 2023-07-11 19:44:03,227 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:44:03,227 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1048_8c9f97026c019506833a.png with policy now 2023-07-11 19:44:03,254 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:44:03,256 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:44:03,256 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:44:03,258 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:44:03,488 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/wni0gcq1-media/images/validation_1048_f14ff48f5c6ee9389577.png 2023-07-11 19:44:03,538 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/z1pwnmth-media/images/validation_1048_8c9f97026c019506833a.png 2023-07-11 19:44:03,559 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/6qpfmeny-media/images/validation_1048_bd23a5c1898f8da42d93.png 2023-07-11 19:44:03,567 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/fv1tw7ic-media/images/validation_1048_04bf65c9392f974c75b8.png 2023-07-11 19:44:06,609 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:44:06,609 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:44:06,609 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1048_04bf65c9392f974c75b8.png 2023-07-11 19:44:06,610 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1048_bd23a5c1898f8da42d93.png 2023-07-11 19:44:06,611 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1048_8c9f97026c019506833a.png 2023-07-11 19:44:06,611 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1048_f14ff48f5c6ee9389577.png 2023-07-11 19:44:06,611 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 19:44:06,811 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:44:06,966 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ecis7oya-media/images/validation_1048_bd23a5c1898f8da42d93.png 2023-07-11 19:44:06,966 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/tdse6uqv-media/images/validation_1048_04bf65c9392f974c75b8.png 2023-07-11 19:44:07,018 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ghldjuuy-media/images/validation_1048_f14ff48f5c6ee9389577.png 2023-07-11 19:44:07,020 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/7rb0cil3-media/images/validation_1048_8c9f97026c019506833a.png 2023-07-11 19:44:07,813 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:44:10,364 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:44:10,366 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:44:10,366 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:44:10,368 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:44:10,824 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:44:12,621 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:44:12,622 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:44:13,067 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:44:13,069 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:44:13,069 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:44:13,070 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:44:13,071 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:44:13,816 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:44:13,816 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:44:13,817 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:44:15,879 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:44:15,881 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:44:15,881 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:44:15,883 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:44:18,634 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:44:18,635 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:44:18,923 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:44:19,335 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:44:19,337 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:44:19,338 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:44:19,339 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:44:22,157 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:44:22,159 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:44:22,159 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:44:22,160 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:44:24,647 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:44:24,647 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:44:24,872 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:44:25,983 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:44:25,984 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:44:25,985 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:44:25,986 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:44:26,265 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:44:26,267 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:44:26,267 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:44:26,269 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:44:28,816 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:44:28,816 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:44:28,817 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:44:30,660 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:44:30,660 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:44:30,868 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:44:35,869 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:44:38,871 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:44:38,871 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1056_30c717b2b7e4a916ae89.png with policy now 2023-07-11 19:44:38,886 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:44:38,886 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1056_1479ab3105077bd77424.png with policy now 2023-07-11 19:44:38,896 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:44:38,896 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1056_d16d0398bcc7dbf04bbf.png with policy now 2023-07-11 19:44:38,904 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:44:38,905 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1056_0ac8c42ee2433dffe939.png with policy now 2023-07-11 19:44:38,945 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:44:38,947 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:44:38,948 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:44:38,949 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:44:39,190 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/r31w5jnl-media/images/validation_1056_30c717b2b7e4a916ae89.png 2023-07-11 19:44:39,201 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/zg1pncn1-media/images/validation_1056_d16d0398bcc7dbf04bbf.png 2023-07-11 19:44:39,214 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/26o84ong-media/images/validation_1056_1479ab3105077bd77424.png 2023-07-11 19:44:39,215 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/x8kffm2j-media/images/validation_1056_0ac8c42ee2433dffe939.png 2023-07-11 19:44:40,826 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:44:41,404 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:44:42,208 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:44:42,685 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:44:42,685 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:44:42,686 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1056_0ac8c42ee2433dffe939.png 2023-07-11 19:44:42,686 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1056_d16d0398bcc7dbf04bbf.png 2023-07-11 19:44:42,687 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1056_30c717b2b7e4a916ae89.png 2023-07-11 19:44:42,687 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1056_1479ab3105077bd77424.png 2023-07-11 19:44:42,688 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 19:44:43,095 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/8xjtx0m6-media/images/validation_1056_d16d0398bcc7dbf04bbf.png 2023-07-11 19:44:43,096 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/jbqcoqkl-media/images/validation_1056_0ac8c42ee2433dffe939.png 2023-07-11 19:44:43,106 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/zkw8mamo-media/images/validation_1056_30c717b2b7e4a916ae89.png 2023-07-11 19:44:43,115 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/um51u856-media/images/validation_1056_1479ab3105077bd77424.png 2023-07-11 19:44:43,817 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:44:43,817 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:44:43,818 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:44:45,557 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:44:45,559 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:44:45,560 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:44:45,561 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:44:47,241 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:44:48,698 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:44:48,698 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:44:49,398 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:44:49,400 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:44:49,400 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:44:49,402 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:44:52,588 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:44:52,590 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:44:52,590 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:44:52,590 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:44:52,592 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:44:54,711 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:44:54,711 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:44:56,038 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:44:56,040 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:44:56,040 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:44:56,042 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:44:57,935 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:44:58,817 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:44:58,817 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:44:58,817 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:44:59,461 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:44:59,463 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:44:59,463 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:44:59,465 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:45:00,723 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:45:00,724 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:45:02,564 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:45:02,566 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:45:02,566 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:45:02,568 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:45:02,846 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:45:02,848 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:45:02,848 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:45:02,850 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:45:03,276 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:45:06,736 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:45:06,736 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:45:08,415 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:45:10,829 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:45:13,817 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:45:13,817 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:45:13,818 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:45:13,874 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:45:15,412 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:45:15,412 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1064_ae3016b396ea28947987.png with policy now 2023-07-11 19:45:15,427 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:45:15,427 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1064_b69fc5a0e25eb7503924.png with policy now 2023-07-11 19:45:15,430 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:45:15,430 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1064_e704101016c7c230c717.png with policy now 2023-07-11 19:45:15,438 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:45:15,438 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1064_1cc5791fea4c65f0fde3.png with policy now 2023-07-11 19:45:15,478 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:45:15,479 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:45:15,480 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:45:15,481 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:45:15,739 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/u9kk5rex-media/images/validation_1064_ae3016b396ea28947987.png 2023-07-11 19:45:15,767 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/rc6buxrg-media/images/validation_1064_1cc5791fea4c65f0fde3.png 2023-07-11 19:45:15,767 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/qzwnkklf-media/images/validation_1064_b69fc5a0e25eb7503924.png 2023-07-11 19:45:15,770 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/7zkle0bz-media/images/validation_1064_e704101016c7c230c717.png 2023-07-11 19:45:18,761 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:45:18,762 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1064_e704101016c7c230c717.png 2023-07-11 19:45:18,762 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1064_b69fc5a0e25eb7503924.png 2023-07-11 19:45:18,762 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1064_ae3016b396ea28947987.png 2023-07-11 19:45:18,763 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1064_1cc5791fea4c65f0fde3.png 2023-07-11 19:45:18,763 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 19:45:19,005 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:45:19,096 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/h4zddws2-media/images/validation_1064_e704101016c7c230c717.png 2023-07-11 19:45:19,113 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/s0ougy1r-media/images/validation_1064_b69fc5a0e25eb7503924.png 2023-07-11 19:45:19,140 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/m92e5j2q-media/images/validation_1064_ae3016b396ea28947987.png 2023-07-11 19:45:19,155 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/l1x0ewh7-media/images/validation_1064_1cc5791fea4c65f0fde3.png 2023-07-11 19:45:19,593 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:45:22,594 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:45:22,596 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:45:22,597 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:45:22,598 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:45:24,542 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:45:24,774 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:45:24,775 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:45:25,778 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:45:25,780 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:45:25,781 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:45:25,782 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:45:28,835 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:45:28,836 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:45:28,837 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:45:29,707 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:45:29,709 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:45:29,709 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:45:29,710 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:45:29,711 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:45:30,787 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:45:30,787 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:45:33,029 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:45:33,031 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:45:33,032 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:45:33,033 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:45:35,144 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:45:36,022 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:45:36,024 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:45:36,024 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:45:36,025 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:45:36,799 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:45:36,800 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:45:40,260 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:45:40,261 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:45:40,261 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:45:40,262 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:45:40,264 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:45:40,541 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:45:40,543 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:45:40,543 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:45:40,545 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:45:40,834 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:45:42,812 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:45:42,812 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:45:43,835 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:45:43,836 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:45:43,836 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:45:45,876 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:45:50,877 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:45:53,171 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:45:53,171 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1072_fed73b1b21a38bd14ea8.png with policy now 2023-07-11 19:45:53,182 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:45:53,182 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1072_4118ab19403ed7a6f06d.png with policy now 2023-07-11 19:45:53,191 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:45:53,191 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1072_3bd02124d49d0b8ff154.png with policy now 2023-07-11 19:45:53,199 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:45:53,199 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1072_41ce97897430def0ca20.png with policy now 2023-07-11 19:45:53,242 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:45:53,244 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:45:53,244 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:45:53,249 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:45:53,488 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/jpnj7h4i-media/images/validation_1072_fed73b1b21a38bd14ea8.png 2023-07-11 19:45:53,498 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/57c0bu5z-media/images/validation_1072_41ce97897430def0ca20.png 2023-07-11 19:45:53,502 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/e8poddad-media/images/validation_1072_4118ab19403ed7a6f06d.png 2023-07-11 19:45:53,509 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/g4bilbi2-media/images/validation_1072_3bd02124d49d0b8ff154.png 2023-07-11 19:45:54,837 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:45:54,837 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1072_fed73b1b21a38bd14ea8.png 2023-07-11 19:45:54,838 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1072_41ce97897430def0ca20.png 2023-07-11 19:45:54,838 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1072_4118ab19403ed7a6f06d.png 2023-07-11 19:45:54,839 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1072_3bd02124d49d0b8ff154.png 2023-07-11 19:45:54,839 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 19:45:55,185 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/wkz76n5m-media/images/validation_1072_41ce97897430def0ca20.png 2023-07-11 19:45:55,193 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/wbd85wle-media/images/validation_1072_fed73b1b21a38bd14ea8.png 2023-07-11 19:45:55,211 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/j95iskvq-media/images/validation_1072_4118ab19403ed7a6f06d.png 2023-07-11 19:45:55,215 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/vxmuq83p-media/images/validation_1072_3bd02124d49d0b8ff154.png 2023-07-11 19:45:55,921 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:45:57,968 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:45:58,836 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:45:58,836 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:45:58,837 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:46:00,850 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:46:01,126 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:46:02,240 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:46:02,242 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:46:02,243 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:46:02,244 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:46:05,613 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:46:05,615 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:46:05,615 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:46:05,617 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:46:06,457 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:46:06,864 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:46:06,865 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:46:08,591 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:46:08,593 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:46:08,593 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:46:08,595 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:46:10,835 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:46:11,538 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:46:11,539 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:46:11,540 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:46:11,540 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:46:11,542 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:46:12,878 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:46:12,879 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:46:13,836 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:46:13,836 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:46:13,837 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:46:14,904 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:46:14,906 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:46:14,906 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:46:14,908 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:46:17,282 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:46:18,210 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:46:18,212 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:46:18,213 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:46:18,390 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:46:18,493 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:46:18,495 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:46:18,495 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:46:18,497 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:46:18,892 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:46:18,892 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:46:22,417 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:46:24,903 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:46:27,419 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:46:28,856 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:46:28,857 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:46:28,857 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:46:31,407 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:46:31,407 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1080_c32133924b5d95fd238b.png with policy now 2023-07-11 19:46:31,449 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:46:31,449 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1080_5c4fecd9dcdd1e11f95a.png with policy now 2023-07-11 19:46:31,450 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:46:31,450 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1080_1d776c3f3b5c17211029.png with policy now 2023-07-11 19:46:31,451 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:46:31,451 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1080_9db4cb0a67fe9ae5cb6b.png with policy now 2023-07-11 19:46:31,477 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:46:31,478 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:46:31,478 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:46:31,479 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:46:31,758 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/5je2j6l5-media/images/validation_1080_c32133924b5d95fd238b.png 2023-07-11 19:46:31,787 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/1pltjjdr-media/images/validation_1080_1d776c3f3b5c17211029.png 2023-07-11 19:46:31,787 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/y85tv42p-media/images/validation_1080_9db4cb0a67fe9ae5cb6b.png 2023-07-11 19:46:31,796 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/qg181xq5-media/images/validation_1080_5c4fecd9dcdd1e11f95a.png 2023-07-11 19:46:32,480 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:46:35,454 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:46:36,929 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:46:36,929 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:46:36,930 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1080_1d776c3f3b5c17211029.png 2023-07-11 19:46:36,930 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1080_9db4cb0a67fe9ae5cb6b.png 2023-07-11 19:46:36,931 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1080_5c4fecd9dcdd1e11f95a.png 2023-07-11 19:46:36,931 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1080_c32133924b5d95fd238b.png 2023-07-11 19:46:36,932 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 19:46:37,295 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/znw9f90w-media/images/validation_1080_9db4cb0a67fe9ae5cb6b.png 2023-07-11 19:46:37,301 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/7xbzvcbb-media/images/validation_1080_c32133924b5d95fd238b.png 2023-07-11 19:46:37,315 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/6e8ehik0-media/images/validation_1080_1d776c3f3b5c17211029.png 2023-07-11 19:46:37,319 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/latncnbu-media/images/validation_1080_5c4fecd9dcdd1e11f95a.png 2023-07-11 19:46:37,977 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:46:38,925 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:46:38,927 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:46:38,928 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:46:38,929 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:46:40,838 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:46:41,648 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:46:41,650 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:46:41,650 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:46:41,652 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:46:42,942 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:46:42,942 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:46:43,605 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:46:43,857 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:46:43,857 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:46:43,858 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:46:45,574 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:46:45,576 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:46:45,577 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:46:45,578 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:46:48,489 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:46:48,491 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:46:48,492 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:46:48,493 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:46:48,955 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:46:48,955 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:46:49,494 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:46:51,524 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:46:51,525 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:46:51,526 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:46:51,527 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:46:54,501 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:46:54,967 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:46:54,968 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:46:55,180 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:46:55,182 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:46:55,182 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:46:55,184 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:46:55,461 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:46:55,463 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:46:55,463 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:46:55,465 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:46:58,857 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:46:58,857 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:46:58,858 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:46:59,894 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:47:00,980 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:47:00,981 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:47:04,895 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:47:08,020 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:47:08,021 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1088_68176a9b6a649b7a461d.png with policy now 2023-07-11 19:47:08,061 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:47:08,061 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1088_cba58a2d69dc3368c094.png with policy now 2023-07-11 19:47:08,062 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:47:08,062 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1088_1676e030f39d2264623e.png with policy now 2023-07-11 19:47:08,063 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:47:08,063 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1088_fd83b25b70ba4257c9b1.png with policy now 2023-07-11 19:47:08,091 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:47:08,092 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:47:08,092 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:47:08,094 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:47:08,345 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/lob22ph7-media/images/validation_1088_68176a9b6a649b7a461d.png 2023-07-11 19:47:08,387 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ajwnn8ff-media/images/validation_1088_fd83b25b70ba4257c9b1.png 2023-07-11 19:47:08,389 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/e7i1oend-media/images/validation_1088_cba58a2d69dc3368c094.png 2023-07-11 19:47:08,390 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/gffezsps-media/images/validation_1088_1676e030f39d2264623e.png 2023-07-11 19:47:10,519 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:47:10,841 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:47:11,618 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:47:13,008 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:47:13,009 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:47:13,009 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1088_fd83b25b70ba4257c9b1.png 2023-07-11 19:47:13,009 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1088_cba58a2d69dc3368c094.png 2023-07-11 19:47:13,010 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1088_1676e030f39d2264623e.png 2023-07-11 19:47:13,011 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1088_68176a9b6a649b7a461d.png 2023-07-11 19:47:13,011 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 19:47:13,374 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/1j3xgll3-media/images/validation_1088_fd83b25b70ba4257c9b1.png 2023-07-11 19:47:13,378 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/zvumsx71-media/images/validation_1088_cba58a2d69dc3368c094.png 2023-07-11 19:47:13,394 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/okd0ofhc-media/images/validation_1088_1676e030f39d2264623e.png 2023-07-11 19:47:13,423 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/loapv3by-media/images/validation_1088_68176a9b6a649b7a461d.png 2023-07-11 19:47:13,857 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:47:13,857 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:47:13,858 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:47:15,899 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:47:15,901 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:47:15,901 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:47:15,902 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:47:15,903 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:47:18,647 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:47:18,648 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:47:18,649 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:47:18,650 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:47:19,021 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:47:19,021 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:47:21,581 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:47:22,182 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:47:22,184 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:47:22,184 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:47:22,186 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:47:25,034 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:47:25,034 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:47:25,779 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:47:25,781 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:47:25,782 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:47:25,783 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:47:27,570 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:47:29,172 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:47:29,172 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:47:29,173 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:47:29,299 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:47:29,301 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:47:29,302 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:47:29,303 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:47:31,047 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:47:31,047 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:47:31,933 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:47:31,935 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:47:31,935 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:47:31,937 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:47:32,214 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:47:32,216 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:47:32,217 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:47:32,218 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:47:32,656 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:47:37,060 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:47:37,060 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:47:37,794 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:47:40,844 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:47:42,846 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:47:44,194 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:47:44,195 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:47:44,195 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:47:44,819 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:47:44,819 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1096_bd01b42ca6177ea47c5a.png with policy now 2023-07-11 19:47:44,833 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:47:44,834 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1096_333e99e73a03c30fe42d.png with policy now 2023-07-11 19:47:44,844 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:47:44,844 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1096_befbd5340fa0acbaf37e.png with policy now 2023-07-11 19:47:44,852 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:47:44,853 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1096_c88fd5a40a05f4bac5c8.png with policy now 2023-07-11 19:47:44,889 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:47:44,891 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:47:44,895 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:47:44,897 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:47:45,148 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/lw9au4t0-media/images/validation_1096_bd01b42ca6177ea47c5a.png 2023-07-11 19:47:45,158 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/yz40x1b1-media/images/validation_1096_befbd5340fa0acbaf37e.png 2023-07-11 19:47:45,160 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/pe37fout-media/images/validation_1096_333e99e73a03c30fe42d.png 2023-07-11 19:47:45,184 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/tuvwy2cg-media/images/validation_1096_c88fd5a40a05f4bac5c8.png 2023-07-11 19:47:48,448 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:47:48,517 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:47:49,085 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:47:49,086 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:47:49,086 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1096_333e99e73a03c30fe42d.png 2023-07-11 19:47:49,087 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1096_befbd5340fa0acbaf37e.png 2023-07-11 19:47:49,087 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1096_c88fd5a40a05f4bac5c8.png 2023-07-11 19:47:49,088 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1096_bd01b42ca6177ea47c5a.png 2023-07-11 19:47:49,088 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 19:47:49,461 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/8bepcxmw-media/images/validation_1096_333e99e73a03c30fe42d.png 2023-07-11 19:47:49,467 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/xr8sp6tc-media/images/validation_1096_bd01b42ca6177ea47c5a.png 2023-07-11 19:47:49,479 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/stmtpts9-media/images/validation_1096_befbd5340fa0acbaf37e.png 2023-07-11 19:47:49,487 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/pzi330fh-media/images/validation_1096_c88fd5a40a05f4bac5c8.png 2023-07-11 19:47:51,547 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:47:51,549 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:47:51,549 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:47:51,551 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:47:53,827 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:47:54,849 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:47:54,852 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:47:54,852 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:47:54,854 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:47:55,099 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:47:55,099 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:47:58,954 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:47:58,956 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:47:58,956 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:47:58,956 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:47:58,958 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:47:59,194 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:47:59,195 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:47:59,195 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:48:01,112 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:48:01,112 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:48:01,338 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:48:01,340 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:48:01,341 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:48:01,342 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:48:04,223 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:48:04,225 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:48:04,226 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:48:04,226 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:48:04,228 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:48:07,124 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:48:07,125 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:48:08,428 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:48:08,430 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:48:08,431 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:48:08,432 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:48:08,710 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:48:08,712 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:48:08,712 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:48:08,714 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:48:09,336 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:48:10,847 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:48:13,136 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:48:13,137 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:48:14,194 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:48:14,195 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:48:14,195 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:48:15,248 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:48:20,249 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:48:21,324 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:48:21,325 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1104_bee308209e779d45bd8b.png with policy now 2023-07-11 19:48:21,339 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:48:21,339 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1104_1f83a28d25fff40729e7.png with policy now 2023-07-11 19:48:21,349 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:48:21,349 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1104_d8dc6a37b8589618b062.png with policy now 2023-07-11 19:48:21,358 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:48:21,359 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1104_637101446275b24ba9b1.png with policy now 2023-07-11 19:48:21,395 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:48:21,396 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:48:21,397 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:48:21,402 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:48:21,669 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/h38gb2hz-media/images/validation_1104_1f83a28d25fff40729e7.png 2023-07-11 19:48:21,676 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/tu4tf57m-media/images/validation_1104_bee308209e779d45bd8b.png 2023-07-11 19:48:21,676 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/bclxj56p-media/images/validation_1104_d8dc6a37b8589618b062.png 2023-07-11 19:48:21,710 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ubx066qu-media/images/validation_1104_637101446275b24ba9b1.png 2023-07-11 19:48:25,164 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:48:25,165 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:48:25,165 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1104_637101446275b24ba9b1.png 2023-07-11 19:48:25,165 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1104_d8dc6a37b8589618b062.png 2023-07-11 19:48:25,166 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1104_bee308209e779d45bd8b.png 2023-07-11 19:48:25,167 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1104_1f83a28d25fff40729e7.png 2023-07-11 19:48:25,167 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 19:48:25,502 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/stndrwu0-media/images/validation_1104_637101446275b24ba9b1.png 2023-07-11 19:48:25,507 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:48:25,508 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:48:25,523 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/injvsbe2-media/images/validation_1104_d8dc6a37b8589618b062.png 2023-07-11 19:48:25,568 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/aaics539-media/images/validation_1104_1f83a28d25fff40729e7.png 2023-07-11 19:48:25,580 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/w4sra1gu-media/images/validation_1104_bee308209e779d45bd8b.png 2023-07-11 19:48:28,647 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:48:28,649 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:48:28,649 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:48:28,651 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:48:29,194 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:48:29,195 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:48:29,196 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:48:31,073 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:48:31,177 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:48:31,178 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:48:32,177 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:48:32,179 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:48:32,180 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:48:32,181 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:48:35,536 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:48:35,538 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:48:35,539 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:48:35,540 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:48:36,769 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:48:37,191 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:48:37,191 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:48:38,286 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:48:38,287 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:48:38,288 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:48:38,289 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:48:40,850 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:48:41,251 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:48:41,252 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:48:41,253 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:48:41,254 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:48:42,192 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:48:43,204 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:48:43,204 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:48:44,505 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:48:44,505 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:48:44,506 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:48:44,631 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:48:44,632 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:48:44,633 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:48:44,634 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:48:44,914 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:48:44,915 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:48:44,915 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:48:44,917 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:48:47,527 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:48:49,216 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:48:49,217 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:48:52,529 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:48:57,530 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:48:57,570 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:48:57,570 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1112_9e12455b1c6a5c5317bb.png with policy now 2023-07-11 19:48:57,585 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:48:57,585 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1112_535e129af2802072858a.png with policy now 2023-07-11 19:48:57,592 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:48:57,592 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1112_e455625d1982cc47ad44.png with policy now 2023-07-11 19:48:57,604 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:48:57,604 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1112_fb29bfacf919ea5e7503.png with policy now 2023-07-11 19:48:57,641 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:48:57,643 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:48:57,644 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:48:57,649 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:48:57,888 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/geuouylu-media/images/validation_1112_9e12455b1c6a5c5317bb.png 2023-07-11 19:48:57,910 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/134gilh2-media/images/validation_1112_e455625d1982cc47ad44.png 2023-07-11 19:48:57,910 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/vsxzunk6-media/images/validation_1112_fb29bfacf919ea5e7503.png 2023-07-11 19:48:57,928 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/zlrgtvms-media/images/validation_1112_535e129af2802072858a.png 2023-07-11 19:48:59,531 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:48:59,531 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:48:59,531 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:49:01,245 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:49:01,245 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:49:01,246 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1112_e455625d1982cc47ad44.png 2023-07-11 19:49:01,246 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1112_fb29bfacf919ea5e7503.png 2023-07-11 19:49:01,247 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1112_9e12455b1c6a5c5317bb.png 2023-07-11 19:49:01,247 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1112_535e129af2802072858a.png 2023-07-11 19:49:01,248 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 19:49:01,584 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/8wi5qawz-media/images/validation_1112_e455625d1982cc47ad44.png 2023-07-11 19:49:01,603 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/t4opb6s9-media/images/validation_1112_fb29bfacf919ea5e7503.png 2023-07-11 19:49:01,613 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/nx0uq080-media/images/validation_1112_9e12455b1c6a5c5317bb.png 2023-07-11 19:49:01,627 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/qk9cjnxc-media/images/validation_1112_535e129af2802072858a.png 2023-07-11 19:49:01,689 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:49:03,270 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:49:04,976 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:49:04,978 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:49:04,978 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:49:04,980 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:49:07,258 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:49:07,259 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:49:07,603 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:49:07,605 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:49:07,605 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:49:07,607 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:49:08,461 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:49:10,529 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:49:10,530 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:49:10,531 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:49:10,532 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:49:10,853 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:49:13,271 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:49:13,272 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:49:14,202 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:49:14,204 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:49:14,205 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:49:14,205 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:49:14,207 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:49:14,530 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:49:14,531 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:49:18,233 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:49:18,235 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:49:18,236 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:49:18,262 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:49:19,284 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:49:19,284 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:49:19,766 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:49:21,256 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:49:21,258 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:49:21,259 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:49:21,260 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:49:21,537 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:49:21,539 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:49:21,540 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:49:21,541 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:49:25,240 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:49:25,296 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:49:25,297 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:49:29,552 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:49:29,552 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:49:29,553 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:49:30,611 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:49:34,323 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:49:34,323 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1120_d29b97b60ac55d690e1b.png with policy now 2023-07-11 19:49:34,335 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:49:34,335 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1120_09953f69ca2d54775598.png with policy now 2023-07-11 19:49:34,347 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:49:34,348 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1120_c92093d2b13b886281aa.png with policy now 2023-07-11 19:49:34,356 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:49:34,356 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1120_7383de8551e064f6ad43.png with policy now 2023-07-11 19:49:34,393 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:49:34,395 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:49:34,395 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:49:34,396 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:49:34,640 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/v3psgp1w-media/images/validation_1120_d29b97b60ac55d690e1b.png 2023-07-11 19:49:34,672 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/a9nli33x-media/images/validation_1120_09953f69ca2d54775598.png 2023-07-11 19:49:34,681 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/pmh949ks-media/images/validation_1120_c92093d2b13b886281aa.png 2023-07-11 19:49:34,696 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/jn3wyj83-media/images/validation_1120_7383de8551e064f6ad43.png 2023-07-11 19:49:35,630 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:49:37,325 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:49:37,326 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:49:37,326 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1120_c92093d2b13b886281aa.png 2023-07-11 19:49:37,327 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1120_d29b97b60ac55d690e1b.png 2023-07-11 19:49:37,327 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1120_09953f69ca2d54775598.png 2023-07-11 19:49:37,328 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1120_7383de8551e064f6ad43.png 2023-07-11 19:49:37,328 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 19:49:37,677 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/dmp7paop-media/images/validation_1120_c92093d2b13b886281aa.png 2023-07-11 19:49:37,691 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/aqkdw082-media/images/validation_1120_7383de8551e064f6ad43.png 2023-07-11 19:49:37,701 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/29ze9h17-media/images/validation_1120_d29b97b60ac55d690e1b.png 2023-07-11 19:49:37,713 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/y7dvspn4-media/images/validation_1120_09953f69ca2d54775598.png 2023-07-11 19:49:38,260 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:49:40,855 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:49:40,860 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:49:41,879 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:49:41,881 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:49:41,882 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:49:41,883 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:49:43,335 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:49:43,335 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:49:44,552 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:49:44,552 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:49:44,553 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:49:45,088 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:49:45,090 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:49:45,090 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:49:45,092 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:49:46,548 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:49:48,335 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:49:48,337 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:49:48,337 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:49:48,339 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:49:49,351 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:49:49,352 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:49:51,280 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:49:51,282 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:49:51,282 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:49:51,284 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:49:51,803 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:49:53,947 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:49:53,948 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:49:53,949 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:49:53,950 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:49:55,364 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:49:55,364 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:49:56,870 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:49:57,398 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:49:57,399 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:49:57,400 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:49:57,402 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:49:57,679 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:49:57,681 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:49:57,681 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:49:57,683 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:49:59,552 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:49:59,553 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:49:59,553 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:50:01,377 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:50:01,377 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:50:02,606 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:50:07,607 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:50:10,318 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:50:10,318 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1128_3302c80927e3dde2eb3d.png with policy now 2023-07-11 19:50:10,332 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:50:10,333 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1128_d8f656ea71301403f7bf.png with policy now 2023-07-11 19:50:10,342 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:50:10,342 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1128_9459ebef2c19962e4b34.png with policy now 2023-07-11 19:50:10,353 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:50:10,354 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1128_0f91e55a8a98dd2e8ef8.png with policy now 2023-07-11 19:50:10,391 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:50:10,394 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:50:10,395 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:50:10,397 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:50:10,636 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/4legx835-media/images/validation_1128_3302c80927e3dde2eb3d.png 2023-07-11 19:50:10,653 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/3p0ba0wr-media/images/validation_1128_d8f656ea71301403f7bf.png 2023-07-11 19:50:10,657 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/egdqgx86-media/images/validation_1128_9459ebef2c19962e4b34.png 2023-07-11 19:50:10,663 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/znhytslm-media/images/validation_1128_0f91e55a8a98dd2e8ef8.png 2023-07-11 19:50:10,858 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:50:13,163 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:50:13,405 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:50:13,405 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:50:13,406 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1128_0f91e55a8a98dd2e8ef8.png 2023-07-11 19:50:13,406 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1128_3302c80927e3dde2eb3d.png 2023-07-11 19:50:13,407 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1128_9459ebef2c19962e4b34.png 2023-07-11 19:50:13,407 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1128_d8f656ea71301403f7bf.png 2023-07-11 19:50:13,408 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 19:50:13,758 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/mj935env-media/images/validation_1128_3302c80927e3dde2eb3d.png 2023-07-11 19:50:13,758 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/02l6c6fs-media/images/validation_1128_0f91e55a8a98dd2e8ef8.png 2023-07-11 19:50:13,771 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/risqiync-media/images/validation_1128_9459ebef2c19962e4b34.png 2023-07-11 19:50:13,772 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/o8zz590h-media/images/validation_1128_d8f656ea71301403f7bf.png 2023-07-11 19:50:13,821 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:50:14,552 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:50:14,553 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:50:14,553 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:50:17,136 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:50:17,138 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:50:17,139 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:50:17,140 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:50:18,365 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:50:19,418 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:50:19,419 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:50:20,865 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:50:20,867 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:50:20,868 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:50:20,869 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:50:23,514 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:50:24,025 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:50:24,027 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:50:24,027 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:50:24,198 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:50:25,431 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:50:25,432 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:50:27,104 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:50:27,106 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:50:27,107 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:50:27,108 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:50:28,961 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:50:29,553 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:50:29,553 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:50:29,554 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:50:29,669 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:50:29,671 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:50:29,672 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:50:29,673 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:50:31,444 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:50:31,445 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:50:33,285 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:50:33,287 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:50:33,288 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:50:33,289 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:50:33,567 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:50:33,568 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:50:33,569 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:50:33,570 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:50:33,977 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:50:37,458 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:50:37,459 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:50:39,134 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:50:40,862 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:50:44,583 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:50:44,583 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:50:44,583 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:50:44,625 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:50:46,189 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:50:46,190 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1136_7271d56ca24970e2e44b.png with policy now 2023-07-11 19:50:46,204 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:50:46,204 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1136_6b3d6e01567109ccdd4f.png with policy now 2023-07-11 19:50:46,214 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:50:46,214 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1136_1b0f4331091bfdab39b6.png with policy now 2023-07-11 19:50:46,225 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:50:46,225 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1136_ed5070884043b0e9388b.png with policy now 2023-07-11 19:50:46,260 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:50:46,261 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:50:46,262 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:50:46,263 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:50:46,576 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/wqr7zl6q-media/images/validation_1136_7271d56ca24970e2e44b.png 2023-07-11 19:50:46,593 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/2nyynrzj-media/images/validation_1136_6b3d6e01567109ccdd4f.png 2023-07-11 19:50:46,601 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/q5vefw14-media/images/validation_1136_1b0f4331091bfdab39b6.png 2023-07-11 19:50:46,617 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/xemfmigl-media/images/validation_1136_ed5070884043b0e9388b.png 2023-07-11 19:50:49,481 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:50:49,481 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:50:49,481 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1136_1b0f4331091bfdab39b6.png 2023-07-11 19:50:49,482 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1136_7271d56ca24970e2e44b.png 2023-07-11 19:50:49,482 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1136_6b3d6e01567109ccdd4f.png 2023-07-11 19:50:49,482 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1136_ed5070884043b0e9388b.png 2023-07-11 19:50:49,483 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 19:50:49,859 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/p1cp8wd6-media/images/validation_1136_1b0f4331091bfdab39b6.png 2023-07-11 19:50:49,861 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/93m2tpz8-media/images/validation_1136_7271d56ca24970e2e44b.png 2023-07-11 19:50:49,893 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/kfnqrgzf-media/images/validation_1136_6b3d6e01567109ccdd4f.png 2023-07-11 19:50:49,910 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ipmvyqxv-media/images/validation_1136_ed5070884043b0e9388b.png 2023-07-11 19:50:50,208 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:50:50,209 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:50:53,573 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:50:53,575 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:50:53,575 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:50:53,577 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:50:55,494 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:50:55,495 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:50:55,655 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:50:57,373 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:50:57,375 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:50:57,375 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:50:57,377 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:51:00,094 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:51:00,094 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:51:00,095 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:51:00,699 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:51:00,700 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:51:00,701 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:51:00,701 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:51:00,703 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:51:01,508 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:51:01,508 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:51:03,763 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:51:03,765 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:51:03,765 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:51:03,767 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:51:05,833 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:51:06,452 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:51:06,454 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:51:06,454 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:51:06,455 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:51:07,522 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:51:07,522 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:51:09,514 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:51:09,516 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:51:09,516 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:51:09,518 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:51:09,795 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:51:09,797 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:51:09,798 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:51:09,799 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:51:10,864 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:51:10,869 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:51:13,535 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:51:13,536 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:51:15,096 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:51:15,096 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:51:15,096 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:51:16,141 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:51:21,142 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:51:22,411 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:51:22,411 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1144_7debe473f9ee916b15a7.png with policy now 2023-07-11 19:51:22,422 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:51:22,422 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1144_aeed065d00db7ef520a7.png with policy now 2023-07-11 19:51:22,434 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:51:22,434 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1144_11f4342cbfa59cc9ac0e.png with policy now 2023-07-11 19:51:22,444 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:51:22,444 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1144_0f579e23872ee306cab9.png with policy now 2023-07-11 19:51:22,478 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:51:22,480 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:51:22,480 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:51:22,482 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:51:22,699 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/sfwo8fuu-media/images/validation_1144_7debe473f9ee916b15a7.png 2023-07-11 19:51:22,729 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/dvkg0ogo-media/images/validation_1144_aeed065d00db7ef520a7.png 2023-07-11 19:51:22,737 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/rfxm5x7k-media/images/validation_1144_0f579e23872ee306cab9.png 2023-07-11 19:51:22,748 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/tzu0gjhn-media/images/validation_1144_11f4342cbfa59cc9ac0e.png 2023-07-11 19:51:25,561 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:51:25,562 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:51:25,562 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1144_aeed065d00db7ef520a7.png 2023-07-11 19:51:25,562 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1144_7debe473f9ee916b15a7.png 2023-07-11 19:51:25,563 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1144_11f4342cbfa59cc9ac0e.png 2023-07-11 19:51:25,563 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1144_0f579e23872ee306cab9.png 2023-07-11 19:51:25,564 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 19:51:25,958 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:51:25,964 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/76okfl9m-media/images/validation_1144_7debe473f9ee916b15a7.png 2023-07-11 19:51:25,966 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/tzkunjzz-media/images/validation_1144_aeed065d00db7ef520a7.png 2023-07-11 19:51:25,978 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/cm6o1twq-media/images/validation_1144_0f579e23872ee306cab9.png 2023-07-11 19:51:25,991 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/zqlh91yd-media/images/validation_1144_11f4342cbfa59cc9ac0e.png 2023-07-11 19:51:26,850 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:51:29,572 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:51:29,574 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:51:29,575 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:51:29,576 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:51:30,096 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:51:30,096 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:51:30,097 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:51:31,575 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:51:31,575 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:51:32,796 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:51:32,879 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:51:32,881 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:51:32,881 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:51:32,883 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:51:36,020 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:51:36,022 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:51:36,022 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:51:36,024 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:51:37,588 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:51:37,588 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:51:37,924 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:51:39,538 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:51:39,540 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:51:39,541 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:51:39,542 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:51:40,867 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:51:43,420 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:51:43,422 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:51:43,422 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:51:43,422 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:51:43,424 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:51:43,600 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:51:43,601 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:51:45,098 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:51:45,099 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:51:45,099 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:51:46,786 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:51:46,788 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:51:46,788 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:51:46,790 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:51:47,069 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:51:47,070 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:51:47,071 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:51:47,072 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:51:48,660 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:51:49,614 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:51:49,615 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:51:53,661 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:51:58,662 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:51:59,725 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:51:59,725 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1152_75f582b863cdbbe4a504.png with policy now 2023-07-11 19:51:59,739 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:51:59,740 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1152_ac6be0e73608e31f23e9.png with policy now 2023-07-11 19:51:59,749 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:51:59,750 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1152_f44e05a7e3e21014ac08.png with policy now 2023-07-11 19:51:59,752 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:51:59,753 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1152_9ee0451c0db170f16218.png with policy now 2023-07-11 19:51:59,792 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:51:59,794 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:51:59,794 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:51:59,795 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:52:00,079 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/1u8w9q1k-media/images/validation_1152_9ee0451c0db170f16218.png 2023-07-11 19:52:00,081 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/6ths5iel-media/images/validation_1152_ac6be0e73608e31f23e9.png 2023-07-11 19:52:00,094 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/bp40k57p-media/images/validation_1152_f44e05a7e3e21014ac08.png 2023-07-11 19:52:00,098 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:52:00,099 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:52:00,100 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ty9qy7jy-media/images/validation_1152_75f582b863cdbbe4a504.png 2023-07-11 19:52:00,100 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:52:01,641 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:52:01,642 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1152_75f582b863cdbbe4a504.png 2023-07-11 19:52:01,642 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1152_ac6be0e73608e31f23e9.png 2023-07-11 19:52:01,643 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1152_9ee0451c0db170f16218.png 2023-07-11 19:52:01,644 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1152_f44e05a7e3e21014ac08.png 2023-07-11 19:52:01,644 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 19:52:01,986 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/j8i4p4c6-media/images/validation_1152_75f582b863cdbbe4a504.png 2023-07-11 19:52:01,993 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/81zs2vm3-media/images/validation_1152_f44e05a7e3e21014ac08.png 2023-07-11 19:52:02,009 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/6c3pekh4-media/images/validation_1152_ac6be0e73608e31f23e9.png 2023-07-11 19:52:02,013 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/pg4t2244-media/images/validation_1152_9ee0451c0db170f16218.png 2023-07-11 19:52:03,385 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:52:04,102 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:52:06,294 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:52:06,296 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:52:06,297 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:52:06,298 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:52:07,656 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:52:07,656 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:52:09,146 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:52:10,870 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:52:10,905 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:52:10,907 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:52:10,908 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:52:10,909 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:52:13,668 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:52:13,669 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:52:13,747 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:52:13,749 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:52:13,749 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:52:13,750 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:52:14,317 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:52:15,098 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:52:15,099 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:52:15,099 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:52:16,383 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:52:16,385 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:52:16,385 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:52:16,388 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:52:19,318 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:52:19,682 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:52:19,683 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:52:20,150 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:52:20,152 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:52:20,152 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:52:20,154 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:52:23,296 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:52:23,297 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:52:23,298 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:52:23,300 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:52:23,577 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:52:23,579 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:52:23,580 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:52:23,581 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:52:25,135 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:52:25,696 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:52:25,697 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:52:30,122 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:52:30,122 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:52:30,123 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:52:30,208 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:52:35,209 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:52:36,145 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:52:36,146 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1160_d1c53487db656ecbc8a8.png with policy now 2023-07-11 19:52:36,156 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:52:36,156 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1160_b5862260f9dfb527cb9e.png with policy now 2023-07-11 19:52:36,165 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:52:36,165 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1160_2f1ca38ee1b0c5d85255.png with policy now 2023-07-11 19:52:36,174 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:52:36,174 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1160_bde0a03bf1ad8419d9c3.png with policy now 2023-07-11 19:52:36,215 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:52:36,216 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:52:36,216 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:52:36,218 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:52:36,457 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/hu6bgcrr-media/images/validation_1160_d1c53487db656ecbc8a8.png 2023-07-11 19:52:36,510 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/30m85nm9-media/images/validation_1160_bde0a03bf1ad8419d9c3.png 2023-07-11 19:52:36,515 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/hq33e4m6-media/images/validation_1160_b5862260f9dfb527cb9e.png 2023-07-11 19:52:36,516 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/xcgxxt3x-media/images/validation_1160_2f1ca38ee1b0c5d85255.png 2023-07-11 19:52:37,725 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:52:37,726 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1160_bde0a03bf1ad8419d9c3.png 2023-07-11 19:52:37,726 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1160_2f1ca38ee1b0c5d85255.png 2023-07-11 19:52:37,727 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1160_d1c53487db656ecbc8a8.png 2023-07-11 19:52:37,727 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1160_b5862260f9dfb527cb9e.png 2023-07-11 19:52:37,728 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 19:52:38,090 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/rrzhv98a-media/images/validation_1160_b5862260f9dfb527cb9e.png 2023-07-11 19:52:38,117 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/4sio3stz-media/images/validation_1160_bde0a03bf1ad8419d9c3.png 2023-07-11 19:52:38,121 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/7nuofy13-media/images/validation_1160_d1c53487db656ecbc8a8.png 2023-07-11 19:52:38,161 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/yhh0xzqm-media/images/validation_1160_2f1ca38ee1b0c5d85255.png 2023-07-11 19:52:40,844 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:52:40,873 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:52:40,894 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:52:43,739 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:52:44,023 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:52:44,025 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:52:44,025 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:52:44,027 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:52:45,214 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:52:45,215 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:52:45,215 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:52:45,965 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:52:47,497 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:52:47,499 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:52:47,499 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:52:47,501 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:52:49,750 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:52:49,751 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:52:50,536 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:52:50,538 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:52:50,539 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:52:50,540 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:52:51,527 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:52:53,719 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:52:53,721 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:52:53,722 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:52:53,723 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:52:55,763 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:52:55,764 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:52:56,529 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:52:57,261 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:52:57,263 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:52:57,263 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:52:57,265 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:52:59,886 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:52:59,888 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:52:59,889 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:52:59,890 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:53:00,167 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:53:00,169 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:53:00,169 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:53:00,171 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:53:00,214 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:53:00,214 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:53:00,215 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:53:01,777 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:53:01,777 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:53:01,793 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:53:06,794 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:53:07,790 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:53:10,876 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:53:11,878 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:53:12,780 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:53:12,781 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1168_7a5ab95838f1b5896fdb.png with policy now 2023-07-11 19:53:12,794 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:53:12,795 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1168_e6f6e202cb1d86be97bc.png with policy now 2023-07-11 19:53:12,802 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:53:12,802 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1168_083b57a36fb9b48aa915.png with policy now 2023-07-11 19:53:12,814 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:53:12,814 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1168_d980d83f952d5c30a538.png with policy now 2023-07-11 19:53:12,850 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:53:12,851 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:53:12,851 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:53:12,854 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:53:13,082 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/73g9o3v5-media/images/validation_1168_7a5ab95838f1b5896fdb.png 2023-07-11 19:53:13,110 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ydwe0f5o-media/images/validation_1168_e6f6e202cb1d86be97bc.png 2023-07-11 19:53:13,124 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/3snu9ezu-media/images/validation_1168_083b57a36fb9b48aa915.png 2023-07-11 19:53:13,129 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/0vp875nw-media/images/validation_1168_d980d83f952d5c30a538.png 2023-07-11 19:53:13,804 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:53:13,805 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1168_7a5ab95838f1b5896fdb.png 2023-07-11 19:53:13,805 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1168_083b57a36fb9b48aa915.png 2023-07-11 19:53:13,806 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1168_e6f6e202cb1d86be97bc.png 2023-07-11 19:53:13,806 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1168_d980d83f952d5c30a538.png 2023-07-11 19:53:13,807 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 19:53:14,150 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/1hcvarm7-media/images/validation_1168_7a5ab95838f1b5896fdb.png 2023-07-11 19:53:14,155 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ep6l4oz0-media/images/validation_1168_083b57a36fb9b48aa915.png 2023-07-11 19:53:14,181 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/l0w237nf-media/images/validation_1168_d980d83f952d5c30a538.png 2023-07-11 19:53:14,182 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/omewha12-media/images/validation_1168_e6f6e202cb1d86be97bc.png 2023-07-11 19:53:15,426 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:53:15,426 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:53:15,427 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:53:16,231 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:53:16,994 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:53:19,818 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:53:19,943 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:53:19,945 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:53:19,945 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:53:19,947 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:53:22,274 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:53:23,221 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:53:23,223 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:53:23,223 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:53:23,324 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:53:25,832 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:53:25,832 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:53:27,089 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:53:27,091 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:53:27,092 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:53:27,093 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:53:27,827 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:53:29,918 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:53:29,920 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:53:29,920 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:53:29,922 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:53:30,426 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:53:30,426 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:53:30,427 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:53:31,891 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:53:31,892 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:53:33,108 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:53:33,109 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:53:33,110 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:53:33,110 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:53:33,112 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:53:36,137 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:53:36,139 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:53:36,139 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:53:36,141 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:53:36,418 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:53:36,420 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:53:36,420 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:53:36,422 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:53:37,905 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:53:37,906 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:53:38,982 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:53:40,878 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:53:43,916 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:53:44,881 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:53:45,426 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:53:45,427 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:53:45,427 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:53:49,036 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:53:49,037 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1176_b76e250604e885c4bd55.png with policy now 2023-07-11 19:53:49,047 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:53:49,047 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1176_d028389a725dd737ece1.png with policy now 2023-07-11 19:53:49,056 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:53:49,056 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1176_df0c9ae2823ce7d96461.png with policy now 2023-07-11 19:53:49,065 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:53:49,065 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1176_e97469ade650ac1ab331.png with policy now 2023-07-11 19:53:49,108 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:53:49,112 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:53:49,112 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:53:49,116 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:53:49,324 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/fgwc3wlr-media/images/validation_1176_b76e250604e885c4bd55.png 2023-07-11 19:53:49,366 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/3lduhsia-media/images/validation_1176_d028389a725dd737ece1.png 2023-07-11 19:53:49,369 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/gnuktik3-media/images/validation_1176_df0c9ae2823ce7d96461.png 2023-07-11 19:53:49,371 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/tvbjz6un-media/images/validation_1176_e97469ade650ac1ab331.png 2023-07-11 19:53:49,932 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:53:49,932 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1176_df0c9ae2823ce7d96461.png 2023-07-11 19:53:49,933 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1176_d028389a725dd737ece1.png 2023-07-11 19:53:49,934 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1176_b76e250604e885c4bd55.png 2023-07-11 19:53:49,934 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1176_e97469ade650ac1ab331.png 2023-07-11 19:53:49,935 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 19:53:50,117 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:53:50,312 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/68xhcd61-media/images/validation_1176_d028389a725dd737ece1.png 2023-07-11 19:53:50,350 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ar0luq0f-media/images/validation_1176_e97469ade650ac1ab331.png 2023-07-11 19:53:50,350 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/44fhh846-media/images/validation_1176_b76e250604e885c4bd55.png 2023-07-11 19:53:50,354 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/vophg83d-media/images/validation_1176_df0c9ae2823ce7d96461.png 2023-07-11 19:53:52,481 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:53:55,541 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:53:55,543 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:53:55,543 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:53:55,544 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:53:55,545 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:53:55,945 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:53:55,946 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:53:59,378 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:53:59,379 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:53:59,380 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:53:59,381 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:54:00,426 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:54:00,427 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:54:00,427 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:54:01,486 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:54:01,959 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:54:01,960 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:54:03,371 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:54:03,373 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:54:03,373 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:54:03,374 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:54:05,891 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:54:05,894 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:54:05,894 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:54:05,896 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:54:06,897 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:54:07,972 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:54:07,972 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:54:09,482 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:54:09,484 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:54:09,485 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:54:09,486 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:54:10,881 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:54:12,490 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:54:12,492 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:54:12,492 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:54:12,493 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:54:12,495 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:54:12,771 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:54:12,773 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:54:12,774 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:54:12,775 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:54:13,985 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:54:13,986 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:54:15,438 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:54:15,438 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:54:15,438 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:54:18,477 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:54:19,999 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:54:23,478 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:54:25,287 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:54:25,288 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1184_974d5757846f0694ffb4.png with policy now 2023-07-11 19:54:25,300 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:54:25,300 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1184_9984128afeb104db29a1.png with policy now 2023-07-11 19:54:25,313 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:54:25,313 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1184_09da65422971f700e437.png with policy now 2023-07-11 19:54:25,322 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:54:25,322 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1184_05a4d2cabac98cd4d1d4.png with policy now 2023-07-11 19:54:25,361 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:54:25,362 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:54:25,363 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:54:25,364 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:54:25,601 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/1uihhtn6-media/images/validation_1184_974d5757846f0694ffb4.png 2023-07-11 19:54:25,635 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/egj83sew-media/images/validation_1184_05a4d2cabac98cd4d1d4.png 2023-07-11 19:54:25,637 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/hqsf8qhw-media/images/validation_1184_9984128afeb104db29a1.png 2023-07-11 19:54:25,637 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/diegpl9k-media/images/validation_1184_09da65422971f700e437.png 2023-07-11 19:54:26,013 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:54:26,014 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1184_05a4d2cabac98cd4d1d4.png 2023-07-11 19:54:26,014 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1184_9984128afeb104db29a1.png 2023-07-11 19:54:26,015 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1184_09da65422971f700e437.png 2023-07-11 19:54:26,015 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1184_974d5757846f0694ffb4.png 2023-07-11 19:54:26,016 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 19:54:26,339 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/6xad0nvi-media/images/validation_1184_05a4d2cabac98cd4d1d4.png 2023-07-11 19:54:26,385 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/7env0ye9-media/images/validation_1184_09da65422971f700e437.png 2023-07-11 19:54:26,387 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ug9gdxs8-media/images/validation_1184_974d5757846f0694ffb4.png 2023-07-11 19:54:26,490 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/6slen1nj-media/images/validation_1184_9984128afeb104db29a1.png 2023-07-11 19:54:29,076 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:54:30,065 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:54:30,438 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:54:30,438 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:54:30,439 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:54:32,027 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:54:33,311 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:54:33,313 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:54:33,313 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:54:33,315 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:54:34,160 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:54:37,219 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:54:37,221 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:54:37,221 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:54:37,223 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:54:38,041 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:54:38,041 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:54:39,578 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:54:40,342 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:54:40,344 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:54:40,344 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:54:40,346 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:54:40,884 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:54:44,053 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:54:44,053 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:54:44,660 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:54:44,662 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:54:44,662 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:54:44,663 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:54:44,664 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:54:45,747 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:54:45,747 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:54:45,748 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:54:47,222 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:54:47,224 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:54:47,225 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:54:47,226 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:54:50,018 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:54:50,020 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:54:50,020 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:54:50,021 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:54:50,022 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:54:50,066 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:54:50,066 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:54:50,302 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:54:50,303 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:54:50,304 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:54:50,305 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:54:55,873 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:54:56,079 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:54:56,080 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:55:00,747 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:55:00,747 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:55:00,748 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:55:01,795 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:55:02,934 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:55:02,934 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1192_14bd1888a41987672706.png with policy now 2023-07-11 19:55:02,946 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:55:02,947 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1192_0b8f02a4056f08fbfc01.png with policy now 2023-07-11 19:55:02,959 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:55:02,960 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1192_62796f1f83e7ccdc060a.png with policy now 2023-07-11 19:55:02,969 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:55:02,969 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1192_d7bfbf7aea35b7bed028.png with policy now 2023-07-11 19:55:03,010 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:55:03,011 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:55:03,011 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:55:03,014 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:55:03,245 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/04aqv7gz-media/images/validation_1192_14bd1888a41987672706.png 2023-07-11 19:55:03,275 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/t48p116z-media/images/validation_1192_62796f1f83e7ccdc060a.png 2023-07-11 19:55:03,277 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/cw7f0ssd-media/images/validation_1192_d7bfbf7aea35b7bed028.png 2023-07-11 19:55:03,278 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/y0w76k5h-media/images/validation_1192_0b8f02a4056f08fbfc01.png 2023-07-11 19:55:06,831 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:55:06,960 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:55:08,109 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:55:08,109 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:55:08,109 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1192_62796f1f83e7ccdc060a.png 2023-07-11 19:55:08,109 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1192_14bd1888a41987672706.png 2023-07-11 19:55:08,110 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1192_0b8f02a4056f08fbfc01.png 2023-07-11 19:55:08,110 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1192_d7bfbf7aea35b7bed028.png 2023-07-11 19:55:08,110 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 19:55:08,649 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/mek11rlq-media/images/validation_1192_14bd1888a41987672706.png 2023-07-11 19:55:08,655 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/7mbbygjo-media/images/validation_1192_62796f1f83e7ccdc060a.png 2023-07-11 19:55:08,674 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/un831wta-media/images/validation_1192_d7bfbf7aea35b7bed028.png 2023-07-11 19:55:08,924 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/23ww8y2a-media/images/validation_1192_0b8f02a4056f08fbfc01.png 2023-07-11 19:55:10,888 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:55:10,929 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:55:10,930 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:55:10,930 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:55:10,931 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:55:12,204 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:55:14,127 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:55:14,127 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:55:14,127 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:55:14,129 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:55:14,129 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:55:14,131 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:55:15,747 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:55:15,747 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:55:15,748 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:55:17,603 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:55:17,605 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:55:17,605 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:55:17,606 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:55:17,607 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:55:20,727 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:55:20,728 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:55:20,728 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:55:20,730 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:55:21,140 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:55:21,141 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:55:23,416 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:55:24,118 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:55:24,119 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:55:24,120 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:55:24,121 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:55:27,239 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:55:27,240 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:55:27,241 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:55:27,243 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:55:27,520 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:55:27,522 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:55:27,523 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:55:27,524 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:55:28,155 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:55:28,155 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:55:29,046 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:55:30,747 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:55:30,748 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:55:30,748 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:55:34,789 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:55:35,168 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:55:39,790 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:55:40,078 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:55:40,078 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1200_16f4a0c9ef346f446c13.png with policy now 2023-07-11 19:55:40,088 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:55:40,088 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1200_e93ddab6d31d607831c0.png with policy now 2023-07-11 19:55:40,097 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:55:40,098 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1200_edcc4e9f600fa42e7eff.png with policy now 2023-07-11 19:55:40,107 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:55:40,107 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1200_acc1bde7ddec3fe3577c.png with policy now 2023-07-11 19:55:40,147 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:55:40,148 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:55:40,148 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:55:40,150 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:55:40,431 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/59ihciry-media/images/validation_1200_16f4a0c9ef346f446c13.png 2023-07-11 19:55:40,466 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/5kxp5j6s-media/images/validation_1200_e93ddab6d31d607831c0.png 2023-07-11 19:55:40,470 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/640bcurj-media/images/validation_1200_edcc4e9f600fa42e7eff.png 2023-07-11 19:55:40,487 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/hifjkj1j-media/images/validation_1200_acc1bde7ddec3fe3577c.png 2023-07-11 19:55:40,890 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:55:42,184 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:55:42,185 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1200_edcc4e9f600fa42e7eff.png 2023-07-11 19:55:42,185 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1200_acc1bde7ddec3fe3577c.png 2023-07-11 19:55:42,186 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1200_16f4a0c9ef346f446c13.png 2023-07-11 19:55:42,186 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1200_e93ddab6d31d607831c0.png 2023-07-11 19:55:42,187 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 19:55:42,547 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/3mcv7wtn-media/images/validation_1200_e93ddab6d31d607831c0.png 2023-07-11 19:55:42,547 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/woput052-media/images/validation_1200_acc1bde7ddec3fe3577c.png 2023-07-11 19:55:42,558 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/0sh09nyp-media/images/validation_1200_edcc4e9f600fa42e7eff.png 2023-07-11 19:55:42,562 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/30ebtp6z-media/images/validation_1200_16f4a0c9ef346f446c13.png 2023-07-11 19:55:43,941 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:55:45,276 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:55:45,747 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:55:45,748 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:55:45,749 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:55:46,718 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:55:46,720 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:55:46,721 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:55:46,722 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:55:49,200 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:55:49,200 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:55:50,144 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:55:50,146 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:55:50,146 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:55:50,148 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:55:51,149 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:55:54,226 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:55:54,228 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:55:54,229 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:55:54,230 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:55:56,216 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:55:56,217 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:55:56,342 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:55:57,852 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:55:57,854 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:55:57,854 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:55:57,856 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:56:00,895 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:56:00,895 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:56:00,896 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:56:01,544 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:56:01,546 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:56:01,546 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:56:01,547 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:56:01,549 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:56:03,228 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:56:03,229 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:56:04,950 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:56:04,952 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:56:04,952 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:56:04,954 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:56:05,232 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:56:05,234 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:56:05,234 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:56:05,236 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:56:06,830 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:56:10,243 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:56:10,243 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:56:10,893 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:56:11,894 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:56:15,904 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:56:15,904 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:56:15,905 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:56:16,963 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:56:17,816 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:56:17,817 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1208_9fe00c3f196682ce8aed.png with policy now 2023-07-11 19:56:17,827 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:56:17,828 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1208_2403b0bdec188a248ed1.png with policy now 2023-07-11 19:56:17,836 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:56:17,836 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1208_595eef34e4d21c6eb4a9.png with policy now 2023-07-11 19:56:17,845 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:56:17,845 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1208_0a244457848c676d98e6.png with policy now 2023-07-11 19:56:17,888 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:56:17,890 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:56:17,890 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:56:17,894 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:56:18,135 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/57qfz69x-media/images/validation_1208_9fe00c3f196682ce8aed.png 2023-07-11 19:56:18,140 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/6s8go9tn-media/images/validation_1208_2403b0bdec188a248ed1.png 2023-07-11 19:56:18,158 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/9tpi62mu-media/images/validation_1208_595eef34e4d21c6eb4a9.png 2023-07-11 19:56:18,160 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/zhg6z2yx-media/images/validation_1208_0a244457848c676d98e6.png 2023-07-11 19:56:21,546 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:56:22,547 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:56:24,273 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:56:24,274 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:56:24,274 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1208_0a244457848c676d98e6.png 2023-07-11 19:56:24,274 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1208_9fe00c3f196682ce8aed.png 2023-07-11 19:56:24,275 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1208_2403b0bdec188a248ed1.png 2023-07-11 19:56:24,276 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1208_595eef34e4d21c6eb4a9.png 2023-07-11 19:56:24,276 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 19:56:24,611 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/afscr5ws-media/images/validation_1208_0a244457848c676d98e6.png 2023-07-11 19:56:24,631 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/2sipzxg8-media/images/validation_1208_2403b0bdec188a248ed1.png 2023-07-11 19:56:24,650 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/few8bqde-media/images/validation_1208_9fe00c3f196682ce8aed.png 2023-07-11 19:56:24,664 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/jms1jwcb-media/images/validation_1208_595eef34e4d21c6eb4a9.png 2023-07-11 19:56:25,386 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:56:25,388 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:56:25,388 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:56:25,390 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:56:27,618 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:56:29,231 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:56:29,233 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:56:29,233 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:56:29,410 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:56:30,904 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:56:30,905 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:56:30,905 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:56:31,287 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:56:31,288 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:56:31,925 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:56:31,927 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:56:31,927 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:56:31,929 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:56:32,930 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:56:35,651 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:56:35,653 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:56:35,654 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:56:35,656 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:56:38,300 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:56:38,301 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:56:38,765 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:56:38,766 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:56:38,767 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:56:38,767 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:56:38,769 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:56:40,896 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:56:41,734 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:56:41,736 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:56:41,736 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:56:41,738 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:56:42,016 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:56:42,018 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:56:42,018 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:56:42,020 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:56:44,644 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:56:45,314 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:56:45,315 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:56:45,904 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:56:45,905 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:56:45,905 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:56:49,964 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:56:54,680 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:56:54,681 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1216_71a11268d66c7c3c3964.png with policy now 2023-07-11 19:56:54,694 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:56:54,694 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1216_ce73942bbae6d5990516.png with policy now 2023-07-11 19:56:54,712 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:56:54,713 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1216_6f69187eb5366064d111.png with policy now 2023-07-11 19:56:54,722 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:56:54,722 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1216_0ce2aa47670fb06b9041.png with policy now 2023-07-11 19:56:54,762 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:56:54,764 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:56:54,764 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:56:54,769 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:56:55,001 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/wt950z0m-media/images/validation_1216_71a11268d66c7c3c3964.png 2023-07-11 19:56:55,022 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ef7lf72b-media/images/validation_1216_ce73942bbae6d5990516.png 2023-07-11 19:56:55,023 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/fzh96jb1-media/images/validation_1216_6f69187eb5366064d111.png 2023-07-11 19:56:55,112 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/j6prr2fx-media/images/validation_1216_0ce2aa47670fb06b9041.png 2023-07-11 19:56:55,771 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:56:59,024 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:56:59,342 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:56:59,342 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:56:59,343 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1216_71a11268d66c7c3c3964.png 2023-07-11 19:56:59,343 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1216_ce73942bbae6d5990516.png 2023-07-11 19:56:59,344 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1216_6f69187eb5366064d111.png 2023-07-11 19:56:59,344 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1216_0ce2aa47670fb06b9041.png 2023-07-11 19:56:59,345 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 19:56:59,702 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/q73juv6s-media/images/validation_1216_0ce2aa47670fb06b9041.png 2023-07-11 19:56:59,713 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/vatu81f1-media/images/validation_1216_ce73942bbae6d5990516.png 2023-07-11 19:56:59,715 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/andr7kb5-media/images/validation_1216_71a11268d66c7c3c3964.png 2023-07-11 19:56:59,731 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ga3iuskm-media/images/validation_1216_6f69187eb5366064d111.png 2023-07-11 19:57:00,974 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:57:00,975 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:57:00,975 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:57:01,030 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:57:01,748 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:57:01,750 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:57:01,750 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:57:01,752 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:57:04,533 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:57:04,535 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:57:04,535 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:57:04,536 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:57:06,356 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:57:06,357 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:57:06,478 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:57:07,968 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:57:07,970 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:57:07,970 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:57:07,972 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:57:10,766 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:57:10,768 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:57:10,768 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:57:10,770 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:57:10,900 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:57:11,546 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:57:13,370 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:57:13,371 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:57:14,753 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:57:14,755 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:57:14,756 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:57:14,757 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:57:16,391 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:57:16,391 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:57:16,392 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:57:17,514 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:57:18,410 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:57:18,412 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:57:18,412 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:57:18,414 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:57:18,695 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:57:18,697 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:57:18,697 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:57:18,702 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:57:20,386 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:57:20,386 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:57:23,232 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:57:27,400 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:57:28,234 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:57:31,286 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:57:31,286 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1224_e37d195d179a05497868.png with policy now 2023-07-11 19:57:31,302 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:57:31,302 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1224_39b1e250a34221277567.png with policy now 2023-07-11 19:57:31,305 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:57:31,305 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1224_c5fa5e45c4aa753c17c9.png with policy now 2023-07-11 19:57:31,314 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:57:31,319 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1224_2628069d13a5bea9e3df.png with policy now 2023-07-11 19:57:31,354 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:57:31,357 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:57:31,357 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:57:31,359 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:57:31,394 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:57:31,395 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:57:31,395 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:57:31,592 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/uc2fnl8e-media/images/validation_1224_e37d195d179a05497868.png 2023-07-11 19:57:31,612 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/w2yrsgd7-media/images/validation_1224_39b1e250a34221277567.png 2023-07-11 19:57:31,619 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/716d7qi2-media/images/validation_1224_2628069d13a5bea9e3df.png 2023-07-11 19:57:31,634 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/pku8rqxc-media/images/validation_1224_c5fa5e45c4aa753c17c9.png 2023-07-11 19:57:33,454 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:57:34,415 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:57:34,415 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:57:34,415 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1224_c5fa5e45c4aa753c17c9.png 2023-07-11 19:57:34,416 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1224_e37d195d179a05497868.png 2023-07-11 19:57:34,416 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1224_39b1e250a34221277567.png 2023-07-11 19:57:34,417 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1224_2628069d13a5bea9e3df.png 2023-07-11 19:57:34,417 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 19:57:34,829 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/g28p2n31-media/images/validation_1224_c5fa5e45c4aa753c17c9.png 2023-07-11 19:57:34,841 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/l04u2pia-media/images/validation_1224_39b1e250a34221277567.png 2023-07-11 19:57:34,847 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/9klmevsk-media/images/validation_1224_2628069d13a5bea9e3df.png 2023-07-11 19:57:34,859 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/kv14w3n2-media/images/validation_1224_e37d195d179a05497868.png 2023-07-11 19:57:35,449 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:57:38,456 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:57:39,679 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:57:39,681 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:57:39,682 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:57:39,683 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:57:40,902 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:57:41,428 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:57:41,429 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:57:42,614 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:57:42,616 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:57:42,616 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:57:42,617 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:57:43,618 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:57:45,579 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:57:45,581 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:57:45,581 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:57:45,583 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:57:46,686 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:57:46,686 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:57:46,687 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:57:48,174 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:57:48,176 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:57:48,176 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:57:48,178 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:57:48,443 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:57:48,443 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:57:49,072 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:57:51,397 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:57:51,398 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:57:51,398 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:57:51,400 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:57:54,727 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:57:54,729 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:57:54,730 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:57:54,730 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:57:54,732 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:57:55,010 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:57:55,012 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:57:55,012 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:57:55,014 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:57:55,453 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:57:55,453 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:58:00,545 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:58:01,708 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:58:01,709 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:58:01,709 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:58:02,468 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:58:05,763 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:58:07,568 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:58:07,568 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1232_4c55aad4840c63f78577.png with policy now 2023-07-11 19:58:07,577 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:58:07,578 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1232_d39d5a329f94acd4f6a4.png with policy now 2023-07-11 19:58:07,586 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:58:07,586 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1232_e6497da0525aea382fbe.png with policy now 2023-07-11 19:58:07,594 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:58:07,595 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1232_87e2c345946d0460c998.png with policy now 2023-07-11 19:58:07,635 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:58:07,637 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:58:07,637 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:58:07,639 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:58:07,879 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/eco7lygf-media/images/validation_1232_d39d5a329f94acd4f6a4.png 2023-07-11 19:58:07,881 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/8mdr67q8-media/images/validation_1232_4c55aad4840c63f78577.png 2023-07-11 19:58:07,899 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/cakqzczn-media/images/validation_1232_e6497da0525aea382fbe.png 2023-07-11 19:58:07,905 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ntuzt75n-media/images/validation_1232_87e2c345946d0460c998.png 2023-07-11 19:58:09,482 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:58:09,483 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:58:09,483 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1232_d39d5a329f94acd4f6a4.png 2023-07-11 19:58:09,484 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1232_4c55aad4840c63f78577.png 2023-07-11 19:58:09,484 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1232_87e2c345946d0460c998.png 2023-07-11 19:58:09,485 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1232_e6497da0525aea382fbe.png 2023-07-11 19:58:09,485 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 19:58:09,824 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/4sky90ih-media/images/validation_1232_4c55aad4840c63f78577.png 2023-07-11 19:58:09,837 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/k0p46iyn-media/images/validation_1232_87e2c345946d0460c998.png 2023-07-11 19:58:09,838 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/yigpawn1-media/images/validation_1232_d39d5a329f94acd4f6a4.png 2023-07-11 19:58:09,860 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/eux3kn68-media/images/validation_1232_e6497da0525aea382fbe.png 2023-07-11 19:58:10,844 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:58:10,904 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:58:11,844 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:58:14,897 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:58:14,899 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:58:14,900 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:58:14,901 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:58:16,391 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:58:16,496 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:58:16,497 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:58:17,249 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:58:17,249 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:58:17,250 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:58:18,075 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:58:18,077 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:58:18,078 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:58:18,080 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:58:21,584 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:58:21,781 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:58:21,782 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:58:21,783 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:58:21,784 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:58:23,511 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:58:23,511 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:58:25,316 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:58:25,318 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:58:25,319 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:58:25,320 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:58:26,740 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:58:28,258 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:58:28,260 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:58:28,260 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:58:28,262 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:58:30,522 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:58:30,523 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:58:31,361 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:58:31,363 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:58:31,363 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:58:31,365 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:58:31,643 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:58:31,645 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:58:31,646 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:58:31,647 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:58:32,115 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:58:32,249 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:58:32,249 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:58:32,250 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:58:37,296 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:58:37,535 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:58:37,536 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:58:40,907 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:58:42,909 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:58:44,319 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:58:44,319 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1240_9502bd66906821a48a2f.png with policy now 2023-07-11 19:58:44,329 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:58:44,329 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1240_c1ba7526a4470eb03bb6.png with policy now 2023-07-11 19:58:44,338 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:58:44,339 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1240_8441de0e32308471748a.png with policy now 2023-07-11 19:58:44,347 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:58:44,347 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1240_4a6eef5fa6860d20cbc1.png with policy now 2023-07-11 19:58:44,389 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:58:44,391 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:58:44,391 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:58:44,393 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:58:44,550 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:58:44,551 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1240_c1ba7526a4470eb03bb6.png 2023-07-11 19:58:44,551 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1240_8441de0e32308471748a.png 2023-07-11 19:58:44,552 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1240_4a6eef5fa6860d20cbc1.png 2023-07-11 19:58:44,552 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1240_9502bd66906821a48a2f.png 2023-07-11 19:58:44,553 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 19:58:44,633 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/njpaf5yx-media/images/validation_1240_c1ba7526a4470eb03bb6.png 2023-07-11 19:58:44,640 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/u82my363-media/images/validation_1240_9502bd66906821a48a2f.png 2023-07-11 19:58:44,649 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/lh4t69p6-media/images/validation_1240_8441de0e32308471748a.png 2023-07-11 19:58:44,657 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/w510buzg-media/images/validation_1240_4a6eef5fa6860d20cbc1.png 2023-07-11 19:58:45,037 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/82kpdpre-media/images/validation_1240_9502bd66906821a48a2f.png 2023-07-11 19:58:45,066 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/jo05mrg4-media/images/validation_1240_c1ba7526a4470eb03bb6.png 2023-07-11 19:58:45,378 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/to8q5les-media/images/validation_1240_8441de0e32308471748a.png 2023-07-11 19:58:45,381 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/f53ifgvu-media/images/validation_1240_4a6eef5fa6860d20cbc1.png 2023-07-11 19:58:47,252 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:58:47,252 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:58:47,253 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:58:47,977 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:58:47,978 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:58:51,549 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:58:51,551 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:58:51,551 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:58:51,553 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:58:51,565 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:58:53,193 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:58:55,157 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:58:55,159 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:58:55,159 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:58:55,161 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:58:58,579 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:58:58,579 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:58:58,703 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:58:58,705 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:58:58,705 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:58:58,706 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:58:58,707 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:59:01,582 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:59:01,584 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:59:01,584 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:59:01,585 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:59:03,077 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:59:03,078 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:59:03,078 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:59:03,832 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:59:05,590 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:59:05,591 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:59:05,833 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:59:05,835 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:59:05,835 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:59:05,837 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:59:08,669 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:59:08,671 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:59:08,671 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:59:08,672 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:59:08,951 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:59:08,953 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:59:08,953 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:59:08,954 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:59:08,955 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:59:10,912 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:59:12,605 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:59:12,605 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:59:14,914 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:59:18,078 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:59:18,078 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:59:18,079 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:59:20,124 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:59:21,530 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:59:21,531 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1248_d2b71281b721da5f3901.png with policy now 2023-07-11 19:59:21,543 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:59:21,544 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1248_d8552d083a3c70763db2.png with policy now 2023-07-11 19:59:21,555 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:59:21,555 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1248_f192f7502e6f7f5405e5.png with policy now 2023-07-11 19:59:21,563 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:59:21,563 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1248_5c1a4d1cff1b24bda178.png with policy now 2023-07-11 19:59:21,603 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:59:21,607 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:59:21,607 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:59:21,609 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:59:21,847 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/0pvej953-media/images/validation_1248_d8552d083a3c70763db2.png 2023-07-11 19:59:21,852 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/goi7uk5s-media/images/validation_1248_d2b71281b721da5f3901.png 2023-07-11 19:59:21,867 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/qondytx9-media/images/validation_1248_5c1a4d1cff1b24bda178.png 2023-07-11 19:59:21,869 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/nrpb9h4v-media/images/validation_1248_f192f7502e6f7f5405e5.png 2023-07-11 19:59:25,284 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:59:25,286 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:59:26,634 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:59:26,634 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:59:26,635 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1248_5c1a4d1cff1b24bda178.png 2023-07-11 19:59:26,635 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1248_d2b71281b721da5f3901.png 2023-07-11 19:59:26,636 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1248_d8552d083a3c70763db2.png 2023-07-11 19:59:26,636 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1248_f192f7502e6f7f5405e5.png 2023-07-11 19:59:26,637 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 19:59:26,993 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/p61sclgv-media/images/validation_1248_5c1a4d1cff1b24bda178.png 2023-07-11 19:59:26,997 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/5bys3c31-media/images/validation_1248_d2b71281b721da5f3901.png 2023-07-11 19:59:27,005 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/hs18u9t1-media/images/validation_1248_d8552d083a3c70763db2.png 2023-07-11 19:59:27,022 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/eana6r51-media/images/validation_1248_f192f7502e6f7f5405e5.png 2023-07-11 19:59:28,931 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:59:28,933 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:59:28,933 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:59:28,935 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:59:30,856 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:59:32,815 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:59:32,817 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:59:32,818 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:59:32,819 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:59:33,078 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:59:33,078 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:59:33,079 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:59:33,648 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:59:33,649 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:59:36,395 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:59:36,397 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:59:36,397 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:59:36,398 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:59:36,399 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:59:39,326 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:59:39,328 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:59:39,328 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:59:39,330 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:59:40,662 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:59:40,663 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:59:40,913 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 19:59:41,971 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:59:41,973 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:59:41,973 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:59:41,973 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:59:41,975 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:59:44,868 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:59:44,870 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:59:44,871 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:59:44,872 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:59:45,150 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:59:45,152 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:59:45,153 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:59:45,154 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:59:47,676 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 19:59:47,676 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 19:59:47,686 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:59:48,078 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 19:59:48,078 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 19:59:48,079 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 19:59:53,128 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 19:59:57,733 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:59:57,733 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1256_910aaa0aeb996f2bff52.png with policy now 2023-07-11 19:59:57,746 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:59:57,746 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1256_6eaa59f4c776007b4577.png with policy now 2023-07-11 19:59:57,758 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:59:57,758 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1256_008ae0ebd6c28b34501c.png with policy now 2023-07-11 19:59:57,765 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 19:59:57,765 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1256_4d5909a879197a9d6947.png with policy now 2023-07-11 19:59:57,804 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 19:59:57,805 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 19:59:57,806 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 19:59:57,807 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 19:59:58,059 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/911y0ave-media/images/validation_1256_910aaa0aeb996f2bff52.png 2023-07-11 19:59:58,068 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/4uqso1ks-media/images/validation_1256_6eaa59f4c776007b4577.png 2023-07-11 19:59:58,072 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/dz9h8yiq-media/images/validation_1256_008ae0ebd6c28b34501c.png 2023-07-11 19:59:58,077 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/uvs85n6x-media/images/validation_1256_4d5909a879197a9d6947.png 2023-07-11 19:59:58,808 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:00:01,705 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:00:01,705 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:00:01,705 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1256_910aaa0aeb996f2bff52.png 2023-07-11 20:00:01,706 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1256_6eaa59f4c776007b4577.png 2023-07-11 20:00:01,707 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1256_008ae0ebd6c28b34501c.png 2023-07-11 20:00:01,707 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1256_4d5909a879197a9d6947.png 2023-07-11 20:00:01,708 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 20:00:01,708 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:00:02,065 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ktluwy08-media/images/validation_1256_910aaa0aeb996f2bff52.png 2023-07-11 20:00:02,073 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/cnz62ebs-media/images/validation_1256_6eaa59f4c776007b4577.png 2023-07-11 20:00:02,079 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/04k4frsm-media/images/validation_1256_4d5909a879197a9d6947.png 2023-07-11 20:00:02,086 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/poygxelq-media/images/validation_1256_008ae0ebd6c28b34501c.png 2023-07-11 20:00:03,078 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:00:03,079 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 20:00:03,079 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 20:00:04,681 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:00:04,847 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:00:04,849 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:00:04,850 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:00:04,851 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:00:08,281 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:00:08,283 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:00:08,283 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:00:08,285 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:00:08,719 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:00:08,720 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:00:09,943 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:00:10,916 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:00:11,325 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:00:11,328 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:00:11,328 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:00:11,330 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:00:14,925 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:00:14,927 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:00:14,927 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:00:14,929 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:00:15,732 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:00:15,733 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:00:15,784 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:00:17,944 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:00:17,946 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:00:17,946 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:00:17,948 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:00:18,078 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:00:18,079 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 20:00:18,080 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 20:00:20,981 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:00:22,074 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:00:22,076 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:00:22,077 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:00:22,078 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:00:22,365 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:00:22,367 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:00:22,367 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:00:22,369 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:00:22,747 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:00:22,748 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:00:26,002 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:00:29,760 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:00:31,003 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:00:33,089 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:00:33,089 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 20:00:33,090 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 20:00:35,007 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:00:35,007 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1264_a34f676857c79531f476.png with policy now 2023-07-11 20:00:35,022 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:00:35,022 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1264_12e521ee796cd4abe6b4.png with policy now 2023-07-11 20:00:35,032 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:00:35,032 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1264_41d26f1ebd22f8010867.png with policy now 2023-07-11 20:00:35,040 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:00:35,040 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1264_3c294db4afe2a101d5c2.png with policy now 2023-07-11 20:00:35,078 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:00:35,079 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:00:35,079 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:00:35,081 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:00:35,503 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/vyt6lx18-media/images/validation_1264_a34f676857c79531f476.png 2023-07-11 20:00:35,505 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ym0114ah-media/images/validation_1264_3c294db4afe2a101d5c2.png 2023-07-11 20:00:35,528 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/l1683qw3-media/images/validation_1264_41d26f1ebd22f8010867.png 2023-07-11 20:00:35,529 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/5f1wwxwt-media/images/validation_1264_12e521ee796cd4abe6b4.png 2023-07-11 20:00:36,081 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:00:36,774 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:00:36,775 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1264_a34f676857c79531f476.png 2023-07-11 20:00:36,775 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1264_41d26f1ebd22f8010867.png 2023-07-11 20:00:36,776 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1264_12e521ee796cd4abe6b4.png 2023-07-11 20:00:36,776 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1264_3c294db4afe2a101d5c2.png 2023-07-11 20:00:36,777 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 20:00:37,194 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/dhu6ynbs-media/images/validation_1264_41d26f1ebd22f8010867.png 2023-07-11 20:00:37,201 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ldqvx3pw-media/images/validation_1264_a34f676857c79531f476.png 2023-07-11 20:00:37,223 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/2l1xl06d-media/images/validation_1264_12e521ee796cd4abe6b4.png 2023-07-11 20:00:37,228 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/pc0f7ofs-media/images/validation_1264_3c294db4afe2a101d5c2.png 2023-07-11 20:00:39,226 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:00:40,919 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:00:41,106 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:00:42,058 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:00:42,060 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:00:42,061 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:00:42,062 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:00:43,788 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:00:43,789 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:00:45,158 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:00:45,160 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:00:45,160 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:00:45,162 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:00:46,615 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:00:48,195 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:00:48,196 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 20:00:48,196 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 20:00:48,324 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:00:48,325 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:00:48,326 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:00:48,327 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:00:50,803 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:00:50,804 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:00:52,029 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:00:52,031 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:00:52,031 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:00:52,032 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:00:52,033 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:00:55,203 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:00:55,204 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:00:55,205 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:00:55,206 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:00:57,650 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:00:57,816 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:00:57,817 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:00:58,837 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:00:58,838 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:00:58,839 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:00:58,840 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:00:59,119 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:00:59,121 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:00:59,121 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:00:59,123 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:01:02,731 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:01:03,195 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:01:03,196 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 20:01:03,197 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 20:01:04,831 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:01:04,832 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:01:08,268 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:01:10,922 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:01:11,723 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:01:11,723 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1272_1d9217684c014aa87e19.png with policy now 2023-07-11 20:01:11,737 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:01:11,738 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1272_217b42b3f8761d16b428.png with policy now 2023-07-11 20:01:11,747 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:01:11,748 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1272_0fe5a4f175505061d59f.png with policy now 2023-07-11 20:01:11,756 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:01:11,756 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1272_dd42d9becd27cc9d54fe.png with policy now 2023-07-11 20:01:11,793 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:01:11,795 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:01:11,795 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:01:11,797 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:01:11,846 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:01:11,847 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1272_dd42d9becd27cc9d54fe.png 2023-07-11 20:01:11,848 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1272_0fe5a4f175505061d59f.png 2023-07-11 20:01:11,848 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1272_1d9217684c014aa87e19.png 2023-07-11 20:01:11,849 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1272_217b42b3f8761d16b428.png 2023-07-11 20:01:11,849 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 20:01:12,069 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ajqgp0ed-media/images/validation_1272_dd42d9becd27cc9d54fe.png 2023-07-11 20:01:12,069 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/0wnhh4gk-media/images/validation_1272_0fe5a4f175505061d59f.png 2023-07-11 20:01:12,080 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/9ethmpin-media/images/validation_1272_217b42b3f8761d16b428.png 2023-07-11 20:01:12,227 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ua8ge3i7-media/images/validation_1272_1d9217684c014aa87e19.png 2023-07-11 20:01:12,442 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/vgqzy3y1-media/images/validation_1272_0fe5a4f175505061d59f.png 2023-07-11 20:01:12,551 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/n9lufc4j-media/images/validation_1272_dd42d9becd27cc9d54fe.png 2023-07-11 20:01:12,555 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/1im726jp-media/images/validation_1272_217b42b3f8761d16b428.png 2023-07-11 20:01:12,788 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/28vtjfoz-media/images/validation_1272_1d9217684c014aa87e19.png 2023-07-11 20:01:13,468 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:01:15,955 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:01:18,196 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:01:18,196 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 20:01:18,197 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 20:01:18,719 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:01:18,860 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:01:19,574 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:01:19,576 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:01:19,577 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:01:19,578 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:01:22,672 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:01:22,674 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:01:22,674 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:01:22,676 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:01:23,847 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:01:25,875 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:01:25,876 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:01:26,473 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:01:26,475 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:01:26,476 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:01:26,477 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:01:29,150 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:01:29,152 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:01:29,152 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:01:29,153 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:01:29,154 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:01:31,401 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:01:31,402 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:01:31,403 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:01:31,404 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:01:32,888 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:01:32,889 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:01:33,196 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:01:33,196 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 20:01:33,197 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 20:01:34,935 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:01:34,936 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:01:34,936 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:01:34,937 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:01:34,938 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:01:35,217 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:01:35,218 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:01:35,219 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:01:35,221 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:01:39,904 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:01:39,904 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:01:40,773 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:01:40,924 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:01:45,927 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:01:47,793 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:01:47,793 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1280_5c8b2ecfb16916c0242c.png with policy now 2023-07-11 20:01:47,803 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:01:47,803 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1280_d62f57953ca77cb73b3f.png with policy now 2023-07-11 20:01:47,812 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:01:47,812 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1280_1cc9b7ef9b70d011ac71.png with policy now 2023-07-11 20:01:47,820 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:01:47,820 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1280_f43945339b235e0f01c6.png with policy now 2023-07-11 20:01:47,863 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:01:47,865 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:01:47,865 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:01:47,867 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:01:48,119 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/b2him95t-media/images/validation_1280_5c8b2ecfb16916c0242c.png 2023-07-11 20:01:48,131 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/de6m41sc-media/images/validation_1280_d62f57953ca77cb73b3f.png 2023-07-11 20:01:48,135 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/nuv7lyp7-media/images/validation_1280_1cc9b7ef9b70d011ac71.png 2023-07-11 20:01:48,152 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/bkzmwsaw-media/images/validation_1280_f43945339b235e0f01c6.png 2023-07-11 20:01:48,196 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:01:48,197 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 20:01:48,197 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 20:01:51,020 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:01:52,122 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:01:53,932 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:01:53,932 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:01:53,932 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1280_5c8b2ecfb16916c0242c.png 2023-07-11 20:01:53,933 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1280_f43945339b235e0f01c6.png 2023-07-11 20:01:53,934 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1280_d62f57953ca77cb73b3f.png 2023-07-11 20:01:53,934 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1280_1cc9b7ef9b70d011ac71.png 2023-07-11 20:01:53,934 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 20:01:54,301 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/udl9qxjs-media/images/validation_1280_f43945339b235e0f01c6.png 2023-07-11 20:01:54,304 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/jfw6wbym-media/images/validation_1280_5c8b2ecfb16916c0242c.png 2023-07-11 20:01:54,317 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/pd6eqokc-media/images/validation_1280_d62f57953ca77cb73b3f.png 2023-07-11 20:01:54,322 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/nfo45bcw-media/images/validation_1280_1cc9b7ef9b70d011ac71.png 2023-07-11 20:01:55,636 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:01:55,638 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:01:55,638 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:01:55,640 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:01:56,518 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:01:58,567 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:01:58,569 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:01:58,569 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:01:58,571 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:02:00,945 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:02:00,946 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:02:02,224 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:02:02,281 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:02:02,283 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:02:02,283 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:02:02,285 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:02:03,416 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:02:03,416 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 20:02:03,417 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 20:02:06,412 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:02:06,414 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:02:06,414 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:02:06,416 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:02:07,831 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:02:07,959 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:02:07,960 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:02:09,643 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:02:09,645 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:02:09,645 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:02:09,647 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:02:10,927 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:02:12,251 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:02:12,253 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:02:12,253 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:02:12,255 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:02:12,534 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:02:12,535 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:02:12,536 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:02:12,537 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:02:13,105 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:02:14,973 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:02:14,973 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:02:18,242 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:02:18,416 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:02:18,416 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 20:02:18,417 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 20:02:23,464 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:02:25,223 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:02:25,223 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1288_ce6487d4a69d3d6621e0.png with policy now 2023-07-11 20:02:25,233 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:02:25,233 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1288_401eb7d8a6e899c795fc.png with policy now 2023-07-11 20:02:25,243 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:02:25,243 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1288_477cc6da95a873dd86c6.png with policy now 2023-07-11 20:02:25,251 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:02:25,251 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1288_d7f23e91466de6777b6a.png with policy now 2023-07-11 20:02:25,294 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:02:25,295 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:02:25,295 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:02:25,297 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:02:25,527 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/qmml28x2-media/images/validation_1288_ce6487d4a69d3d6621e0.png 2023-07-11 20:02:25,551 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/t0k94gn4-media/images/validation_1288_401eb7d8a6e899c795fc.png 2023-07-11 20:02:25,566 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/3tvh20kz-media/images/validation_1288_d7f23e91466de6777b6a.png 2023-07-11 20:02:25,571 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/sbm3j0vv-media/images/validation_1288_477cc6da95a873dd86c6.png 2023-07-11 20:02:28,991 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:02:29,002 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:02:29,003 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:02:29,003 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1288_ce6487d4a69d3d6621e0.png 2023-07-11 20:02:29,004 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1288_477cc6da95a873dd86c6.png 2023-07-11 20:02:29,004 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1288_d7f23e91466de6777b6a.png 2023-07-11 20:02:29,005 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1288_401eb7d8a6e899c795fc.png 2023-07-11 20:02:29,005 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 20:02:29,074 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:02:29,351 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/sxppb7ce-media/images/validation_1288_ce6487d4a69d3d6621e0.png 2023-07-11 20:02:29,357 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/8qcotmqm-media/images/validation_1288_477cc6da95a873dd86c6.png 2023-07-11 20:02:29,372 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/27zsacpv-media/images/validation_1288_d7f23e91466de6777b6a.png 2023-07-11 20:02:29,425 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/l69ygjvo-media/images/validation_1288_401eb7d8a6e899c795fc.png 2023-07-11 20:02:32,319 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:02:32,320 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:02:32,321 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:02:32,322 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:02:33,416 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:02:33,417 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 20:02:33,418 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 20:02:34,222 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:02:35,297 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:02:35,299 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:02:35,299 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:02:35,301 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:02:36,016 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:02:36,017 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:02:38,568 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:02:38,570 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:02:38,571 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:02:38,572 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:02:39,449 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:02:40,930 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:02:42,169 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:02:42,171 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:02:42,171 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:02:42,194 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:02:43,031 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:02:43,032 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:02:45,084 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:02:45,086 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:02:45,086 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:02:45,087 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:02:45,088 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:02:48,417 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:02:48,417 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 20:02:48,418 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 20:02:48,705 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:02:48,707 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:02:48,707 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:02:48,709 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:02:48,988 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:02:48,990 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:02:48,990 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:02:48,992 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:02:50,045 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:02:50,045 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:02:50,535 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:02:55,536 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:03:00,537 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:03:01,550 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:03:01,551 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1296_db936d1e5ef08167a9b9.png with policy now 2023-07-11 20:03:01,565 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:03:01,565 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1296_42e08916c70b880ffc82.png with policy now 2023-07-11 20:03:01,574 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:03:01,574 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1296_2a52a9d5fd9d9e666456.png with policy now 2023-07-11 20:03:01,583 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:03:01,583 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1296_23cc3b9e181168d6ed71.png with policy now 2023-07-11 20:03:01,617 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:03:01,619 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:03:01,619 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:03:01,620 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:03:01,877 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/w547ash6-media/images/validation_1296_db936d1e5ef08167a9b9.png 2023-07-11 20:03:01,890 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/080lso98-media/images/validation_1296_42e08916c70b880ffc82.png 2023-07-11 20:03:01,892 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/o89q7ip8-media/images/validation_1296_2a52a9d5fd9d9e666456.png 2023-07-11 20:03:01,892 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/idt8yhkl-media/images/validation_1296_23cc3b9e181168d6ed71.png 2023-07-11 20:03:03,589 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:03:03,589 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 20:03:03,590 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 20:03:04,079 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:03:04,079 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:03:04,080 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1296_23cc3b9e181168d6ed71.png 2023-07-11 20:03:04,080 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1296_42e08916c70b880ffc82.png 2023-07-11 20:03:04,081 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1296_2a52a9d5fd9d9e666456.png 2023-07-11 20:03:04,082 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1296_db936d1e5ef08167a9b9.png 2023-07-11 20:03:04,082 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 20:03:04,463 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/fiyuv1r6-media/images/validation_1296_42e08916c70b880ffc82.png 2023-07-11 20:03:04,499 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/64uo87e1-media/images/validation_1296_2a52a9d5fd9d9e666456.png 2023-07-11 20:03:04,520 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/j0nnzcrp-media/images/validation_1296_db936d1e5ef08167a9b9.png 2023-07-11 20:03:04,683 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/o7i57s6w-media/images/validation_1296_23cc3b9e181168d6ed71.png 2023-07-11 20:03:05,797 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:03:05,798 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:03:09,279 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:03:09,281 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:03:09,281 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:03:09,283 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:03:10,933 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:03:10,938 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:03:11,093 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:03:11,093 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:03:11,702 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:03:11,704 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:03:11,704 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:03:11,706 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:03:15,014 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:03:15,016 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:03:15,016 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:03:15,018 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:03:16,757 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:03:18,108 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:03:18,109 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:03:18,777 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:03:18,777 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 20:03:18,778 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 20:03:18,905 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:03:18,906 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:03:18,906 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:03:18,908 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:03:22,135 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:03:22,893 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:03:22,894 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:03:22,895 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:03:22,896 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:03:25,123 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:03:25,124 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:03:26,689 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:03:26,690 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:03:26,690 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:03:26,692 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:03:26,969 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:03:26,971 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:03:26,971 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:03:26,973 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:03:27,392 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:03:32,137 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:03:32,138 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:03:32,530 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:03:33,777 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:03:33,778 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 20:03:33,778 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 20:03:37,815 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:03:39,656 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:03:39,656 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1304_5455e556b43dc90874e7.png with policy now 2023-07-11 20:03:39,697 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:03:39,697 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1304_bb39219de51d2ce5bad3.png with policy now 2023-07-11 20:03:39,698 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:03:39,698 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1304_500c611fcafe0a66ed45.png with policy now 2023-07-11 20:03:39,699 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:03:39,699 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1304_1d25ec0bcdc8cc75bf15.png with policy now 2023-07-11 20:03:39,728 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:03:39,730 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:03:39,730 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:03:39,732 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:03:39,977 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/rkhdwjzg-media/images/validation_1304_5455e556b43dc90874e7.png 2023-07-11 20:03:40,021 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/3sbb8urd-media/images/validation_1304_bb39219de51d2ce5bad3.png 2023-07-11 20:03:40,021 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/3m9h425v-media/images/validation_1304_1d25ec0bcdc8cc75bf15.png 2023-07-11 20:03:40,170 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/hvzmulf5-media/images/validation_1304_500c611fcafe0a66ed45.png 2023-07-11 20:03:40,936 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:03:43,086 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:03:43,087 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:03:45,942 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:03:45,944 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:03:45,945 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:03:45,946 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:03:46,166 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:03:46,167 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:03:46,167 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1304_1d25ec0bcdc8cc75bf15.png 2023-07-11 20:03:46,168 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1304_500c611fcafe0a66ed45.png 2023-07-11 20:03:46,168 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1304_bb39219de51d2ce5bad3.png 2023-07-11 20:03:46,169 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1304_5455e556b43dc90874e7.png 2023-07-11 20:03:46,169 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 20:03:46,506 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/4avnnduc-media/images/validation_1304_500c611fcafe0a66ed45.png 2023-07-11 20:03:46,522 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/6fxhicms-media/images/validation_1304_5455e556b43dc90874e7.png 2023-07-11 20:03:46,743 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/1nonw2wj-media/images/validation_1304_1d25ec0bcdc8cc75bf15.png 2023-07-11 20:03:46,746 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/sfanhfnd-media/images/validation_1304_bb39219de51d2ce5bad3.png 2023-07-11 20:03:48,979 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:03:48,980 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 20:03:48,980 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 20:03:49,023 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:03:50,276 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:03:50,277 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:03:50,277 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:03:50,279 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:03:53,181 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:03:53,181 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:03:53,817 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:03:53,818 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:03:53,819 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:03:53,820 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:03:54,606 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:03:56,856 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:03:56,857 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:03:56,857 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:03:56,859 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:03:59,892 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:04:00,026 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:04:00,027 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:04:00,028 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:04:00,029 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:04:00,195 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:04:00,196 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:04:03,090 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:04:03,091 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:04:03,091 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:04:03,093 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:04:03,374 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:04:03,375 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:04:03,376 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:04:03,377 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:04:03,979 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:04:03,980 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 20:04:03,980 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 20:04:05,040 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:04:07,293 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:04:07,293 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:04:10,041 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:04:10,939 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:04:15,924 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:04:15,925 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1312_8123dc447f99405e7c54.png with policy now 2023-07-11 20:04:15,926 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:04:15,939 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:04:15,939 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1312_3ec5f179893340066c13.png with policy now 2023-07-11 20:04:15,949 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:04:15,949 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1312_c4b40ef60ee62f599312.png with policy now 2023-07-11 20:04:15,952 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:04:15,954 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1312_2f48e342607d482d7214.png with policy now 2023-07-11 20:04:15,994 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:04:15,995 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:04:15,996 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:04:15,997 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:04:16,270 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/chy0zagc-media/images/validation_1312_c4b40ef60ee62f599312.png 2023-07-11 20:04:16,277 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/o83fcezb-media/images/validation_1312_3ec5f179893340066c13.png 2023-07-11 20:04:16,287 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/49uil04v-media/images/validation_1312_8123dc447f99405e7c54.png 2023-07-11 20:04:16,451 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/4m8syc6m-media/images/validation_1312_2f48e342607d482d7214.png 2023-07-11 20:04:18,980 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:04:18,980 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 20:04:18,981 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 20:04:18,987 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:04:21,323 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:04:21,323 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:04:21,324 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1312_3ec5f179893340066c13.png 2023-07-11 20:04:21,324 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1312_2f48e342607d482d7214.png 2023-07-11 20:04:21,325 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1312_8123dc447f99405e7c54.png 2023-07-11 20:04:21,325 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1312_c4b40ef60ee62f599312.png 2023-07-11 20:04:21,326 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 20:04:21,676 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:04:21,680 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/bex5a4sk-media/images/validation_1312_8123dc447f99405e7c54.png 2023-07-11 20:04:21,684 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/joe8dv1n-media/images/validation_1312_2f48e342607d482d7214.png 2023-07-11 20:04:21,688 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/9jzbnzto-media/images/validation_1312_3ec5f179893340066c13.png 2023-07-11 20:04:21,715 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ngw385t6-media/images/validation_1312_c4b40ef60ee62f599312.png 2023-07-11 20:04:22,493 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:04:22,495 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:04:22,496 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:04:22,497 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:04:25,817 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:04:25,819 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:04:25,819 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:04:25,821 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:04:26,704 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:04:28,333 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:04:28,333 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:04:29,014 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:04:29,015 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:04:29,016 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:04:29,017 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:04:31,762 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:04:31,763 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:04:31,763 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:04:31,764 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:04:31,765 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:04:33,980 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:04:33,981 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 20:04:33,981 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 20:04:34,884 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:04:34,886 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:04:34,886 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:04:34,888 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:04:35,348 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:04:35,348 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:04:37,347 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:04:37,995 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:04:37,997 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:04:37,997 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:04:37,999 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:04:38,280 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:04:38,282 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:04:38,282 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:04:38,284 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:04:40,942 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:04:42,361 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:04:42,362 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:04:42,944 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:04:47,945 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:04:48,981 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:04:48,981 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 20:04:48,982 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 20:04:51,051 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:04:51,051 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1320_9444d2a3d4d06dc520ad.png with policy now 2023-07-11 20:04:51,061 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:04:51,061 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1320_808df0a2d3159d683778.png with policy now 2023-07-11 20:04:51,070 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:04:51,071 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1320_8cc1e0a2f932c625950d.png with policy now 2023-07-11 20:04:51,079 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:04:51,079 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1320_bc6cc209488d5a0dd512.png with policy now 2023-07-11 20:04:51,120 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:04:51,121 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:04:51,121 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:04:51,123 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:04:51,537 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/vivcglvs-media/images/validation_1320_bc6cc209488d5a0dd512.png 2023-07-11 20:04:51,541 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/wuuq3g9s-media/images/validation_1320_808df0a2d3159d683778.png 2023-07-11 20:04:51,565 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/uzg44trn-media/images/validation_1320_8cc1e0a2f932c625950d.png 2023-07-11 20:04:51,565 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/4bmqtr6s-media/images/validation_1320_9444d2a3d4d06dc520ad.png 2023-07-11 20:04:53,076 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:04:55,157 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:04:56,391 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:04:56,391 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:04:56,392 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1320_8cc1e0a2f932c625950d.png 2023-07-11 20:04:56,392 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1320_9444d2a3d4d06dc520ad.png 2023-07-11 20:04:56,393 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1320_bc6cc209488d5a0dd512.png 2023-07-11 20:04:56,393 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1320_808df0a2d3159d683778.png 2023-07-11 20:04:56,394 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 20:04:56,749 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/a5n0wj19-media/images/validation_1320_bc6cc209488d5a0dd512.png 2023-07-11 20:04:56,758 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/n98b2ds3-media/images/validation_1320_8cc1e0a2f932c625950d.png 2023-07-11 20:04:56,774 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/sbhrdkzp-media/images/validation_1320_808df0a2d3159d683778.png 2023-07-11 20:04:56,999 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ei9854e5-media/images/validation_1320_9444d2a3d4d06dc520ad.png 2023-07-11 20:04:57,585 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:04:57,586 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:04:57,587 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:04:57,588 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:04:58,519 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:05:01,481 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:05:01,483 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:05:01,483 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:05:01,485 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:05:03,405 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:05:03,405 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:05:03,878 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:05:03,981 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 20:05:03,981 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 20:05:04,762 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:05:04,763 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:05:04,764 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:05:04,765 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:05:08,110 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:05:08,112 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:05:08,113 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:05:08,215 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:05:09,097 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:05:10,419 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:05:10,420 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:05:10,945 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:05:10,963 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:05:10,965 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:05:10,966 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:05:10,967 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:05:13,838 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:05:13,839 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:05:13,840 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:05:13,841 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:05:14,121 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:05:14,123 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:05:14,123 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:05:14,124 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:05:14,126 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:05:17,433 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:05:17,434 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:05:18,995 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:05:18,995 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 20:05:18,996 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 20:05:20,043 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:05:25,045 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:05:26,715 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:05:26,715 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1328_d5ddf8aec98ceaf31a6d.png with policy now 2023-07-11 20:05:26,725 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:05:26,725 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1328_2a8e50b9a2f0ea3c709b.png with policy now 2023-07-11 20:05:26,734 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:05:26,735 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1328_3644a270a89ab0e7f320.png with policy now 2023-07-11 20:05:26,744 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:05:26,745 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1328_66a0e97f0510c0041e71.png with policy now 2023-07-11 20:05:26,790 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:05:26,792 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:05:26,792 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:05:26,794 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:05:27,040 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/e4mvxwq5-media/images/validation_1328_d5ddf8aec98ceaf31a6d.png 2023-07-11 20:05:27,052 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/9w15b9ll-media/images/validation_1328_66a0e97f0510c0041e71.png 2023-07-11 20:05:27,064 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/gutb7jw0-media/images/validation_1328_2a8e50b9a2f0ea3c709b.png 2023-07-11 20:05:27,071 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/67ihffc2-media/images/validation_1328_3644a270a89ab0e7f320.png 2023-07-11 20:05:30,517 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:05:30,518 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:05:31,463 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:05:31,464 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:05:31,464 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1328_2a8e50b9a2f0ea3c709b.png 2023-07-11 20:05:31,465 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1328_d5ddf8aec98ceaf31a6d.png 2023-07-11 20:05:31,465 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1328_66a0e97f0510c0041e71.png 2023-07-11 20:05:31,466 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1328_3644a270a89ab0e7f320.png 2023-07-11 20:05:31,466 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 20:05:31,806 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/npl3y3ly-media/images/validation_1328_66a0e97f0510c0041e71.png 2023-07-11 20:05:31,817 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/r62flqdw-media/images/validation_1328_2a8e50b9a2f0ea3c709b.png 2023-07-11 20:05:31,819 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/l7h5h14t-media/images/validation_1328_d5ddf8aec98ceaf31a6d.png 2023-07-11 20:05:31,841 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/2ynubs3z-media/images/validation_1328_3644a270a89ab0e7f320.png 2023-07-11 20:05:34,038 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:05:34,038 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 20:05:34,039 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 20:05:34,168 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:05:34,170 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:05:34,170 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:05:34,172 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:05:35,699 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:05:37,689 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:05:37,691 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:05:37,691 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:05:37,693 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:05:38,477 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:05:38,478 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:05:40,947 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:05:40,952 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:05:41,142 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:05:41,144 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:05:41,145 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:05:41,146 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:05:44,315 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:05:44,317 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:05:44,317 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:05:44,318 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:05:45,492 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:05:45,492 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:05:46,480 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:05:47,568 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:05:47,569 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:05:47,570 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:05:47,571 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:05:49,306 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:05:49,306 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 20:05:49,307 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 20:05:51,183 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:05:51,185 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:05:51,185 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:05:51,467 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:05:51,576 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:05:51,577 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:05:51,577 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:05:51,577 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:05:51,580 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:05:52,505 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:05:52,506 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:05:56,997 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:06:01,998 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:06:04,010 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:06:04,010 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1336_1e36f4db41c7e9a383a2.png with policy now 2023-07-11 20:06:04,025 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:06:04,025 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1336_bd08d38f68d5827ddd30.png with policy now 2023-07-11 20:06:04,029 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:06:04,029 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1336_1c736decbf969e547f0e.png with policy now 2023-07-11 20:06:04,042 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:06:04,042 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1336_5c456d466b0c671052e3.png with policy now 2023-07-11 20:06:04,079 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:06:04,080 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:06:04,081 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:06:04,082 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:06:04,306 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 20:06:04,306 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:06:04,307 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 20:06:04,357 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/hpdygm9w-media/images/validation_1336_1c736decbf969e547f0e.png 2023-07-11 20:06:04,368 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/jyuhhbvf-media/images/validation_1336_1e36f4db41c7e9a383a2.png 2023-07-11 20:06:04,372 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/eyx4napp-media/images/validation_1336_5c456d466b0c671052e3.png 2023-07-11 20:06:04,387 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/9871eovz-media/images/validation_1336_bd08d38f68d5827ddd30.png 2023-07-11 20:06:06,537 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:06:06,538 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:06:06,538 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1336_1c736decbf969e547f0e.png 2023-07-11 20:06:06,538 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1336_1e36f4db41c7e9a383a2.png 2023-07-11 20:06:06,539 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1336_bd08d38f68d5827ddd30.png 2023-07-11 20:06:06,539 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1336_5c456d466b0c671052e3.png 2023-07-11 20:06:06,540 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 20:06:06,862 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/v83kk6s0-media/images/validation_1336_1c736decbf969e547f0e.png 2023-07-11 20:06:06,867 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/998chh6o-media/images/validation_1336_1e36f4db41c7e9a383a2.png 2023-07-11 20:06:06,891 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/wt5007rv-media/images/validation_1336_5c456d466b0c671052e3.png 2023-07-11 20:06:06,896 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/4ztcaqml-media/images/validation_1336_bd08d38f68d5827ddd30.png 2023-07-11 20:06:07,823 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:06:07,824 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:06:10,695 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:06:10,697 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:06:10,697 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:06:10,699 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:06:10,951 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:06:13,552 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:06:13,553 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:06:13,616 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:06:14,621 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:06:14,622 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:06:14,623 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:06:14,624 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:06:17,804 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:06:17,806 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:06:17,807 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:06:17,808 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:06:18,782 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:06:19,306 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 20:06:19,307 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:06:19,307 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 20:06:20,568 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:06:20,569 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:06:21,391 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:06:21,393 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:06:21,394 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:06:21,395 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:06:23,798 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:06:24,515 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:06:24,517 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:06:24,517 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:06:24,519 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:06:27,579 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:06:27,580 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:06:27,582 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:06:27,584 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:06:27,585 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:06:27,586 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:06:27,866 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:06:27,868 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:06:27,869 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:06:27,870 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:06:29,493 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:06:34,307 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:06:34,307 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 20:06:34,308 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 20:06:34,594 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:06:34,594 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:06:35,402 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:06:40,403 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:06:40,492 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:06:40,492 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1344_fdc245de4607a1ec4488.png with policy now 2023-07-11 20:06:40,505 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:06:40,505 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1344_365a301a6ed3b1ea3869.png with policy now 2023-07-11 20:06:40,516 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:06:40,517 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1344_48edd71e0c11119f6254.png with policy now 2023-07-11 20:06:40,526 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:06:40,526 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1344_1b37e5405be2f6007744.png with policy now 2023-07-11 20:06:40,563 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:06:40,572 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:06:40,573 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:06:40,576 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:06:40,827 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/uai1i091-media/images/validation_1344_fdc245de4607a1ec4488.png 2023-07-11 20:06:40,846 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/1ubjrb09-media/images/validation_1344_48edd71e0c11119f6254.png 2023-07-11 20:06:40,854 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/wwj7qkdo-media/images/validation_1344_1b37e5405be2f6007744.png 2023-07-11 20:06:40,866 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/z19c47ot-media/images/validation_1344_365a301a6ed3b1ea3869.png 2023-07-11 20:06:40,953 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:06:41,610 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:06:41,610 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1344_48edd71e0c11119f6254.png 2023-07-11 20:06:41,611 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1344_365a301a6ed3b1ea3869.png 2023-07-11 20:06:41,612 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1344_fdc245de4607a1ec4488.png 2023-07-11 20:06:41,612 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1344_1b37e5405be2f6007744.png 2023-07-11 20:06:41,612 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 20:06:41,932 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/witzd3e7-media/images/validation_1344_48edd71e0c11119f6254.png 2023-07-11 20:06:41,955 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/t5vkv37e-media/images/validation_1344_fdc245de4607a1ec4488.png 2023-07-11 20:06:41,963 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/0d7xigyd-media/images/validation_1344_365a301a6ed3b1ea3869.png 2023-07-11 20:06:42,312 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/y2r5bdka-media/images/validation_1344_1b37e5405be2f6007744.png 2023-07-11 20:06:44,007 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:06:45,749 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:06:47,377 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:06:47,378 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:06:47,379 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:06:47,381 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:06:48,625 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:06:48,625 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:06:49,307 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 20:06:49,307 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:06:49,308 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 20:06:50,491 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:06:50,493 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:06:50,494 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:06:50,495 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:06:51,248 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:06:53,987 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:06:53,988 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:06:53,989 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:06:53,991 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:06:55,641 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:06:55,641 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:06:56,856 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:06:56,858 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:06:56,858 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:06:56,858 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:06:56,860 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:06:59,942 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:06:59,944 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:06:59,944 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:06:59,946 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:07:01,895 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:07:02,655 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:07:02,656 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:07:03,634 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:07:03,636 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:07:03,636 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:07:03,638 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:07:03,918 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:07:03,920 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:07:03,920 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:07:03,922 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:07:04,307 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 20:07:04,307 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 20:07:07,678 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:07:09,670 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:07:09,670 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:07:10,956 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:07:12,958 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:07:16,685 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:07:16,685 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1352_4092798bf460bac8f7a8.png with policy now 2023-07-11 20:07:16,695 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:07:16,695 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1352_8fd1600004e6bf64927d.png with policy now 2023-07-11 20:07:16,705 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:07:16,705 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1352_eca533871e5a572dfd60.png with policy now 2023-07-11 20:07:16,714 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:07:16,714 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1352_9573be96ed0d456f9e97.png with policy now 2023-07-11 20:07:16,758 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:07:16,760 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:07:16,760 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:07:16,762 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:07:17,002 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/c6ebsmrh-media/images/validation_1352_4092798bf460bac8f7a8.png 2023-07-11 20:07:17,047 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/w85dv6v1-media/images/validation_1352_9573be96ed0d456f9e97.png 2023-07-11 20:07:17,070 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/t84ldt03-media/images/validation_1352_eca533871e5a572dfd60.png 2023-07-11 20:07:17,131 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/4yohbllf-media/images/validation_1352_8fd1600004e6bf64927d.png 2023-07-11 20:07:18,120 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:07:19,307 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 20:07:19,307 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:07:19,308 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 20:07:20,825 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:07:23,699 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:07:23,700 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:07:23,700 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1352_4092798bf460bac8f7a8.png 2023-07-11 20:07:23,701 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1352_8fd1600004e6bf64927d.png 2023-07-11 20:07:23,701 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1352_eca533871e5a572dfd60.png 2023-07-11 20:07:23,702 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1352_9573be96ed0d456f9e97.png 2023-07-11 20:07:23,702 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 20:07:23,717 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:07:24,056 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/hs6irrru-media/images/validation_1352_8fd1600004e6bf64927d.png 2023-07-11 20:07:24,056 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/6ma4ye9b-media/images/validation_1352_eca533871e5a572dfd60.png 2023-07-11 20:07:24,059 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/kd6ntrw1-media/images/validation_1352_4092798bf460bac8f7a8.png 2023-07-11 20:07:24,082 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/vfzfq557-media/images/validation_1352_9573be96ed0d456f9e97.png 2023-07-11 20:07:24,919 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:07:24,921 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:07:24,921 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:07:24,923 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:07:28,148 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:07:28,150 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:07:28,150 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:07:28,152 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:07:28,832 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:07:30,714 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:07:30,714 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:07:31,057 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:07:31,058 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:07:31,059 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:07:31,060 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:07:34,330 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:07:34,330 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 20:07:34,331 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 20:07:34,381 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:07:34,459 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:07:34,460 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:07:34,460 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:07:34,462 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:07:37,727 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:07:37,728 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:07:38,050 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:07:38,052 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:07:38,053 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:07:38,054 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:07:40,000 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:07:40,959 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:07:41,681 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:07:41,682 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:07:41,683 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:07:41,684 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:07:41,974 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:07:41,976 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:07:41,977 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:07:41,978 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:07:44,741 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:07:44,742 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:07:45,573 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:07:49,330 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 20:07:49,330 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:07:49,331 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 20:07:51,373 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:07:54,647 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:07:54,647 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1360_61604253d2d4fc950a9b.png with policy now 2023-07-11 20:07:54,662 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:07:54,662 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1360_1346866ddc9966bbc550.png with policy now 2023-07-11 20:07:54,672 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:07:54,672 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1360_905182c9f1a3424d43d9.png with policy now 2023-07-11 20:07:54,681 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:07:54,682 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1360_130544406f3b3910ce09.png with policy now 2023-07-11 20:07:54,718 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:07:54,719 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:07:54,720 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:07:54,721 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:07:54,959 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/p6khbmlh-media/images/validation_1360_61604253d2d4fc950a9b.png 2023-07-11 20:07:55,002 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/o1rhx6z3-media/images/validation_1360_130544406f3b3910ce09.png 2023-07-11 20:07:55,010 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/xws6hqd4-media/images/validation_1360_905182c9f1a3424d43d9.png 2023-07-11 20:07:55,010 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/5ki0ah9a-media/images/validation_1360_1346866ddc9966bbc550.png 2023-07-11 20:07:57,042 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:07:58,773 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:07:58,774 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:07:58,774 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1360_905182c9f1a3424d43d9.png 2023-07-11 20:07:58,775 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1360_61604253d2d4fc950a9b.png 2023-07-11 20:07:58,775 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1360_130544406f3b3910ce09.png 2023-07-11 20:07:58,776 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1360_1346866ddc9966bbc550.png 2023-07-11 20:07:58,776 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 20:07:58,938 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:07:59,124 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/yq8kij8k-media/images/validation_1360_61604253d2d4fc950a9b.png 2023-07-11 20:07:59,127 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/bpi7rdjq-media/images/validation_1360_905182c9f1a3424d43d9.png 2023-07-11 20:07:59,134 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/fc4c5y3e-media/images/validation_1360_130544406f3b3910ce09.png 2023-07-11 20:07:59,154 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/46ibtg5r-media/images/validation_1360_1346866ddc9966bbc550.png 2023-07-11 20:08:02,426 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:08:02,427 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:08:02,427 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:08:02,428 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:08:02,430 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:08:04,330 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 20:08:04,331 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 20:08:05,529 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:08:05,530 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:08:05,531 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:08:05,532 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:08:05,789 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:08:05,789 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:08:07,579 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:08:08,950 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:08:08,952 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:08:08,952 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:08:08,954 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:08:10,961 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:08:12,803 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:08:12,804 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:08:12,850 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:08:12,851 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:08:12,851 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:08:12,852 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:08:12,853 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:08:16,062 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:08:16,065 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:08:16,065 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:08:16,067 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:08:18,497 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:08:19,128 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:08:19,130 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:08:19,130 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:08:19,131 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:08:19,330 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 20:08:19,330 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 20:08:19,452 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:08:19,454 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:08:19,454 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:08:19,456 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:08:19,818 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:08:19,818 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:08:23,970 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:08:26,833 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:08:28,971 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:08:31,973 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:08:31,973 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1368_26163caf1003b87c0917.png with policy now 2023-07-11 20:08:31,984 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:08:31,984 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1368_ed8be6871a61d2c59c72.png with policy now 2023-07-11 20:08:31,993 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:08:31,993 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1368_8ccd1f2533b3ff46050a.png with policy now 2023-07-11 20:08:32,002 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:08:32,002 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1368_c0a22e845a09ff0cd3c5.png with policy now 2023-07-11 20:08:32,045 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:08:32,047 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:08:32,047 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:08:32,049 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:08:32,294 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/bofuy230-media/images/validation_1368_26163caf1003b87c0917.png 2023-07-11 20:08:32,314 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/h6iltn7o-media/images/validation_1368_ed8be6871a61d2c59c72.png 2023-07-11 20:08:32,319 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/hsanyg3v-media/images/validation_1368_8ccd1f2533b3ff46050a.png 2023-07-11 20:08:32,327 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/m447eh8t-media/images/validation_1368_c0a22e845a09ff0cd3c5.png 2023-07-11 20:08:33,850 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:08:33,850 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1368_ed8be6871a61d2c59c72.png 2023-07-11 20:08:33,851 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1368_8ccd1f2533b3ff46050a.png 2023-07-11 20:08:33,851 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1368_26163caf1003b87c0917.png 2023-07-11 20:08:33,852 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1368_c0a22e845a09ff0cd3c5.png 2023-07-11 20:08:33,852 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 20:08:34,183 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/d03hu4wr-media/images/validation_1368_8ccd1f2533b3ff46050a.png 2023-07-11 20:08:34,196 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:08:34,199 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/owx3px5d-media/images/validation_1368_26163caf1003b87c0917.png 2023-07-11 20:08:34,207 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/non8pl1w-media/images/validation_1368_c0a22e845a09ff0cd3c5.png 2023-07-11 20:08:34,214 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/onv2x2gi-media/images/validation_1368_ed8be6871a61d2c59c72.png 2023-07-11 20:08:34,331 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 20:08:34,331 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:08:34,332 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 20:08:35,850 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:08:38,969 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:08:38,971 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:08:38,971 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:08:38,973 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:08:39,974 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:08:40,865 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:08:40,865 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:08:40,963 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:08:42,512 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:08:42,514 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:08:42,514 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:08:42,516 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:08:45,077 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:08:45,710 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:08:45,712 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:08:45,713 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:08:45,714 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:08:47,880 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:08:47,881 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:08:49,355 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:08:49,356 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 20:08:49,356 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 20:08:49,362 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:08:49,407 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:08:49,407 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:08:49,409 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:08:50,347 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:08:52,730 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:08:52,732 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:08:52,732 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:08:52,734 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:08:54,895 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:08:54,896 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:08:55,427 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:08:55,429 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:08:55,429 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:08:55,430 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:08:55,431 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:08:55,709 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:08:55,711 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:08:55,711 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:08:55,713 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:09:01,359 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:09:01,910 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:09:01,910 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:09:04,387 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:09:04,387 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 20:09:04,387 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 20:09:06,431 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:09:08,349 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:09:08,349 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1376_9731a8706e6ed0046c32.png with policy now 2023-07-11 20:09:08,360 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:09:08,360 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1376_a8c45d7ccc7b5c5f5a87.png with policy now 2023-07-11 20:09:08,370 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:09:08,370 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1376_1bed837a55c2e8de98f6.png with policy now 2023-07-11 20:09:08,381 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:09:08,381 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1376_a1dbd4bbb6ca33baecc1.png with policy now 2023-07-11 20:09:08,448 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:09:08,449 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:09:08,450 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:09:08,451 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:09:08,799 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/5xc1qvr9-media/images/validation_1376_a8c45d7ccc7b5c5f5a87.png 2023-07-11 20:09:08,836 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/xuxh0byc-media/images/validation_1376_9731a8706e6ed0046c32.png 2023-07-11 20:09:08,865 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/olbbfl6x-media/images/validation_1376_a1dbd4bbb6ca33baecc1.png 2023-07-11 20:09:08,926 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:09:08,927 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1376_a8c45d7ccc7b5c5f5a87.png 2023-07-11 20:09:08,927 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1376_9731a8706e6ed0046c32.png 2023-07-11 20:09:08,928 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1376_a1dbd4bbb6ca33baecc1.png 2023-07-11 20:09:08,928 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1376_1bed837a55c2e8de98f6.png 2023-07-11 20:09:08,929 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 20:09:08,958 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/1lwht09j-media/images/validation_1376_1bed837a55c2e8de98f6.png 2023-07-11 20:09:09,313 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/v7gscer7-media/images/validation_1376_9731a8706e6ed0046c32.png 2023-07-11 20:09:09,357 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ambdesu8-media/images/validation_1376_a1dbd4bbb6ca33baecc1.png 2023-07-11 20:09:09,358 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/pt2jpygs-media/images/validation_1376_1bed837a55c2e8de98f6.png 2023-07-11 20:09:09,538 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/gz2f4bb7-media/images/validation_1376_a8c45d7ccc7b5c5f5a87.png 2023-07-11 20:09:10,965 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:09:12,136 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:09:12,137 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:09:15,321 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:09:15,323 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:09:15,323 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:09:15,325 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:09:15,941 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:09:15,942 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:09:17,654 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:09:18,777 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:09:18,779 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:09:18,779 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:09:18,781 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:09:19,387 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 20:09:19,387 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:09:19,387 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 20:09:21,547 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:09:21,549 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:09:21,549 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:09:21,551 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:09:22,957 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:09:22,957 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:09:22,988 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:09:24,410 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:09:24,412 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:09:24,413 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:09:24,414 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:09:27,505 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:09:27,507 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:09:27,507 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:09:27,509 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:09:28,510 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:09:29,972 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:09:29,972 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:09:31,053 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:09:31,054 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:09:31,055 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:09:31,057 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:09:31,335 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:09:31,336 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:09:31,337 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:09:31,338 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:09:33,908 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:09:34,412 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 20:09:34,412 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:09:34,413 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 20:09:36,986 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:09:36,987 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:09:39,460 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:09:40,968 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:09:43,945 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:09:43,945 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1384_52b65704857359088b8a.png with policy now 2023-07-11 20:09:43,960 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:09:43,960 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1384_ca0a6b436b4b86c17ede.png with policy now 2023-07-11 20:09:43,970 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:09:43,970 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1384_a0a8d6a6682689911e9c.png with policy now 2023-07-11 20:09:43,979 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:09:43,979 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1384_91c90db7ff100b331d46.png with policy now 2023-07-11 20:09:44,004 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1384_52b65704857359088b8a.png 2023-07-11 20:09:44,009 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1384_ca0a6b436b4b86c17ede.png 2023-07-11 20:09:44,010 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1384_91c90db7ff100b331d46.png 2023-07-11 20:09:44,010 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1384_a0a8d6a6682689911e9c.png 2023-07-11 20:09:44,011 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 20:09:44,019 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:09:44,022 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:09:44,022 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:09:44,025 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:09:44,254 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/y77d74g6-media/images/validation_1384_52b65704857359088b8a.png 2023-07-11 20:09:44,282 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/hqcxkkva-media/images/validation_1384_a0a8d6a6682689911e9c.png 2023-07-11 20:09:44,286 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/r9ggszhc-media/images/validation_1384_91c90db7ff100b331d46.png 2023-07-11 20:09:44,293 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/rhrqmmld-media/images/validation_1384_ca0a6b436b4b86c17ede.png 2023-07-11 20:09:44,604 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/f000ksk5-media/images/validation_1384_52b65704857359088b8a.png 2023-07-11 20:09:44,641 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/hjrionws-media/images/validation_1384_a0a8d6a6682689911e9c.png 2023-07-11 20:09:44,651 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/pyc8204w-media/images/validation_1384_91c90db7ff100b331d46.png 2023-07-11 20:09:44,983 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/vvxqtbf9-media/images/validation_1384_ca0a6b436b4b86c17ede.png 2023-07-11 20:09:45,026 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:09:47,909 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:09:49,412 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 20:09:49,412 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 20:09:50,058 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:09:51,018 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:09:51,019 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:09:52,239 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:09:52,241 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:09:52,241 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:09:52,243 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:09:55,236 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:09:55,238 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:09:55,238 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:09:55,239 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:09:55,241 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:09:58,033 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:09:58,033 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:09:58,101 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:09:58,103 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:09:58,103 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:09:58,105 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:10:00,370 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:10:01,088 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:10:01,090 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:10:01,090 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:10:01,092 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:10:04,341 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:10:04,343 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:10:04,344 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:10:04,345 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:10:04,412 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 20:10:04,413 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 20:10:05,048 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:10:05,048 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:10:05,796 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:10:07,664 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:10:07,665 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:10:07,666 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:10:07,668 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:10:07,946 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:10:07,948 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:10:07,948 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:10:07,950 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:10:10,970 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:10:10,975 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:10:12,068 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:10:12,070 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:10:15,973 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:10:19,412 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 20:10:19,413 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 20:10:20,485 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:10:20,485 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1392_a4ad4ffc5207f63facbc.png with policy now 2023-07-11 20:10:20,499 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:10:20,499 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1392_eca6ba6691c330e7631e.png with policy now 2023-07-11 20:10:20,509 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:10:20,509 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1392_ca30155883656ac69b8a.png with policy now 2023-07-11 20:10:20,517 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:10:20,517 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1392_f0607d0b692156045759.png with policy now 2023-07-11 20:10:20,555 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:10:20,557 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:10:20,558 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:10:20,559 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:10:20,788 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/o28f2xfs-media/images/validation_1392_a4ad4ffc5207f63facbc.png 2023-07-11 20:10:20,811 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/x0l5628b-media/images/validation_1392_ca30155883656ac69b8a.png 2023-07-11 20:10:20,830 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/wsmv8dts-media/images/validation_1392_eca6ba6691c330e7631e.png 2023-07-11 20:10:20,832 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/g9ul9eva-media/images/validation_1392_f0607d0b692156045759.png 2023-07-11 20:10:21,560 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:10:24,141 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:10:26,101 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:10:26,101 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:10:26,101 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1392_f0607d0b692156045759.png 2023-07-11 20:10:26,102 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1392_ca30155883656ac69b8a.png 2023-07-11 20:10:26,103 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1392_eca6ba6691c330e7631e.png 2023-07-11 20:10:26,103 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1392_a4ad4ffc5207f63facbc.png 2023-07-11 20:10:26,104 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 20:10:26,474 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/i5fwgsa8-media/images/validation_1392_f0607d0b692156045759.png 2023-07-11 20:10:26,488 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/kyqsg8ks-media/images/validation_1392_eca6ba6691c330e7631e.png 2023-07-11 20:10:26,488 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/lje37rhj-media/images/validation_1392_ca30155883656ac69b8a.png 2023-07-11 20:10:26,490 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/2gchjag1-media/images/validation_1392_a4ad4ffc5207f63facbc.png 2023-07-11 20:10:26,917 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:10:26,919 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:10:26,919 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:10:26,919 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:10:26,921 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:10:30,461 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:10:30,463 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:10:30,463 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:10:30,465 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:10:32,285 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:10:33,116 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:10:33,116 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:10:33,700 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:10:33,702 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:10:33,702 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:10:33,704 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:10:34,441 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:10:34,441 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 20:10:34,442 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 20:10:37,822 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:10:37,961 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:10:37,962 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:10:37,963 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:10:37,965 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:10:40,973 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:10:41,129 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:10:41,130 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:10:41,557 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:10:41,559 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:10:41,559 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:10:41,561 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:10:43,199 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:10:44,943 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:10:44,945 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:10:44,945 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:10:44,947 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:10:45,224 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:10:45,226 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:10:45,226 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:10:45,228 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:10:48,789 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:10:49,276 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:10:49,277 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:10:49,441 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 20:10:49,441 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:10:49,442 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 20:10:54,488 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:10:57,816 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:10:57,816 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1400_4c95c187c0312a8c87f2.png with policy now 2023-07-11 20:10:57,830 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:10:57,830 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1400_a6f4367b333bbbefc831.png with policy now 2023-07-11 20:10:57,841 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:10:57,841 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1400_8023481f74cd92744fdd.png with policy now 2023-07-11 20:10:57,843 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:10:57,843 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1400_5bf1b64dcf28d3495823.png with policy now 2023-07-11 20:10:57,884 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:10:57,885 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:10:57,886 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:10:57,887 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:10:58,150 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/cahojutd-media/images/validation_1400_4c95c187c0312a8c87f2.png 2023-07-11 20:10:58,164 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/2l0co5fv-media/images/validation_1400_8023481f74cd92744fdd.png 2023-07-11 20:10:58,176 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/3nzeuw5w-media/images/validation_1400_5bf1b64dcf28d3495823.png 2023-07-11 20:10:58,195 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/k42b1vq9-media/images/validation_1400_a6f4367b333bbbefc831.png 2023-07-11 20:10:59,954 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:11:01,549 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:11:04,578 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:11:04,578 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 20:11:04,579 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 20:11:04,704 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:11:04,705 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:11:04,706 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:11:04,707 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:11:05,308 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:11:05,308 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:11:05,308 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1400_a6f4367b333bbbefc831.png 2023-07-11 20:11:05,309 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1400_5bf1b64dcf28d3495823.png 2023-07-11 20:11:05,310 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1400_4c95c187c0312a8c87f2.png 2023-07-11 20:11:05,310 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1400_8023481f74cd92744fdd.png 2023-07-11 20:11:05,310 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 20:11:05,424 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:11:05,662 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/nvfjztby-media/images/validation_1400_5bf1b64dcf28d3495823.png 2023-07-11 20:11:05,665 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/o9akyzhr-media/images/validation_1400_8023481f74cd92744fdd.png 2023-07-11 20:11:05,676 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/b89hqnmq-media/images/validation_1400_4c95c187c0312a8c87f2.png 2023-07-11 20:11:05,693 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/1ve7inzc-media/images/validation_1400_a6f4367b333bbbefc831.png 2023-07-11 20:11:07,717 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:11:07,719 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:11:07,719 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:11:07,721 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:11:10,976 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:11:10,981 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:11:11,156 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:11:11,158 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:11:11,158 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:11:11,160 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:11:13,325 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:11:13,325 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:11:14,513 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:11:14,515 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:11:14,515 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:11:14,517 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:11:16,479 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:11:17,816 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:11:17,818 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:11:17,818 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:11:17,820 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:11:20,244 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:11:20,244 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 20:11:20,245 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 20:11:21,340 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:11:21,340 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:11:21,451 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:11:21,452 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:11:21,453 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:11:21,454 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:11:21,732 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:11:21,734 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:11:21,734 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:11:21,735 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:11:21,737 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:11:27,323 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:11:29,354 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:11:29,355 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:11:32,325 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:11:34,362 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:11:34,363 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1408_64af9f65d337fc4f7fc0.png with policy now 2023-07-11 20:11:34,377 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:11:34,378 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1408_a9dee32d07729bbe5ab6.png with policy now 2023-07-11 20:11:34,387 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:11:34,387 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1408_673f682e941b2195ef5f.png with policy now 2023-07-11 20:11:34,397 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:11:34,397 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1408_d9062fb94fef4cfadbd2.png with policy now 2023-07-11 20:11:34,431 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:11:34,433 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:11:34,433 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:11:34,435 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:11:34,706 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/0y3w4mgx-media/images/validation_1408_64af9f65d337fc4f7fc0.png 2023-07-11 20:11:34,713 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/31nej6f8-media/images/validation_1408_a9dee32d07729bbe5ab6.png 2023-07-11 20:11:34,715 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/yhc4fwwq-media/images/validation_1408_673f682e941b2195ef5f.png 2023-07-11 20:11:34,726 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/hg0g3fvs-media/images/validation_1408_d9062fb94fef4cfadbd2.png 2023-07-11 20:11:35,244 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:11:35,245 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 20:11:35,245 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 20:11:37,371 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:11:37,371 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:11:37,372 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1408_d9062fb94fef4cfadbd2.png 2023-07-11 20:11:37,372 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1408_64af9f65d337fc4f7fc0.png 2023-07-11 20:11:37,373 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1408_a9dee32d07729bbe5ab6.png 2023-07-11 20:11:37,373 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1408_673f682e941b2195ef5f.png 2023-07-11 20:11:37,374 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 20:11:37,578 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:11:37,709 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ekqh38cp-media/images/validation_1408_d9062fb94fef4cfadbd2.png 2023-07-11 20:11:37,742 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/wgxc9733-media/images/validation_1408_a9dee32d07729bbe5ab6.png 2023-07-11 20:11:37,744 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/nfp33edu-media/images/validation_1408_673f682e941b2195ef5f.png 2023-07-11 20:11:37,757 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/2naynpbv-media/images/validation_1408_64af9f65d337fc4f7fc0.png 2023-07-11 20:11:38,491 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:11:40,979 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:11:41,826 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:11:41,827 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:11:41,827 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:11:41,829 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:11:43,252 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:11:44,943 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:11:44,945 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:11:44,945 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:11:44,947 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:11:45,388 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:11:45,388 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:11:47,910 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:11:47,912 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:11:47,912 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:11:47,914 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:11:48,903 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:11:50,423 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:11:50,423 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 20:11:50,424 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 20:11:51,768 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:11:51,770 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:11:51,770 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:11:51,772 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:11:53,403 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:11:53,404 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:11:54,338 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:11:55,232 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:11:55,234 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:11:55,234 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:11:55,236 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:11:58,688 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:11:58,690 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:11:58,690 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:11:58,692 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:11:58,972 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:11:58,973 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:11:58,974 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:11:58,975 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:11:59,370 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:12:01,418 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:12:01,418 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:12:04,505 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:12:05,461 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:12:05,461 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 20:12:05,462 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 20:12:09,742 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:12:10,982 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:12:14,984 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:12:19,985 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:12:20,461 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 20:12:20,461 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:12:20,462 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 20:12:25,507 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:12:30,508 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:12:35,461 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 20:12:35,461 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:12:35,462 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 20:12:35,530 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:12:40,531 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:12:40,985 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:12:45,987 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:12:50,461 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 20:12:50,462 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 20:12:51,507 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:12:56,508 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:13:01,509 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:13:05,461 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 20:13:05,462 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 20:13:07,505 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:13:10,989 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:13:12,991 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:13:17,992 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:13:20,462 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 20:13:20,462 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 20:13:23,505 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:13:28,506 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:13:33,508 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:13:35,462 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 20:13:35,462 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 20:13:39,502 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:13:40,990 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:13:44,993 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:13:49,994 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:13:50,462 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 20:13:50,463 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 20:13:55,508 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:14:00,509 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:14:05,462 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 20:14:05,463 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 20:14:06,509 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:14:10,993 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:14:11,995 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:14:16,996 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:14:20,462 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 20:14:20,463 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 20:14:22,509 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:14:27,510 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:14:32,511 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:14:35,463 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 20:14:35,463 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 20:14:37,633 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:14:40,996 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:14:42,998 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:14:47,999 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:14:50,463 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 20:14:50,463 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 20:14:53,524 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:14:58,525 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:15:03,525 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:15:05,463 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 20:15:05,464 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 20:15:09,523 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:15:10,999 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:15:13,722 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:15:13,722 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1416_42fdf771bb26d71377ce.png with policy now 2023-07-11 20:15:13,736 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:15:13,736 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1416_3aead0846726e7166380.png with policy now 2023-07-11 20:15:13,746 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:15:13,746 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1416_8e95d8013f7d1cb53ee7.png with policy now 2023-07-11 20:15:13,755 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:15:13,755 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1416_9a51711877bf807947e8.png with policy now 2023-07-11 20:15:13,791 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:15:13,792 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:15:13,792 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:15:13,794 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:15:14,038 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:15:14,039 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1416_9a51711877bf807947e8.png 2023-07-11 20:15:14,039 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1416_8e95d8013f7d1cb53ee7.png 2023-07-11 20:15:14,040 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1416_42fdf771bb26d71377ce.png 2023-07-11 20:15:14,041 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1416_3aead0846726e7166380.png 2023-07-11 20:15:14,041 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 20:15:14,060 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/yll05oof-media/images/validation_1416_42fdf771bb26d71377ce.png 2023-07-11 20:15:14,068 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/2y8nwge5-media/images/validation_1416_8e95d8013f7d1cb53ee7.png 2023-07-11 20:15:14,089 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/hn0o2plx-media/images/validation_1416_3aead0846726e7166380.png 2023-07-11 20:15:14,114 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/mkcnc7r3-media/images/validation_1416_9a51711877bf807947e8.png 2023-07-11 20:15:14,470 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/9y0bm9lf-media/images/validation_1416_42fdf771bb26d71377ce.png 2023-07-11 20:15:14,510 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/2ro3qdmv-media/images/validation_1416_3aead0846726e7166380.png 2023-07-11 20:15:14,605 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/fqgtzss4-media/images/validation_1416_8e95d8013f7d1cb53ee7.png 2023-07-11 20:15:14,794 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:15:14,852 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/j1aimabk-media/images/validation_1416_9a51711877bf807947e8.png 2023-07-11 20:15:17,598 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:15:20,161 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:15:20,463 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 20:15:20,464 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:15:20,464 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 20:15:21,234 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:15:21,236 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:15:21,236 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:15:21,238 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:15:22,053 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:15:22,054 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:15:23,686 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:15:23,688 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:15:23,688 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:15:23,690 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:15:25,358 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:15:27,073 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:15:27,075 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:15:27,075 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:15:27,090 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:15:30,069 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:15:30,069 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:15:30,589 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:15:30,781 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:15:30,783 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:15:30,783 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:15:30,785 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:15:34,231 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:15:34,232 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:15:34,233 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:15:34,234 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:15:35,464 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 20:15:35,464 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:15:35,465 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 20:15:36,506 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:15:38,084 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:15:38,084 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:15:38,449 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:15:38,451 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:15:38,451 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:15:38,453 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:15:38,732 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:15:38,734 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:15:38,734 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:15:38,735 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:15:41,003 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:15:42,005 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:15:46,099 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:15:46,099 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:15:47,006 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:15:50,464 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:15:50,464 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 20:15:50,465 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 20:15:51,274 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:15:51,275 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1424_7ce5c6417cd370810ef9.png with policy now 2023-07-11 20:15:51,317 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:15:51,317 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1424_53e42d50be50609b55e6.png with policy now 2023-07-11 20:15:51,384 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:15:51,384 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1424_bc80783a000c38fd4ffb.png with policy now 2023-07-11 20:15:51,395 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:15:51,395 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1424_ac49a027c2b42beeb91d.png with policy now 2023-07-11 20:15:51,434 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:15:51,435 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:15:51,436 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:15:51,438 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:15:51,576 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/hoihcx48-media/images/validation_1424_7ce5c6417cd370810ef9.png 2023-07-11 20:15:51,626 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/o2n34xhd-media/images/validation_1424_53e42d50be50609b55e6.png 2023-07-11 20:15:51,710 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/wkw6zeeg-media/images/validation_1424_ac49a027c2b42beeb91d.png 2023-07-11 20:15:51,713 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/6vy17nno-media/images/validation_1424_bc80783a000c38fd4ffb.png 2023-07-11 20:15:52,439 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:15:54,116 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:15:54,116 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:15:54,117 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1424_bc80783a000c38fd4ffb.png 2023-07-11 20:15:54,117 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1424_7ce5c6417cd370810ef9.png 2023-07-11 20:15:54,118 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1424_ac49a027c2b42beeb91d.png 2023-07-11 20:15:54,118 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1424_53e42d50be50609b55e6.png 2023-07-11 20:15:54,119 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 20:15:54,438 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/3xvj9cn6-media/images/validation_1424_bc80783a000c38fd4ffb.png 2023-07-11 20:15:54,460 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/7sty0214-media/images/validation_1424_7ce5c6417cd370810ef9.png 2023-07-11 20:15:54,471 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/8ldbmne2-media/images/validation_1424_ac49a027c2b42beeb91d.png 2023-07-11 20:15:54,487 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/jdv5nlqa-media/images/validation_1424_53e42d50be50609b55e6.png 2023-07-11 20:15:54,703 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:15:57,749 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:15:57,900 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:15:57,902 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:15:57,902 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:15:57,904 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:16:01,049 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:16:01,050 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:16:01,051 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:16:01,052 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:16:02,129 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:16:02,130 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:16:02,803 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:16:04,289 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:16:04,291 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:16:04,291 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:16:04,292 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:16:05,539 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:16:05,539 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 20:16:05,539 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 20:16:07,267 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:16:07,268 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:16:07,269 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:16:07,270 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:16:08,096 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:16:10,143 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:16:10,143 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:16:11,004 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:16:11,315 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:16:11,317 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:16:11,317 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:16:11,318 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:16:13,802 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:16:14,717 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:16:14,719 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:16:14,719 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:16:14,721 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:16:15,000 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:16:15,002 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:16:15,003 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:16:15,004 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:16:18,157 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:16:18,158 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:16:19,568 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:16:20,559 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:16:20,559 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 20:16:20,560 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 20:16:24,595 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:16:27,539 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:16:27,540 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1432_c09b2e9e85aa46b447a1.png with policy now 2023-07-11 20:16:27,581 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:16:27,581 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1432_bd3dcb8ce785525d5487.png with policy now 2023-07-11 20:16:27,582 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:16:27,582 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1432_a5ace0cb3df2538fb498.png with policy now 2023-07-11 20:16:27,582 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:16:27,583 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1432_43ab5d4df5c3f9be760e.png with policy now 2023-07-11 20:16:27,608 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:16:27,610 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:16:27,610 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:16:27,611 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:16:27,851 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/6u33fxu3-media/images/validation_1432_c09b2e9e85aa46b447a1.png 2023-07-11 20:16:27,904 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/34u984jf-media/images/validation_1432_a5ace0cb3df2538fb498.png 2023-07-11 20:16:27,912 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/1rko3p6y-media/images/validation_1432_bd3dcb8ce785525d5487.png 2023-07-11 20:16:27,925 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/eb4s7dkm-media/images/validation_1432_43ab5d4df5c3f9be760e.png 2023-07-11 20:16:29,782 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:16:31,984 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:16:34,187 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:16:34,188 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:16:34,188 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1432_a5ace0cb3df2538fb498.png 2023-07-11 20:16:34,188 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1432_43ab5d4df5c3f9be760e.png 2023-07-11 20:16:34,189 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1432_bd3dcb8ce785525d5487.png 2023-07-11 20:16:34,190 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1432_c09b2e9e85aa46b447a1.png 2023-07-11 20:16:34,190 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 20:16:34,588 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/q0qn1sov-media/images/validation_1432_a5ace0cb3df2538fb498.png 2023-07-11 20:16:34,605 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/1xp31e3z-media/images/validation_1432_c09b2e9e85aa46b447a1.png 2023-07-11 20:16:34,614 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/u915gl6a-media/images/validation_1432_bd3dcb8ce785525d5487.png 2023-07-11 20:16:34,722 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:16:34,724 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:16:34,724 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:16:34,725 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:16:34,778 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/s8s6wqwu-media/images/validation_1432_43ab5d4df5c3f9be760e.png 2023-07-11 20:16:35,565 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:16:35,566 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 20:16:35,567 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 20:16:35,603 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:16:37,823 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:16:37,825 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:16:37,825 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:16:37,827 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:16:41,007 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:16:41,012 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:16:41,595 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:16:41,597 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:16:41,597 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:16:41,599 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:16:42,202 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:16:42,202 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:16:44,897 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:16:44,899 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:16:44,899 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:16:44,973 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:16:46,585 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:16:48,516 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:16:48,518 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:16:48,518 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:16:48,553 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:16:50,217 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:16:50,217 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:16:50,803 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:16:50,803 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 20:16:50,804 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 20:16:51,588 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:16:51,590 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:16:51,591 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:16:51,591 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:16:51,593 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:16:51,870 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:16:51,872 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:16:51,872 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:16:51,895 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:16:57,396 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:16:58,233 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:16:58,233 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:17:02,397 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:17:04,445 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:17:04,445 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1440_c3be6299a9293bc4d02e.png with policy now 2023-07-11 20:17:04,455 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:17:04,456 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1440_e6f44d029048b1c1f306.png with policy now 2023-07-11 20:17:04,464 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:17:04,464 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1440_f4b00c37b355d60cde54.png with policy now 2023-07-11 20:17:04,473 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:17:04,473 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1440_36f9aa3dad42c2857b92.png with policy now 2023-07-11 20:17:04,516 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:17:04,521 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:17:04,522 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:17:04,531 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:17:04,771 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/7vmlqjm7-media/images/validation_1440_e6f44d029048b1c1f306.png 2023-07-11 20:17:04,784 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/zblnyk75-media/images/validation_1440_f4b00c37b355d60cde54.png 2023-07-11 20:17:04,805 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/3qshzb3d-media/images/validation_1440_36f9aa3dad42c2857b92.png 2023-07-11 20:17:04,935 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/fkz9is18-media/images/validation_1440_c3be6299a9293bc4d02e.png 2023-07-11 20:17:05,994 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:17:05,994 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 20:17:05,995 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 20:17:06,250 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:17:06,251 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1440_c3be6299a9293bc4d02e.png 2023-07-11 20:17:06,251 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1440_36f9aa3dad42c2857b92.png 2023-07-11 20:17:06,252 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1440_f4b00c37b355d60cde54.png 2023-07-11 20:17:06,252 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1440_e6f44d029048b1c1f306.png 2023-07-11 20:17:06,253 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 20:17:06,585 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/bt3s5n5l-media/images/validation_1440_f4b00c37b355d60cde54.png 2023-07-11 20:17:06,612 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/nnbyk873-media/images/validation_1440_36f9aa3dad42c2857b92.png 2023-07-11 20:17:06,627 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/aia8q07s-media/images/validation_1440_e6f44d029048b1c1f306.png 2023-07-11 20:17:06,647 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/gquleeyv-media/images/validation_1440_c3be6299a9293bc4d02e.png 2023-07-11 20:17:07,621 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:17:08,664 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:17:11,010 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:17:12,144 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:17:12,146 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:17:12,146 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:17:12,148 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:17:13,149 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:17:14,270 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:17:14,271 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:17:15,491 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:17:15,492 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:17:15,493 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:17:15,850 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:17:18,604 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:17:18,786 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:17:18,787 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:17:18,788 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:17:18,804 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:17:21,240 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:17:21,240 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 20:17:21,241 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 20:17:22,284 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:17:22,285 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:17:22,975 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:17:22,976 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:17:22,977 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:17:22,979 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:17:24,029 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:17:25,104 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:17:25,105 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:17:25,106 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:17:25,108 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:17:28,245 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:17:28,247 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:17:28,247 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:17:28,257 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:17:28,530 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:17:28,531 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:17:28,532 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:17:28,568 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:17:29,568 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:17:30,300 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:17:30,301 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:17:34,569 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:17:36,240 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:17:36,241 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 20:17:36,242 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 20:17:38,317 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:17:40,369 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:17:41,013 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:17:46,015 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:17:48,471 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:17:48,472 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1448_fa3457d0fce8bba4f078.png with policy now 2023-07-11 20:17:48,575 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:17:48,575 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1448_4314f1796f83f2dc2195.png with policy now 2023-07-11 20:17:48,642 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:17:48,642 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1448_74b0779cd28b2472d148.png with policy now 2023-07-11 20:17:48,693 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:17:48,694 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1448_268b0859bbaca6723e6a.png with policy now 2023-07-11 20:17:48,740 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:17:48,741 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:17:48,742 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:17:48,756 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:17:48,799 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/r7mhq068-media/images/validation_1448_fa3457d0fce8bba4f078.png 2023-07-11 20:17:48,872 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/pnhddde8-media/images/validation_1448_4314f1796f83f2dc2195.png 2023-07-11 20:17:48,962 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/lcsbq03d-media/images/validation_1448_74b0779cd28b2472d148.png 2023-07-11 20:17:49,010 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/aowydm1w-media/images/validation_1448_268b0859bbaca6723e6a.png 2023-07-11 20:17:51,083 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:17:51,241 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:17:51,241 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 20:17:51,241 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 20:17:52,372 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:17:54,348 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:17:54,349 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:17:54,349 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1448_4314f1796f83f2dc2195.png 2023-07-11 20:17:54,349 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1448_fa3457d0fce8bba4f078.png 2023-07-11 20:17:54,350 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1448_268b0859bbaca6723e6a.png 2023-07-11 20:17:54,351 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1448_74b0779cd28b2472d148.png 2023-07-11 20:17:54,351 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 20:17:54,695 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/5ym9sp0t-media/images/validation_1448_fa3457d0fce8bba4f078.png 2023-07-11 20:17:54,706 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/xhc3ng14-media/images/validation_1448_4314f1796f83f2dc2195.png 2023-07-11 20:17:54,706 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/598v6qm3-media/images/validation_1448_268b0859bbaca6723e6a.png 2023-07-11 20:17:54,710 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/q2es1o6y-media/images/validation_1448_74b0779cd28b2472d148.png 2023-07-11 20:17:55,669 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:17:55,671 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:17:55,671 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:17:55,673 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:17:56,591 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:17:59,141 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:17:59,143 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:17:59,144 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:17:59,145 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:18:02,030 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:18:02,362 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:18:02,362 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:18:02,869 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:18:02,871 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:18:02,871 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:18:02,873 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:18:06,099 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:18:06,101 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:18:06,101 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:18:06,103 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:18:06,241 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:18:06,241 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 20:18:06,242 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 20:18:07,070 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:18:10,377 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:18:10,377 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:18:10,468 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:18:10,470 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:18:10,470 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:18:10,472 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:18:11,015 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:18:12,774 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:18:13,402 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:18:13,404 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:18:13,404 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:18:13,406 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:18:13,685 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:18:13,687 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:18:13,687 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:18:13,689 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:18:18,284 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:18:18,392 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:18:18,392 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:18:21,269 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:18:21,269 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 20:18:21,270 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 20:18:23,305 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:18:26,337 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:18:26,337 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1456_1ea2e8f00cd9d7016b4c.png with policy now 2023-07-11 20:18:26,385 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:18:26,385 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1456_2cc115b8258f18162f1a.png with policy now 2023-07-11 20:18:26,420 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1456_2cc115b8258f18162f1a.png 2023-07-11 20:18:26,421 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1456_d34a21e6b56035864f15.png 2023-07-11 20:18:26,421 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1456_1ea2e8f00cd9d7016b4c.png 2023-07-11 20:18:26,422 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 20:18:26,461 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:18:26,461 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1456_d34a21e6b56035864f15.png with policy now 2023-07-11 20:18:26,505 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:18:26,505 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1456_79e985401ee07adaf747.png with policy now 2023-07-11 20:18:26,552 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:18:26,553 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:18:26,557 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:18:26,568 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:18:26,646 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/z6p49mjv-media/images/validation_1456_1ea2e8f00cd9d7016b4c.png 2023-07-11 20:18:26,704 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/b74a0tso-media/images/validation_1456_2cc115b8258f18162f1a.png 2023-07-11 20:18:26,770 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/k9jknshq-media/images/validation_1456_d34a21e6b56035864f15.png 2023-07-11 20:18:26,799 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ff87sh20-media/images/validation_1456_79e985401ee07adaf747.png 2023-07-11 20:18:27,068 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/x8oi1ucc-media/images/validation_1456_1ea2e8f00cd9d7016b4c.png 2023-07-11 20:18:27,153 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/q3unx73z-media/images/validation_1456_2cc115b8258f18162f1a.png 2023-07-11 20:18:27,262 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/n6ablzkv-media/images/validation_1456_d34a21e6b56035864f15.png 2023-07-11 20:18:28,791 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:18:29,362 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:18:32,790 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:18:32,791 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:18:32,792 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:18:32,801 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:18:34,439 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:18:34,440 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:18:34,440 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1456_79e985401ee07adaf747.png 2023-07-11 20:18:34,441 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 20:18:34,655 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:18:34,821 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/d54ld06j-media/images/validation_1456_79e985401ee07adaf747.png 2023-07-11 20:18:36,338 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:18:36,338 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 20:18:36,339 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 20:18:36,508 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:18:36,510 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:18:36,510 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:18:36,512 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:18:39,689 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:18:39,690 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:18:39,691 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:18:39,691 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:18:39,693 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:18:41,018 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:18:42,454 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:18:42,454 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:18:43,850 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:18:43,852 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:18:43,852 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:18:43,854 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:18:45,449 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:18:46,863 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:18:46,865 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:18:46,865 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:18:46,909 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:18:50,024 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:18:50,026 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:18:50,026 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:18:50,035 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:18:50,308 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:18:50,310 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:18:50,311 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:18:50,335 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:18:50,480 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:18:50,480 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:18:50,757 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:18:51,442 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:18:51,442 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 20:18:51,443 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 20:18:56,479 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:18:58,499 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:19:01,480 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:19:03,035 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:19:03,036 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1464_3e948f3d4a9690e3e832.png with policy now 2023-07-11 20:19:03,093 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:19:03,094 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1464_ef2794b66f8354aeb131.png with policy now 2023-07-11 20:19:03,166 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:19:03,166 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1464_19080724f5041cc8877a.png with policy now 2023-07-11 20:19:03,220 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:19:03,220 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1464_0ca40d1889e8c981f83b.png with policy now 2023-07-11 20:19:03,267 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:19:03,268 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:19:03,269 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:19:03,286 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:19:03,358 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/9ylqi6kt-media/images/validation_1464_3e948f3d4a9690e3e832.png 2023-07-11 20:19:03,413 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/3rqz2oio-media/images/validation_1464_ef2794b66f8354aeb131.png 2023-07-11 20:19:03,486 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/njzgk8vq-media/images/validation_1464_19080724f5041cc8877a.png 2023-07-11 20:19:03,530 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/o5b4vixb-media/images/validation_1464_0ca40d1889e8c981f83b.png 2023-07-11 20:19:06,442 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:19:06,442 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 20:19:06,442 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 20:19:06,498 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:19:06,516 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:19:06,516 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:19:06,516 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1464_0ca40d1889e8c981f83b.png 2023-07-11 20:19:06,517 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1464_19080724f5041cc8877a.png 2023-07-11 20:19:06,518 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1464_ef2794b66f8354aeb131.png 2023-07-11 20:19:06,518 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1464_3e948f3d4a9690e3e832.png 2023-07-11 20:19:06,519 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 20:19:06,852 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/pzmpnt8t-media/images/validation_1464_19080724f5041cc8877a.png 2023-07-11 20:19:06,860 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/9gbpmqtl-media/images/validation_1464_0ca40d1889e8c981f83b.png 2023-07-11 20:19:06,873 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/uervv1bv-media/images/validation_1464_3e948f3d4a9690e3e832.png 2023-07-11 20:19:06,910 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/pgb98pjx-media/images/validation_1464_ef2794b66f8354aeb131.png 2023-07-11 20:19:07,374 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:19:10,676 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:19:10,678 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:19:10,678 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:19:10,680 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:19:11,021 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:19:11,683 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:19:14,531 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:19:14,531 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:19:15,043 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:19:15,045 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:19:15,045 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:19:15,046 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:19:16,837 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:19:18,758 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:19:18,760 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:19:18,760 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:19:18,810 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:19:21,442 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:19:21,442 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 20:19:21,443 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 20:19:21,696 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:19:21,698 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:19:21,698 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:19:21,715 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:19:22,419 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:19:22,545 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:19:22,546 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:19:25,035 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:19:25,037 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:19:25,038 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:19:25,039 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:19:27,691 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:19:28,351 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:19:28,353 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:19:28,354 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:19:28,356 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:19:28,634 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:19:28,636 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:19:28,637 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:19:28,653 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:19:30,562 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:19:30,562 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:19:33,247 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:19:36,462 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:19:36,462 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 20:19:36,463 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 20:19:38,502 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:19:41,024 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:19:41,287 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:19:41,287 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1472_e88bd6f6b5daeddb588f.png with policy now 2023-07-11 20:19:41,358 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:19:41,359 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1472_f3df87bdd58ea75008e9.png with policy now 2023-07-11 20:19:41,442 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:19:41,442 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1472_ec9e711d039b6031a67b.png with policy now 2023-07-11 20:19:41,509 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:19:41,510 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1472_97171e5e266bd9576202.png with policy now 2023-07-11 20:19:41,559 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:19:41,561 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:19:41,561 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:19:41,592 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/q4kv96g7-media/images/validation_1472_e88bd6f6b5daeddb588f.png 2023-07-11 20:19:41,600 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:19:41,670 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/vwatwxgq-media/images/validation_1472_f3df87bdd58ea75008e9.png 2023-07-11 20:19:41,757 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ohd510lt-media/images/validation_1472_ec9e711d039b6031a67b.png 2023-07-11 20:19:41,806 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/pi9ank3c-media/images/validation_1472_97171e5e266bd9576202.png 2023-07-11 20:19:44,151 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:19:45,601 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:19:46,592 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:19:46,593 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:19:46,593 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1472_97171e5e266bd9576202.png 2023-07-11 20:19:46,593 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1472_ec9e711d039b6031a67b.png 2023-07-11 20:19:46,594 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1472_f3df87bdd58ea75008e9.png 2023-07-11 20:19:46,595 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1472_e88bd6f6b5daeddb588f.png 2023-07-11 20:19:46,595 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 20:19:46,931 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/7nyz4k9y-media/images/validation_1472_97171e5e266bd9576202.png 2023-07-11 20:19:46,941 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/o8hkhcfx-media/images/validation_1472_ec9e711d039b6031a67b.png 2023-07-11 20:19:46,951 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ciri3k3y-media/images/validation_1472_e88bd6f6b5daeddb588f.png 2023-07-11 20:19:46,973 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ayt2mmwm-media/images/validation_1472_f3df87bdd58ea75008e9.png 2023-07-11 20:19:48,967 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:19:48,969 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:19:48,970 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:19:48,992 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:19:49,589 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:19:51,462 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:19:51,463 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 20:19:51,463 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 20:19:51,723 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:19:51,731 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:19:51,731 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:19:51,762 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:19:54,611 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:19:54,612 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:19:54,873 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:19:55,564 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:19:55,565 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:19:55,566 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:19:55,575 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:19:58,667 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:19:58,669 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:19:58,669 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:19:58,704 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:20:00,103 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:20:01,882 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:20:01,883 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:20:01,884 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:20:01,900 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:20:02,628 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:20:02,629 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:20:05,773 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:20:05,775 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:20:05,775 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:20:05,776 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:20:05,792 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:20:06,056 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:20:06,058 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:20:06,058 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:20:06,088 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:20:06,462 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 20:20:06,462 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 20:20:10,650 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:20:10,650 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:20:11,027 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:20:11,032 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:20:16,029 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:20:18,838 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:20:18,839 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1480_c13f5759d561a664b48e.png with policy now 2023-07-11 20:20:18,929 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:20:18,930 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1480_03ac52985174dcc37f37.png with policy now 2023-07-11 20:20:19,006 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:20:19,006 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1480_b5501bb70ba4c344c2a7.png with policy now 2023-07-11 20:20:19,073 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:20:19,073 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1480_389503b866d34e567119.png with policy now 2023-07-11 20:20:19,120 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:20:19,121 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:20:19,122 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:20:19,138 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:20:19,159 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/jcz9vt6f-media/images/validation_1480_c13f5759d561a664b48e.png 2023-07-11 20:20:19,227 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/01op16ap-media/images/validation_1480_03ac52985174dcc37f37.png 2023-07-11 20:20:19,318 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/az3q73b5-media/images/validation_1480_b5501bb70ba4c344c2a7.png 2023-07-11 20:20:19,390 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/x8jicawt-media/images/validation_1480_389503b866d34e567119.png 2023-07-11 20:20:21,218 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:20:21,462 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 20:20:21,463 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:20:21,463 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 20:20:22,665 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:20:26,815 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:20:26,817 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:20:26,817 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:20:26,817 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:20:27,202 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:20:27,204 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:20:27,205 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:20:27,205 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1480_03ac52985174dcc37f37.png 2023-07-11 20:20:27,206 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1480_389503b866d34e567119.png 2023-07-11 20:20:27,207 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1480_b5501bb70ba4c344c2a7.png 2023-07-11 20:20:27,207 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1480_c13f5759d561a664b48e.png 2023-07-11 20:20:27,208 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 20:20:27,546 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/csxuhs80-media/images/validation_1480_389503b866d34e567119.png 2023-07-11 20:20:27,555 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ggnsa5sr-media/images/validation_1480_b5501bb70ba4c344c2a7.png 2023-07-11 20:20:27,561 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/znmxdohq-media/images/validation_1480_03ac52985174dcc37f37.png 2023-07-11 20:20:27,566 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ctgtajia-media/images/validation_1480_c13f5759d561a664b48e.png 2023-07-11 20:20:29,667 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:20:29,669 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:20:29,669 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:20:29,674 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:20:32,377 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:20:33,150 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:20:33,152 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:20:33,153 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:20:33,179 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:20:35,219 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:20:35,220 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:20:36,463 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 20:20:36,463 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:20:36,463 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 20:20:37,506 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:20:37,527 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:20:37,528 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:20:37,529 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:20:37,546 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:20:40,750 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:20:40,751 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:20:40,752 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:20:40,779 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:20:41,029 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:20:42,945 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:20:43,234 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:20:43,234 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:20:43,595 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:20:43,597 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:20:43,597 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:20:43,617 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:20:43,877 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:20:43,879 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:20:43,879 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:20:43,881 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:20:48,421 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:20:51,254 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:20:51,255 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:20:51,467 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:20:51,467 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 20:20:51,467 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 20:20:53,512 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:20:56,591 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:20:56,591 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1488_d8f7bbf4bace619f94c8.png with policy now 2023-07-11 20:20:56,687 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:20:56,687 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1488_bad17279d6c78d388689.png with policy now 2023-07-11 20:20:56,754 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:20:56,754 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1488_945435b42cdef1f698ac.png with policy now 2023-07-11 20:20:56,823 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:20:56,824 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1488_7a327cfc0f0d14000d66.png with policy now 2023-07-11 20:20:56,870 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:20:56,872 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:20:56,872 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:20:56,915 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:20:56,925 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/jvbnp20s-media/images/validation_1488_d8f7bbf4bace619f94c8.png 2023-07-11 20:20:57,007 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/j7pfax90-media/images/validation_1488_bad17279d6c78d388689.png 2023-07-11 20:20:57,097 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/hv1a390q-media/images/validation_1488_945435b42cdef1f698ac.png 2023-07-11 20:20:57,129 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/pz95rk4u-media/images/validation_1488_7a327cfc0f0d14000d66.png 2023-07-11 20:20:58,792 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:20:59,301 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:20:59,301 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:20:59,302 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1488_945435b42cdef1f698ac.png 2023-07-11 20:20:59,302 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1488_bad17279d6c78d388689.png 2023-07-11 20:20:59,303 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1488_d8f7bbf4bace619f94c8.png 2023-07-11 20:20:59,303 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1488_7a327cfc0f0d14000d66.png 2023-07-11 20:20:59,304 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 20:20:59,642 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/tv3us4nw-media/images/validation_1488_945435b42cdef1f698ac.png 2023-07-11 20:20:59,654 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/3uaf02ol-media/images/validation_1488_d8f7bbf4bace619f94c8.png 2023-07-11 20:20:59,659 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/kbkpo212-media/images/validation_1488_7a327cfc0f0d14000d66.png 2023-07-11 20:20:59,668 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ojeb1uwy-media/images/validation_1488_bad17279d6c78d388689.png 2023-07-11 20:21:00,139 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:21:03,646 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:21:03,648 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:21:03,648 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:21:03,664 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:21:04,484 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:21:06,157 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:21:06,159 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:21:06,159 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:21:06,181 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:21:06,466 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 20:21:06,467 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 20:21:07,320 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:21:07,320 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:21:09,488 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:21:10,254 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:21:10,256 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:21:10,256 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:21:10,272 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:21:11,032 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:21:13,766 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:21:13,768 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:21:13,768 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:21:13,779 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:21:14,504 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:21:15,338 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:21:15,338 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:21:16,863 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:21:16,865 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:21:16,865 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:21:16,885 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:21:19,707 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:21:20,279 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:21:20,281 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:21:20,281 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:21:20,297 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:21:20,560 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:21:20,562 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:21:20,563 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:21:20,572 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:21:21,467 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 20:21:21,467 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 20:21:23,369 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:21:23,369 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:21:25,506 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:21:30,507 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:21:33,347 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:21:33,348 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1496_861caf343730e243e96c.png with policy now 2023-07-11 20:21:33,407 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:21:33,407 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1496_792b9ac80f3c91021509.png with policy now 2023-07-11 20:21:33,477 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:21:33,477 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1496_5999e1ed8ed0b37baa88.png with policy now 2023-07-11 20:21:33,525 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:21:33,525 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1496_f3cbc6942eee44231d8a.png with policy now 2023-07-11 20:21:33,573 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:21:33,574 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:21:33,575 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:21:33,610 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:21:33,679 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/agojoy8d-media/images/validation_1496_861caf343730e243e96c.png 2023-07-11 20:21:33,730 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/023py960-media/images/validation_1496_792b9ac80f3c91021509.png 2023-07-11 20:21:33,795 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ypmb0xks-media/images/validation_1496_5999e1ed8ed0b37baa88.png 2023-07-11 20:21:33,847 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/a2kp24x5-media/images/validation_1496_f3cbc6942eee44231d8a.png 2023-07-11 20:21:35,829 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:21:36,467 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 20:21:36,467 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:21:36,468 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 20:21:36,784 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:21:39,403 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:21:39,403 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:21:39,404 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1496_5999e1ed8ed0b37baa88.png 2023-07-11 20:21:39,404 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1496_f3cbc6942eee44231d8a.png 2023-07-11 20:21:39,405 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1496_861caf343730e243e96c.png 2023-07-11 20:21:39,405 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1496_792b9ac80f3c91021509.png 2023-07-11 20:21:39,406 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 20:21:39,601 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:21:39,603 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:21:39,604 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:21:39,623 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:21:39,743 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/vpjyi8y6-media/images/validation_1496_861caf343730e243e96c.png 2023-07-11 20:21:39,749 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/jjbrhuup-media/images/validation_1496_f3cbc6942eee44231d8a.png 2023-07-11 20:21:39,751 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/5hxwcgz8-media/images/validation_1496_5999e1ed8ed0b37baa88.png 2023-07-11 20:21:39,764 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/4txutqku-media/images/validation_1496_792b9ac80f3c91021509.png 2023-07-11 20:21:41,035 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:21:41,040 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:21:43,075 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:21:43,077 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:21:43,077 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:21:43,092 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:21:46,484 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:21:47,417 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:21:47,418 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:21:47,419 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:21:51,467 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: stop_status 2023-07-11 20:21:56,468 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:22:06,696 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:22:07,461 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:22:07,461 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:22:07,461 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:22:07,461 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:22:07,462 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: stop_status 2023-07-11 20:22:07,462 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:22:10,332 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:22:10,334 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:22:10,335 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:22:10,336 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:22:11,038 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:22:11,728 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:22:13,077 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:22:13,077 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:22:13,078 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:22:13,078 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:22:13,080 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:22:14,711 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:22:14,712 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:22:16,599 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:22:16,601 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:22:16,601 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:22:16,603 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:22:16,880 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:22:16,882 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:22:16,882 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:22:16,884 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:22:17,067 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:22:18,539 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:22:22,100 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:22:22,726 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:22:22,728 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:22:23,540 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:22:27,130 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:22:28,541 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:22:29,520 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:22:29,521 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1504_1629e05d318559a66a72.png with policy now 2023-07-11 20:22:29,531 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:22:29,531 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1504_53154f693252da98516a.png with policy now 2023-07-11 20:22:29,544 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:22:29,545 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1504_fb85857165e5c2b83d35.png with policy now 2023-07-11 20:22:29,555 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:22:29,556 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1504_b30643dac647697eb3e3.png with policy now 2023-07-11 20:22:29,592 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:22:29,595 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:22:29,596 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:22:29,599 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:22:29,836 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ryey5mnf-media/images/validation_1504_1629e05d318559a66a72.png 2023-07-11 20:22:29,854 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/8sdzfuir-media/images/validation_1504_fb85857165e5c2b83d35.png 2023-07-11 20:22:29,859 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/fdcrajbk-media/images/validation_1504_53154f693252da98516a.png 2023-07-11 20:22:29,879 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/z5hux4rz-media/images/validation_1504_b30643dac647697eb3e3.png 2023-07-11 20:22:30,744 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:22:30,745 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1504_1629e05d318559a66a72.png 2023-07-11 20:22:30,745 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1504_53154f693252da98516a.png 2023-07-11 20:22:30,746 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1504_fb85857165e5c2b83d35.png 2023-07-11 20:22:30,746 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1504_b30643dac647697eb3e3.png 2023-07-11 20:22:30,747 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 20:22:31,076 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/xzmf7mcs-media/images/validation_1504_1629e05d318559a66a72.png 2023-07-11 20:22:31,088 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/tnxfjpbo-media/images/validation_1504_fb85857165e5c2b83d35.png 2023-07-11 20:22:31,102 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/l1pszumc-media/images/validation_1504_53154f693252da98516a.png 2023-07-11 20:22:31,132 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/piq5qysm-media/images/validation_1504_b30643dac647697eb3e3.png 2023-07-11 20:22:32,816 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:22:34,353 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:22:34,355 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:22:37,357 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:22:37,359 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:22:37,360 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:22:37,361 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:22:38,654 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:22:38,758 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:22:38,761 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:22:40,151 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:22:40,154 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:22:40,154 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:22:40,154 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:22:40,156 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:22:41,041 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:22:44,393 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:22:44,520 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:22:44,523 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:22:44,523 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:22:44,526 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:22:45,416 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:22:46,774 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:22:46,774 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:22:48,003 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:22:48,004 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:22:48,005 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:22:48,006 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:22:49,877 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:22:50,488 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:22:51,267 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:22:51,269 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:22:51,269 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:22:51,271 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:22:54,196 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:22:54,198 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:22:54,198 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:22:54,200 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:22:54,477 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:22:54,479 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:22:54,480 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:22:54,481 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:22:54,789 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:22:54,790 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:22:55,143 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:22:56,064 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:23:00,159 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:23:01,065 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:23:02,852 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:23:05,198 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:23:06,066 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:23:07,107 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:23:07,108 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1512_76d0a37e65b519005e16.png with policy now 2023-07-11 20:23:07,122 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:23:07,122 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1512_9628924d5a8c1579e38e.png with policy now 2023-07-11 20:23:07,129 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:23:07,129 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1512_ac8f638669bb1938fe1e.png with policy now 2023-07-11 20:23:07,138 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:23:07,139 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1512_dc90df180ea054ba86a6.png with policy now 2023-07-11 20:23:07,177 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:23:07,179 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:23:07,179 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:23:07,181 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:23:07,431 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/2xfe7fsb-media/images/validation_1512_76d0a37e65b519005e16.png 2023-07-11 20:23:07,443 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ihcuc5ms-media/images/validation_1512_dc90df180ea054ba86a6.png 2023-07-11 20:23:07,446 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/c0o8kzhg-media/images/validation_1512_9628924d5a8c1579e38e.png 2023-07-11 20:23:07,448 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/uxeq430z-media/images/validation_1512_ac8f638669bb1938fe1e.png 2023-07-11 20:23:10,479 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:23:10,873 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:23:10,873 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:23:10,874 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1512_ac8f638669bb1938fe1e.png 2023-07-11 20:23:10,875 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1512_76d0a37e65b519005e16.png 2023-07-11 20:23:10,875 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1512_dc90df180ea054ba86a6.png 2023-07-11 20:23:10,876 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1512_9628924d5a8c1579e38e.png 2023-07-11 20:23:10,876 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 20:23:11,016 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:23:11,044 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:23:11,210 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/qhwbkf4i-media/images/validation_1512_ac8f638669bb1938fe1e.png 2023-07-11 20:23:11,223 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/o4s05ffc-media/images/validation_1512_dc90df180ea054ba86a6.png 2023-07-11 20:23:11,245 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/aj0wp28n-media/images/validation_1512_9628924d5a8c1579e38e.png 2023-07-11 20:23:11,266 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/hv5c4hse-media/images/validation_1512_76d0a37e65b519005e16.png 2023-07-11 20:23:11,752 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:23:13,944 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:23:13,946 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:23:13,947 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:23:13,948 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:23:15,888 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:23:16,925 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:23:16,927 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:23:16,927 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:23:16,928 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:23:16,929 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:23:18,891 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:23:18,892 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:23:20,897 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:23:20,899 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:23:20,899 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:23:20,901 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:23:20,920 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:23:22,464 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:23:23,715 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:23:23,717 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:23:23,717 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:23:23,719 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:23:26,639 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:23:26,640 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:23:26,904 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:23:26,904 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:23:27,106 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:23:27,108 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:23:27,108 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:23:27,110 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:23:28,014 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:23:30,360 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:23:30,362 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:23:30,363 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:23:30,364 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:23:30,644 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:23:30,646 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:23:30,646 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:23:30,648 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:23:32,232 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:23:33,225 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:23:34,919 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:23:34,920 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:23:37,247 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:23:38,227 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:23:41,047 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:23:42,285 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:23:43,280 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:23:43,280 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1520_759fd5f584726cd89a08.png with policy now 2023-07-11 20:23:43,281 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:23:43,291 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:23:43,291 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1520_cbf2c6d45cd68fb8ce0d.png with policy now 2023-07-11 20:23:43,305 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:23:43,305 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1520_da08c97000f72b3ea979.png with policy now 2023-07-11 20:23:43,315 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:23:43,315 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1520_4acf2882ee3fdf5b3a10.png with policy now 2023-07-11 20:23:43,381 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:23:43,382 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:23:43,383 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:23:43,384 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:23:43,592 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/i069k5d0-media/images/validation_1520_759fd5f584726cd89a08.png 2023-07-11 20:23:43,611 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/puy5ox41-media/images/validation_1520_cbf2c6d45cd68fb8ce0d.png 2023-07-11 20:23:43,631 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/h55oh791-media/images/validation_1520_da08c97000f72b3ea979.png 2023-07-11 20:23:43,656 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/zx96s2cm-media/images/validation_1520_4acf2882ee3fdf5b3a10.png 2023-07-11 20:23:47,624 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:23:47,630 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:23:48,305 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:23:50,888 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:23:50,890 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:23:50,890 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:23:50,892 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:23:50,952 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:23:50,953 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:23:50,953 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1520_759fd5f584726cd89a08.png 2023-07-11 20:23:50,953 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1520_cbf2c6d45cd68fb8ce0d.png 2023-07-11 20:23:50,954 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1520_4acf2882ee3fdf5b3a10.png 2023-07-11 20:23:50,955 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1520_da08c97000f72b3ea979.png 2023-07-11 20:23:50,955 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 20:23:51,316 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/sr6ochd3-media/images/validation_1520_759fd5f584726cd89a08.png 2023-07-11 20:23:51,329 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/uvap0w5x-media/images/validation_1520_cbf2c6d45cd68fb8ce0d.png 2023-07-11 20:23:51,338 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/nxalyy4n-media/images/validation_1520_da08c97000f72b3ea979.png 2023-07-11 20:23:51,344 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/irpp4wp6-media/images/validation_1520_4acf2882ee3fdf5b3a10.png 2023-07-11 20:23:52,775 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:23:53,362 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:23:53,937 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:23:53,939 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:23:53,940 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:23:53,941 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:23:57,042 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:23:57,043 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:23:57,043 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:23:57,045 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:23:58,361 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:23:58,485 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:23:58,969 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:23:58,969 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:24:00,387 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:24:00,389 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:24:00,389 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:24:00,391 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:24:04,111 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:24:04,489 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:24:04,615 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:24:04,617 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:24:04,617 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:24:04,619 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:24:06,982 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:24:06,983 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:24:07,894 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:24:07,896 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:24:07,897 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:24:07,898 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:24:08,176 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:24:08,178 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:24:08,178 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:24:08,180 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:24:09,784 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:24:10,123 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:24:10,123 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:24:11,049 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:24:14,997 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:24:14,997 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:24:15,052 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:24:15,155 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:24:20,053 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:24:20,192 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:24:20,866 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:24:20,866 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1528_27dd1112c1ee345cd5a7.png with policy now 2023-07-11 20:24:20,877 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:24:20,878 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1528_175b3055df42cae41485.png with policy now 2023-07-11 20:24:20,891 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:24:20,891 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1528_823ba070d344cad76878.png with policy now 2023-07-11 20:24:20,901 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:24:20,901 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1528_2e0f5eda4c827c82d01f.png with policy now 2023-07-11 20:24:20,965 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:24:20,967 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:24:20,968 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:24:20,970 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:24:21,178 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/96p01u4m-media/images/validation_1528_27dd1112c1ee345cd5a7.png 2023-07-11 20:24:21,192 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/5i8kkbu8-media/images/validation_1528_823ba070d344cad76878.png 2023-07-11 20:24:21,194 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ll83jotn-media/images/validation_1528_2e0f5eda4c827c82d01f.png 2023-07-11 20:24:21,296 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/nizu3dec-media/images/validation_1528_175b3055df42cae41485.png 2023-07-11 20:24:23,015 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:24:23,015 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1528_175b3055df42cae41485.png 2023-07-11 20:24:23,016 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1528_27dd1112c1ee345cd5a7.png 2023-07-11 20:24:23,017 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1528_2e0f5eda4c827c82d01f.png 2023-07-11 20:24:23,017 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1528_823ba070d344cad76878.png 2023-07-11 20:24:23,018 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 20:24:23,376 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/z8nx79rf-media/images/validation_1528_175b3055df42cae41485.png 2023-07-11 20:24:23,392 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ppzbvv9c-media/images/validation_1528_27dd1112c1ee345cd5a7.png 2023-07-11 20:24:23,396 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/wvb2p3mr-media/images/validation_1528_2e0f5eda4c827c82d01f.png 2023-07-11 20:24:23,400 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/er5q2y8v-media/images/validation_1528_823ba070d344cad76878.png 2023-07-11 20:24:24,611 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:24:25,613 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:24:25,790 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:24:28,295 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:24:28,297 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:24:28,298 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:24:28,299 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:24:30,650 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:24:30,818 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:24:31,031 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:24:31,032 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:24:32,081 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:24:32,083 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:24:32,083 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:24:32,085 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:24:35,330 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:24:35,331 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:24:35,331 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:24:35,334 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:24:36,058 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:24:36,178 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:24:38,607 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:24:38,609 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:24:38,609 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:24:38,611 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:24:39,047 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:24:39,047 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:24:41,052 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:24:41,989 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:24:41,989 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:24:42,054 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:24:42,117 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:24:42,118 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:24:42,119 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:24:42,120 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:24:45,117 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:24:45,118 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:24:45,119 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:24:45,120 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:24:45,400 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:24:45,402 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:24:45,403 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:24:45,404 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:24:46,998 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:24:47,062 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:24:47,063 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:24:47,958 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:24:52,049 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:24:52,959 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:24:55,076 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:24:57,084 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:24:57,960 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:24:57,992 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:24:57,992 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1536_08a161065bf6a4856204.png with policy now 2023-07-11 20:24:58,006 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:24:58,006 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1536_d45ac8dd546b5c439b82.png with policy now 2023-07-11 20:24:58,016 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:24:58,017 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1536_71bad97180edd0fb5ddd.png with policy now 2023-07-11 20:24:58,024 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:24:58,024 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1536_3fc0a6e0fd102232715a.png with policy now 2023-07-11 20:24:58,062 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:24:58,063 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:24:58,064 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:24:58,065 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:24:58,410 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/l843n88w-media/images/validation_1536_08a161065bf6a4856204.png 2023-07-11 20:24:58,419 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/m6lchcji-media/images/validation_1536_71bad97180edd0fb5ddd.png 2023-07-11 20:24:58,462 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ln7jivfb-media/images/validation_1536_3fc0a6e0fd102232715a.png 2023-07-11 20:24:58,514 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/bdza48mc-media/images/validation_1536_d45ac8dd546b5c439b82.png 2023-07-11 20:25:01,904 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:25:02,854 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:25:02,977 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:25:03,095 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:25:03,095 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:25:03,096 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1536_d45ac8dd546b5c439b82.png 2023-07-11 20:25:03,096 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1536_08a161065bf6a4856204.png 2023-07-11 20:25:03,097 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1536_3fc0a6e0fd102232715a.png 2023-07-11 20:25:03,097 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1536_71bad97180edd0fb5ddd.png 2023-07-11 20:25:03,098 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 20:25:03,480 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/xtncgzyl-media/images/validation_1536_d45ac8dd546b5c439b82.png 2023-07-11 20:25:03,492 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/lvcflv26-media/images/validation_1536_71bad97180edd0fb5ddd.png 2023-07-11 20:25:03,499 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/skfbgn3i-media/images/validation_1536_08a161065bf6a4856204.png 2023-07-11 20:25:03,506 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/vdtl5x9b-media/images/validation_1536_3fc0a6e0fd102232715a.png 2023-07-11 20:25:05,708 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:25:05,709 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:25:05,709 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:25:05,711 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:25:08,098 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:25:08,200 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:25:08,200 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:25:08,782 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:25:08,784 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:25:08,784 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:25:08,786 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:25:11,055 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:25:11,110 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:25:11,111 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:25:12,136 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:25:12,138 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:25:12,138 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:25:12,140 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:25:13,208 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:25:13,513 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:25:15,470 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:25:15,472 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:25:15,472 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:25:15,474 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:25:18,342 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:25:18,548 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:25:18,550 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:25:18,550 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:25:18,550 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:25:18,552 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:25:19,123 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:25:19,124 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:25:22,483 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:25:22,484 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:25:22,485 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:25:22,487 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:25:22,764 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:25:22,766 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:25:22,766 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:25:22,768 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:25:23,358 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:25:24,349 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:25:27,139 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:25:27,139 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:25:28,636 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:25:29,350 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:25:33,677 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:25:34,351 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:25:35,346 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:25:35,346 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1544_3c1a45e0401e280d0a21.png with policy now 2023-07-11 20:25:35,361 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:25:35,361 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1544_de785629b33cb381b74b.png with policy now 2023-07-11 20:25:35,370 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:25:35,370 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1544_f2506e5ffc80bfd166ad.png with policy now 2023-07-11 20:25:35,379 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:25:35,379 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1544_bf2932c9751754389a5c.png with policy now 2023-07-11 20:25:35,417 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:25:35,419 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:25:35,420 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:25:35,422 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:25:35,671 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ne972c9w-media/images/validation_1544_3c1a45e0401e280d0a21.png 2023-07-11 20:25:35,691 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/8sihkvop-media/images/validation_1544_de785629b33cb381b74b.png 2023-07-11 20:25:35,691 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/msto9uf2-media/images/validation_1544_bf2932c9751754389a5c.png 2023-07-11 20:25:35,700 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ayxjzgyu-media/images/validation_1544_f2506e5ffc80bfd166ad.png 2023-07-11 20:25:39,380 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:25:39,381 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:25:39,447 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:25:39,448 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:25:41,058 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:25:42,898 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:25:42,900 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:25:42,901 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:25:42,902 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:25:43,171 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:25:43,171 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:25:43,172 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1544_bf2932c9751754389a5c.png 2023-07-11 20:25:43,172 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1544_f2506e5ffc80bfd166ad.png 2023-07-11 20:25:43,173 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1544_de785629b33cb381b74b.png 2023-07-11 20:25:43,174 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1544_3c1a45e0401e280d0a21.png 2023-07-11 20:25:43,174 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 20:25:43,535 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/08hru4w9-media/images/validation_1544_f2506e5ffc80bfd166ad.png 2023-07-11 20:25:43,543 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/in24dmfp-media/images/validation_1544_de785629b33cb381b74b.png 2023-07-11 20:25:43,554 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/b3jg58cz-media/images/validation_1544_bf2932c9751754389a5c.png 2023-07-11 20:25:43,564 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/c02ujtdr-media/images/validation_1544_3c1a45e0401e280d0a21.png 2023-07-11 20:25:44,413 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:25:45,365 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:25:46,253 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:25:46,255 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:25:46,256 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:25:46,257 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:25:49,688 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:25:49,690 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:25:49,690 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:25:49,692 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:25:49,761 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:25:51,186 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:25:51,187 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:25:51,336 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:25:53,263 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:25:53,265 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:25:53,265 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:25:53,266 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:25:55,380 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:25:56,504 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:25:56,694 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:25:56,696 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:25:56,697 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:25:56,698 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:25:59,200 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:25:59,200 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:25:59,517 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:25:59,520 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:25:59,520 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:25:59,521 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:25:59,800 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:25:59,802 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:25:59,802 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:25:59,804 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:26:00,567 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:26:02,477 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:26:05,592 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:26:07,215 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:26:07,216 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:26:07,478 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:26:10,629 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:26:11,061 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:26:12,540 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:26:12,540 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1552_8d1e832ac023537c101c.png with policy now 2023-07-11 20:26:12,541 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:26:12,555 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:26:12,555 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1552_75bb106df0919e3df204.png with policy now 2023-07-11 20:26:12,564 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:26:12,565 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1552_aa4cc079d3d9191a8770.png with policy now 2023-07-11 20:26:12,573 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:26:12,573 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1552_b723e293e89f77fe239f.png with policy now 2023-07-11 20:26:12,616 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:26:12,618 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:26:12,618 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:26:12,621 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:26:12,868 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/rx9sqlb0-media/images/validation_1552_75bb106df0919e3df204.png 2023-07-11 20:26:12,873 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/0s09u3gp-media/images/validation_1552_8d1e832ac023537c101c.png 2023-07-11 20:26:12,890 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/sqwo91fg-media/images/validation_1552_aa4cc079d3d9191a8770.png 2023-07-11 20:26:12,890 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/eng0q32w-media/images/validation_1552_b723e293e89f77fe239f.png 2023-07-11 20:26:15,232 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:26:15,233 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1552_b723e293e89f77fe239f.png 2023-07-11 20:26:15,233 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1552_aa4cc079d3d9191a8770.png 2023-07-11 20:26:15,234 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1552_75bb106df0919e3df204.png 2023-07-11 20:26:15,235 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1552_8d1e832ac023537c101c.png 2023-07-11 20:26:15,235 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 20:26:15,574 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/r48el73a-media/images/validation_1552_b723e293e89f77fe239f.png 2023-07-11 20:26:15,589 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/yqkuia7s-media/images/validation_1552_aa4cc079d3d9191a8770.png 2023-07-11 20:26:15,596 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/s296a1oc-media/images/validation_1552_8d1e832ac023537c101c.png 2023-07-11 20:26:15,617 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/8yn4ctk9-media/images/validation_1552_75bb106df0919e3df204.png 2023-07-11 20:26:15,647 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:26:15,994 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:26:18,000 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:26:18,587 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:26:18,588 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:26:18,589 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:26:18,590 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:26:21,078 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:26:22,045 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:26:22,046 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:26:22,047 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:26:22,048 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:26:23,249 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:26:23,249 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:26:23,765 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:26:25,461 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:26:25,463 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:26:25,463 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:26:25,465 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:26:26,079 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:26:29,365 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:26:29,367 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:26:29,367 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:26:29,367 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:26:29,369 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:26:31,265 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:26:31,266 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:26:31,586 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:26:33,106 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:26:33,108 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:26:33,109 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:26:33,110 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:26:34,962 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:26:35,894 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:26:35,895 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:26:35,896 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:26:35,898 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:26:36,175 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:26:36,177 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:26:36,177 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:26:36,179 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:26:36,842 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:26:39,282 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:26:39,282 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:26:40,905 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:26:41,064 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:26:41,879 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:26:46,066 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:26:46,900 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:26:48,951 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:26:48,951 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1560_080450d9e4c3f1b32250.png with policy now 2023-07-11 20:26:48,965 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:26:48,966 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1560_c97c1d61c8724e14c00b.png with policy now 2023-07-11 20:26:48,976 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:26:48,977 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1560_7ff55320d57bf225ae68.png with policy now 2023-07-11 20:26:48,985 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:26:48,985 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1560_92d3ec020ffb72174fba.png with policy now 2023-07-11 20:26:49,022 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:26:49,024 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:26:49,024 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:26:49,026 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:26:49,278 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/0lm8kpe5-media/images/validation_1560_080450d9e4c3f1b32250.png 2023-07-11 20:26:49,318 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/nu1xqvmc-media/images/validation_1560_92d3ec020ffb72174fba.png 2023-07-11 20:26:49,318 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ffm03n2i-media/images/validation_1560_c97c1d61c8724e14c00b.png 2023-07-11 20:26:49,350 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/eqngcal2-media/images/validation_1560_7ff55320d57bf225ae68.png 2023-07-11 20:26:51,690 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:26:52,081 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:26:52,537 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:26:55,316 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:26:55,316 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:26:55,317 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1560_080450d9e4c3f1b32250.png 2023-07-11 20:26:55,317 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1560_c97c1d61c8724e14c00b.png 2023-07-11 20:26:55,318 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1560_7ff55320d57bf225ae68.png 2023-07-11 20:26:55,318 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1560_92d3ec020ffb72174fba.png 2023-07-11 20:26:55,319 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 20:26:55,637 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ggols1wc-media/images/validation_1560_080450d9e4c3f1b32250.png 2023-07-11 20:26:55,663 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ndaxlp2g-media/images/validation_1560_c97c1d61c8724e14c00b.png 2023-07-11 20:26:55,665 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/30qvdrb1-media/images/validation_1560_92d3ec020ffb72174fba.png 2023-07-11 20:26:55,719 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/kv2kohpc-media/images/validation_1560_7ff55320d57bf225ae68.png 2023-07-11 20:26:56,181 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:26:56,183 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:26:56,183 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:26:56,185 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:26:57,186 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:26:58,882 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:26:59,702 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:26:59,704 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:26:59,704 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:26:59,706 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:27:02,284 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:27:03,332 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:27:03,333 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:27:03,357 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:27:03,358 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:27:03,359 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:27:03,360 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:27:04,317 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:27:06,511 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:27:06,513 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:27:06,513 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:27:06,515 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:27:07,744 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:27:08,970 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:27:08,972 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:27:08,973 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:27:08,974 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:27:09,383 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:27:11,067 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:27:11,348 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:27:11,349 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:27:12,490 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:27:12,492 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:27:12,492 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:27:12,494 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:27:12,772 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:27:12,774 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:27:12,774 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:27:12,775 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:27:12,776 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:27:14,767 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:27:18,463 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:27:19,365 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:27:19,366 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:27:19,796 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:27:23,464 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:27:24,838 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:27:25,507 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:27:25,507 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1568_67b6d1876f24f6e8c3e4.png with policy now 2023-07-11 20:27:25,521 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:27:25,522 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1568_b803421655b7dc310cf8.png with policy now 2023-07-11 20:27:25,533 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:27:25,533 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1568_45dc21f5d3d178d54339.png with policy now 2023-07-11 20:27:25,541 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:27:25,541 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1568_a041609e0e3c44c0a850.png with policy now 2023-07-11 20:27:25,578 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:27:25,579 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:27:25,580 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:27:25,581 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:27:25,873 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/dvq47djc-media/images/validation_1568_67b6d1876f24f6e8c3e4.png 2023-07-11 20:27:25,879 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/0bwb2t7x-media/images/validation_1568_b803421655b7dc310cf8.png 2023-07-11 20:27:25,905 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/44huq539-media/images/validation_1568_a041609e0e3c44c0a850.png 2023-07-11 20:27:25,933 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ylymbu4d-media/images/validation_1568_45dc21f5d3d178d54339.png 2023-07-11 20:27:27,383 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:27:27,384 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1568_b803421655b7dc310cf8.png 2023-07-11 20:27:27,384 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1568_45dc21f5d3d178d54339.png 2023-07-11 20:27:27,385 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1568_a041609e0e3c44c0a850.png 2023-07-11 20:27:27,385 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1568_67b6d1876f24f6e8c3e4.png 2023-07-11 20:27:27,386 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 20:27:27,740 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/6366p22n-media/images/validation_1568_b803421655b7dc310cf8.png 2023-07-11 20:27:27,751 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ai33mmd3-media/images/validation_1568_a041609e0e3c44c0a850.png 2023-07-11 20:27:27,755 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/wtl9p2iw-media/images/validation_1568_67b6d1876f24f6e8c3e4.png 2023-07-11 20:27:27,958 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/z7e3pe7d-media/images/validation_1568_45dc21f5d3d178d54339.png 2023-07-11 20:27:28,860 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:27:28,861 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:27:29,961 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:27:33,208 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:27:33,210 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:27:33,210 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:27:33,212 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:27:33,903 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:27:35,399 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:27:35,399 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:27:36,684 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:27:36,809 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:27:36,811 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:27:36,812 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:27:36,814 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:27:39,415 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:27:40,273 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:27:40,275 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:27:40,275 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:27:40,297 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:27:41,070 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:27:41,826 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:27:41,826 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:27:43,304 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:27:43,306 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:27:43,306 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:27:43,308 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:27:43,415 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:27:43,415 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:27:44,565 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:27:45,690 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:27:45,692 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:27:45,692 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:27:45,694 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:27:47,486 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:27:49,575 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:27:49,577 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:27:49,577 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:27:49,577 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:27:49,579 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:27:49,856 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:27:49,858 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:27:49,858 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:27:49,860 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:27:51,429 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:27:51,429 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:27:52,675 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:27:52,676 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:27:55,396 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:27:57,695 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:27:59,444 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:28:00,397 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:28:02,446 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:28:02,446 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1576_2104b767ef4f17684879.png with policy now 2023-07-11 20:28:02,459 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:28:02,459 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1576_c1fc87dd1314b4adf472.png with policy now 2023-07-11 20:28:02,471 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:28:02,471 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1576_4f1e1d5fc289cbc9fa89.png with policy now 2023-07-11 20:28:02,480 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:28:02,481 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1576_a69d5aa10180a8853110.png with policy now 2023-07-11 20:28:02,519 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:28:02,522 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:28:02,522 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:28:02,524 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:28:02,738 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:28:02,758 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/d4v9idri-media/images/validation_1576_2104b767ef4f17684879.png 2023-07-11 20:28:02,781 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/9opr66ml-media/images/validation_1576_a69d5aa10180a8853110.png 2023-07-11 20:28:02,783 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/0145is5e-media/images/validation_1576_c1fc87dd1314b4adf472.png 2023-07-11 20:28:02,789 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/q0dky8wd-media/images/validation_1576_4f1e1d5fc289cbc9fa89.png 2023-07-11 20:28:06,098 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:28:06,099 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:28:07,461 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:28:07,462 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:28:07,462 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1576_4f1e1d5fc289cbc9fa89.png 2023-07-11 20:28:07,463 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1576_a69d5aa10180a8853110.png 2023-07-11 20:28:07,463 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1576_2104b767ef4f17684879.png 2023-07-11 20:28:07,464 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1576_c1fc87dd1314b4adf472.png 2023-07-11 20:28:07,464 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 20:28:07,871 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/uiu2uzue-media/images/validation_1576_c1fc87dd1314b4adf472.png 2023-07-11 20:28:08,003 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/vv28y6qy-media/images/validation_1576_2104b767ef4f17684879.png 2023-07-11 20:28:08,008 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/omgw619p-media/images/validation_1576_a69d5aa10180a8853110.png 2023-07-11 20:28:08,016 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/f0iqb0qe-media/images/validation_1576_4f1e1d5fc289cbc9fa89.png 2023-07-11 20:28:08,290 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:28:09,093 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:28:09,095 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:28:09,095 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:28:09,097 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:28:11,072 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:28:11,508 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:28:12,459 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:28:12,461 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:28:12,461 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:28:12,463 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:28:13,631 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:28:15,005 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:28:15,007 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:28:15,008 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:28:15,009 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:28:15,477 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:28:15,478 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:28:16,928 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:28:17,569 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:28:17,571 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:28:17,571 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:28:17,573 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:28:18,808 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:28:22,017 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:28:22,018 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:28:22,019 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:28:22,019 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:28:22,022 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:28:23,493 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:28:23,493 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:28:23,889 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:28:25,858 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:28:25,860 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:28:25,861 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:28:25,862 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:28:26,140 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:28:26,142 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:28:26,142 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:28:26,144 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:28:27,726 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:28:28,890 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:28:31,509 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:28:31,510 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:28:32,727 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:28:33,928 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:28:37,728 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:28:38,702 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:28:38,702 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1584_fc7f4c75961dd33ba081.png with policy now 2023-07-11 20:28:38,712 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:28:38,712 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1584_6d080ebd872af391cc12.png with policy now 2023-07-11 20:28:38,721 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:28:38,722 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1584_5060670cca612a4e440f.png with policy now 2023-07-11 20:28:38,731 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:28:38,731 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1584_4047a2d99cc326c5a57b.png with policy now 2023-07-11 20:28:38,773 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:28:38,775 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:28:38,775 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:28:38,776 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:28:38,951 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:28:39,014 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/9aanbf50-media/images/validation_1584_fc7f4c75961dd33ba081.png 2023-07-11 20:28:39,023 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/m2escbg0-media/images/validation_1584_6d080ebd872af391cc12.png 2023-07-11 20:28:39,033 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/4gs40lim-media/images/validation_1584_5060670cca612a4e440f.png 2023-07-11 20:28:39,071 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/3n72b3hz-media/images/validation_1584_4047a2d99cc326c5a57b.png 2023-07-11 20:28:39,527 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:28:39,528 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1584_4047a2d99cc326c5a57b.png 2023-07-11 20:28:39,528 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1584_6d080ebd872af391cc12.png 2023-07-11 20:28:39,529 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1584_fc7f4c75961dd33ba081.png 2023-07-11 20:28:39,529 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1584_5060670cca612a4e440f.png 2023-07-11 20:28:39,530 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 20:28:39,931 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/lp0curkx-media/images/validation_1584_4047a2d99cc326c5a57b.png 2023-07-11 20:28:39,935 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/xu9nbgyd-media/images/validation_1584_6d080ebd872af391cc12.png 2023-07-11 20:28:39,957 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/6ymazels-media/images/validation_1584_5060670cca612a4e440f.png 2023-07-11 20:28:39,972 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/vxew1owt-media/images/validation_1584_fc7f4c75961dd33ba081.png 2023-07-11 20:28:41,075 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:28:42,252 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:28:42,894 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:28:44,127 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:28:45,452 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:28:45,454 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:28:45,454 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:28:45,456 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:28:47,543 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:28:47,543 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:28:48,165 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:28:48,637 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:28:48,639 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:28:48,639 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:28:48,641 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:28:49,559 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:28:52,190 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:28:52,192 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:28:52,193 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:28:52,194 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:28:53,623 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:28:54,756 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:28:55,210 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:28:55,213 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:28:55,213 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:28:55,215 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:28:55,559 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:28:55,559 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:28:59,112 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:28:59,170 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:28:59,172 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:28:59,172 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:28:59,174 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:29:00,495 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:29:02,277 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:29:02,279 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:29:02,279 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:29:02,281 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:29:02,559 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:29:02,561 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:29:02,561 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:29:02,563 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:29:03,575 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:29:03,576 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:29:04,220 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:29:05,530 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:29:09,221 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:29:10,546 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:29:11,078 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:29:11,590 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:29:15,080 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:29:15,291 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:29:15,292 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1592_f48579df1612af3504f2.png with policy now 2023-07-11 20:29:15,306 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:29:15,306 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1592_97278c134a95be731717.png with policy now 2023-07-11 20:29:15,316 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:29:15,316 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1592_92ac374f16616a3f52d5.png with policy now 2023-07-11 20:29:15,326 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:29:15,326 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1592_00402f5cbaef44c29659.png with policy now 2023-07-11 20:29:15,362 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:29:15,363 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:29:15,363 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:29:15,365 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:29:15,590 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:29:15,615 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/3hjftwn9-media/images/validation_1592_f48579df1612af3504f2.png 2023-07-11 20:29:15,636 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/37anuubl-media/images/validation_1592_00402f5cbaef44c29659.png 2023-07-11 20:29:15,674 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/7ybi9czn-media/images/validation_1592_92ac374f16616a3f52d5.png 2023-07-11 20:29:15,674 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/heib7qlk-media/images/validation_1592_97278c134a95be731717.png 2023-07-11 20:29:18,603 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:29:19,608 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:29:19,609 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:29:19,609 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1592_97278c134a95be731717.png 2023-07-11 20:29:19,609 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1592_f48579df1612af3504f2.png 2023-07-11 20:29:19,610 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1592_92ac374f16616a3f52d5.png 2023-07-11 20:29:19,610 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1592_00402f5cbaef44c29659.png 2023-07-11 20:29:19,611 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 20:29:19,987 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/unrg2j5o-media/images/validation_1592_92ac374f16616a3f52d5.png 2023-07-11 20:29:19,989 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/8vx3vr65-media/images/validation_1592_00402f5cbaef44c29659.png 2023-07-11 20:29:19,991 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/9hbfvytx-media/images/validation_1592_97278c134a95be731717.png 2023-07-11 20:29:20,019 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/4dfkncwz-media/images/validation_1592_f48579df1612af3504f2.png 2023-07-11 20:29:20,885 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:29:21,002 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:29:22,317 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:29:22,319 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:29:22,319 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:29:22,321 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:29:25,737 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:29:25,739 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:29:25,740 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:29:25,741 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:29:26,284 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:29:26,285 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:29:26,405 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:29:27,625 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:29:27,625 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:29:28,687 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:29:28,689 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:29:28,689 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:29:28,692 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:29:31,452 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:29:31,895 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:29:32,925 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:29:32,927 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:29:32,927 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:29:32,929 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:29:36,641 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:29:36,641 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:29:36,678 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:29:36,741 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:29:36,742 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:29:36,743 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:29:36,744 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:29:37,614 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:29:39,896 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:29:39,897 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:29:39,898 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:29:39,900 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:29:40,178 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:29:40,179 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:29:40,180 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:29:40,181 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:29:41,081 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:29:42,083 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:29:42,621 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:29:45,658 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:29:45,658 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:29:47,084 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:29:47,665 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:29:52,087 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:29:52,685 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:29:52,756 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:29:52,756 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1600_699d0df245df08e0830f.png with policy now 2023-07-11 20:29:52,766 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:29:52,767 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1600_0f6bf7c1ef77ecb39e6c.png with policy now 2023-07-11 20:29:52,776 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:29:52,776 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1600_deb755fc1e7c3773dac4.png with policy now 2023-07-11 20:29:52,785 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:29:52,785 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1600_1759c1e65a73f7f13991.png with policy now 2023-07-11 20:29:52,832 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:29:52,833 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:29:52,834 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:29:52,835 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:29:53,058 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/lruh21zm-media/images/validation_1600_699d0df245df08e0830f.png 2023-07-11 20:29:53,075 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/fceagtiz-media/images/validation_1600_0f6bf7c1ef77ecb39e6c.png 2023-07-11 20:29:53,091 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/m81mcqco-media/images/validation_1600_deb755fc1e7c3773dac4.png 2023-07-11 20:29:53,105 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/t2an81i3-media/images/validation_1600_1759c1e65a73f7f13991.png 2023-07-11 20:29:54,676 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:29:54,676 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1600_699d0df245df08e0830f.png 2023-07-11 20:29:54,677 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1600_deb755fc1e7c3773dac4.png 2023-07-11 20:29:54,678 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1600_1759c1e65a73f7f13991.png 2023-07-11 20:29:54,678 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1600_0f6bf7c1ef77ecb39e6c.png 2023-07-11 20:29:54,679 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 20:29:55,037 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/4sbrnwzm-media/images/validation_1600_deb755fc1e7c3773dac4.png 2023-07-11 20:29:55,049 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/o3qhd17x-media/images/validation_1600_699d0df245df08e0830f.png 2023-07-11 20:29:55,057 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/7teb2nl5-media/images/validation_1600_0f6bf7c1ef77ecb39e6c.png 2023-07-11 20:29:55,319 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/z2arwask-media/images/validation_1600_1759c1e65a73f7f13991.png 2023-07-11 20:29:56,544 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:29:57,546 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:29:59,169 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:30:00,807 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:30:00,809 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:30:00,809 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:30:00,811 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:30:02,715 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:30:03,692 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:30:03,693 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:30:03,854 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:30:03,856 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:30:03,857 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:30:03,858 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:30:04,191 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:30:07,403 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:30:07,405 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:30:07,405 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:30:07,407 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:30:08,262 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:30:09,786 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:30:10,205 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:30:10,207 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:30:10,207 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:30:10,209 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:30:11,084 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:30:12,710 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:30:12,711 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:30:12,934 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:30:12,936 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:30:12,936 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:30:12,937 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:30:13,938 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:30:15,803 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:30:16,788 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:30:16,789 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:30:16,790 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:30:16,792 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:30:17,069 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:30:17,071 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:30:17,071 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:30:17,073 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:30:19,654 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:30:20,827 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:30:21,725 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:30:21,726 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:30:24,655 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:30:25,854 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:30:29,621 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:30:29,621 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1608_da4f2d17fb83c3abf79b.png with policy now 2023-07-11 20:30:29,632 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:30:29,633 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1608_6d2196a2c690a306717d.png with policy now 2023-07-11 20:30:29,645 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:30:29,646 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1608_7ae5a05f8fc5b0d59741.png with policy now 2023-07-11 20:30:29,657 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:30:29,657 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1608_47f23ae6bc7899d2f737.png with policy now 2023-07-11 20:30:29,658 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:30:29,693 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:30:29,694 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:30:29,695 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:30:29,697 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:30:29,937 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/xg4ddi3n-media/images/validation_1608_da4f2d17fb83c3abf79b.png 2023-07-11 20:30:29,971 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/mcppqae3-media/images/validation_1608_7ae5a05f8fc5b0d59741.png 2023-07-11 20:30:29,979 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/0ykhikwz-media/images/validation_1608_6d2196a2c690a306717d.png 2023-07-11 20:30:29,989 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/zaratq1d-media/images/validation_1608_47f23ae6bc7899d2f737.png 2023-07-11 20:30:30,743 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:30:30,744 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1608_6d2196a2c690a306717d.png 2023-07-11 20:30:30,744 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1608_da4f2d17fb83c3abf79b.png 2023-07-11 20:30:30,745 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1608_47f23ae6bc7899d2f737.png 2023-07-11 20:30:30,745 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1608_7ae5a05f8fc5b0d59741.png 2023-07-11 20:30:30,746 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 20:30:31,101 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/84shukuc-media/images/validation_1608_da4f2d17fb83c3abf79b.png 2023-07-11 20:30:31,104 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/8t9tn42m-media/images/validation_1608_6d2196a2c690a306717d.png 2023-07-11 20:30:31,112 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/8pj60qcc-media/images/validation_1608_47f23ae6bc7899d2f737.png 2023-07-11 20:30:31,123 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:30:31,159 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/hwl5jgr9-media/images/validation_1608_7ae5a05f8fc5b0d59741.png 2023-07-11 20:30:33,678 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:30:34,679 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:30:36,813 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:30:36,814 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:30:36,814 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:30:36,816 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:30:36,967 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:30:39,758 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:30:39,759 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:30:40,405 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:30:40,406 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:30:40,407 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:30:40,407 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:30:40,409 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:30:41,087 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:30:42,612 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:30:42,612 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:30:43,174 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:30:43,176 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:30:43,176 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:30:43,178 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:30:45,964 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:30:46,766 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:30:46,768 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:30:46,768 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:30:46,770 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:30:48,201 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:30:48,774 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:30:48,775 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:30:49,972 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:30:49,973 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:30:49,974 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:30:49,975 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:30:50,976 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:30:53,437 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:30:53,439 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:30:53,439 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:30:53,441 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:30:53,594 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:30:53,720 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:30:53,722 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:30:53,723 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:30:53,724 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:30:56,308 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:30:57,790 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:30:57,791 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:30:58,661 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:31:01,309 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:31:03,709 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:31:06,309 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:31:06,350 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:31:06,350 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1616_4162b0a1073cede299c3.png with policy now 2023-07-11 20:31:06,364 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:31:06,365 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1616_23f6c16a47fdbf946082.png with policy now 2023-07-11 20:31:06,374 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:31:06,375 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1616_08b89fa693ea9f058799.png with policy now 2023-07-11 20:31:06,384 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:31:06,384 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1616_5cc14ce0bb3ca15eff14.png with policy now 2023-07-11 20:31:06,420 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:31:06,425 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:31:06,426 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:31:06,429 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:31:06,675 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/u91kf9td-media/images/validation_1616_23f6c16a47fdbf946082.png 2023-07-11 20:31:06,676 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/sx34asjl-media/images/validation_1616_4162b0a1073cede299c3.png 2023-07-11 20:31:06,689 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/g2wns4mm-media/images/validation_1616_08b89fa693ea9f058799.png 2023-07-11 20:31:06,702 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/hqig9vte-media/images/validation_1616_5cc14ce0bb3ca15eff14.png 2023-07-11 20:31:06,809 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:31:06,809 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1616_23f6c16a47fdbf946082.png 2023-07-11 20:31:06,810 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1616_5cc14ce0bb3ca15eff14.png 2023-07-11 20:31:06,810 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1616_08b89fa693ea9f058799.png 2023-07-11 20:31:06,811 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1616_4162b0a1073cede299c3.png 2023-07-11 20:31:06,811 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 20:31:07,158 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/m35a8csb-media/images/validation_1616_23f6c16a47fdbf946082.png 2023-07-11 20:31:07,160 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/v5y36eco-media/images/validation_1616_5cc14ce0bb3ca15eff14.png 2023-07-11 20:31:07,168 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/j37urti9-media/images/validation_1616_08b89fa693ea9f058799.png 2023-07-11 20:31:07,183 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/7lkf9qph-media/images/validation_1616_4162b0a1073cede299c3.png 2023-07-11 20:31:08,722 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:31:10,282 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:31:11,090 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:31:11,441 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:31:14,148 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:31:14,150 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:31:14,150 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:31:14,152 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:31:14,904 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:31:15,826 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:31:15,826 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:31:16,947 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:31:16,948 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:31:16,948 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:31:16,949 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:31:16,951 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:31:19,941 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:31:20,246 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:31:20,248 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:31:20,249 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:31:20,250 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:31:22,140 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:31:22,914 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:31:22,915 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:31:22,916 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:31:22,918 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:31:24,842 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:31:24,843 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:31:25,723 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:31:25,723 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:31:26,524 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:31:26,526 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:31:26,526 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:31:26,528 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:31:27,529 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:31:30,235 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:31:30,237 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:31:30,238 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:31:30,239 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:31:30,520 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:31:30,521 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:31:30,522 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:31:30,523 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:31:30,726 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:31:33,092 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:31:33,858 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:31:33,858 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:31:35,755 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:31:38,093 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:31:40,805 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:31:41,092 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:31:43,094 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:31:43,121 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:31:43,121 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1624_c5c2b67c71ebaa893e3d.png with policy now 2023-07-11 20:31:43,132 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:31:43,132 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1624_87fd4f364cf3c1a5d743.png with policy now 2023-07-11 20:31:43,141 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:31:43,141 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1624_2d21b8cf80456a450416.png with policy now 2023-07-11 20:31:43,150 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:31:43,150 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1624_f2a4008163e3b0de1c67.png with policy now 2023-07-11 20:31:43,194 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:31:43,195 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:31:43,196 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:31:43,198 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:31:43,447 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/tb6k6w02-media/images/validation_1624_87fd4f364cf3c1a5d743.png 2023-07-11 20:31:43,451 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/z58b8x5m-media/images/validation_1624_2d21b8cf80456a450416.png 2023-07-11 20:31:43,456 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/7nop0nls-media/images/validation_1624_f2a4008163e3b0de1c67.png 2023-07-11 20:31:43,457 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/2uijezdp-media/images/validation_1624_c5c2b67c71ebaa893e3d.png 2023-07-11 20:31:46,460 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:31:47,348 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:31:48,705 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:31:50,473 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:31:50,475 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:31:50,476 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:31:50,477 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:31:51,892 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:31:51,893 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:31:51,893 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1624_c5c2b67c71ebaa893e3d.png 2023-07-11 20:31:51,893 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1624_f2a4008163e3b0de1c67.png 2023-07-11 20:31:51,894 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1624_87fd4f364cf3c1a5d743.png 2023-07-11 20:31:51,895 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1624_2d21b8cf80456a450416.png 2023-07-11 20:31:51,895 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 20:31:52,235 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/6w0zb3re-media/images/validation_1624_f2a4008163e3b0de1c67.png 2023-07-11 20:31:52,242 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/mthl8x1t-media/images/validation_1624_c5c2b67c71ebaa893e3d.png 2023-07-11 20:31:52,261 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/gkhcy4au-media/images/validation_1624_87fd4f364cf3c1a5d743.png 2023-07-11 20:31:52,263 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/h7xpvh17-media/images/validation_1624_2d21b8cf80456a450416.png 2023-07-11 20:31:52,346 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:31:52,546 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:31:52,548 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:31:52,549 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:31:52,550 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:31:53,815 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:31:56,763 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:31:56,764 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:31:56,764 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:31:56,766 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:31:57,779 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:31:57,779 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:31:59,577 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:32:00,501 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:32:00,503 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:32:00,503 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:32:00,504 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:32:00,909 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:32:00,909 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:32:03,165 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:32:03,293 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:32:03,295 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:32:03,295 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:32:03,296 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:32:04,753 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:32:06,594 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:32:06,595 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:32:06,596 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:32:06,597 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:32:06,879 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:32:06,881 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:32:06,881 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:32:06,883 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:32:08,692 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:32:09,925 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:32:09,926 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:32:10,420 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:32:11,095 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:32:13,745 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:32:16,098 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:32:18,782 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:32:19,408 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:32:19,409 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1632_87c3258c1b9a48214036.png with policy now 2023-07-11 20:32:19,420 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:32:19,421 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1632_4f352d796eb262af149d.png with policy now 2023-07-11 20:32:19,433 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:32:19,433 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1632_147308a3bc862cc2ef12.png with policy now 2023-07-11 20:32:19,442 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:32:19,442 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1632_ea4f6f4c802c31284d78.png with policy now 2023-07-11 20:32:19,482 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:32:19,485 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:32:19,486 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:32:19,488 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:32:19,725 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/sdaxsjtn-media/images/validation_1632_87c3258c1b9a48214036.png 2023-07-11 20:32:19,733 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/83wmmoxk-media/images/validation_1632_4f352d796eb262af149d.png 2023-07-11 20:32:19,753 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/nybp55ky-media/images/validation_1632_ea4f6f4c802c31284d78.png 2023-07-11 20:32:19,760 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/6az3f13v-media/images/validation_1632_147308a3bc862cc2ef12.png 2023-07-11 20:32:21,724 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:32:23,322 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:32:23,891 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:32:26,716 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:32:26,718 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:32:26,718 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:32:26,719 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:32:27,721 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:32:27,960 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:32:27,961 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:32:27,961 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1632_ea4f6f4c802c31284d78.png 2023-07-11 20:32:27,962 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1632_147308a3bc862cc2ef12.png 2023-07-11 20:32:27,962 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1632_4f352d796eb262af149d.png 2023-07-11 20:32:27,963 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1632_87c3258c1b9a48214036.png 2023-07-11 20:32:27,963 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 20:32:28,331 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/xc6y14f8-media/images/validation_1632_ea4f6f4c802c31284d78.png 2023-07-11 20:32:28,336 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/gycmlv31-media/images/validation_1632_147308a3bc862cc2ef12.png 2023-07-11 20:32:28,342 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/crbr3c7x-media/images/validation_1632_87c3258c1b9a48214036.png 2023-07-11 20:32:28,375 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ere8ofpc-media/images/validation_1632_4f352d796eb262af149d.png 2023-07-11 20:32:28,978 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:32:29,506 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:32:29,507 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:32:29,508 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:32:29,509 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:32:32,386 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:32:32,388 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:32:32,388 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:32:32,390 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:32:33,206 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:32:34,085 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:32:35,894 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:32:35,896 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:32:35,896 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:32:35,898 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:32:36,974 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:32:36,974 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:32:38,261 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:32:39,048 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:32:39,050 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:32:39,050 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:32:39,052 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:32:39,896 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:32:39,897 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:32:41,098 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:32:43,285 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:32:43,379 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:32:43,381 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:32:43,382 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:32:43,384 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:32:43,663 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:32:43,666 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:32:43,666 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:32:43,668 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:32:45,253 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:32:45,990 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:32:45,990 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:32:49,220 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:32:50,267 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:32:54,221 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:32:55,006 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:32:55,313 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:32:56,259 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:32:56,259 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1640_3d4aeb5bd69764009277.png with policy now 2023-07-11 20:32:56,269 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:32:56,269 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1640_0dd87108f2fea6cda5da.png with policy now 2023-07-11 20:32:56,278 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:32:56,279 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1640_134a552e8977908b132a.png with policy now 2023-07-11 20:32:56,287 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:32:56,287 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1640_41d647d9e8df50b6d062.png with policy now 2023-07-11 20:32:56,328 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:32:56,333 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:32:56,334 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:32:56,336 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:32:56,576 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/m7eu1avn-media/images/validation_1640_3d4aeb5bd69764009277.png 2023-07-11 20:32:56,594 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/q4phol6s-media/images/validation_1640_41d647d9e8df50b6d062.png 2023-07-11 20:32:56,595 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/dmq3xp2a-media/images/validation_1640_134a552e8977908b132a.png 2023-07-11 20:32:56,613 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/7jyhepxd-media/images/validation_1640_0dd87108f2fea6cda5da.png 2023-07-11 20:32:59,383 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:33:00,441 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:33:01,018 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:33:04,024 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:33:04,025 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:33:04,025 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1640_134a552e8977908b132a.png 2023-07-11 20:33:04,026 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1640_3d4aeb5bd69764009277.png 2023-07-11 20:33:04,026 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1640_41d647d9e8df50b6d062.png 2023-07-11 20:33:04,027 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1640_0dd87108f2fea6cda5da.png 2023-07-11 20:33:04,027 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 20:33:04,107 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:33:04,109 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:33:04,109 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:33:04,110 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:33:04,366 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/hkinpkei-media/images/validation_1640_3d4aeb5bd69764009277.png 2023-07-11 20:33:04,379 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/wy6057g4-media/images/validation_1640_0dd87108f2fea6cda5da.png 2023-07-11 20:33:04,434 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/39y5fdy5-media/images/validation_1640_41d647d9e8df50b6d062.png 2023-07-11 20:33:04,446 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:33:04,600 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/5km8pdjy-media/images/validation_1640_134a552e8977908b132a.png 2023-07-11 20:33:06,021 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:33:06,671 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:33:06,673 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:33:06,673 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:33:06,676 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:33:09,640 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:33:10,729 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:33:10,731 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:33:10,731 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:33:10,733 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:33:11,101 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:33:11,595 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:33:13,042 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:33:13,042 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:33:14,198 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:33:14,200 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:33:14,201 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:33:14,202 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:33:15,204 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:33:16,623 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:33:17,866 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:33:17,868 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:33:17,868 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:33:17,870 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:33:20,305 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:33:21,070 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:33:21,072 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:33:21,073 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:33:21,305 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:33:21,354 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:33:21,356 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:33:21,356 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:33:21,357 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:33:21,944 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:33:22,058 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:33:22,059 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:33:25,964 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:33:26,990 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:33:30,965 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:33:31,075 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:33:32,009 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:33:33,974 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:33:33,974 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1648_ae897ffe1c6081601953.png with policy now 2023-07-11 20:33:33,985 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:33:33,985 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1648_3801cc005fba469011fe.png with policy now 2023-07-11 20:33:33,994 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:33:33,994 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1648_cab21230e1e43c4dd2f2.png with policy now 2023-07-11 20:33:34,002 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:33:34,003 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1648_1de3e9f61db08c1fa525.png with policy now 2023-07-11 20:33:34,044 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:33:34,045 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:33:34,046 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:33:34,047 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:33:34,349 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/62a5hf0x-media/images/validation_1648_ae897ffe1c6081601953.png 2023-07-11 20:33:34,365 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/9app8eij-media/images/validation_1648_cab21230e1e43c4dd2f2.png 2023-07-11 20:33:34,383 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/0mtlpcer-media/images/validation_1648_1de3e9f61db08c1fa525.png 2023-07-11 20:33:34,452 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/u742nt6z-media/images/validation_1648_3801cc005fba469011fe.png 2023-07-11 20:33:36,420 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:33:37,842 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:33:38,795 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:33:40,093 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:33:40,093 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:33:40,094 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1648_cab21230e1e43c4dd2f2.png 2023-07-11 20:33:40,094 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1648_1de3e9f61db08c1fa525.png 2023-07-11 20:33:40,095 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1648_ae897ffe1c6081601953.png 2023-07-11 20:33:40,095 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1648_3801cc005fba469011fe.png 2023-07-11 20:33:40,096 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 20:33:40,456 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/dvyp3bdz-media/images/validation_1648_1de3e9f61db08c1fa525.png 2023-07-11 20:33:40,456 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/n0f526qg-media/images/validation_1648_ae897ffe1c6081601953.png 2023-07-11 20:33:40,458 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/mno3mb86-media/images/validation_1648_cab21230e1e43c4dd2f2.png 2023-07-11 20:33:40,460 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/fvmfhtnx-media/images/validation_1648_3801cc005fba469011fe.png 2023-07-11 20:33:41,104 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:33:41,873 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:33:41,875 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:33:41,875 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:33:41,876 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:33:41,878 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:33:42,867 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:33:44,969 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:33:44,971 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:33:44,971 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:33:44,973 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:33:47,573 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:33:48,219 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:33:48,347 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:33:48,349 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:33:48,349 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:33:48,351 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:33:49,110 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:33:49,110 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:33:52,024 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:33:52,026 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:33:52,026 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:33:52,028 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:33:53,029 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:33:53,733 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:33:55,239 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:33:55,241 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:33:55,242 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:33:55,243 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:33:58,126 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:33:58,126 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:33:58,761 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:33:58,878 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:33:58,889 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:33:58,891 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:33:58,891 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:33:58,893 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:33:59,172 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:33:59,174 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:33:59,175 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:33:59,176 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:34:03,801 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:34:04,732 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:34:07,143 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:34:07,143 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:34:08,845 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:34:09,733 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:34:11,107 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:34:11,720 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:34:11,721 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1656_8c3aa89e0dc3c0cfd217.png with policy now 2023-07-11 20:34:11,731 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:34:11,731 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1656_e4d7864777608c38c24e.png with policy now 2023-07-11 20:34:11,741 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:34:11,741 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1656_e5c0e9634965058135d3.png with policy now 2023-07-11 20:34:11,749 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:34:11,750 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1656_55996a575df11d966c0a.png with policy now 2023-07-11 20:34:11,793 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:34:11,795 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:34:11,795 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:34:11,797 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:34:12,033 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/kzwb1jdx-media/images/validation_1656_8c3aa89e0dc3c0cfd217.png 2023-07-11 20:34:12,054 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/223argdl-media/images/validation_1656_e4d7864777608c38c24e.png 2023-07-11 20:34:12,055 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/fd8no3qt-media/images/validation_1656_55996a575df11d966c0a.png 2023-07-11 20:34:12,077 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/sdpbnzeo-media/images/validation_1656_e5c0e9634965058135d3.png 2023-07-11 20:34:13,887 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:34:15,541 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:34:15,661 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:34:16,161 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:34:16,161 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:34:16,162 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1656_e4d7864777608c38c24e.png 2023-07-11 20:34:16,162 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1656_e5c0e9634965058135d3.png 2023-07-11 20:34:16,163 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1656_55996a575df11d966c0a.png 2023-07-11 20:34:16,163 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1656_8c3aa89e0dc3c0cfd217.png 2023-07-11 20:34:16,164 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 20:34:16,509 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/jqgaefx2-media/images/validation_1656_e5c0e9634965058135d3.png 2023-07-11 20:34:16,517 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/s0u0i7x7-media/images/validation_1656_55996a575df11d966c0a.png 2023-07-11 20:34:16,524 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/tg2aizc6-media/images/validation_1656_8c3aa89e0dc3c0cfd217.png 2023-07-11 20:34:16,526 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/zsj20em4-media/images/validation_1656_e4d7864777608c38c24e.png 2023-07-11 20:34:19,200 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:34:19,202 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:34:19,203 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:34:19,204 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:34:19,459 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:34:20,686 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:34:22,630 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:34:22,632 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:34:22,633 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:34:22,634 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:34:25,152 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:34:25,153 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:34:25,181 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:34:25,181 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:34:26,154 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:34:26,325 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:34:26,327 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:34:26,327 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:34:26,329 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:34:29,694 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:34:29,696 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:34:29,696 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:34:29,698 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:34:31,261 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:34:31,356 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:34:32,942 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:34:32,943 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:34:32,944 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:34:32,945 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:34:34,198 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:34:34,198 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:34:36,336 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:34:36,424 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:34:36,462 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:34:36,463 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:34:36,464 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:34:36,465 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:34:36,744 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:34:36,746 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:34:36,746 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:34:36,748 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:34:41,110 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:34:41,364 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:34:42,112 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:34:43,342 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:34:43,342 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:34:46,383 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:34:47,113 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:34:49,312 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:34:49,313 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1664_66dbe96487807e61c639.png with policy now 2023-07-11 20:34:49,323 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:34:49,324 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1664_97cb7ab131b642c3b7bc.png with policy now 2023-07-11 20:34:49,340 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:34:49,340 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1664_2ae67e741780fdc97d63.png with policy now 2023-07-11 20:34:49,341 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:34:49,342 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1664_ef66c85637afc9c7e435.png with policy now 2023-07-11 20:34:49,385 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:34:49,386 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:34:49,387 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:34:49,388 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:34:49,631 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/xas55o18-media/images/validation_1664_66dbe96487807e61c639.png 2023-07-11 20:34:49,637 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/9q5c4h1u-media/images/validation_1664_97cb7ab131b642c3b7bc.png 2023-07-11 20:34:49,651 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/1nw5kzit-media/images/validation_1664_2ae67e741780fdc97d63.png 2023-07-11 20:34:49,659 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/6bu02ro3-media/images/validation_1664_ef66c85637afc9c7e435.png 2023-07-11 20:34:52,016 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:34:52,139 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:34:52,359 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:34:52,360 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1664_66dbe96487807e61c639.png 2023-07-11 20:34:52,360 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1664_ef66c85637afc9c7e435.png 2023-07-11 20:34:52,361 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1664_2ae67e741780fdc97d63.png 2023-07-11 20:34:52,361 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1664_97cb7ab131b642c3b7bc.png 2023-07-11 20:34:52,361 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 20:34:52,701 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/q02h3nms-media/images/validation_1664_66dbe96487807e61c639.png 2023-07-11 20:34:52,725 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/2xci5xlt-media/images/validation_1664_ef66c85637afc9c7e435.png 2023-07-11 20:34:52,729 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/x5x10e6u-media/images/validation_1664_2ae67e741780fdc97d63.png 2023-07-11 20:34:52,748 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/6pc6x4bz-media/images/validation_1664_97cb7ab131b642c3b7bc.png 2023-07-11 20:34:54,109 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:34:57,656 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:34:57,656 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:34:57,732 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:34:57,782 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:34:57,784 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:34:57,785 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:34:57,786 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:35:00,770 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:35:00,772 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:35:00,772 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:35:00,774 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:35:01,376 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:35:01,376 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:35:03,601 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:35:03,603 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:35:03,603 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:35:03,603 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:35:03,605 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:35:03,645 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:35:07,801 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:35:07,803 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:35:07,804 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:35:07,805 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:35:08,860 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:35:08,983 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:35:10,391 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:35:10,392 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:35:10,891 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:35:10,892 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:35:10,893 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:35:10,894 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:35:11,113 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:35:14,167 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:35:14,169 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:35:14,169 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:35:14,170 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:35:14,171 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:35:14,324 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:35:14,324 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:35:14,449 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:35:14,451 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:35:14,451 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:35:14,453 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:35:19,338 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:35:19,407 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:35:19,408 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:35:20,068 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:35:24,360 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:35:25,069 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:35:27,048 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:35:27,048 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1672_f950a91f8f5d756c5764.png with policy now 2023-07-11 20:35:27,059 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:35:27,059 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1672_379652fb1beafd6c8e09.png with policy now 2023-07-11 20:35:27,068 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:35:27,068 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1672_02de7ad6917c63f45a9a.png with policy now 2023-07-11 20:35:27,077 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:35:27,077 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1672_32cff90f0713a31af381.png with policy now 2023-07-11 20:35:27,119 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:35:27,121 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:35:27,121 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:35:27,122 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:35:27,407 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/aav8mvwb-media/images/validation_1672_f950a91f8f5d756c5764.png 2023-07-11 20:35:27,419 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/zgd9k7mf-media/images/validation_1672_379652fb1beafd6c8e09.png 2023-07-11 20:35:27,425 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/gce279fe-media/images/validation_1672_02de7ad6917c63f45a9a.png 2023-07-11 20:35:27,458 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/o3cltqtf-media/images/validation_1672_32cff90f0713a31af381.png 2023-07-11 20:35:28,425 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:35:28,425 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1672_32cff90f0713a31af381.png 2023-07-11 20:35:28,426 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1672_379652fb1beafd6c8e09.png 2023-07-11 20:35:28,426 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1672_02de7ad6917c63f45a9a.png 2023-07-11 20:35:28,426 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1672_f950a91f8f5d756c5764.png 2023-07-11 20:35:28,427 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 20:35:28,808 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/85w2ejuz-media/images/validation_1672_379652fb1beafd6c8e09.png 2023-07-11 20:35:28,810 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/6oflx0z9-media/images/validation_1672_f950a91f8f5d756c5764.png 2023-07-11 20:35:28,810 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/1s9b9xo9-media/images/validation_1672_32cff90f0713a31af381.png 2023-07-11 20:35:28,853 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/uxxs06kn-media/images/validation_1672_02de7ad6917c63f45a9a.png 2023-07-11 20:35:29,380 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:35:30,487 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:35:31,092 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:35:33,911 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:35:33,913 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:35:33,913 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:35:33,915 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:35:34,834 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:35:35,798 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:35:37,440 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:35:37,441 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:35:37,569 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:35:37,571 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:35:37,571 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:35:37,573 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:35:40,901 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:35:40,979 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:35:40,979 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:35:41,108 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:35:41,110 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:35:41,110 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:35:41,152 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:35:41,153 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:35:44,331 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:35:44,334 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:35:44,334 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:35:44,336 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:35:46,344 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:35:46,442 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:35:46,457 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:35:46,458 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:35:48,004 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:35:48,006 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:35:48,006 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:35:48,008 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:35:50,723 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:35:50,725 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:35:50,725 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:35:50,727 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:35:51,006 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:35:51,007 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:35:51,008 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:35:51,009 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:35:51,456 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:35:51,597 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:35:55,474 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:35:55,475 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:35:56,595 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:35:56,616 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:36:01,596 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:36:01,633 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:36:03,680 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:36:03,681 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1680_e0d685807d82b1fe3333.png with policy now 2023-07-11 20:36:03,691 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:36:03,691 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1680_34a40e5cb3d7a0052b50.png with policy now 2023-07-11 20:36:03,700 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:36:03,700 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1680_3d0c633b59f3f6e54d50.png with policy now 2023-07-11 20:36:03,708 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:36:03,708 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1680_3015d5a2cea98cfefb90.png with policy now 2023-07-11 20:36:03,755 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:36:03,756 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:36:03,757 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:36:03,762 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:36:04,006 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/vmu6388c-media/images/validation_1680_3d0c633b59f3f6e54d50.png 2023-07-11 20:36:04,016 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/1fughqf0-media/images/validation_1680_3015d5a2cea98cfefb90.png 2023-07-11 20:36:04,016 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/yc5edepu-media/images/validation_1680_34a40e5cb3d7a0052b50.png 2023-07-11 20:36:04,099 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ffkny5ys-media/images/validation_1680_e0d685807d82b1fe3333.png 2023-07-11 20:36:04,493 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:36:04,493 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1680_3015d5a2cea98cfefb90.png 2023-07-11 20:36:04,494 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1680_e0d685807d82b1fe3333.png 2023-07-11 20:36:04,494 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1680_34a40e5cb3d7a0052b50.png 2023-07-11 20:36:04,495 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1680_3d0c633b59f3f6e54d50.png 2023-07-11 20:36:04,495 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 20:36:04,826 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/2ze1r3ci-media/images/validation_1680_3015d5a2cea98cfefb90.png 2023-07-11 20:36:04,861 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/68h8aet4-media/images/validation_1680_3d0c633b59f3f6e54d50.png 2023-07-11 20:36:04,870 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/pk60u2fe-media/images/validation_1680_34a40e5cb3d7a0052b50.png 2023-07-11 20:36:04,975 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/u8n7z3hs-media/images/validation_1680_e0d685807d82b1fe3333.png 2023-07-11 20:36:06,846 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:36:06,943 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:36:07,615 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:36:10,561 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:36:10,563 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:36:10,563 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:36:10,565 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:36:11,117 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:36:12,715 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:36:12,733 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:36:13,508 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:36:13,509 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:36:14,418 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:36:14,420 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:36:14,420 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:36:14,421 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:36:17,897 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:36:17,898 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:36:17,899 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:36:17,899 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:36:17,901 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:36:18,256 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:36:21,197 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:36:21,199 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:36:21,199 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:36:21,200 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:36:22,524 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:36:22,525 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:36:22,904 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:36:23,814 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:36:24,406 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:36:24,408 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:36:24,408 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:36:24,410 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:36:27,356 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:36:27,358 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:36:27,358 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:36:27,360 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:36:27,638 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:36:27,639 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:36:27,639 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:36:27,640 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:36:28,028 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:36:29,228 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:36:29,229 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:36:31,541 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:36:31,542 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:36:33,165 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:36:34,282 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:36:38,166 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:36:39,328 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:36:40,195 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:36:40,195 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1688_59c336d7ac9a01cc96ed.png with policy now 2023-07-11 20:36:40,206 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:36:40,206 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1688_78b53a833655e5e7ddfd.png with policy now 2023-07-11 20:36:40,215 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:36:40,215 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1688_47be1430323289df7f7b.png with policy now 2023-07-11 20:36:40,224 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:36:40,225 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1688_afc0af5c54864aff9b73.png with policy now 2023-07-11 20:36:40,265 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:36:40,266 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:36:40,266 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:36:40,268 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:36:40,529 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/p6gdvvgg-media/images/validation_1688_59c336d7ac9a01cc96ed.png 2023-07-11 20:36:40,551 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/z5ztosoj-media/images/validation_1688_47be1430323289df7f7b.png 2023-07-11 20:36:40,562 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:36:40,562 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1688_78b53a833655e5e7ddfd.png 2023-07-11 20:36:40,563 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1688_afc0af5c54864aff9b73.png 2023-07-11 20:36:40,564 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/i8e0xqnr-media/images/validation_1688_afc0af5c54864aff9b73.png 2023-07-11 20:36:40,564 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1688_59c336d7ac9a01cc96ed.png 2023-07-11 20:36:40,565 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1688_47be1430323289df7f7b.png 2023-07-11 20:36:40,565 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 20:36:40,570 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/dyi8rcpa-media/images/validation_1688_78b53a833655e5e7ddfd.png 2023-07-11 20:36:40,929 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/2j1p0x5h-media/images/validation_1688_47be1430323289df7f7b.png 2023-07-11 20:36:40,930 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/w39n9aln-media/images/validation_1688_78b53a833655e5e7ddfd.png 2023-07-11 20:36:40,931 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/3uikwb6r-media/images/validation_1688_59c336d7ac9a01cc96ed.png 2023-07-11 20:36:40,940 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/47sknmfr-media/images/validation_1688_afc0af5c54864aff9b73.png 2023-07-11 20:36:41,120 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:36:43,576 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:36:44,362 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:36:45,227 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:36:47,777 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:36:47,778 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:36:47,779 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:36:47,781 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:36:49,507 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:36:49,580 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:36:49,580 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:36:50,921 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:36:51,047 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:36:51,048 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:36:51,048 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:36:51,050 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:36:54,487 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:36:54,488 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:36:54,489 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:36:54,491 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:36:55,492 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:36:56,468 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:36:56,468 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:36:58,301 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:36:58,303 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:36:58,303 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:36:58,305 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:36:58,596 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:36:58,596 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:37:00,658 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:37:01,312 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:37:01,314 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:37:01,314 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:37:01,316 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:37:01,821 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:37:04,305 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:37:04,307 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:37:04,307 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:37:04,309 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:37:04,588 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:37:04,590 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:37:04,590 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:37:04,592 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:37:06,360 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:37:07,535 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:37:07,612 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:37:07,612 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:37:11,123 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:37:12,125 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:37:12,575 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:37:17,126 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:37:17,395 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:37:17,395 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1696_50897643f723faf8a1d8.png with policy now 2023-07-11 20:37:17,406 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:37:17,406 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1696_e4fbe946ca939c5f2907.png with policy now 2023-07-11 20:37:17,415 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:37:17,415 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1696_ac0d4f4b95333e8b92f1.png with policy now 2023-07-11 20:37:17,427 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:37:17,427 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1696_d73227af624327a56f89.png with policy now 2023-07-11 20:37:17,464 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:37:17,466 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:37:17,466 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:37:17,470 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:37:17,614 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:37:17,734 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/1n4jd5l3-media/images/validation_1696_50897643f723faf8a1d8.png 2023-07-11 20:37:17,755 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/xpp3lj46-media/images/validation_1696_e4fbe946ca939c5f2907.png 2023-07-11 20:37:17,758 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/5ng0pby9-media/images/validation_1696_d73227af624327a56f89.png 2023-07-11 20:37:17,763 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/mkbev9uc-media/images/validation_1696_ac0d4f4b95333e8b92f1.png 2023-07-11 20:37:21,087 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:37:22,400 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:37:22,636 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:37:24,125 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:37:24,127 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:37:24,127 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:37:24,424 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:37:25,646 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:37:25,647 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:37:25,647 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1696_e4fbe946ca939c5f2907.png 2023-07-11 20:37:25,648 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1696_ac0d4f4b95333e8b92f1.png 2023-07-11 20:37:25,648 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1696_50897643f723faf8a1d8.png 2023-07-11 20:37:25,649 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1696_d73227af624327a56f89.png 2023-07-11 20:37:25,649 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 20:37:26,003 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/34g87ys0-media/images/validation_1696_e4fbe946ca939c5f2907.png 2023-07-11 20:37:26,011 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/xo9uqp09-media/images/validation_1696_ac0d4f4b95333e8b92f1.png 2023-07-11 20:37:26,029 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/pppzh84t-media/images/validation_1696_50897643f723faf8a1d8.png 2023-07-11 20:37:26,074 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/9ot2c1z5-media/images/validation_1696_d73227af624327a56f89.png 2023-07-11 20:37:28,057 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:37:28,057 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:37:28,102 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:37:28,178 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:37:28,179 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:37:28,180 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:37:28,181 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:37:30,677 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:37:30,679 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:37:30,679 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:37:30,681 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:37:33,569 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:37:33,701 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:37:33,702 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:37:33,703 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:37:33,703 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:37:33,705 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:37:34,663 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:37:34,664 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:37:37,871 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:37:37,873 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:37:37,873 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:37:37,874 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:37:39,099 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:37:39,226 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:37:40,527 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:37:40,529 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:37:40,529 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:37:40,531 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:37:40,811 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:37:40,812 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:37:40,813 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:37:40,814 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:37:41,126 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:37:43,679 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:37:43,680 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:37:44,111 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:37:44,404 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:37:49,140 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:37:49,405 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:37:53,476 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:37:53,476 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1704_e1dcbbc29cdb423960a2.png with policy now 2023-07-11 20:37:53,490 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:37:53,491 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1704_314bebb79cb782edfd8f.png with policy now 2023-07-11 20:37:53,500 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:37:53,500 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1704_42a268d779b7667ccc2e.png with policy now 2023-07-11 20:37:53,508 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:37:53,509 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1704_e0d8a90f9602209761f6.png with policy now 2023-07-11 20:37:53,547 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:37:53,548 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:37:53,548 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:37:53,553 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:37:53,775 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/csr3w1st-media/images/validation_1704_e1dcbbc29cdb423960a2.png 2023-07-11 20:37:53,800 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/cs3z87sn-media/images/validation_1704_314bebb79cb782edfd8f.png 2023-07-11 20:37:53,833 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/pvggl00s-media/images/validation_1704_42a268d779b7667ccc2e.png 2023-07-11 20:37:53,834 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/lvtofr8n-media/images/validation_1704_e0d8a90f9602209761f6.png 2023-07-11 20:37:54,161 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:37:54,554 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:37:57,059 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:37:59,491 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:37:59,591 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:38:00,144 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:38:00,146 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:38:00,147 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:38:00,148 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:38:01,714 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:38:01,714 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:38:01,715 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1704_314bebb79cb782edfd8f.png 2023-07-11 20:38:01,715 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1704_42a268d779b7667ccc2e.png 2023-07-11 20:38:01,716 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1704_e0d8a90f9602209761f6.png 2023-07-11 20:38:01,716 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1704_e1dcbbc29cdb423960a2.png 2023-07-11 20:38:01,717 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 20:38:02,131 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/jxbqnhil-media/images/validation_1704_42a268d779b7667ccc2e.png 2023-07-11 20:38:02,136 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ni1tvxzd-media/images/validation_1704_e0d8a90f9602209761f6.png 2023-07-11 20:38:02,157 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/o17uhfgy-media/images/validation_1704_314bebb79cb782edfd8f.png 2023-07-11 20:38:02,173 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/pgc5kc7q-media/images/validation_1704_e1dcbbc29cdb423960a2.png 2023-07-11 20:38:03,115 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:38:03,117 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:38:03,118 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:38:03,120 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:38:04,643 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:38:04,644 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:38:06,803 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:38:06,805 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:38:06,805 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:38:06,807 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:38:09,800 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:38:09,841 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:38:10,573 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:38:10,575 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:38:10,575 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:38:10,577 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:38:10,729 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:38:10,730 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:38:11,129 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:38:14,467 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:38:14,469 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:38:14,470 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:38:14,471 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:38:15,338 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:38:15,578 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:38:17,384 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:38:17,386 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:38:17,387 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:38:17,388 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:38:17,669 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:38:17,671 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:38:17,671 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:38:17,673 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:38:19,746 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:38:19,746 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:38:20,395 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:38:21,223 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:38:25,411 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:38:26,224 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:38:30,236 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:38:30,237 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1712_50df1a6368b684b0eabd.png with policy now 2023-07-11 20:38:30,251 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:38:30,252 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1712_bb78706228e3f64446a2.png with policy now 2023-07-11 20:38:30,262 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:38:30,262 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1712_27d02e0fde32d79961e2.png with policy now 2023-07-11 20:38:30,264 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:38:30,265 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1712_b2393282cb27ff5aaf28.png with policy now 2023-07-11 20:38:30,308 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:38:30,310 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:38:30,310 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:38:30,315 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:38:30,457 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:38:30,561 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/rc9v35ug-media/images/validation_1712_50df1a6368b684b0eabd.png 2023-07-11 20:38:30,594 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ax71mh07-media/images/validation_1712_b2393282cb27ff5aaf28.png 2023-07-11 20:38:30,609 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/x5r0q2f5-media/images/validation_1712_27d02e0fde32d79961e2.png 2023-07-11 20:38:30,609 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/2vohxa42-media/images/validation_1712_bb78706228e3f64446a2.png 2023-07-11 20:38:31,316 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:38:33,932 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:38:36,086 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:38:36,770 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:38:36,771 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:38:36,772 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:38:36,772 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:38:36,774 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:38:37,781 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:38:37,782 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:38:37,782 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1712_bb78706228e3f64446a2.png 2023-07-11 20:38:37,783 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1712_27d02e0fde32d79961e2.png 2023-07-11 20:38:37,783 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1712_50df1a6368b684b0eabd.png 2023-07-11 20:38:37,784 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1712_b2393282cb27ff5aaf28.png 2023-07-11 20:38:37,784 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 20:38:39,580 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/s76ovyaw-media/images/validation_1712_50df1a6368b684b0eabd.png 2023-07-11 20:38:39,619 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/37fl1opt-media/images/validation_1712_b2393282cb27ff5aaf28.png 2023-07-11 20:38:40,073 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/bzbeda7y-media/images/validation_1712_27d02e0fde32d79961e2.png 2023-07-11 20:38:40,214 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/15oyev8u-media/images/validation_1712_bb78706228e3f64446a2.png 2023-07-11 20:38:40,361 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:38:40,362 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:38:40,362 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:38:40,364 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:38:41,118 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:38:41,133 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:38:41,779 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:38:43,086 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:38:43,087 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:38:43,088 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:38:43,089 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:38:45,910 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:38:45,912 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:38:45,913 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:38:45,914 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:38:46,784 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:38:46,787 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:38:46,800 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:38:46,801 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:38:48,869 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:38:48,871 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:38:48,871 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:38:48,873 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:38:51,895 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:38:52,679 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:38:52,869 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:38:52,871 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:38:52,871 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:38:52,873 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:38:53,151 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:38:53,152 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:38:53,153 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:38:53,154 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:38:55,817 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:38:55,818 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:38:56,908 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:38:57,744 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:39:01,955 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:39:02,745 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:39:05,765 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:39:05,765 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1720_679437e5e8fa85b1b9fa.png with policy now 2023-07-11 20:39:05,780 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:39:05,780 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1720_973ea290aaf93bd8b3d2.png with policy now 2023-07-11 20:39:05,790 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:39:05,790 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1720_5a01ec75660827b76fe7.png with policy now 2023-07-11 20:39:05,799 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:39:05,799 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1720_c173e21a79da12ee1883.png with policy now 2023-07-11 20:39:05,839 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:39:05,842 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:39:05,842 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:39:05,848 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:39:06,082 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/otjxwidw-media/images/validation_1720_679437e5e8fa85b1b9fa.png 2023-07-11 20:39:06,093 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/j09rey90-media/images/validation_1720_973ea290aaf93bd8b3d2.png 2023-07-11 20:39:06,115 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/2jir1285-media/images/validation_1720_c173e21a79da12ee1883.png 2023-07-11 20:39:06,141 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/c5a62bbq-media/images/validation_1720_5a01ec75660827b76fe7.png 2023-07-11 20:39:07,009 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:39:07,806 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:39:09,271 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:39:11,134 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:39:12,143 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:39:12,566 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:39:12,568 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:39:12,568 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:39:12,570 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:39:13,375 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:39:13,853 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:39:13,854 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:39:13,854 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1720_679437e5e8fa85b1b9fa.png 2023-07-11 20:39:13,855 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1720_c173e21a79da12ee1883.png 2023-07-11 20:39:13,855 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1720_973ea290aaf93bd8b3d2.png 2023-07-11 20:39:13,856 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1720_5a01ec75660827b76fe7.png 2023-07-11 20:39:13,856 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 20:39:14,188 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ukj22ljc-media/images/validation_1720_679437e5e8fa85b1b9fa.png 2023-07-11 20:39:14,201 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/gjir4gvp-media/images/validation_1720_c173e21a79da12ee1883.png 2023-07-11 20:39:14,203 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/85bfh08m-media/images/validation_1720_5a01ec75660827b76fe7.png 2023-07-11 20:39:14,229 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/0rzb3e5f-media/images/validation_1720_973ea290aaf93bd8b3d2.png 2023-07-11 20:39:15,125 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:39:15,126 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:39:15,127 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:39:15,128 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:39:17,526 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:39:18,219 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:39:18,221 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:39:18,221 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:39:18,223 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:39:19,224 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:39:22,480 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:39:22,482 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:39:22,482 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:39:22,484 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:39:22,865 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:39:22,871 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:39:22,872 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:39:24,341 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:39:25,955 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:39:25,957 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:39:25,957 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:39:25,959 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:39:28,369 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:39:28,369 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:39:29,489 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:39:29,764 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:39:29,766 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:39:29,767 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:39:29,768 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:39:30,047 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:39:30,049 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:39:30,050 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:39:30,051 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:39:31,887 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:39:31,888 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:39:33,673 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:39:34,589 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:39:38,707 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:39:39,590 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:39:41,139 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:39:42,639 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:39:42,639 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1728_590c097e334a7ac52f79.png with policy now 2023-07-11 20:39:42,653 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:39:42,653 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1728_f006b6af0fba27faf76e.png with policy now 2023-07-11 20:39:42,664 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:39:42,664 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1728_89f458f901461e553af6.png with policy now 2023-07-11 20:39:42,672 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:39:42,673 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1728_61df962c93eb4afff8f0.png with policy now 2023-07-11 20:39:42,712 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:39:42,713 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:39:42,713 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:39:42,715 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:39:42,967 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/lrwsq0v3-media/images/validation_1728_590c097e334a7ac52f79.png 2023-07-11 20:39:42,983 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/vhd1xhy0-media/images/validation_1728_61df962c93eb4afff8f0.png 2023-07-11 20:39:42,986 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/zuju1daa-media/images/validation_1728_89f458f901461e553af6.png 2023-07-11 20:39:42,998 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/2in2t8t6-media/images/validation_1728_f006b6af0fba27faf76e.png 2023-07-11 20:39:43,748 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:39:45,021 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:39:47,397 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:39:49,575 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:39:49,924 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:39:49,925 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:39:49,925 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1728_590c097e334a7ac52f79.png 2023-07-11 20:39:49,926 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1728_89f458f901461e553af6.png 2023-07-11 20:39:49,926 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1728_f006b6af0fba27faf76e.png 2023-07-11 20:39:49,927 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1728_61df962c93eb4afff8f0.png 2023-07-11 20:39:49,927 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 20:39:50,292 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/idrbpfua-media/images/validation_1728_f006b6af0fba27faf76e.png 2023-07-11 20:39:50,310 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/3uvngk54-media/images/validation_1728_61df962c93eb4afff8f0.png 2023-07-11 20:39:50,312 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/h8kgo1uh-media/images/validation_1728_590c097e334a7ac52f79.png 2023-07-11 20:39:50,324 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/f7ounvvb-media/images/validation_1728_89f458f901461e553af6.png 2023-07-11 20:39:50,698 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:39:51,098 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:39:51,100 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:39:51,100 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:39:51,102 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:39:54,106 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:39:54,108 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:39:54,108 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:39:54,110 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:39:54,977 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:39:55,852 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:39:57,563 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:39:57,565 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:39:57,565 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:39:57,567 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:39:58,942 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:39:58,943 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:40:00,899 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:40:01,028 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:40:01,030 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:40:01,030 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:40:01,031 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:40:01,032 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:40:04,302 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:40:04,304 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:40:04,304 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:40:04,306 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:40:06,298 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:40:06,661 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:40:07,589 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:40:07,591 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:40:07,592 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:40:07,593 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:40:07,872 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:40:07,874 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:40:07,874 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:40:07,876 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:40:07,958 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:40:07,959 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:40:11,140 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:40:11,349 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:40:12,142 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:40:16,520 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:40:16,976 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:40:17,143 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:40:20,528 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:40:20,529 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1736_23c66688cd6e5a6be908.png with policy now 2023-07-11 20:40:20,539 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:40:20,539 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1736_81a3ccf88a1ed0a25cef.png with policy now 2023-07-11 20:40:20,553 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:40:20,553 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1736_1029fe384454cadd60b1.png with policy now 2023-07-11 20:40:20,562 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:40:20,562 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1736_9f5d69e25ec444d8827e.png with policy now 2023-07-11 20:40:20,602 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:40:20,603 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:40:20,603 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:40:20,606 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:40:20,815 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/1k6jo3xs-media/images/validation_1736_23c66688cd6e5a6be908.png 2023-07-11 20:40:20,869 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/hqq6ia5e-media/images/validation_1736_1029fe384454cadd60b1.png 2023-07-11 20:40:20,874 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ucjqhw34-media/images/validation_1736_9f5d69e25ec444d8827e.png 2023-07-11 20:40:20,898 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/fdy1713b-media/images/validation_1736_81a3ccf88a1ed0a25cef.png 2023-07-11 20:40:21,543 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:40:22,193 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:40:24,058 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:40:25,994 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:40:25,995 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:40:25,995 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1736_1029fe384454cadd60b1.png 2023-07-11 20:40:25,995 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1736_9f5d69e25ec444d8827e.png 2023-07-11 20:40:25,996 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1736_23c66688cd6e5a6be908.png 2023-07-11 20:40:25,996 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1736_81a3ccf88a1ed0a25cef.png 2023-07-11 20:40:25,997 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 20:40:26,330 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/m0vsp0wp-media/images/validation_1736_9f5d69e25ec444d8827e.png 2023-07-11 20:40:26,351 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/tb4kjwdu-media/images/validation_1736_23c66688cd6e5a6be908.png 2023-07-11 20:40:26,373 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/q7r4tpfs-media/images/validation_1736_81a3ccf88a1ed0a25cef.png 2023-07-11 20:40:26,381 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/78vwk6w9-media/images/validation_1736_1029fe384454cadd60b1.png 2023-07-11 20:40:26,929 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:40:27,740 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:40:27,742 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:40:27,743 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:40:27,743 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:40:27,745 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:40:31,204 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:40:31,206 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:40:31,207 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:40:31,208 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:40:32,079 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:40:33,271 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:40:34,001 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:40:34,003 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:40:34,004 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:40:34,005 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:40:35,007 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:40:35,008 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:40:37,534 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:40:37,535 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:40:37,536 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:40:37,538 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:40:37,562 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:40:38,539 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:40:40,876 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:40:40,878 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:40:40,878 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:40:40,880 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:40:41,143 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:40:43,005 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:40:43,005 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:40:43,825 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:40:44,022 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:40:44,023 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:40:44,818 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:40:44,820 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:40:44,821 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:40:44,822 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:40:45,100 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:40:45,102 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:40:45,103 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:40:45,104 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:40:48,032 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:40:49,710 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:40:53,039 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:40:53,039 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:40:53,049 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:40:54,710 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:40:57,783 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:40:57,783 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1744_d7f69c8955653068af1c.png with policy now 2023-07-11 20:40:57,793 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:40:57,793 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1744_5a12380a065557e23f86.png with policy now 2023-07-11 20:40:57,802 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:40:57,802 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1744_9f5c0597dd918b1fd1cd.png with policy now 2023-07-11 20:40:57,811 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:40:57,811 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1744_ce9d3bc192f9ffe8e69b.png with policy now 2023-07-11 20:40:57,851 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:40:57,853 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:40:57,854 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:40:57,856 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:40:58,089 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:40:58,106 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/byredgga-media/images/validation_1744_d7f69c8955653068af1c.png 2023-07-11 20:40:58,117 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/xr3otw4b-media/images/validation_1744_9f5c0597dd918b1fd1cd.png 2023-07-11 20:40:58,117 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/bmm0kr7o-media/images/validation_1744_5a12380a065557e23f86.png 2023-07-11 20:40:58,133 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/6zxia56m-media/images/validation_1744_ce9d3bc192f9ffe8e69b.png 2023-07-11 20:40:59,714 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:41:01,538 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:41:02,058 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:41:02,058 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:41:02,058 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1744_d7f69c8955653068af1c.png 2023-07-11 20:41:02,059 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1744_5a12380a065557e23f86.png 2023-07-11 20:41:02,059 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1744_ce9d3bc192f9ffe8e69b.png 2023-07-11 20:41:02,060 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1744_9f5c0597dd918b1fd1cd.png 2023-07-11 20:41:02,061 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 20:41:02,411 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/qne9yzlc-media/images/validation_1744_d7f69c8955653068af1c.png 2023-07-11 20:41:02,425 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/88wtc54j-media/images/validation_1744_5a12380a065557e23f86.png 2023-07-11 20:41:02,454 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/r6uy42tf-media/images/validation_1744_9f5c0597dd918b1fd1cd.png 2023-07-11 20:41:02,469 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/r62u5vuh-media/images/validation_1744_ce9d3bc192f9ffe8e69b.png 2023-07-11 20:41:03,612 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:41:04,839 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:41:04,841 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:41:04,841 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:41:04,841 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:41:04,843 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:41:07,931 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:41:07,933 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:41:07,933 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:41:07,935 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:41:08,803 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:41:10,369 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:41:11,071 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:41:11,071 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:41:11,146 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:41:11,184 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:41:11,186 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:41:11,186 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:41:11,187 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:41:13,971 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:41:13,972 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:41:14,039 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:41:14,041 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:41:14,041 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:41:14,043 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:41:15,447 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:41:17,849 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:41:17,851 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:41:17,851 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:41:17,853 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:41:20,049 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:41:20,085 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:41:20,085 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:41:21,173 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:41:22,584 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:41:22,586 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:41:22,586 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:41:22,588 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:41:22,871 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:41:22,873 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:41:22,873 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:41:22,875 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:41:25,174 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:41:26,460 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:41:29,103 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:41:29,103 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:41:30,228 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:41:31,461 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:41:35,247 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:41:35,477 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:41:35,477 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1752_41340b7ef5a230ba2c1d.png with policy now 2023-07-11 20:41:35,487 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:41:35,488 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1752_2d2c4c4207dded6ac310.png with policy now 2023-07-11 20:41:35,501 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:41:35,501 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1752_1fca70034e43e1efd16b.png with policy now 2023-07-11 20:41:35,510 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:41:35,510 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1752_c0be450d736d315a7f7f.png with policy now 2023-07-11 20:41:35,548 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:41:35,549 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:41:35,550 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:41:35,551 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:41:35,778 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/0kwfmq8o-media/images/validation_1752_41340b7ef5a230ba2c1d.png 2023-07-11 20:41:35,788 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/5mg6p22n-media/images/validation_1752_2d2c4c4207dded6ac310.png 2023-07-11 20:41:35,816 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ssugsb04-media/images/validation_1752_c0be450d736d315a7f7f.png 2023-07-11 20:41:35,823 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/3tgmhuit-media/images/validation_1752_1fca70034e43e1efd16b.png 2023-07-11 20:41:36,552 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:41:38,122 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:41:38,122 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:41:38,123 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1752_1fca70034e43e1efd16b.png 2023-07-11 20:41:38,123 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1752_41340b7ef5a230ba2c1d.png 2023-07-11 20:41:38,124 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1752_c0be450d736d315a7f7f.png 2023-07-11 20:41:38,124 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1752_2d2c4c4207dded6ac310.png 2023-07-11 20:41:38,125 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 20:41:39,356 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:41:40,251 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:41:40,384 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ypnf22dz-media/images/validation_1752_c0be450d736d315a7f7f.png 2023-07-11 20:41:40,566 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/jzxtfn1j-media/images/validation_1752_41340b7ef5a230ba2c1d.png 2023-07-11 20:41:40,604 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/w43h4gov-media/images/validation_1752_1fca70034e43e1efd16b.png 2023-07-11 20:41:41,149 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:41:41,293 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ztm5eips-media/images/validation_1752_2d2c4c4207dded6ac310.png 2023-07-11 20:41:42,075 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:41:42,077 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:41:42,077 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:41:42,078 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:41:42,079 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:41:45,359 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:41:45,361 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:41:45,361 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:41:45,363 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:41:45,821 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:41:47,147 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:41:47,148 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:41:47,197 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:41:48,319 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:41:48,321 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:41:48,321 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:41:48,323 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:41:51,228 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:41:52,038 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:41:52,040 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:41:52,040 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:41:52,042 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:41:52,685 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:41:54,900 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:41:54,902 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:41:54,902 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:41:54,904 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:41:56,164 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:41:56,165 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:41:56,437 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:41:58,495 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:41:58,510 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:41:58,512 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:41:58,512 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:41:58,514 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:41:58,792 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:41:58,794 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:41:58,794 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:41:58,796 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:42:01,438 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:42:04,341 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:42:05,182 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:42:05,182 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:42:06,487 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:42:09,342 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:42:11,151 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:42:11,332 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:42:11,332 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1760_f9e33a1e2eb7c8bf304f.png with policy now 2023-07-11 20:42:11,347 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:42:11,347 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1760_963ce470d918d6ff96d4.png with policy now 2023-07-11 20:42:11,357 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:42:11,357 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1760_2b0a2a64666aeb10c8d5.png with policy now 2023-07-11 20:42:11,366 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:42:11,366 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1760_1a916468c5f244822b28.png with policy now 2023-07-11 20:42:11,405 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:42:11,406 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:42:11,407 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:42:11,408 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:42:11,508 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:42:11,670 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/u7na7dud-media/images/validation_1760_963ce470d918d6ff96d4.png 2023-07-11 20:42:11,670 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/izyubqmi-media/images/validation_1760_1a916468c5f244822b28.png 2023-07-11 20:42:11,674 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ehluqlq5-media/images/validation_1760_f9e33a1e2eb7c8bf304f.png 2023-07-11 20:42:11,695 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/4oop7jdf-media/images/validation_1760_2b0a2a64666aeb10c8d5.png 2023-07-11 20:42:14,203 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:42:14,203 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:42:14,204 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1760_2b0a2a64666aeb10c8d5.png 2023-07-11 20:42:14,204 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1760_f9e33a1e2eb7c8bf304f.png 2023-07-11 20:42:14,205 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1760_963ce470d918d6ff96d4.png 2023-07-11 20:42:14,205 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1760_1a916468c5f244822b28.png 2023-07-11 20:42:14,205 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 20:42:14,376 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:42:14,587 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/5mbf3ie2-media/images/validation_1760_1a916468c5f244822b28.png 2023-07-11 20:42:14,598 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ci1n7xbz-media/images/validation_1760_2b0a2a64666aeb10c8d5.png 2023-07-11 20:42:14,611 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/kzlj1qrg-media/images/validation_1760_963ce470d918d6ff96d4.png 2023-07-11 20:42:14,625 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/1t6e8g03-media/images/validation_1760_f9e33a1e2eb7c8bf304f.png 2023-07-11 20:42:15,183 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:42:17,425 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:42:18,248 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:42:18,249 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:42:18,249 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:42:18,251 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:42:19,475 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:42:20,761 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:42:20,763 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:42:20,763 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:42:20,765 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:42:22,813 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:42:23,220 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:42:23,220 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:42:23,871 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:42:23,873 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:42:23,873 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:42:23,875 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:42:24,507 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:42:26,995 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:42:26,998 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:42:26,999 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:42:27,000 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:42:27,870 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:42:29,710 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:42:30,396 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:42:30,398 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:42:30,399 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:42:30,400 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:42:32,236 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:42:32,237 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:42:33,451 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:42:34,042 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:42:34,044 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:42:34,044 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:42:34,046 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:42:34,323 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:42:34,325 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:42:34,326 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:42:34,327 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:42:34,814 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:42:38,526 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:42:39,953 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:42:41,154 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:42:41,252 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:42:41,253 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:42:43,554 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:42:45,157 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:42:47,001 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:42:47,002 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1768_2dae9d2ec2b54e62fa2a.png with policy now 2023-07-11 20:42:47,017 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:42:47,017 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1768_cfcc80a8ccc0659ac78e.png with policy now 2023-07-11 20:42:47,027 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:42:47,027 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1768_bfde8140b1fc4f9db23e.png with policy now 2023-07-11 20:42:47,036 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:42:47,036 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1768_bad2b0cfefb3891be2e4.png with policy now 2023-07-11 20:42:47,091 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:42:47,092 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:42:47,093 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:42:47,094 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:42:47,325 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/3h41at3u-media/images/validation_1768_2dae9d2ec2b54e62fa2a.png 2023-07-11 20:42:47,367 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/kqvqoobz-media/images/validation_1768_bfde8140b1fc4f9db23e.png 2023-07-11 20:42:47,369 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/pri1ko03-media/images/validation_1768_cfcc80a8ccc0659ac78e.png 2023-07-11 20:42:47,375 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/i9t98hha-media/images/validation_1768_bad2b0cfefb3891be2e4.png 2023-07-11 20:42:49,346 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:42:50,198 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:42:50,272 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:42:50,273 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:42:50,273 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1768_cfcc80a8ccc0659ac78e.png 2023-07-11 20:42:50,274 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1768_bfde8140b1fc4f9db23e.png 2023-07-11 20:42:50,274 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1768_bad2b0cfefb3891be2e4.png 2023-07-11 20:42:50,275 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1768_2dae9d2ec2b54e62fa2a.png 2023-07-11 20:42:50,275 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 20:42:50,626 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/fnvllqcc-media/images/validation_1768_cfcc80a8ccc0659ac78e.png 2023-07-11 20:42:50,629 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/3fynctly-media/images/validation_1768_bfde8140b1fc4f9db23e.png 2023-07-11 20:42:50,644 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/2layh5dp-media/images/validation_1768_bad2b0cfefb3891be2e4.png 2023-07-11 20:42:50,658 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/wkkqd5ye-media/images/validation_1768_2dae9d2ec2b54e62fa2a.png 2023-07-11 20:42:50,887 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:42:54,195 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:42:54,197 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:42:54,197 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:42:54,199 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:42:55,281 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:42:55,402 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:42:57,081 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:42:57,083 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:42:57,083 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:42:57,085 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:42:59,290 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:42:59,290 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:43:00,319 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:43:00,320 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:43:00,326 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:43:00,327 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:43:00,328 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:43:00,329 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:43:00,964 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:43:04,152 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:43:04,153 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:43:04,154 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:43:04,155 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:43:05,739 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:43:06,640 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:43:07,452 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:43:07,454 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:43:07,454 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:43:07,456 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:43:08,307 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:43:08,307 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:43:11,020 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:43:11,146 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:43:11,148 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:43:11,148 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:43:11,150 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:43:11,157 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:43:11,430 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:43:11,432 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:43:11,433 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:43:11,434 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:43:11,905 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:43:16,021 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:43:17,044 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:43:17,327 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:43:17,327 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:43:21,057 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:43:22,045 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:43:24,023 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:43:24,024 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1776_7c1cf0a21ce762d708cc.png with policy now 2023-07-11 20:43:24,038 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:43:24,039 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1776_a6253244e0fcd16b6309.png with policy now 2023-07-11 20:43:24,060 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:43:24,061 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1776_1dfeb68dada29a861d6d.png with policy now 2023-07-11 20:43:24,070 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:43:24,071 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1776_145e5db4ddf1294e74ca.png with policy now 2023-07-11 20:43:24,112 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:43:24,114 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:43:24,115 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:43:24,117 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:43:24,374 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/tabu6le8-media/images/validation_1776_7c1cf0a21ce762d708cc.png 2023-07-11 20:43:24,396 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ou97z9xe-media/images/validation_1776_a6253244e0fcd16b6309.png 2023-07-11 20:43:24,407 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/qadi01e9-media/images/validation_1776_145e5db4ddf1294e74ca.png 2023-07-11 20:43:24,420 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/pfsjbvlu-media/images/validation_1776_1dfeb68dada29a861d6d.png 2023-07-11 20:43:26,249 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:43:26,342 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:43:26,343 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1776_a6253244e0fcd16b6309.png 2023-07-11 20:43:26,343 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1776_1dfeb68dada29a861d6d.png 2023-07-11 20:43:26,344 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1776_7c1cf0a21ce762d708cc.png 2023-07-11 20:43:26,345 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1776_145e5db4ddf1294e74ca.png 2023-07-11 20:43:26,345 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 20:43:26,699 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/pqowu1xi-media/images/validation_1776_a6253244e0fcd16b6309.png 2023-07-11 20:43:26,705 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/fe16av2b-media/images/validation_1776_7c1cf0a21ce762d708cc.png 2023-07-11 20:43:26,715 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/dz0o7ups-media/images/validation_1776_145e5db4ddf1294e74ca.png 2023-07-11 20:43:26,725 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/epmz3exd-media/images/validation_1776_1dfeb68dada29a861d6d.png 2023-07-11 20:43:27,683 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:43:27,709 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:43:30,921 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:43:30,923 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:43:30,923 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:43:30,925 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:43:32,064 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:43:33,574 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:43:34,471 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:43:34,473 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:43:34,473 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:43:34,475 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:43:35,357 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:43:35,358 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:43:37,465 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:43:37,732 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:43:37,734 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:43:37,734 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:43:37,735 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:43:39,007 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:43:40,524 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:43:40,526 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:43:40,526 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:43:40,527 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:43:41,160 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:43:42,737 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:43:43,871 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:43:43,873 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:43:43,873 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:43:43,875 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:43:44,373 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:43:44,374 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:43:44,774 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:43:47,891 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:43:47,892 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:43:47,893 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:43:47,894 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:43:48,173 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:43:48,175 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:43:48,175 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:43:48,177 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:43:48,253 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:43:50,715 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:43:53,311 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:43:53,391 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:43:53,391 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:43:55,716 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:43:58,331 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:44:00,717 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:44:00,781 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:44:00,781 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1784_2f7a93c89a8e73332a71.png with policy now 2023-07-11 20:44:00,801 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:44:00,801 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1784_40c1334312f0eaa4a409.png with policy now 2023-07-11 20:44:00,810 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:44:00,811 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1784_0b7130c5c030a08cc234.png with policy now 2023-07-11 20:44:00,819 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:44:00,819 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1784_7e32e1185f32b992ee09.png with policy now 2023-07-11 20:44:00,860 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:44:00,862 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:44:00,862 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:44:00,864 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:44:01,090 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/joxox3je-media/images/validation_1784_2f7a93c89a8e73332a71.png 2023-07-11 20:44:01,141 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/39za5tum-media/images/validation_1784_40c1334312f0eaa4a409.png 2023-07-11 20:44:01,155 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/oqkqr3xh-media/images/validation_1784_0b7130c5c030a08cc234.png 2023-07-11 20:44:01,159 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/nhpqnygy-media/images/validation_1784_7e32e1185f32b992ee09.png 2023-07-11 20:44:02,409 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:44:02,410 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1784_40c1334312f0eaa4a409.png 2023-07-11 20:44:02,410 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1784_2f7a93c89a8e73332a71.png 2023-07-11 20:44:02,411 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1784_7e32e1185f32b992ee09.png 2023-07-11 20:44:02,411 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1784_0b7130c5c030a08cc234.png 2023-07-11 20:44:02,412 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 20:44:02,772 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ta31qdzh-media/images/validation_1784_2f7a93c89a8e73332a71.png 2023-07-11 20:44:02,774 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/5fgiyep5-media/images/validation_1784_7e32e1185f32b992ee09.png 2023-07-11 20:44:02,777 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ea1f4p8q-media/images/validation_1784_40c1334312f0eaa4a409.png 2023-07-11 20:44:02,793 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/rv9hblcx-media/images/validation_1784_0b7130c5c030a08cc234.png 2023-07-11 20:44:03,354 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:44:04,895 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:44:05,897 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:44:08,310 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:44:08,312 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:44:08,312 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:44:08,314 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:44:09,182 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:44:10,964 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:44:11,163 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:44:11,429 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:44:11,429 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:44:11,745 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:44:11,747 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:44:11,747 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:44:11,749 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:44:14,184 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:44:15,418 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:44:15,420 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:44:15,420 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:44:15,422 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:44:16,132 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:44:18,815 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:44:18,817 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:44:18,817 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:44:18,819 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:44:19,689 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:44:20,447 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:44:20,448 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:44:21,249 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:44:22,046 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:44:22,048 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:44:22,049 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:44:22,050 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:44:24,708 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:44:25,529 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:44:25,531 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:44:25,531 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:44:25,533 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:44:25,811 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:44:25,813 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:44:25,813 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:44:25,815 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:44:26,254 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:44:29,463 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:44:29,464 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:44:29,737 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:44:31,392 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:44:34,769 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:44:36,393 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:44:38,416 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:44:38,416 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1792_6c8c3c85979fc0112886.png with policy now 2023-07-11 20:44:38,438 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:44:38,438 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1792_03a513699d32547cfcce.png with policy now 2023-07-11 20:44:38,473 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:44:38,473 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1792_e03fe724b53ff007f509.png with policy now 2023-07-11 20:44:38,486 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1792_6c8c3c85979fc0112886.png 2023-07-11 20:44:38,496 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:44:38,497 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1792_5119d4d7a2257dd71e9b.png with policy now 2023-07-11 20:44:38,497 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1792_e03fe724b53ff007f509.png 2023-07-11 20:44:38,501 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1792_03a513699d32547cfcce.png 2023-07-11 20:44:38,502 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 20:44:38,551 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:44:38,552 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:44:38,552 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:44:38,554 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:44:38,742 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/199lk5ir-media/images/validation_1792_6c8c3c85979fc0112886.png 2023-07-11 20:44:38,808 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/243gncz7-media/images/validation_1792_03a513699d32547cfcce.png 2023-07-11 20:44:38,831 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/0yflqcvm-media/images/validation_1792_e03fe724b53ff007f509.png 2023-07-11 20:44:38,889 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/1h8wjlmw-media/images/validation_1792_5119d4d7a2257dd71e9b.png 2023-07-11 20:44:39,122 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/bpjrbtfc-media/images/validation_1792_6c8c3c85979fc0112886.png 2023-07-11 20:44:39,391 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/jkxx7x2k-media/images/validation_1792_e03fe724b53ff007f509.png 2023-07-11 20:44:39,415 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/qwkqezer-media/images/validation_1792_03a513699d32547cfcce.png 2023-07-11 20:44:39,824 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:44:41,166 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:44:41,393 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:44:41,959 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:44:44,691 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:44:44,692 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:44:44,693 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:44:44,694 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:44:44,912 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:44:47,261 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:44:47,506 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:44:47,506 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:44:47,506 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1792_5119d4d7a2257dd71e9b.png 2023-07-11 20:44:47,507 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 20:44:47,839 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/k1x6ko1q-media/images/validation_1792_5119d4d7a2257dd71e9b.png 2023-07-11 20:44:48,793 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:44:48,795 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:44:48,795 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:44:48,797 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:44:50,770 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:44:52,266 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:44:52,268 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:44:52,268 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:44:52,269 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:44:52,271 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:44:55,971 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:44:55,972 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:44:55,973 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:44:55,974 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:44:56,139 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:44:56,523 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:44:56,524 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:44:57,727 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:44:59,960 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:44:59,961 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:44:59,962 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:44:59,963 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:45:01,182 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:45:01,182 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:45:02,755 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:45:02,757 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:45:02,757 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:45:02,758 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:45:02,760 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:45:03,038 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:45:03,040 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:45:03,040 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:45:03,042 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:45:06,189 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:45:06,539 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:45:06,540 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:45:08,634 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:45:11,169 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:45:11,218 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:45:14,171 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:45:15,637 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:45:15,637 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1800_29829117e355f84867a9.png with policy now 2023-07-11 20:45:15,686 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:45:15,687 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1800_1d80862a2e619ee149af.png with policy now 2023-07-11 20:45:15,699 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:45:15,700 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1800_ba06bc64108c60c3d249.png with policy now 2023-07-11 20:45:15,710 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:45:15,710 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1800_595d783a674b31607abe.png with policy now 2023-07-11 20:45:15,747 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:45:15,748 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:45:15,749 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:45:15,756 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:45:15,958 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/aaj186d5-media/images/validation_1800_29829117e355f84867a9.png 2023-07-11 20:45:16,005 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/bvbmyyk8-media/images/validation_1800_ba06bc64108c60c3d249.png 2023-07-11 20:45:16,010 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/w0kd2ix5-media/images/validation_1800_1d80862a2e619ee149af.png 2023-07-11 20:45:16,029 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/h2w9f1h5-media/images/validation_1800_595d783a674b31607abe.png 2023-07-11 20:45:16,239 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:45:16,558 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:45:16,559 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1800_595d783a674b31607abe.png 2023-07-11 20:45:16,559 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1800_1d80862a2e619ee149af.png 2023-07-11 20:45:16,560 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1800_ba06bc64108c60c3d249.png 2023-07-11 20:45:16,560 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1800_29829117e355f84867a9.png 2023-07-11 20:45:16,561 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 20:45:16,906 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/fkx7t7pr-media/images/validation_1800_595d783a674b31607abe.png 2023-07-11 20:45:16,922 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/281kcp1n-media/images/validation_1800_1d80862a2e619ee149af.png 2023-07-11 20:45:16,934 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/3yrxak62-media/images/validation_1800_29829117e355f84867a9.png 2023-07-11 20:45:16,938 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/vg8iggu9-media/images/validation_1800_ba06bc64108c60c3d249.png 2023-07-11 20:45:19,538 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:45:19,539 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:45:21,240 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:45:22,830 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:45:22,832 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:45:22,832 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:45:22,834 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:45:24,774 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:45:25,788 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:45:25,789 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:45:25,790 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:45:25,791 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:45:26,575 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:45:26,575 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:45:26,876 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:45:28,999 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:45:29,001 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:45:29,001 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:45:29,003 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:45:30,585 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:45:32,217 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:45:32,343 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:45:32,345 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:45:32,345 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:45:32,347 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:45:36,178 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:45:36,234 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:45:36,235 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:45:36,236 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:45:36,237 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:45:36,679 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:45:36,679 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:45:37,228 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:45:39,269 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:45:39,271 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:45:39,272 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:45:39,273 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:45:39,552 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:45:39,554 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:45:39,555 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:45:39,556 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:45:41,172 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:45:42,173 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:45:42,527 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:45:46,696 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:45:46,696 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:45:47,175 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:45:47,558 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:45:52,176 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:45:52,200 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:45:52,200 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1808_2c08b8408a4ed78c3242.png with policy now 2023-07-11 20:45:52,242 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:45:52,242 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1808_4e784e29fddbb72791ca.png with policy now 2023-07-11 20:45:52,253 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:45:52,253 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1808_60edf5874ce5e54fff8a.png with policy now 2023-07-11 20:45:52,277 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:45:52,278 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1808_e42f3a5cff465c0326eb.png with policy now 2023-07-11 20:45:52,322 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:45:52,323 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:45:52,324 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:45:52,329 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:45:52,517 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/kyu68lnx-media/images/validation_1808_2c08b8408a4ed78c3242.png 2023-07-11 20:45:52,566 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/9mptq1x4-media/images/validation_1808_4e784e29fddbb72791ca.png 2023-07-11 20:45:52,568 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:45:52,581 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/0rxhv6fk-media/images/validation_1808_e42f3a5cff465c0326eb.png 2023-07-11 20:45:52,581 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/p0itklth-media/images/validation_1808_60edf5874ce5e54fff8a.png 2023-07-11 20:45:56,013 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:45:56,716 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:45:56,716 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:45:56,716 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1808_2c08b8408a4ed78c3242.png 2023-07-11 20:45:56,717 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1808_4e784e29fddbb72791ca.png 2023-07-11 20:45:56,717 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1808_e42f3a5cff465c0326eb.png 2023-07-11 20:45:56,718 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1808_60edf5874ce5e54fff8a.png 2023-07-11 20:45:56,718 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 20:45:57,059 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/m3vi0fpg-media/images/validation_1808_2c08b8408a4ed78c3242.png 2023-07-11 20:45:57,076 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/5txvw1gp-media/images/validation_1808_60edf5874ce5e54fff8a.png 2023-07-11 20:45:57,081 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/nytrjs14-media/images/validation_1808_4e784e29fddbb72791ca.png 2023-07-11 20:45:57,093 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/3djgoe9x-media/images/validation_1808_e42f3a5cff465c0326eb.png 2023-07-11 20:45:57,393 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:45:57,715 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:45:58,626 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:45:58,628 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:45:58,628 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:45:58,630 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:46:02,160 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:46:02,162 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:46:02,162 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:46:02,365 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:46:03,364 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:46:03,712 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:46:05,341 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:46:05,343 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:46:05,343 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:46:05,345 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:46:06,734 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:46:06,735 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:46:08,816 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:46:08,904 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:46:09,513 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:46:09,515 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:46:09,515 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:46:09,517 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:46:11,174 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:46:12,743 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:46:12,745 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:46:12,745 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:46:12,747 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:46:13,845 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:46:14,593 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:46:16,085 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:46:16,087 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:46:16,087 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:46:16,184 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:46:16,367 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:46:16,368 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:46:16,369 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:46:16,370 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:46:16,751 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:46:16,751 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:46:18,960 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:46:19,962 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:46:24,021 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:46:24,963 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:46:26,816 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:46:29,040 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:46:29,040 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1816_6ce5b89f865c878ca7ce.png with policy now 2023-07-11 20:46:29,048 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:46:29,059 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:46:29,059 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1816_db34427fe5dc5f158ded.png with policy now 2023-07-11 20:46:29,105 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:46:29,105 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1816_0b84c7e55854de6abb35.png with policy now 2023-07-11 20:46:29,124 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:46:29,124 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1816_c0afacd8e639ac88c950.png with policy now 2023-07-11 20:46:29,157 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:46:29,159 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:46:29,159 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:46:29,161 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:46:29,366 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/m74f1q2m-media/images/validation_1816_6ce5b89f865c878ca7ce.png 2023-07-11 20:46:29,387 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/iona4qj9-media/images/validation_1816_db34427fe5dc5f158ded.png 2023-07-11 20:46:29,420 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/6hywl2n9-media/images/validation_1816_0b84c7e55854de6abb35.png 2023-07-11 20:46:29,434 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/157z645q-media/images/validation_1816_c0afacd8e639ac88c950.png 2023-07-11 20:46:30,163 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:46:32,729 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:46:35,085 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:46:35,206 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:46:35,720 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:46:35,722 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:46:35,723 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:46:35,724 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:46:36,834 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:46:36,835 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:46:36,835 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1816_c0afacd8e639ac88c950.png 2023-07-11 20:46:36,835 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1816_6ce5b89f865c878ca7ce.png 2023-07-11 20:46:36,836 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1816_db34427fe5dc5f158ded.png 2023-07-11 20:46:36,836 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1816_0b84c7e55854de6abb35.png 2023-07-11 20:46:36,837 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 20:46:37,187 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/axxrizh3-media/images/validation_1816_6ce5b89f865c878ca7ce.png 2023-07-11 20:46:37,193 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/9yzhdrsr-media/images/validation_1816_c0afacd8e639ac88c950.png 2023-07-11 20:46:37,210 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/q00fxl0t-media/images/validation_1816_0b84c7e55854de6abb35.png 2023-07-11 20:46:37,216 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/xb9xh4a1-media/images/validation_1816_db34427fe5dc5f158ded.png 2023-07-11 20:46:39,061 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:46:39,063 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:46:39,063 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:46:39,065 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:46:40,334 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:46:40,418 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:46:41,177 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:46:43,499 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:46:43,501 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:46:43,502 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:46:43,503 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:46:45,704 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:46:46,404 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:46:46,572 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:46:46,573 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:46:46,573 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:46:46,575 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:46:46,852 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:46:46,852 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:46:50,558 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:46:50,559 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:46:50,560 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:46:50,561 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:46:51,408 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:46:51,429 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:46:54,369 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:46:54,371 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:46:54,371 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:46:54,373 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:46:54,651 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:46:54,653 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:46:54,654 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:46:54,655 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:46:56,431 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:46:56,868 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:46:56,868 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:46:57,294 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:47:01,468 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:47:02,295 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:47:06,508 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:47:07,296 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:47:07,314 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:47:07,315 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1824_de127351776d646ede0d.png with policy now 2023-07-11 20:47:07,325 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:47:07,325 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1824_2018a3ead09df8a1c987.png with policy now 2023-07-11 20:47:07,345 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:47:07,345 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1824_9cf90d8af441b007efe6.png with policy now 2023-07-11 20:47:07,354 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:47:07,354 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1824_ffbfc753df3dec5c3208.png with policy now 2023-07-11 20:47:07,397 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:47:07,399 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:47:07,399 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:47:07,400 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:47:07,640 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/5kpxj5wq-media/images/validation_1824_2018a3ead09df8a1c987.png 2023-07-11 20:47:07,662 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/1d7gq6s7-media/images/validation_1824_ffbfc753df3dec5c3208.png 2023-07-11 20:47:07,666 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/5j4uwh9t-media/images/validation_1824_9cf90d8af441b007efe6.png 2023-07-11 20:47:07,733 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/zub3oh83-media/images/validation_1824_de127351776d646ede0d.png 2023-07-11 20:47:10,972 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:47:11,180 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:47:11,734 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:47:12,415 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:47:14,420 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:47:14,422 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:47:14,422 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:47:14,424 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:47:16,905 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:47:16,905 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:47:16,906 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1824_2018a3ead09df8a1c987.png 2023-07-11 20:47:16,906 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1824_9cf90d8af441b007efe6.png 2023-07-11 20:47:16,907 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1824_de127351776d646ede0d.png 2023-07-11 20:47:16,907 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1824_ffbfc753df3dec5c3208.png 2023-07-11 20:47:16,908 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 20:47:17,251 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/pxdcm92o-media/images/validation_1824_2018a3ead09df8a1c987.png 2023-07-11 20:47:17,259 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/st6z3xxt-media/images/validation_1824_de127351776d646ede0d.png 2023-07-11 20:47:17,269 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/a90crkil-media/images/validation_1824_9cf90d8af441b007efe6.png 2023-07-11 20:47:17,273 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/nicxkd5p-media/images/validation_1824_ffbfc753df3dec5c3208.png 2023-07-11 20:47:17,482 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:47:17,482 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:47:17,611 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:47:17,613 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:47:17,613 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:47:17,613 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:47:17,615 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:47:20,637 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:47:20,639 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:47:20,639 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:47:20,641 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:47:22,693 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:47:23,142 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:47:23,380 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:47:23,381 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:47:23,382 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:47:23,383 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:47:26,923 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:47:26,923 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:47:27,477 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:47:27,479 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:47:27,479 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:47:27,481 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:47:28,222 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:47:28,555 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:47:30,796 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:47:30,798 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:47:30,799 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:47:30,801 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:47:31,080 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:47:31,081 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:47:31,082 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:47:31,083 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:47:33,683 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:47:33,748 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:47:36,939 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:47:36,940 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:47:38,732 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:47:38,749 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:47:41,182 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:47:43,753 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:47:43,837 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:47:43,837 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1832_9cd399d776df586c9c02.png with policy now 2023-07-11 20:47:43,838 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:47:43,859 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:47:43,860 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1832_5e23f48d1d42652a0b10.png with policy now 2023-07-11 20:47:43,925 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:47:43,925 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1832_7a8f27b94cbeac050b37.png with policy now 2023-07-11 20:47:43,938 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:47:43,938 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1832_1bfea247631943876000.png with policy now 2023-07-11 20:47:43,979 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:47:43,981 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:47:43,982 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:47:43,984 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:47:44,136 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/cx0oh95o-media/images/validation_1832_9cd399d776df586c9c02.png 2023-07-11 20:47:44,171 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/6fb881v3-media/images/validation_1832_5e23f48d1d42652a0b10.png 2023-07-11 20:47:44,238 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/x59ksicp-media/images/validation_1832_7a8f27b94cbeac050b37.png 2023-07-11 20:47:44,265 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/cw5qhxqf-media/images/validation_1832_1bfea247631943876000.png 2023-07-11 20:47:46,958 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:47:46,959 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:47:46,959 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1832_7a8f27b94cbeac050b37.png 2023-07-11 20:47:46,959 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1832_9cd399d776df586c9c02.png 2023-07-11 20:47:46,960 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1832_1bfea247631943876000.png 2023-07-11 20:47:46,961 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1832_5e23f48d1d42652a0b10.png 2023-07-11 20:47:46,961 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 20:47:47,311 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/gsth0foi-media/images/validation_1832_7a8f27b94cbeac050b37.png 2023-07-11 20:47:47,311 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/4a09qv02-media/images/validation_1832_9cd399d776df586c9c02.png 2023-07-11 20:47:47,330 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/kssdzc31-media/images/validation_1832_1bfea247631943876000.png 2023-07-11 20:47:47,340 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/kc2rxbwl-media/images/validation_1832_5e23f48d1d42652a0b10.png 2023-07-11 20:47:47,941 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:47:48,943 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:47:48,967 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:47:51,131 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:47:51,133 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:47:51,134 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:47:51,135 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:47:53,998 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:47:54,694 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:47:54,823 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:47:54,825 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:47:54,825 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:47:54,827 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:47:56,976 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:47:56,976 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:47:57,739 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:47:57,741 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:47:57,741 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:47:57,743 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:47:59,410 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:47:59,696 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:48:01,076 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:48:01,077 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:48:01,078 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:48:01,079 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:48:04,665 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:48:04,723 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:48:04,852 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:48:04,853 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:48:04,854 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:48:04,855 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:48:06,993 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:48:06,993 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:48:07,582 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:48:07,584 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:48:07,584 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:48:07,586 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:48:07,863 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:48:07,865 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:48:07,866 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:48:07,867 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:48:09,756 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:48:10,484 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:48:11,185 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:48:14,785 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:48:16,188 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:48:17,011 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:48:17,011 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:48:19,820 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:48:20,484 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:48:20,485 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1840_2e537d6ae5539024016c.png with policy now 2023-07-11 20:48:20,495 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:48:20,495 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1840_eacff8e0f39e8da08765.png with policy now 2023-07-11 20:48:20,524 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:48:20,524 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1840_beb7c5efe82f32f2c89f.png with policy now 2023-07-11 20:48:20,536 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:48:20,536 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1840_4fb89ef66c812201003f.png with policy now 2023-07-11 20:48:20,577 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:48:20,578 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:48:20,579 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:48:20,580 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:48:20,810 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/0rxh3byz-media/images/validation_1840_eacff8e0f39e8da08765.png 2023-07-11 20:48:20,817 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/b8si2t34-media/images/validation_1840_2e537d6ae5539024016c.png 2023-07-11 20:48:20,835 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/a0kgvi54-media/images/validation_1840_beb7c5efe82f32f2c89f.png 2023-07-11 20:48:20,847 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/gv6oy1g1-media/images/validation_1840_4fb89ef66c812201003f.png 2023-07-11 20:48:21,562 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:48:24,003 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:48:25,129 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:48:26,595 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:48:26,596 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:48:26,597 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:48:26,597 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:48:26,599 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:48:27,030 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:48:27,031 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:48:27,031 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1840_2e537d6ae5539024016c.png 2023-07-11 20:48:27,032 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1840_4fb89ef66c812201003f.png 2023-07-11 20:48:27,032 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1840_eacff8e0f39e8da08765.png 2023-07-11 20:48:27,033 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1840_beb7c5efe82f32f2c89f.png 2023-07-11 20:48:27,033 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 20:48:27,384 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/33bhwm9r-media/images/validation_1840_eacff8e0f39e8da08765.png 2023-07-11 20:48:27,390 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/dlgrkqke-media/images/validation_1840_beb7c5efe82f32f2c89f.png 2023-07-11 20:48:27,395 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/kx8exmfa-media/images/validation_1840_2e537d6ae5539024016c.png 2023-07-11 20:48:27,414 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/wwdwxl6b-media/images/validation_1840_4fb89ef66c812201003f.png 2023-07-11 20:48:29,914 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:48:29,915 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:48:29,916 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:48:29,917 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:48:30,467 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:48:32,081 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:48:33,819 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:48:33,820 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:48:33,820 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:48:33,822 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:48:36,346 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:48:37,048 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:48:37,048 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:48:37,373 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:48:37,375 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:48:37,375 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:48:37,375 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:48:37,377 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:48:40,742 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:48:40,744 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:48:40,744 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:48:40,746 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:48:41,188 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:48:41,681 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:48:42,482 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:48:44,511 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:48:44,513 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:48:44,513 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:48:44,518 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:48:44,796 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:48:44,797 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:48:44,798 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:48:44,799 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:48:46,726 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:48:47,061 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:48:47,062 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:48:48,351 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:48:51,748 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:48:53,352 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:48:56,793 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:48:57,382 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:48:57,382 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1848_08204171975de6aa75a9.png with policy now 2023-07-11 20:48:57,397 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:48:57,397 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1848_98bfc990a3604dddcca6.png with policy now 2023-07-11 20:48:57,407 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:48:57,407 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1848_d9322c759e78020c14f9.png with policy now 2023-07-11 20:48:57,414 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:48:57,414 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1848_1be533c9b8642ccbd69e.png with policy now 2023-07-11 20:48:57,451 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:48:57,452 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:48:57,452 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:48:57,454 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:48:57,726 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/yu6n9mku-media/images/validation_1848_08204171975de6aa75a9.png 2023-07-11 20:48:57,753 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/54tw7xfc-media/images/validation_1848_1be533c9b8642ccbd69e.png 2023-07-11 20:48:57,767 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/yfx9sb5z-media/images/validation_1848_98bfc990a3604dddcca6.png 2023-07-11 20:48:57,784 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/wsf28727-media/images/validation_1848_d9322c759e78020c14f9.png 2023-07-11 20:48:58,420 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:49:00,785 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:49:02,203 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:49:02,204 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:49:03,821 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:49:03,823 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:49:03,823 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:49:03,823 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:49:03,825 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:49:07,099 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:49:07,099 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:49:07,100 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1848_08204171975de6aa75a9.png 2023-07-11 20:49:07,100 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1848_98bfc990a3604dddcca6.png 2023-07-11 20:49:07,101 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1848_d9322c759e78020c14f9.png 2023-07-11 20:49:07,101 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1848_1be533c9b8642ccbd69e.png 2023-07-11 20:49:07,101 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 20:49:07,403 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:49:07,405 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:49:07,405 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:49:07,407 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:49:07,445 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/2acsonhh-media/images/validation_1848_08204171975de6aa75a9.png 2023-07-11 20:49:07,459 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/en6ko4is-media/images/validation_1848_1be533c9b8642ccbd69e.png 2023-07-11 20:49:07,459 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/7g96puyq-media/images/validation_1848_d9322c759e78020c14f9.png 2023-07-11 20:49:07,467 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/euvhmbzk-media/images/validation_1848_98bfc990a3604dddcca6.png 2023-07-11 20:49:08,094 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:49:08,883 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:49:10,827 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:49:10,829 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:49:10,829 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:49:10,831 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:49:11,191 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:49:13,238 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:49:13,929 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:49:14,033 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:49:14,035 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:49:14,035 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:49:14,037 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:49:17,484 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:49:17,485 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:49:18,013 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:49:18,015 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:49:18,015 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:49:18,017 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:49:19,068 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:49:19,924 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:49:21,826 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:49:21,828 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:49:21,828 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:49:21,830 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:49:22,109 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:49:22,111 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:49:22,111 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:49:22,113 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:49:24,382 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:49:25,748 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:49:27,502 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:49:27,503 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:49:29,406 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:49:30,749 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:49:34,454 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:49:34,804 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:49:34,804 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1856_fc9e1244d8b7441727e9.png with policy now 2023-07-11 20:49:34,828 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:49:34,829 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1856_27b928ab82c5575c233c.png with policy now 2023-07-11 20:49:34,838 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:49:34,838 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1856_acd7add95c524a43efc7.png with policy now 2023-07-11 20:49:34,847 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:49:34,847 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1856_9f028c76839d73eab2ab.png with policy now 2023-07-11 20:49:34,888 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:49:34,889 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:49:34,890 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:49:34,891 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:49:35,197 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/rkco542v-media/images/validation_1856_fc9e1244d8b7441727e9.png 2023-07-11 20:49:35,226 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/19a231ff-media/images/validation_1856_9f028c76839d73eab2ab.png 2023-07-11 20:49:35,235 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/5g5pwrvt-media/images/validation_1856_27b928ab82c5575c233c.png 2023-07-11 20:49:35,260 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/9tmtjpo0-media/images/validation_1856_acd7add95c524a43efc7.png 2023-07-11 20:49:35,892 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:49:37,522 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:49:37,523 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:49:37,523 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1856_acd7add95c524a43efc7.png 2023-07-11 20:49:37,524 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1856_27b928ab82c5575c233c.png 2023-07-11 20:49:37,524 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1856_fc9e1244d8b7441727e9.png 2023-07-11 20:49:37,525 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1856_9f028c76839d73eab2ab.png 2023-07-11 20:49:37,525 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 20:49:37,890 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/pd8swd2p-media/images/validation_1856_fc9e1244d8b7441727e9.png 2023-07-11 20:49:37,892 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/4ierbuva-media/images/validation_1856_9f028c76839d73eab2ab.png 2023-07-11 20:49:37,916 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/n6omws74-media/images/validation_1856_27b928ab82c5575c233c.png 2023-07-11 20:49:37,979 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/xi62dx5b-media/images/validation_1856_acd7add95c524a43efc7.png 2023-07-11 20:49:38,313 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:49:40,196 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:49:41,193 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:49:41,197 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:49:41,883 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:49:41,885 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:49:41,885 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:49:41,887 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:49:44,893 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:49:44,895 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:49:44,895 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:49:44,897 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:49:45,822 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:49:46,479 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:49:47,540 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:49:47,540 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:49:48,777 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:49:48,778 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:49:48,779 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:49:48,780 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:49:51,008 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:49:51,946 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:49:52,844 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:49:52,846 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:49:52,846 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:49:52,848 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:49:56,021 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:49:56,077 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:49:56,079 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:49:56,079 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:49:56,080 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:49:57,557 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:49:57,558 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:49:57,710 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:49:59,380 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:49:59,382 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:49:59,382 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:49:59,384 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:49:59,662 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:49:59,663 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:49:59,664 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:49:59,665 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:50:01,252 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:50:01,253 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:50:03,268 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:50:06,286 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:50:07,575 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:50:07,576 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:50:08,269 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:50:11,195 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:50:11,332 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:50:12,279 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:50:12,279 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1864_ec496ac094d6a7c53ab6.png with policy now 2023-07-11 20:50:12,293 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:50:12,294 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1864_2cb8e9d19416aa855ae3.png with policy now 2023-07-11 20:50:12,303 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:50:12,304 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1864_bb9b59dfd7940d706691.png with policy now 2023-07-11 20:50:12,312 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:50:12,313 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1864_3af6cca038957854954d.png with policy now 2023-07-11 20:50:12,349 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:50:12,353 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:50:12,353 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:50:12,360 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:50:12,596 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/irr7cvo4-media/images/validation_1864_ec496ac094d6a7c53ab6.png 2023-07-11 20:50:12,613 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/7afu1drj-media/images/validation_1864_2cb8e9d19416aa855ae3.png 2023-07-11 20:50:12,620 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/mxyio9fi-media/images/validation_1864_bb9b59dfd7940d706691.png 2023-07-11 20:50:12,626 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/w61iaif5-media/images/validation_1864_3af6cca038957854954d.png 2023-07-11 20:50:13,361 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:50:15,789 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:50:16,661 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:50:17,595 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:50:17,595 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:50:17,596 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1864_3af6cca038957854954d.png 2023-07-11 20:50:17,596 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1864_ec496ac094d6a7c53ab6.png 2023-07-11 20:50:17,597 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1864_bb9b59dfd7940d706691.png 2023-07-11 20:50:17,597 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1864_2cb8e9d19416aa855ae3.png 2023-07-11 20:50:17,598 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 20:50:17,954 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/3gptqo5q-media/images/validation_1864_3af6cca038957854954d.png 2023-07-11 20:50:17,973 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/82kw8nt8-media/images/validation_1864_ec496ac094d6a7c53ab6.png 2023-07-11 20:50:17,976 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/pgzh9h5y-media/images/validation_1864_2cb8e9d19416aa855ae3.png 2023-07-11 20:50:17,981 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/bjl2sl90-media/images/validation_1864_bb9b59dfd7940d706691.png 2023-07-11 20:50:18,630 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:50:18,632 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:50:18,632 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:50:18,633 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:50:18,634 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:50:21,679 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:50:21,736 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:50:21,738 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:50:21,738 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:50:21,740 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:50:24,382 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:50:25,718 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:50:25,720 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:50:25,720 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:50:25,722 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:50:26,767 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:50:27,612 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:50:27,613 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:50:29,006 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:50:29,008 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:50:29,008 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:50:29,010 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:50:29,820 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:50:32,103 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:50:32,104 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:50:32,333 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:50:32,334 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:50:32,334 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:50:32,336 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:50:35,557 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:50:35,559 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:50:35,559 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:50:35,559 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:50:35,561 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:50:35,839 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:50:35,841 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:50:35,841 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:50:35,842 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:50:37,105 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:50:37,630 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:50:37,631 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:50:41,198 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:50:41,203 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:50:42,141 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:50:46,201 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:50:47,185 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:50:48,452 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:50:48,452 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1872_cdeb27bcabdc8cb919a1.png with policy now 2023-07-11 20:50:48,467 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:50:48,467 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1872_65231e5433fd842a9973.png with policy now 2023-07-11 20:50:48,477 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:50:48,477 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1872_31e589344f5ef8978c9f.png with policy now 2023-07-11 20:50:48,496 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:50:48,496 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1872_c36006f7e46b670362b4.png with policy now 2023-07-11 20:50:48,544 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:50:48,547 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:50:48,547 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:50:48,549 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:50:48,764 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/mkmnr1l1-media/images/validation_1872_cdeb27bcabdc8cb919a1.png 2023-07-11 20:50:48,805 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/rnuls9j1-media/images/validation_1872_c36006f7e46b670362b4.png 2023-07-11 20:50:48,806 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ytzg5dm9-media/images/validation_1872_65231e5433fd842a9973.png 2023-07-11 20:50:48,808 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/f9snzffr-media/images/validation_1872_31e589344f5ef8978c9f.png 2023-07-11 20:50:51,548 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:50:51,667 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:50:52,541 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:50:54,847 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:50:54,848 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:50:54,849 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:50:54,850 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:50:57,195 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:50:57,674 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:50:57,675 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:50:57,676 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1872_65231e5433fd842a9973.png 2023-07-11 20:50:57,676 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1872_cdeb27bcabdc8cb919a1.png 2023-07-11 20:50:57,677 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1872_c36006f7e46b670362b4.png 2023-07-11 20:50:57,677 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1872_31e589344f5ef8978c9f.png 2023-07-11 20:50:57,678 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 20:50:57,955 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:50:58,034 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/6w8vfay9-media/images/validation_1872_c36006f7e46b670362b4.png 2023-07-11 20:50:58,045 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/4v0u0xah-media/images/validation_1872_65231e5433fd842a9973.png 2023-07-11 20:50:58,053 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/cqk18ldw-media/images/validation_1872_cdeb27bcabdc8cb919a1.png 2023-07-11 20:50:58,114 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/0sfvinsw-media/images/validation_1872_31e589344f5ef8978c9f.png 2023-07-11 20:50:58,780 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:50:58,782 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:50:58,782 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:50:58,784 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:51:02,178 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:51:02,180 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:51:02,180 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:51:02,182 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:51:03,183 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:51:03,385 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:51:06,193 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:51:06,195 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:51:06,195 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:51:06,327 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:51:07,695 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:51:07,695 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:51:09,033 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:51:09,149 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:51:09,162 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:51:09,163 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:51:09,164 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:51:09,165 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:51:11,200 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:51:12,530 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:51:12,532 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:51:12,532 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:51:12,534 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:51:12,812 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:51:12,813 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:51:12,814 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:51:12,815 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:51:14,035 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:51:14,351 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:51:17,711 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:51:17,712 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:51:19,070 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:51:19,352 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:51:24,120 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:51:24,353 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:51:25,421 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:51:25,421 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1880_59266af13de06f4cd472.png with policy now 2023-07-11 20:51:25,438 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:51:25,438 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1880_75325ad11ccd36e3bca1.png with policy now 2023-07-11 20:51:25,450 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:51:25,451 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1880_c848d9be75adb5960650.png with policy now 2023-07-11 20:51:25,460 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:51:25,460 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1880_16f97c54584e8c176782.png with policy now 2023-07-11 20:51:25,498 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:51:25,499 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:51:25,500 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:51:25,501 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:51:25,793 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/yejn46qu-media/images/validation_1880_16f97c54584e8c176782.png 2023-07-11 20:51:25,795 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/1iu8yypz-media/images/validation_1880_59266af13de06f4cd472.png 2023-07-11 20:51:25,811 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/6we810wx-media/images/validation_1880_c848d9be75adb5960650.png 2023-07-11 20:51:25,813 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/cy1emacu-media/images/validation_1880_75325ad11ccd36e3bca1.png 2023-07-11 20:51:27,731 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:51:27,731 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1880_75325ad11ccd36e3bca1.png 2023-07-11 20:51:27,732 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1880_16f97c54584e8c176782.png 2023-07-11 20:51:27,733 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1880_c848d9be75adb5960650.png 2023-07-11 20:51:27,733 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1880_59266af13de06f4cd472.png 2023-07-11 20:51:27,734 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 20:51:28,241 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/kbfdkpvb-media/images/validation_1880_75325ad11ccd36e3bca1.png 2023-07-11 20:51:28,288 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/w4a4obv3-media/images/validation_1880_16f97c54584e8c176782.png 2023-07-11 20:51:28,309 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/foj7yqz8-media/images/validation_1880_59266af13de06f4cd472.png 2023-07-11 20:51:28,311 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/7aribtyn-media/images/validation_1880_c848d9be75adb5960650.png 2023-07-11 20:51:29,405 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:51:29,476 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:51:29,477 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:51:29,606 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:51:33,215 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:51:33,216 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:51:33,217 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:51:33,218 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:51:34,465 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:51:35,087 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:51:35,663 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:51:35,664 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:51:35,666 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:51:35,668 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:51:37,749 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:51:37,749 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:51:39,459 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:51:39,461 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:51:39,461 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:51:39,463 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:51:40,297 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:51:40,349 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:51:41,203 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:51:42,773 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:51:42,775 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:51:42,775 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:51:42,777 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:51:45,306 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:51:46,623 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:51:46,623 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:51:46,749 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:51:46,751 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:51:46,751 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:51:46,753 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:51:47,767 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:51:47,768 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:51:49,870 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:51:49,872 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:51:49,873 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:51:49,874 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:51:50,153 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:51:50,155 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:51:50,155 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:51:50,157 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:51:50,564 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:51:51,743 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:51:55,703 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:51:56,791 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:51:57,785 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:51:57,786 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:52:00,704 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:52:01,829 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:52:02,747 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:52:02,747 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1888_7b75911ccdb678b3a976.png with policy now 2023-07-11 20:52:02,757 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:52:02,757 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1888_86d54203ecfaca6291d0.png with policy now 2023-07-11 20:52:02,783 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:52:02,783 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1888_8ae9cfb1af9ccf7c1a1d.png with policy now 2023-07-11 20:52:02,795 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:52:02,795 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1888_3adb0b07a95c158b84ad.png with policy now 2023-07-11 20:52:02,834 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:52:02,835 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:52:02,836 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:52:02,838 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:52:03,087 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/5c8kh0is-media/images/validation_1888_86d54203ecfaca6291d0.png 2023-07-11 20:52:03,093 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/r9zze4vr-media/images/validation_1888_3adb0b07a95c158b84ad.png 2023-07-11 20:52:03,096 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/7b3gla6n-media/images/validation_1888_8ae9cfb1af9ccf7c1a1d.png 2023-07-11 20:52:03,105 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/qw8vzzfh-media/images/validation_1888_7b75911ccdb678b3a976.png 2023-07-11 20:52:05,762 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:52:06,971 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:52:07,100 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:52:07,806 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:52:07,806 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:52:07,807 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1888_3adb0b07a95c158b84ad.png 2023-07-11 20:52:07,807 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1888_8ae9cfb1af9ccf7c1a1d.png 2023-07-11 20:52:07,808 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1888_7b75911ccdb678b3a976.png 2023-07-11 20:52:07,808 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1888_86d54203ecfaca6291d0.png 2023-07-11 20:52:07,808 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 20:52:08,156 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/toe7m9g5-media/images/validation_1888_3adb0b07a95c158b84ad.png 2023-07-11 20:52:08,166 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/7vhbe1c0-media/images/validation_1888_8ae9cfb1af9ccf7c1a1d.png 2023-07-11 20:52:08,170 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/i0nxprh7-media/images/validation_1888_7b75911ccdb678b3a976.png 2023-07-11 20:52:08,185 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/6h5iqlqb-media/images/validation_1888_86d54203ecfaca6291d0.png 2023-07-11 20:52:09,773 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:52:09,775 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:52:09,775 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:52:09,777 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:52:11,205 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:52:11,210 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:52:12,283 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:52:13,009 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:52:13,010 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:52:13,011 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:52:13,012 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:52:15,764 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:52:15,765 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:52:15,766 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:52:15,767 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:52:16,769 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:52:17,824 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:52:17,824 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:52:17,825 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:52:17,825 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:52:19,392 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:52:19,394 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:52:19,394 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:52:19,396 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:52:21,840 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:52:22,579 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:52:22,581 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:52:22,582 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:52:22,583 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:52:23,067 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:52:25,942 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:52:25,944 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:52:25,945 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:52:25,946 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:52:26,226 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:52:26,228 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:52:26,228 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:52:26,230 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:52:27,754 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:52:27,840 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:52:27,841 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:52:28,071 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:52:32,755 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:52:33,094 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:52:37,757 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:52:38,128 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:52:38,803 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:52:38,804 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1896_a000f6436ad45ec3f551.png with policy now 2023-07-11 20:52:38,817 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:52:38,817 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1896_43076fbb48e6ed29a61c.png with policy now 2023-07-11 20:52:38,828 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:52:38,828 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1896_27c5036394cf299adb47.png with policy now 2023-07-11 20:52:38,870 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:52:38,870 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1896_29f4e682c32068be6ee6.png with policy now 2023-07-11 20:52:38,909 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:52:38,910 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:52:38,911 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:52:38,912 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:52:39,106 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/hugh9mfc-media/images/validation_1896_a000f6436ad45ec3f551.png 2023-07-11 20:52:39,157 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/gumwlaqq-media/images/validation_1896_27c5036394cf299adb47.png 2023-07-11 20:52:39,170 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/urg2xwzz-media/images/validation_1896_43076fbb48e6ed29a61c.png 2023-07-11 20:52:39,337 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/a9yrbp7d-media/images/validation_1896_29f4e682c32068be6ee6.png 2023-07-11 20:52:41,207 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:52:42,647 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:52:43,543 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:52:43,649 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:52:46,584 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:52:46,586 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:52:46,586 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:52:46,588 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:52:47,879 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:52:47,879 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:52:47,880 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1896_29f4e682c32068be6ee6.png 2023-07-11 20:52:47,880 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1896_27c5036394cf299adb47.png 2023-07-11 20:52:47,881 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1896_a000f6436ad45ec3f551.png 2023-07-11 20:52:47,881 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1896_43076fbb48e6ed29a61c.png 2023-07-11 20:52:47,882 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 20:52:48,236 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/6b4oe8uy-media/images/validation_1896_a000f6436ad45ec3f551.png 2023-07-11 20:52:48,246 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ajg07cl1-media/images/validation_1896_43076fbb48e6ed29a61c.png 2023-07-11 20:52:48,257 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/xvg9gyri-media/images/validation_1896_27c5036394cf299adb47.png 2023-07-11 20:52:48,449 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/1uixrvce-media/images/validation_1896_29f4e682c32068be6ee6.png 2023-07-11 20:52:48,845 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:52:48,847 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:52:48,847 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:52:48,847 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:52:48,849 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:52:49,577 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:52:51,875 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:52:51,877 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:52:51,877 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:52:51,880 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:52:54,607 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:52:54,788 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:52:55,584 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:52:55,586 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:52:55,587 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:52:55,588 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:52:57,897 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:52:57,898 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:52:59,189 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:52:59,191 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:52:59,192 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:52:59,193 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:52:59,811 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:53:00,064 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:53:01,751 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:53:01,752 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:53:01,753 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:53:01,755 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:53:02,033 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:53:02,035 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:53:02,035 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:53:02,037 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:53:05,094 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:53:05,582 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:53:07,915 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:53:07,916 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:53:10,163 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:53:10,583 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:53:11,210 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:53:14,630 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:53:14,630 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1904_a8928482b1db701915ac.png with policy now 2023-07-11 20:53:14,640 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:53:14,641 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1904_234baf3f279d560d92d2.png with policy now 2023-07-11 20:53:14,650 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:53:14,650 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1904_c1886dcd749cbc60adfd.png with policy now 2023-07-11 20:53:14,659 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:53:14,659 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1904_35be072a700b769038db.png with policy now 2023-07-11 20:53:14,699 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:53:14,701 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:53:14,701 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:53:14,702 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:53:14,968 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/du567vb6-media/images/validation_1904_234baf3f279d560d92d2.png 2023-07-11 20:53:14,972 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/cnglq4v5-media/images/validation_1904_35be072a700b769038db.png 2023-07-11 20:53:14,981 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ro24mdmo-media/images/validation_1904_c1886dcd749cbc60adfd.png 2023-07-11 20:53:15,024 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/9zo087nm-media/images/validation_1904_a8928482b1db701915ac.png 2023-07-11 20:53:15,187 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:53:15,703 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:53:17,935 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:53:17,935 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:53:17,936 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1904_234baf3f279d560d92d2.png 2023-07-11 20:53:17,936 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1904_a8928482b1db701915ac.png 2023-07-11 20:53:17,937 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1904_c1886dcd749cbc60adfd.png 2023-07-11 20:53:17,937 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1904_35be072a700b769038db.png 2023-07-11 20:53:17,938 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 20:53:18,306 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/knoy8egv-media/images/validation_1904_a8928482b1db701915ac.png 2023-07-11 20:53:18,318 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ioxupmkc-media/images/validation_1904_c1886dcd749cbc60adfd.png 2023-07-11 20:53:18,322 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/qpz62pa3-media/images/validation_1904_35be072a700b769038db.png 2023-07-11 20:53:18,335 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/lfrlvcoh-media/images/validation_1904_234baf3f279d560d92d2.png 2023-07-11 20:53:18,458 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:53:20,865 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:53:20,882 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:53:21,838 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:53:21,840 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:53:21,840 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:53:21,842 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:53:24,897 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:53:24,899 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:53:24,899 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:53:24,901 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:53:26,025 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:53:26,672 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:53:27,952 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:53:27,953 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:53:28,242 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:53:28,243 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:53:28,244 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:53:28,245 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:53:31,249 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:53:31,249 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:53:32,055 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:53:32,057 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:53:32,057 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:53:32,058 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:53:32,060 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:53:34,903 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:53:34,905 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:53:34,906 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:53:34,907 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:53:36,776 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:53:37,385 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:53:37,971 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:53:37,971 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:53:38,283 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:53:38,285 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:53:38,285 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:53:38,287 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:53:38,565 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:53:38,567 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:53:38,567 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:53:38,569 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:53:41,214 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:53:41,812 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:53:43,216 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:53:46,849 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:53:47,989 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:53:47,989 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:53:48,217 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:53:51,195 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:53:51,196 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1912_cdb5377e850db598c266.png with policy now 2023-07-11 20:53:51,211 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:53:51,211 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1912_3af03303490f4370396f.png with policy now 2023-07-11 20:53:51,221 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:53:51,221 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1912_3adb61482ec02a0583d5.png with policy now 2023-07-11 20:53:51,234 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:53:51,234 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1912_9da4544d95a30d67cb32.png with policy now 2023-07-11 20:53:51,275 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:53:51,278 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:53:51,278 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:53:51,280 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:53:51,524 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/75sr9ip7-media/images/validation_1912_cdb5377e850db598c266.png 2023-07-11 20:53:51,545 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/4mhunkwq-media/images/validation_1912_3adb61482ec02a0583d5.png 2023-07-11 20:53:51,549 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/1ipmioei-media/images/validation_1912_9da4544d95a30d67cb32.png 2023-07-11 20:53:51,575 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ski0jo8o-media/images/validation_1912_3af03303490f4370396f.png 2023-07-11 20:53:52,377 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:53:53,227 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:53:54,833 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:53:57,935 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:53:58,010 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:53:58,011 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:53:58,011 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1912_cdb5377e850db598c266.png 2023-07-11 20:53:58,012 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1912_3af03303490f4370396f.png 2023-07-11 20:53:58,012 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1912_3adb61482ec02a0583d5.png 2023-07-11 20:53:58,013 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1912_9da4544d95a30d67cb32.png 2023-07-11 20:53:58,013 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 20:53:58,336 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/1kn2el1g-media/images/validation_1912_cdb5377e850db598c266.png 2023-07-11 20:53:58,372 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/02bu4m8u-media/images/validation_1912_3adb61482ec02a0583d5.png 2023-07-11 20:53:58,375 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/4hg5sz67-media/images/validation_1912_9da4544d95a30d67cb32.png 2023-07-11 20:53:58,407 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/udogmsjq-media/images/validation_1912_3af03303490f4370396f.png 2023-07-11 20:53:59,057 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:53:59,162 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:53:59,164 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:53:59,164 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:53:59,166 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:54:02,064 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:54:02,066 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:54:02,066 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:54:02,068 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:54:03,271 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:54:04,192 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:54:04,945 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:54:04,947 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:54:04,947 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:54:04,949 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:54:07,639 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:54:07,641 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:54:07,641 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:54:07,643 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:54:08,028 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:54:08,029 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:54:08,272 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:54:09,616 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:54:10,778 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:54:10,780 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:54:10,780 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:54:10,782 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:54:11,215 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:54:13,275 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:54:13,508 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:54:13,510 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:54:13,510 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:54:13,512 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:54:13,796 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:54:13,798 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:54:13,798 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:54:13,800 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:54:15,347 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:54:18,046 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:54:18,047 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:54:18,313 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:54:18,313 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:54:20,348 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:54:23,336 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:54:25,349 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:54:26,352 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:54:26,352 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1920_d5a4ff7de7a7153f2941.png with policy now 2023-07-11 20:54:26,366 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:54:26,366 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1920_0d10a7dbf7251bbada04.png with policy now 2023-07-11 20:54:26,376 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:54:26,376 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1920_d289710871a92ba999e8.png with policy now 2023-07-11 20:54:26,406 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:54:26,406 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1920_6d4d45fd2048c6be252f.png with policy now 2023-07-11 20:54:26,447 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:54:26,448 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:54:26,448 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:54:26,450 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:54:26,670 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/z6g104ho-media/images/validation_1920_d5a4ff7de7a7153f2941.png 2023-07-11 20:54:26,688 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/4n24id57-media/images/validation_1920_0d10a7dbf7251bbada04.png 2023-07-11 20:54:26,690 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ly0onqbu-media/images/validation_1920_d289710871a92ba999e8.png 2023-07-11 20:54:26,717 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/7tw1u0dl-media/images/validation_1920_6d4d45fd2048c6be252f.png 2023-07-11 20:54:28,066 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:54:28,067 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1920_d5a4ff7de7a7153f2941.png 2023-07-11 20:54:28,067 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1920_6d4d45fd2048c6be252f.png 2023-07-11 20:54:28,067 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1920_d289710871a92ba999e8.png 2023-07-11 20:54:28,068 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1920_0d10a7dbf7251bbada04.png 2023-07-11 20:54:28,068 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 20:54:28,410 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/zdwd0778-media/images/validation_1920_6d4d45fd2048c6be252f.png 2023-07-11 20:54:28,423 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/8pykuquu-media/images/validation_1920_d289710871a92ba999e8.png 2023-07-11 20:54:28,442 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/nkdagpbm-media/images/validation_1920_0d10a7dbf7251bbada04.png 2023-07-11 20:54:28,683 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/xu618fau-media/images/validation_1920_d5a4ff7de7a7153f2941.png 2023-07-11 20:54:28,892 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:54:30,315 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:54:31,317 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:54:33,671 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:54:33,673 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:54:33,673 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:54:33,675 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:54:34,388 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:54:36,637 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:54:36,638 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:54:36,639 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:54:36,639 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:54:36,641 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:54:38,085 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:54:38,085 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:54:40,086 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:54:40,215 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:54:40,216 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:54:40,216 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:54:40,218 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:54:41,219 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:54:42,097 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:54:43,812 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:54:43,814 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:54:43,814 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:54:43,816 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:54:45,301 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:54:47,019 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:54:47,021 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:54:47,021 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:54:47,023 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:54:47,979 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:54:48,104 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:54:48,104 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:54:50,411 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:54:50,539 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:54:50,541 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:54:50,541 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:54:50,543 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:54:50,823 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:54:50,825 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:54:50,825 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:54:50,827 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:54:53,376 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:54:55,413 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:54:58,121 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:54:58,122 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:54:58,377 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:55:00,451 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:55:03,380 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:55:03,427 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:55:03,428 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1928_a008fe9b5be650d7d99f.png with policy now 2023-07-11 20:55:03,438 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:55:03,438 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1928_4bd865ae05d0074a2375.png with policy now 2023-07-11 20:55:03,447 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:55:03,447 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1928_78729b8aaac1e6b08697.png with policy now 2023-07-11 20:55:03,464 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:55:03,464 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1928_af8fa5edbca9692cffc2.png with policy now 2023-07-11 20:55:03,503 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:55:03,504 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:55:03,505 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:55:03,507 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:55:03,765 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/nejv2t8l-media/images/validation_1928_4bd865ae05d0074a2375.png 2023-07-11 20:55:03,771 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/e89vbbns-media/images/validation_1928_a008fe9b5be650d7d99f.png 2023-07-11 20:55:03,788 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/22cz99pm-media/images/validation_1928_78729b8aaac1e6b08697.png 2023-07-11 20:55:03,815 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ao9q5wtb-media/images/validation_1928_af8fa5edbca9692cffc2.png 2023-07-11 20:55:06,073 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:55:07,293 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:55:08,143 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:55:08,143 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:55:08,143 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1928_78729b8aaac1e6b08697.png 2023-07-11 20:55:08,144 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1928_af8fa5edbca9692cffc2.png 2023-07-11 20:55:08,145 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1928_4bd865ae05d0074a2375.png 2023-07-11 20:55:08,145 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1928_a008fe9b5be650d7d99f.png 2023-07-11 20:55:08,146 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 20:55:08,711 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/fobo0wbz-media/images/validation_1928_78729b8aaac1e6b08697.png 2023-07-11 20:55:08,712 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/vyzkc51z-media/images/validation_1928_4bd865ae05d0074a2375.png 2023-07-11 20:55:08,769 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/2lt4zlh6-media/images/validation_1928_af8fa5edbca9692cffc2.png 2023-07-11 20:55:08,776 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:55:08,911 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/rall110x-media/images/validation_1928_a008fe9b5be650d7d99f.png 2023-07-11 20:55:10,273 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:55:10,275 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:55:10,276 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:55:10,277 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:55:11,221 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:55:11,352 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:55:12,918 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:55:12,920 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:55:12,920 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:55:12,922 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:55:14,019 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:55:15,856 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:55:15,858 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:55:15,858 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:55:15,860 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:55:16,733 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:55:18,161 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:55:18,161 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:55:19,527 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:55:19,795 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:55:19,797 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:55:19,797 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:55:19,799 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:55:21,789 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:55:23,337 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:55:23,339 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:55:23,339 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:55:23,340 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:55:25,158 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:55:26,866 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:55:26,995 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:55:26,997 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:55:26,998 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:55:26,999 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:55:27,281 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:55:27,283 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:55:27,283 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:55:27,285 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:55:28,179 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:55:28,180 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:55:30,861 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:55:31,874 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:55:35,863 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:55:36,910 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:55:38,197 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:55:39,879 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:55:39,879 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1936_35429031d120f2b96126.png with policy now 2023-07-11 20:55:39,889 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:55:39,889 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1936_f3a3fadee46766d2470a.png with policy now 2023-07-11 20:55:39,898 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:55:39,898 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1936_dcec368a7622b3d8c0be.png with policy now 2023-07-11 20:55:39,907 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:55:39,907 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1936_852acc36599591503eef.png with policy now 2023-07-11 20:55:39,953 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:55:39,954 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:55:39,954 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:55:39,957 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:55:40,180 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/3l5l2a52-media/images/validation_1936_35429031d120f2b96126.png 2023-07-11 20:55:40,212 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/043wd9nc-media/images/validation_1936_f3a3fadee46766d2470a.png 2023-07-11 20:55:40,227 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ubm2n2tt-media/images/validation_1936_dcec368a7622b3d8c0be.png 2023-07-11 20:55:40,235 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/gl9u4bwt-media/images/validation_1936_852acc36599591503eef.png 2023-07-11 20:55:40,960 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:55:41,225 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:55:42,579 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:55:44,213 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:55:46,436 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:55:47,554 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:55:47,556 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:55:47,556 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:55:47,558 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:55:47,592 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:55:48,218 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:55:48,219 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:55:48,219 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1936_35429031d120f2b96126.png 2023-07-11 20:55:48,219 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1936_dcec368a7622b3d8c0be.png 2023-07-11 20:55:48,220 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1936_852acc36599591503eef.png 2023-07-11 20:55:48,221 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1936_f3a3fadee46766d2470a.png 2023-07-11 20:55:48,221 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 20:55:48,669 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/00ixd4wy-media/images/validation_1936_852acc36599591503eef.png 2023-07-11 20:55:48,677 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/i4k0d9yn-media/images/validation_1936_dcec368a7622b3d8c0be.png 2023-07-11 20:55:48,733 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/eioy2nz1-media/images/validation_1936_35429031d120f2b96126.png 2023-07-11 20:55:48,795 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/3vlm2k6k-media/images/validation_1936_f3a3fadee46766d2470a.png 2023-07-11 20:55:50,610 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:55:50,612 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:55:50,612 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:55:50,614 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:55:52,402 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:55:53,614 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:55:53,744 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:55:53,745 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:55:53,745 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:55:53,747 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:55:57,640 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:55:57,642 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:55:57,642 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:55:57,643 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:55:57,644 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:55:58,237 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:55:58,237 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:55:59,352 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:55:59,352 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:56:00,854 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:56:00,856 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:56:00,856 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:56:00,858 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:56:02,725 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:56:04,353 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:56:04,575 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:56:04,576 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:56:04,577 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:56:04,578 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:56:04,857 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:56:04,858 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:56:04,859 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:56:04,860 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:56:08,255 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:56:08,255 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:56:08,426 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:56:09,377 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:56:11,226 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:56:14,229 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:56:14,391 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:56:17,418 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:56:17,418 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1944_78808717363c62ee0546.png with policy now 2023-07-11 20:56:17,432 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:56:17,433 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1944_8177942062418e290d7d.png with policy now 2023-07-11 20:56:17,442 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:56:17,443 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1944_35adf9bd55486e1c564c.png with policy now 2023-07-11 20:56:17,451 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:56:17,452 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1944_fa0c0398f5b77951fef8.png with policy now 2023-07-11 20:56:17,489 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:56:17,491 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:56:17,491 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:56:17,493 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:56:17,733 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/3r5k22hm-media/images/validation_1944_78808717363c62ee0546.png 2023-07-11 20:56:17,746 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/sm7ae0zx-media/images/validation_1944_35adf9bd55486e1c564c.png 2023-07-11 20:56:17,757 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/mm98j7qx-media/images/validation_1944_8177942062418e290d7d.png 2023-07-11 20:56:17,768 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/iwqicnqc-media/images/validation_1944_fa0c0398f5b77951fef8.png 2023-07-11 20:56:18,274 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:56:18,275 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1944_8177942062418e290d7d.png 2023-07-11 20:56:18,275 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1944_78808717363c62ee0546.png 2023-07-11 20:56:18,276 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1944_fa0c0398f5b77951fef8.png 2023-07-11 20:56:18,276 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1944_35adf9bd55486e1c564c.png 2023-07-11 20:56:18,277 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 20:56:18,741 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/2a0039xl-media/images/validation_1944_fa0c0398f5b77951fef8.png 2023-07-11 20:56:18,819 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/keyd9a6y-media/images/validation_1944_8177942062418e290d7d.png 2023-07-11 20:56:18,822 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/6dhb8yqj-media/images/validation_1944_78808717363c62ee0546.png 2023-07-11 20:56:18,873 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/5q1dnlgg-media/images/validation_1944_35adf9bd55486e1c564c.png 2023-07-11 20:56:19,415 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:56:19,621 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:56:20,870 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:56:23,987 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:56:23,988 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:56:23,989 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:56:23,991 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:56:24,580 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:56:24,694 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:56:26,435 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:56:26,436 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:56:26,437 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:56:26,438 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:56:28,292 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:56:28,293 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:56:29,632 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:56:30,278 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:56:30,279 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:56:30,279 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:56:30,280 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:56:30,281 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:56:33,583 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:56:33,586 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:56:33,586 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:56:33,588 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:56:35,303 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:56:35,427 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:56:37,928 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:56:37,929 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:56:37,930 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:56:37,931 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:56:38,312 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:56:38,312 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:56:41,059 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:56:41,187 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:56:41,188 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:56:41,188 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:56:41,189 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:56:41,191 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:56:41,229 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:56:41,471 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:56:41,473 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:56:41,473 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:56:41,475 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:56:46,111 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:56:47,014 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:56:48,329 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:56:48,330 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:56:51,136 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:56:52,015 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:56:54,076 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:56:54,077 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1952_01c4c69e0d5b531f80da.png with policy now 2023-07-11 20:56:54,091 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:56:54,091 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1952_b9e6d46bbf3ea0d7f03d.png with policy now 2023-07-11 20:56:54,100 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:56:54,101 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1952_6ad546e10ab053b15547.png with policy now 2023-07-11 20:56:54,103 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:56:54,103 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1952_9eff640395fdb57cd9c7.png with policy now 2023-07-11 20:56:54,142 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:56:54,144 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:56:54,144 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:56:54,145 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:56:54,400 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/0zn366dt-media/images/validation_1952_01c4c69e0d5b531f80da.png 2023-07-11 20:56:54,412 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/coz6wc75-media/images/validation_1952_6ad546e10ab053b15547.png 2023-07-11 20:56:54,426 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/3rn57j9g-media/images/validation_1952_9eff640395fdb57cd9c7.png 2023-07-11 20:56:54,437 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/6c7sezjq-media/images/validation_1952_b9e6d46bbf3ea0d7f03d.png 2023-07-11 20:56:56,649 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:56:57,334 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:56:58,333 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:56:58,352 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:56:58,353 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:56:58,353 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1952_9eff640395fdb57cd9c7.png 2023-07-11 20:56:58,353 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1952_b9e6d46bbf3ea0d7f03d.png 2023-07-11 20:56:58,354 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1952_6ad546e10ab053b15547.png 2023-07-11 20:56:58,355 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1952_01c4c69e0d5b531f80da.png 2023-07-11 20:56:58,355 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 20:56:58,716 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/0qjf5g2k-media/images/validation_1952_9eff640395fdb57cd9c7.png 2023-07-11 20:56:58,849 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/wwek7ntq-media/images/validation_1952_6ad546e10ab053b15547.png 2023-07-11 20:56:58,920 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/o54ec84x-media/images/validation_1952_b9e6d46bbf3ea0d7f03d.png 2023-07-11 20:56:58,939 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/thrfhsq4-media/images/validation_1952_01c4c69e0d5b531f80da.png 2023-07-11 20:57:01,773 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:57:01,903 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:57:01,905 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:57:01,905 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:57:01,907 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:57:02,769 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:57:04,632 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:57:04,634 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:57:04,635 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:57:04,636 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:57:06,920 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:57:08,370 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:57:08,370 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:57:08,374 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:57:08,459 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:57:08,461 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:57:08,461 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:57:08,463 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:57:11,232 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:57:11,534 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:57:11,536 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:57:11,536 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:57:11,538 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:57:12,304 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:57:13,691 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:57:14,486 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:57:14,488 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:57:14,488 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:57:14,489 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:57:17,378 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:57:17,379 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:57:17,379 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:57:17,381 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:57:17,662 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:57:17,663 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:57:17,664 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:57:17,665 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:57:17,710 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:57:17,710 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:57:18,388 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:57:18,388 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:57:19,246 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:57:22,803 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:57:24,248 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:57:27,827 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:57:28,406 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:57:29,248 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:57:30,288 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:57:30,288 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1960_f69ac92a12eeba479470.png with policy now 2023-07-11 20:57:30,313 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:57:30,313 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1960_d0a34bec2dd9fe1ba19d.png with policy now 2023-07-11 20:57:30,327 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:57:30,327 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1960_b35c1f83e65423b07bda.png with policy now 2023-07-11 20:57:30,344 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:57:30,344 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1960_9263ee13eaef00dc177f.png with policy now 2023-07-11 20:57:30,387 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:57:30,389 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:57:30,389 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:57:30,391 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:57:30,644 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/9veyjaii-media/images/validation_1960_f69ac92a12eeba479470.png 2023-07-11 20:57:30,693 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ef9ludrf-media/images/validation_1960_9263ee13eaef00dc177f.png 2023-07-11 20:57:30,702 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/v2gvmno0-media/images/validation_1960_d0a34bec2dd9fe1ba19d.png 2023-07-11 20:57:30,704 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/4xl2jf4z-media/images/validation_1960_b35c1f83e65423b07bda.png 2023-07-11 20:57:33,708 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:57:33,708 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:57:34,386 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:57:34,663 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:57:38,427 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:57:38,427 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:57:38,428 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1960_f69ac92a12eeba479470.png 2023-07-11 20:57:38,428 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1960_b35c1f83e65423b07bda.png 2023-07-11 20:57:38,429 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1960_d0a34bec2dd9fe1ba19d.png 2023-07-11 20:57:38,429 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1960_9263ee13eaef00dc177f.png 2023-07-11 20:57:38,430 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 20:57:38,477 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:57:38,478 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:57:38,478 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:57:38,480 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:57:38,807 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/8oc3lv61-media/images/validation_1960_9263ee13eaef00dc177f.png 2023-07-11 20:57:38,870 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/0sx8tbnu-media/images/validation_1960_f69ac92a12eeba479470.png 2023-07-11 20:57:38,917 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/6lb9tiku-media/images/validation_1960_d0a34bec2dd9fe1ba19d.png 2023-07-11 20:57:38,946 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/vh0arnxu-media/images/validation_1960_b35c1f83e65423b07bda.png 2023-07-11 20:57:39,889 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:57:40,013 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:57:41,235 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:57:42,227 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:57:42,228 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:57:42,229 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:57:42,231 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:57:45,627 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:57:45,627 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:57:45,755 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:57:45,756 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:57:45,757 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:57:45,757 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:57:45,759 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:57:48,445 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:57:48,445 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:57:49,363 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:57:49,364 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:57:49,365 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:57:49,366 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:57:50,769 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:57:50,918 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:57:52,198 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:57:52,200 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:57:52,200 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:57:52,202 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:57:54,808 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:57:54,809 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:57:54,809 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:57:54,811 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:57:55,091 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:57:55,093 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:57:55,093 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:57:55,095 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:57:56,119 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:57:56,617 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:57:58,463 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:57:58,464 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:58:01,160 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:58:01,619 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:58:06,208 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:58:06,620 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:58:07,645 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:58:07,646 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1968_ff19fe998502f08ef355.png with policy now 2023-07-11 20:58:07,656 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:58:07,657 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1968_b94497f03241b794f556.png with policy now 2023-07-11 20:58:07,669 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:58:07,670 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1968_de75a859cd98957c62e9.png with policy now 2023-07-11 20:58:07,679 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:58:07,679 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1968_212445186f3f2a7ed98e.png with policy now 2023-07-11 20:58:07,715 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:58:07,716 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:58:07,717 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:58:07,718 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:58:07,973 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/lf9m4lm9-media/images/validation_1968_ff19fe998502f08ef355.png 2023-07-11 20:58:07,999 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/97y1rwqg-media/images/validation_1968_de75a859cd98957c62e9.png 2023-07-11 20:58:08,001 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/n6n2khdk-media/images/validation_1968_212445186f3f2a7ed98e.png 2023-07-11 20:58:08,018 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/qvvobawh-media/images/validation_1968_b94497f03241b794f556.png 2023-07-11 20:58:08,483 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:58:08,484 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1968_de75a859cd98957c62e9.png 2023-07-11 20:58:08,484 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1968_b94497f03241b794f556.png 2023-07-11 20:58:08,485 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1968_ff19fe998502f08ef355.png 2023-07-11 20:58:08,486 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1968_212445186f3f2a7ed98e.png 2023-07-11 20:58:08,486 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 20:58:08,939 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/r4br60za-media/images/validation_1968_de75a859cd98957c62e9.png 2023-07-11 20:58:09,005 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/uxusqs9a-media/images/validation_1968_ff19fe998502f08ef355.png 2023-07-11 20:58:09,013 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/x2pfznwm-media/images/validation_1968_b94497f03241b794f556.png 2023-07-11 20:58:09,054 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/8381cm25-media/images/validation_1968_212445186f3f2a7ed98e.png 2023-07-11 20:58:11,188 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:58:11,238 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:58:11,247 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:58:11,918 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:58:14,278 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:58:14,279 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:58:14,280 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:58:14,281 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:58:16,633 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:58:16,633 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:58:17,415 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:58:17,416 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:58:17,417 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:58:17,417 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:58:17,419 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:58:18,499 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:58:18,500 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:58:21,280 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:58:21,282 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:58:21,282 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:58:21,285 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:58:21,751 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:58:22,594 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:58:24,656 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:58:24,657 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:58:24,658 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:58:24,659 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:58:27,329 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:58:28,116 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:58:28,117 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:58:28,118 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:58:28,119 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:58:28,382 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:58:28,517 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:58:28,517 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:58:30,829 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:58:30,830 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:58:30,830 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:58:30,832 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:58:31,115 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:58:31,117 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:58:31,118 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:58:31,119 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:58:32,701 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:58:32,701 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:58:33,756 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:58:37,723 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:58:38,527 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:58:38,528 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:58:38,757 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:58:41,241 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:58:42,770 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:58:43,795 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:58:43,795 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1976_ca58948c6f5e8d1e6c38.png with policy now 2023-07-11 20:58:43,796 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:58:43,809 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:58:43,810 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1976_2891304fc5901c3c1e71.png with policy now 2023-07-11 20:58:43,820 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:58:43,820 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1976_0df481310c9adbf409ba.png with policy now 2023-07-11 20:58:43,829 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:58:43,830 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1976_ec9a8a5c1f221d207e64.png with policy now 2023-07-11 20:58:43,866 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:58:43,867 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:58:43,867 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:58:43,869 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:58:44,151 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/8a59qa04-media/images/validation_1976_ca58948c6f5e8d1e6c38.png 2023-07-11 20:58:44,151 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/38vy5kig-media/images/validation_1976_2891304fc5901c3c1e71.png 2023-07-11 20:58:44,160 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/7j2u5aa6-media/images/validation_1976_ec9a8a5c1f221d207e64.png 2023-07-11 20:58:44,175 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/5xy3r7kr-media/images/validation_1976_0df481310c9adbf409ba.png 2023-07-11 20:58:48,086 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:58:48,547 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:58:48,548 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:58:48,548 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1976_0df481310c9adbf409ba.png 2023-07-11 20:58:48,549 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1976_ec9a8a5c1f221d207e64.png 2023-07-11 20:58:48,549 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1976_2891304fc5901c3c1e71.png 2023-07-11 20:58:48,550 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1976_ca58948c6f5e8d1e6c38.png 2023-07-11 20:58:48,550 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 20:58:48,904 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/j3z5i472-media/images/validation_1976_ca58948c6f5e8d1e6c38.png 2023-07-11 20:58:49,002 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:58:49,002 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:58:49,026 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/3wwdj0ey-media/images/validation_1976_ec9a8a5c1f221d207e64.png 2023-07-11 20:58:49,069 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/d667og72-media/images/validation_1976_0df481310c9adbf409ba.png 2023-07-11 20:58:49,087 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:58:49,093 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ea9wyasf-media/images/validation_1976_2891304fc5901c3c1e71.png 2023-07-11 20:58:51,690 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:58:51,692 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:58:51,693 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:58:51,694 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:58:54,344 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:58:54,564 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:58:55,082 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:58:55,084 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:58:55,085 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:58:55,086 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:58:57,987 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:58:57,989 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:58:57,989 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:58:57,991 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:58:58,565 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:58:58,566 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:58:59,748 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:58:59,855 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:59:01,439 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:59:01,440 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:59:01,440 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:59:01,442 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:59:05,838 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:59:06,261 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:59:06,391 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:59:06,392 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:59:06,392 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:59:06,394 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:59:08,581 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:59:08,582 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:59:09,626 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:59:09,628 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:59:09,628 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:59:09,630 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:59:09,908 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:59:09,910 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:59:09,910 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:59:09,912 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:59:11,244 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:59:11,249 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:59:11,499 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:59:16,247 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:59:16,533 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:59:18,600 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:59:18,601 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:59:21,248 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:59:21,571 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:59:22,459 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:59:22,459 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1984_cf4f7819de6a9d73ca74.png with policy now 2023-07-11 20:59:22,471 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:59:22,471 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1984_597e25f12127dac13b2a.png with policy now 2023-07-11 20:59:22,480 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:59:22,480 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1984_c4207687ea8ecd4f4a45.png with policy now 2023-07-11 20:59:22,488 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:59:22,488 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1984_61b9f23963df182bd12d.png with policy now 2023-07-11 20:59:22,531 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:59:22,534 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:59:22,534 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:59:22,540 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:59:22,767 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/bn7rwccr-media/images/validation_1984_cf4f7819de6a9d73ca74.png 2023-07-11 20:59:22,780 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/3uk5omir-media/images/validation_1984_597e25f12127dac13b2a.png 2023-07-11 20:59:22,794 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/e2wn22ya-media/images/validation_1984_61b9f23963df182bd12d.png 2023-07-11 20:59:22,804 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/xwjsjmjy-media/images/validation_1984_c4207687ea8ecd4f4a45.png 2023-07-11 20:59:25,977 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:59:26,849 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:59:26,979 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:59:28,622 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:59:28,622 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:59:28,623 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1984_61b9f23963df182bd12d.png 2023-07-11 20:59:28,623 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1984_597e25f12127dac13b2a.png 2023-07-11 20:59:28,624 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1984_c4207687ea8ecd4f4a45.png 2023-07-11 20:59:28,624 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1984_cf4f7819de6a9d73ca74.png 2023-07-11 20:59:28,625 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 20:59:28,854 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:59:28,855 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:59:28,856 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:59:28,857 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:59:29,099 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/3l1gmvhh-media/images/validation_1984_597e25f12127dac13b2a.png 2023-07-11 20:59:29,156 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/6grhu8cd-media/images/validation_1984_c4207687ea8ecd4f4a45.png 2023-07-11 20:59:29,165 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ymg7kwtc-media/images/validation_1984_61b9f23963df182bd12d.png 2023-07-11 20:59:29,202 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/rl2o5ccp-media/images/validation_1984_cf4f7819de6a9d73ca74.png 2023-07-11 20:59:32,582 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:59:32,626 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:59:32,710 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:59:32,712 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:59:32,712 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:59:32,714 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:59:35,776 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:59:35,778 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:59:35,779 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:59:35,780 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:59:37,638 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:59:37,972 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:59:38,638 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:59:38,639 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:59:39,666 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:59:39,668 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:59:39,668 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:59:39,670 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:59:41,247 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 20:59:42,947 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:59:42,949 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:59:42,949 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:59:42,950 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:59:42,951 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:59:43,539 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:59:46,275 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:59:46,277 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:59:46,277 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:59:46,279 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:59:46,558 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:59:46,560 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:59:46,560 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:59:46,562 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:59:48,137 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:59:48,654 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 20:59:48,654 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 20:59:48,697 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:59:53,138 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:59:53,716 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:59:58,139 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 20:59:58,759 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 20:59:59,183 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:59:59,183 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1992_f8c5d5ec8577a44a670e.png with policy now 2023-07-11 20:59:59,198 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:59:59,198 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1992_cf518b56981a6ad11247.png with policy now 2023-07-11 20:59:59,208 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:59:59,208 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1992_92aeaed1bdf499b11f79.png with policy now 2023-07-11 20:59:59,215 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 20:59:59,215 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_1992_281a32d10f5b0365fb14.png with policy now 2023-07-11 20:59:59,252 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 20:59:59,254 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 20:59:59,254 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 20:59:59,256 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 20:59:59,499 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ft5n9heo-media/images/validation_1992_f8c5d5ec8577a44a670e.png 2023-07-11 20:59:59,542 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/yapv4acb-media/images/validation_1992_cf518b56981a6ad11247.png 2023-07-11 20:59:59,547 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/fww3brqd-media/images/validation_1992_281a32d10f5b0365fb14.png 2023-07-11 20:59:59,562 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/98m4cgxm-media/images/validation_1992_92aeaed1bdf499b11f79.png 2023-07-11 21:00:02,828 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:00:03,830 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:00:03,839 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:00:03,839 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:00:06,275 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:00:06,277 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:00:06,278 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:00:06,337 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:00:08,693 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:00:08,693 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:00:08,694 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1992_f8c5d5ec8577a44a670e.png 2023-07-11 21:00:08,694 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1992_92aeaed1bdf499b11f79.png 2023-07-11 21:00:08,695 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1992_281a32d10f5b0365fb14.png 2023-07-11 21:00:08,695 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1992_cf518b56981a6ad11247.png 2023-07-11 21:00:08,696 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 21:00:09,063 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ww0srlcj-media/images/validation_1992_f8c5d5ec8577a44a670e.png 2023-07-11 21:00:09,249 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/q9nhadag-media/images/validation_1992_281a32d10f5b0365fb14.png 2023-07-11 21:00:09,256 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/dydk0s8f-media/images/validation_1992_cf518b56981a6ad11247.png 2023-07-11 21:00:09,271 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/6z3wakug-media/images/validation_1992_92aeaed1bdf499b11f79.png 2023-07-11 21:00:09,281 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:00:09,404 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:00:10,667 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:00:10,669 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:00:10,670 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:00:10,671 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:00:11,250 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 21:00:14,243 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:00:14,244 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:00:14,245 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:00:14,247 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:00:14,552 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:00:15,185 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:00:17,898 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:00:17,900 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:00:17,900 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:00:17,902 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:00:18,709 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:00:18,710 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:00:19,770 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:00:20,476 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:00:21,205 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:00:21,207 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:00:21,207 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:00:21,209 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:00:24,543 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:00:24,545 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:00:24,546 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:00:24,547 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:00:24,826 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:00:24,828 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:00:24,829 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:00:24,830 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:00:25,418 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:00:25,569 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:00:29,728 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:00:29,728 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:00:30,530 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:00:30,570 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:00:35,548 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:00:35,571 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:00:37,677 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:00:37,677 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2000_56be1cccfe16e31d1eda.png with policy now 2023-07-11 21:00:37,688 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:00:37,689 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2000_5747cd6b7dfc06af840d.png with policy now 2023-07-11 21:00:37,711 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:00:37,711 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2000_11513f1b9a12e97321f9.png with policy now 2023-07-11 21:00:37,720 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:00:37,721 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2000_4ee817ff55a50642bd59.png with policy now 2023-07-11 21:00:37,762 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:00:37,764 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:00:37,764 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:00:37,767 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:00:37,997 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/la44vifi-media/images/validation_2000_56be1cccfe16e31d1eda.png 2023-07-11 21:00:38,023 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/cwp3tfmn-media/images/validation_2000_4ee817ff55a50642bd59.png 2023-07-11 21:00:38,024 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/7mwqkc0n-media/images/validation_2000_11513f1b9a12e97321f9.png 2023-07-11 21:00:38,029 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/yyh6hkin-media/images/validation_2000_5747cd6b7dfc06af840d.png 2023-07-11 21:00:40,748 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:00:40,748 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:00:40,749 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2000_4ee817ff55a50642bd59.png 2023-07-11 21:00:40,749 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2000_56be1cccfe16e31d1eda.png 2023-07-11 21:00:40,750 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2000_11513f1b9a12e97321f9.png 2023-07-11 21:00:40,750 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2000_5747cd6b7dfc06af840d.png 2023-07-11 21:00:40,751 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 21:00:40,913 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:00:41,023 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:00:41,217 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/w3wce0yo-media/images/validation_2000_56be1cccfe16e31d1eda.png 2023-07-11 21:00:41,253 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 21:00:41,321 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/5mnsycep-media/images/validation_2000_11513f1b9a12e97321f9.png 2023-07-11 21:00:41,321 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/3nadfgei-media/images/validation_2000_5747cd6b7dfc06af840d.png 2023-07-11 21:00:41,325 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/9qvlvcve-media/images/validation_2000_4ee817ff55a50642bd59.png 2023-07-11 21:00:41,741 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:00:44,731 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:00:44,733 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:00:44,733 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:00:44,735 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:00:46,436 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:00:46,437 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:00:46,559 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:00:48,778 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:00:48,780 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:00:48,780 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:00:48,782 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:00:51,573 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:00:51,651 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:00:51,766 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:00:51,766 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:00:52,894 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:00:52,896 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:00:52,896 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:00:52,898 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:00:56,244 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:00:56,245 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:00:56,246 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:00:56,251 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:00:56,768 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:00:56,769 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:00:59,312 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:00:59,314 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:00:59,314 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:00:59,316 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:01:01,788 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:01:02,597 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:01:02,599 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:01:02,599 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:01:02,600 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:01:02,602 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:01:02,784 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:01:02,785 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:01:02,880 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:01:02,882 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:01:02,882 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:01:02,884 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:01:06,915 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:01:08,403 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:01:11,270 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 21:01:11,944 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:01:13,800 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:01:13,801 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:01:14,272 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:01:15,390 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:01:15,390 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2008_edeeab747d59b9ede014.png with policy now 2023-07-11 21:01:15,405 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:01:15,406 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2008_a0785c61ee79cf436c16.png with policy now 2023-07-11 21:01:15,415 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:01:15,415 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2008_9c383de17af136bf0d80.png with policy now 2023-07-11 21:01:15,424 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:01:15,424 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2008_67662513522c4b9f8fd5.png with policy now 2023-07-11 21:01:15,460 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:01:15,461 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:01:15,462 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:01:15,463 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:01:15,828 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/wzuk5f5x-media/images/validation_2008_9c383de17af136bf0d80.png 2023-07-11 21:01:15,833 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/j1lb79zj-media/images/validation_2008_67662513522c4b9f8fd5.png 2023-07-11 21:01:15,839 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/tfn2257u-media/images/validation_2008_edeeab747d59b9ede014.png 2023-07-11 21:01:15,858 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/s82fapa8-media/images/validation_2008_a0785c61ee79cf436c16.png 2023-07-11 21:01:17,838 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:01:17,838 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:01:19,404 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:01:19,405 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:01:22,408 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:01:22,409 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:01:22,409 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:01:22,411 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:01:23,479 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:01:24,822 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:01:24,823 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:01:24,823 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2008_edeeab747d59b9ede014.png 2023-07-11 21:01:24,823 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2008_9c383de17af136bf0d80.png 2023-07-11 21:01:24,824 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2008_a0785c61ee79cf436c16.png 2023-07-11 21:01:24,825 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2008_67662513522c4b9f8fd5.png 2023-07-11 21:01:24,825 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 21:01:25,275 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/3lgcpl4x-media/images/validation_2008_9c383de17af136bf0d80.png 2023-07-11 21:01:25,349 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/e1f4rc4y-media/images/validation_2008_a0785c61ee79cf436c16.png 2023-07-11 21:01:25,355 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/u2pr8ijv-media/images/validation_2008_edeeab747d59b9ede014.png 2023-07-11 21:01:25,375 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/owqkohl9-media/images/validation_2008_67662513522c4b9f8fd5.png 2023-07-11 21:01:25,375 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:01:25,911 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:01:25,913 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:01:25,913 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:01:25,915 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:01:28,559 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:01:29,123 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:01:29,125 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:01:29,125 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:01:29,456 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:01:30,658 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:01:31,897 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:01:31,899 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:01:31,900 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:01:31,901 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:01:33,676 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:01:35,775 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:01:35,777 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:01:35,777 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:01:35,777 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:01:35,779 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:01:35,841 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:01:35,841 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:01:38,488 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:01:38,489 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:01:38,489 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:01:38,491 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:01:38,678 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:01:38,771 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:01:38,773 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:01:38,774 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:01:38,775 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:01:41,258 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 21:01:41,260 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:01:43,723 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:01:46,261 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:01:46,859 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:01:46,859 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:01:48,745 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:01:51,262 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:01:51,512 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:01:51,513 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2016_7b8a29624a01f6e3b5aa.png with policy now 2023-07-11 21:01:51,523 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:01:51,523 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2016_42552ac0dc72fef56307.png with policy now 2023-07-11 21:01:51,533 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:01:51,533 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2016_caa85d659cf2405bcefd.png with policy now 2023-07-11 21:01:51,542 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:01:51,542 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2016_b666077c67c4ed2ef995.png with policy now 2023-07-11 21:01:51,583 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:01:51,584 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:01:51,585 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:01:51,589 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:01:51,822 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/r3i515uo-media/images/validation_2016_7b8a29624a01f6e3b5aa.png 2023-07-11 21:01:51,849 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/re8zxzxm-media/images/validation_2016_b666077c67c4ed2ef995.png 2023-07-11 21:01:51,863 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/x3tv5sqz-media/images/validation_2016_42552ac0dc72fef56307.png 2023-07-11 21:01:51,877 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ym7jkdxl-media/images/validation_2016_caa85d659cf2405bcefd.png 2023-07-11 21:01:53,769 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:01:55,643 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:01:56,294 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:01:57,882 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:01:57,882 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:01:57,882 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2016_42552ac0dc72fef56307.png 2023-07-11 21:01:57,883 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2016_caa85d659cf2405bcefd.png 2023-07-11 21:01:57,883 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2016_b666077c67c4ed2ef995.png 2023-07-11 21:01:57,884 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2016_7b8a29624a01f6e3b5aa.png 2023-07-11 21:01:57,884 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 21:01:58,352 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/8ukbjsje-media/images/validation_2016_b666077c67c4ed2ef995.png 2023-07-11 21:01:58,409 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/k211ra22-media/images/validation_2016_caa85d659cf2405bcefd.png 2023-07-11 21:01:58,413 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/5omgo8gm-media/images/validation_2016_7b8a29624a01f6e3b5aa.png 2023-07-11 21:01:58,425 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/2nxbmebf-media/images/validation_2016_42552ac0dc72fef56307.png 2023-07-11 21:01:58,899 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:01:59,028 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:01:59,030 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:01:59,030 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:01:59,032 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:02:01,618 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:02:01,620 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:02:01,621 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:02:01,621 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:02:01,623 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:02:04,084 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:02:04,456 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:02:04,458 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:02:04,458 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:02:04,460 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:02:06,692 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:02:07,463 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:02:07,465 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:02:07,465 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:02:07,466 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:02:08,902 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:02:08,902 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:02:09,504 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:02:11,021 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:02:11,022 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:02:11,023 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:02:11,024 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:02:11,261 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 21:02:11,774 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:02:13,952 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:02:13,954 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:02:13,954 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:02:13,956 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:02:14,234 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:02:14,236 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:02:14,236 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:02:14,238 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:02:14,892 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:02:17,768 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:02:19,920 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:02:19,920 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:02:19,921 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:02:22,769 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:02:24,949 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:02:26,739 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:02:26,740 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2024_39ba76f44c6320920da5.png with policy now 2023-07-11 21:02:26,753 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:02:26,753 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2024_42449f3583b7d0ee5299.png with policy now 2023-07-11 21:02:26,764 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:02:26,765 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2024_58e7ab91d438b3e02f2c.png with policy now 2023-07-11 21:02:26,767 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:02:26,768 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2024_5cd6da1a61c49184c69c.png with policy now 2023-07-11 21:02:26,809 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:02:26,811 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:02:26,811 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:02:26,822 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:02:27,076 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/8ltiatki-media/images/validation_2024_42449f3583b7d0ee5299.png 2023-07-11 21:02:27,076 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/3tatq3e5-media/images/validation_2024_39ba76f44c6320920da5.png 2023-07-11 21:02:27,097 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/2tm81y2a-media/images/validation_2024_58e7ab91d438b3e02f2c.png 2023-07-11 21:02:27,110 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/dqlbxhiy-media/images/validation_2024_5cd6da1a61c49184c69c.png 2023-07-11 21:02:27,823 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:02:30,055 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:02:30,760 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:02:30,945 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:02:30,945 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:02:30,946 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2024_39ba76f44c6320920da5.png 2023-07-11 21:02:30,946 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2024_5cd6da1a61c49184c69c.png 2023-07-11 21:02:30,947 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2024_42449f3583b7d0ee5299.png 2023-07-11 21:02:30,947 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2024_58e7ab91d438b3e02f2c.png 2023-07-11 21:02:30,948 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 21:02:31,296 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/c0xm2aw5-media/images/validation_2024_58e7ab91d438b3e02f2c.png 2023-07-11 21:02:31,386 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ekbe157f-media/images/validation_2024_39ba76f44c6320920da5.png 2023-07-11 21:02:31,386 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/y8fo46xz-media/images/validation_2024_5cd6da1a61c49184c69c.png 2023-07-11 21:02:31,500 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/6o8fxr9o-media/images/validation_2024_42449f3583b7d0ee5299.png 2023-07-11 21:02:32,995 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:02:33,687 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:02:33,688 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:02:33,689 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:02:33,690 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:02:35,098 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:02:35,099 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:02:36,442 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:02:36,444 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:02:36,444 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:02:36,446 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:02:38,022 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:02:39,382 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:02:39,383 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:02:39,384 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:02:39,385 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:02:40,317 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:02:41,263 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 21:02:41,957 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:02:41,958 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:02:44,014 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:02:44,277 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:02:44,278 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:02:44,279 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:02:44,280 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:02:46,337 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:02:47,786 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:02:47,788 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:02:47,788 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:02:47,790 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:02:49,137 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:02:50,625 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:02:50,626 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:02:50,627 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:02:50,628 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:02:50,910 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:02:50,912 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:02:50,912 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:02:50,914 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:02:52,016 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:02:52,975 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:02:52,975 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:02:54,480 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:02:57,060 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:02:59,481 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:03:02,105 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:03:03,529 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:03:03,529 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2032_e9dca02fa7653317e9d3.png with policy now 2023-07-11 21:03:03,576 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:03:03,576 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2032_0966289b7bf679cabff2.png with policy now 2023-07-11 21:03:03,586 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:03:03,586 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2032_ab427828257bfc2f48cd.png with policy now 2023-07-11 21:03:03,594 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:03:03,595 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2032_f92644e9ab4ae89453ab.png with policy now 2023-07-11 21:03:03,637 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:03:03,639 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:03:03,640 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:03:03,644 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:03:03,851 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/5l39xhkn-media/images/validation_2032_e9dca02fa7653317e9d3.png 2023-07-11 21:03:03,888 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/k1fyuiuo-media/images/validation_2032_0966289b7bf679cabff2.png 2023-07-11 21:03:03,907 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/39bprt69-media/images/validation_2032_ab427828257bfc2f48cd.png 2023-07-11 21:03:03,915 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/q1lq50fb-media/images/validation_2032_f92644e9ab4ae89453ab.png 2023-07-11 21:03:03,995 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:03:03,996 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2032_f92644e9ab4ae89453ab.png 2023-07-11 21:03:03,996 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2032_ab427828257bfc2f48cd.png 2023-07-11 21:03:03,997 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2032_e9dca02fa7653317e9d3.png 2023-07-11 21:03:03,997 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2032_0966289b7bf679cabff2.png 2023-07-11 21:03:03,998 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 21:03:04,366 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/lbhkhpvb-media/images/validation_2032_f92644e9ab4ae89453ab.png 2023-07-11 21:03:04,480 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/8kq3wzzm-media/images/validation_2032_ab427828257bfc2f48cd.png 2023-07-11 21:03:04,486 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/fd7q3srx-media/images/validation_2032_e9dca02fa7653317e9d3.png 2023-07-11 21:03:04,566 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/sorl9tk2-media/images/validation_2032_0966289b7bf679cabff2.png 2023-07-11 21:03:04,645 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:03:07,120 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:03:07,262 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:03:10,279 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:03:10,281 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:03:10,282 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:03:10,282 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:03:10,284 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:03:11,266 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 21:03:12,274 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:03:13,121 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:03:13,123 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:03:13,123 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:03:13,125 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:03:15,013 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:03:15,013 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:03:16,130 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:03:16,830 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:03:16,832 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:03:16,833 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:03:16,834 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:03:17,515 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:03:20,581 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:03:20,582 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:03:20,583 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:03:20,586 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:03:21,301 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:03:22,517 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:03:24,071 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:03:24,073 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:03:24,073 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:03:24,075 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:03:26,027 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:03:26,027 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:03:26,838 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:03:26,840 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:03:26,840 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:03:26,841 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:03:26,843 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:03:27,120 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:03:27,121 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:03:27,122 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:03:27,123 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:03:27,527 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:03:32,613 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:03:32,672 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:03:37,045 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:03:37,046 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:03:37,655 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:03:37,673 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:03:39,727 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:03:39,727 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2040_1b5809dee200dda60230.png with policy now 2023-07-11 21:03:39,755 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:03:39,756 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2040_f6836c5b36ab58e6c0fb.png with policy now 2023-07-11 21:03:39,769 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:03:39,769 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2040_11ffc1e2966f22ed98b2.png with policy now 2023-07-11 21:03:39,778 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:03:39,779 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2040_c4c3d69e91379eeb2517.png with policy now 2023-07-11 21:03:39,814 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:03:39,815 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:03:39,816 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:03:39,820 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:03:40,043 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/9jvn5u7j-media/images/validation_2040_1b5809dee200dda60230.png 2023-07-11 21:03:40,096 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/tyw7rq92-media/images/validation_2040_c4c3d69e91379eeb2517.png 2023-07-11 21:03:40,102 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/nj0d6wtg-media/images/validation_2040_f6836c5b36ab58e6c0fb.png 2023-07-11 21:03:40,116 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/h0h8j7fo-media/images/validation_2040_11ffc1e2966f22ed98b2.png 2023-07-11 21:03:41,269 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 21:03:42,838 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:03:42,954 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:03:43,737 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:03:47,128 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:03:47,130 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:03:47,131 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:03:47,133 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:03:47,872 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:03:47,955 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:03:48,065 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:03:48,066 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:03:48,066 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2040_c4c3d69e91379eeb2517.png 2023-07-11 21:03:48,067 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2040_11ffc1e2966f22ed98b2.png 2023-07-11 21:03:48,067 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2040_f6836c5b36ab58e6c0fb.png 2023-07-11 21:03:48,068 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2040_1b5809dee200dda60230.png 2023-07-11 21:03:48,068 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 21:03:48,409 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/5mkwr9bw-media/images/validation_2040_c4c3d69e91379eeb2517.png 2023-07-11 21:03:48,416 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/kymnidux-media/images/validation_2040_f6836c5b36ab58e6c0fb.png 2023-07-11 21:03:48,597 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/h0c681d5-media/images/validation_2040_1b5809dee200dda60230.png 2023-07-11 21:03:48,612 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/nw6orswz-media/images/validation_2040_11ffc1e2966f22ed98b2.png 2023-07-11 21:03:50,709 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:03:50,711 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:03:50,711 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:03:50,713 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:03:53,112 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:03:53,720 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:03:53,857 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:03:53,859 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:03:53,859 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:03:53,861 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:03:56,838 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:03:56,840 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:03:56,841 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:03:56,842 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:03:58,465 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:03:59,085 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:03:59,085 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:03:59,318 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:04:00,160 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:04:00,162 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:04:00,162 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:04:00,164 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:04:03,471 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:04:03,598 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:04:03,600 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:04:03,601 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:04:03,602 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:04:03,881 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:04:03,882 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:04:03,883 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:04:03,885 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:04:04,434 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:04:08,800 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:04:09,435 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:04:10,103 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:04:10,104 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:04:11,272 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 21:04:13,821 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:04:15,274 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:04:16,429 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:04:16,429 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2048_123cf925aafba429d41f.png with policy now 2023-07-11 21:04:16,444 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:04:16,445 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2048_2fac4432914c0d06816d.png with policy now 2023-07-11 21:04:16,464 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:04:16,464 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2048_b964563afa4fd9125c79.png with policy now 2023-07-11 21:04:16,479 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:04:16,479 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2048_9132f798001f5c0ccc6d.png with policy now 2023-07-11 21:04:16,520 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:04:16,521 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:04:16,522 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:04:16,523 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:04:16,755 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/2f4z598c-media/images/validation_2048_123cf925aafba429d41f.png 2023-07-11 21:04:16,766 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/12v6wwz5-media/images/validation_2048_2fac4432914c0d06816d.png 2023-07-11 21:04:16,819 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/3git99p8-media/images/validation_2048_b964563afa4fd9125c79.png 2023-07-11 21:04:16,845 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/1kda4v1k-media/images/validation_2048_9132f798001f5c0ccc6d.png 2023-07-11 21:04:18,967 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:04:18,968 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:04:20,366 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:04:20,367 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:04:21,125 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:04:21,125 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:04:21,126 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2048_2fac4432914c0d06816d.png 2023-07-11 21:04:21,126 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2048_123cf925aafba429d41f.png 2023-07-11 21:04:21,127 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2048_b964563afa4fd9125c79.png 2023-07-11 21:04:21,127 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2048_9132f798001f5c0ccc6d.png 2023-07-11 21:04:21,128 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 21:04:21,474 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/xhfg8sq6-media/images/validation_2048_2fac4432914c0d06816d.png 2023-07-11 21:04:21,490 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/57y27nsd-media/images/validation_2048_123cf925aafba429d41f.png 2023-07-11 21:04:21,502 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/wjyo5tsr-media/images/validation_2048_9132f798001f5c0ccc6d.png 2023-07-11 21:04:21,595 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/vrsdyz34-media/images/validation_2048_b964563afa4fd9125c79.png 2023-07-11 21:04:23,868 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:04:23,871 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:04:23,871 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:04:23,873 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:04:24,584 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:04:25,885 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:04:26,740 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:04:26,742 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:04:26,742 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:04:26,744 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:04:29,836 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:04:29,842 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:04:29,843 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:04:29,844 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:04:29,846 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:04:31,393 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:04:32,142 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:04:32,143 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:04:33,433 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:04:33,434 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:04:33,435 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:04:33,436 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:04:35,245 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:04:35,246 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:04:36,435 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:04:37,218 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:04:37,220 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:04:37,220 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:04:37,222 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:04:40,312 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:04:40,819 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:04:40,821 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:04:40,821 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:04:40,823 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:04:41,101 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:04:41,103 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:04:41,103 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:04:41,105 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:04:41,275 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 21:04:41,593 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:04:43,162 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:04:43,163 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:04:45,355 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:04:46,732 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:04:50,386 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:04:51,733 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:04:53,829 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:04:53,830 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2056_f2d52946f625bac82075.png with policy now 2023-07-11 21:04:53,841 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:04:53,841 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2056_ae649897728e92928855.png with policy now 2023-07-11 21:04:53,850 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:04:53,850 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2056_c30db4d32c0339da583c.png with policy now 2023-07-11 21:04:53,859 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:04:53,859 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2056_119cb34b947beaab2613.png with policy now 2023-07-11 21:04:53,901 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:04:53,905 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:04:53,905 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:04:53,910 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:04:54,144 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/029h8gkz-media/images/validation_2056_f2d52946f625bac82075.png 2023-07-11 21:04:54,159 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/g7b25f8o-media/images/validation_2056_ae649897728e92928855.png 2023-07-11 21:04:54,164 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/osjy9xw8-media/images/validation_2056_c30db4d32c0339da583c.png 2023-07-11 21:04:54,168 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/xnc7w9zc-media/images/validation_2056_119cb34b947beaab2613.png 2023-07-11 21:04:54,184 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:04:54,184 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2056_119cb34b947beaab2613.png 2023-07-11 21:04:54,185 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2056_ae649897728e92928855.png 2023-07-11 21:04:54,185 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2056_c30db4d32c0339da583c.png 2023-07-11 21:04:54,186 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2056_f2d52946f625bac82075.png 2023-07-11 21:04:54,186 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 21:04:54,571 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/5a2422om-media/images/validation_2056_f2d52946f625bac82075.png 2023-07-11 21:04:54,656 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/o0dnzooo-media/images/validation_2056_c30db4d32c0339da583c.png 2023-07-11 21:04:54,719 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/otis9swb-media/images/validation_2056_ae649897728e92928855.png 2023-07-11 21:04:54,740 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ldc84r2h-media/images/validation_2056_119cb34b947beaab2613.png 2023-07-11 21:04:55,467 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:04:57,191 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:04:57,192 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:05:01,173 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:05:01,230 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:05:01,232 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:05:01,232 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:05:01,234 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:05:02,205 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:05:04,880 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:05:04,881 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:05:04,882 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:05:04,883 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:05:05,202 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:05:05,202 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:05:06,204 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:05:07,507 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:05:08,164 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:05:08,166 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:05:08,166 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:05:08,168 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:05:11,278 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 21:05:11,555 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:05:11,557 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:05:11,557 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:05:11,559 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:05:12,012 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:05:13,134 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:05:14,726 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:05:14,727 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:05:14,727 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:05:14,729 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:05:16,221 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:05:16,221 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:05:17,059 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:05:18,183 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:05:18,283 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:05:18,285 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:05:18,285 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:05:18,287 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:05:18,567 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:05:18,569 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:05:18,569 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:05:18,571 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:05:22,168 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:05:24,151 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:05:27,197 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:05:27,241 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:05:27,241 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:05:29,152 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:05:31,122 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:05:31,122 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2064_ea44735eac8f71037fce.png with policy now 2023-07-11 21:05:31,134 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:05:31,134 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2064_b712e152cd356cd14c7a.png with policy now 2023-07-11 21:05:31,146 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:05:31,146 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2064_519a4aa69108743c7ec6.png with policy now 2023-07-11 21:05:31,148 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:05:31,148 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2064_31a98ecd5357f46a64be.png with policy now 2023-07-11 21:05:31,189 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:05:31,190 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:05:31,191 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:05:31,192 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:05:31,453 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/aemr3q6a-media/images/validation_2064_ea44735eac8f71037fce.png 2023-07-11 21:05:31,478 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/2em0gy8t-media/images/validation_2064_31a98ecd5357f46a64be.png 2023-07-11 21:05:31,478 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/682k71p6-media/images/validation_2064_519a4aa69108743c7ec6.png 2023-07-11 21:05:31,479 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/53apgx83-media/images/validation_2064_b712e152cd356cd14c7a.png 2023-07-11 21:05:32,212 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:05:34,152 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:05:34,153 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:05:37,535 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:05:37,664 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:05:37,666 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:05:37,666 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:05:37,668 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:05:38,265 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:05:38,265 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:05:38,265 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2064_ea44735eac8f71037fce.png 2023-07-11 21:05:38,266 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2064_519a4aa69108743c7ec6.png 2023-07-11 21:05:38,267 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2064_b712e152cd356cd14c7a.png 2023-07-11 21:05:38,267 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2064_31a98ecd5357f46a64be.png 2023-07-11 21:05:38,267 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 21:05:38,612 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/i7j6whtn-media/images/validation_2064_519a4aa69108743c7ec6.png 2023-07-11 21:05:38,642 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/grkljo5s-media/images/validation_2064_31a98ecd5357f46a64be.png 2023-07-11 21:05:38,802 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/auia6z5s-media/images/validation_2064_b712e152cd356cd14c7a.png 2023-07-11 21:05:38,947 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/38ykw63l-media/images/validation_2064_ea44735eac8f71037fce.png 2023-07-11 21:05:39,312 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:05:41,281 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 21:05:41,345 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:05:41,346 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:05:41,347 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:05:41,348 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:05:42,620 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:05:44,434 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:05:44,437 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:05:44,437 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:05:44,437 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:05:44,439 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:05:47,819 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:05:47,948 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:05:47,950 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:05:47,951 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:05:47,952 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:05:49,283 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:05:49,284 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:05:49,558 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:05:51,997 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:05:51,999 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:05:51,999 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:05:52,007 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:05:52,871 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:05:55,287 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:05:55,289 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:05:55,289 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:05:55,289 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:05:55,291 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:05:55,569 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:05:55,571 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:05:55,571 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:05:55,573 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:05:58,182 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:06:00,301 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:06:00,301 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:06:01,236 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:06:03,200 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:06:06,238 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:06:08,237 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:06:08,258 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:06:08,259 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2072_ddac369b4ded67607317.png with policy now 2023-07-11 21:06:08,273 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:06:08,274 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2072_1bc4c894c72890f665a0.png with policy now 2023-07-11 21:06:08,277 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:06:08,282 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2072_e0f0cd526f1fbf142c07.png with policy now 2023-07-11 21:06:08,290 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:06:08,291 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2072_a7d9f847844b98f32c62.png with policy now 2023-07-11 21:06:08,325 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:06:08,327 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:06:08,327 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:06:08,330 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:06:08,565 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ij9erfba-media/images/validation_2072_ddac369b4ded67607317.png 2023-07-11 21:06:08,594 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ktvaw4pw-media/images/validation_2072_e0f0cd526f1fbf142c07.png 2023-07-11 21:06:08,609 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/hpk9zrzk-media/images/validation_2072_a7d9f847844b98f32c62.png 2023-07-11 21:06:08,610 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/pcx3xrg8-media/images/validation_2072_1bc4c894c72890f665a0.png 2023-07-11 21:06:11,284 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 21:06:11,288 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:06:11,323 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:06:11,323 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:06:11,323 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2072_a7d9f847844b98f32c62.png 2023-07-11 21:06:11,324 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2072_ddac369b4ded67607317.png 2023-07-11 21:06:11,324 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2072_1bc4c894c72890f665a0.png 2023-07-11 21:06:11,325 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2072_e0f0cd526f1fbf142c07.png 2023-07-11 21:06:11,325 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 21:06:11,685 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/dn1aqe0x-media/images/validation_2072_e0f0cd526f1fbf142c07.png 2023-07-11 21:06:11,699 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/db1hof1y-media/images/validation_2072_1bc4c894c72890f665a0.png 2023-07-11 21:06:11,768 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/19x21vv3-media/images/validation_2072_ddac369b4ded67607317.png 2023-07-11 21:06:11,857 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/7y3kzb8a-media/images/validation_2072_a7d9f847844b98f32c62.png 2023-07-11 21:06:12,829 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:06:13,248 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:06:15,780 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:06:15,782 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:06:15,783 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:06:15,784 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:06:16,484 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:06:18,474 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:06:19,606 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:06:19,608 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:06:19,609 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:06:19,610 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:06:22,042 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:06:22,341 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:06:22,342 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:06:22,602 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:06:22,604 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:06:22,605 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:06:22,606 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:06:23,487 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:06:25,780 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:06:25,782 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:06:25,782 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:06:25,784 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:06:27,391 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:06:28,524 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:06:29,632 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:06:29,634 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:06:29,635 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:06:29,636 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:06:32,847 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:06:32,850 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:06:32,850 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:06:32,851 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:06:32,852 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:06:33,131 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:06:33,133 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:06:33,133 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:06:33,135 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:06:33,361 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:06:33,361 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:06:33,850 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:06:38,694 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:06:39,269 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:06:41,288 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 21:06:44,290 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:06:44,318 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:06:44,380 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:06:45,745 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:06:45,745 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2080_234274db4799f23af905.png with policy now 2023-07-11 21:06:45,760 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:06:45,760 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2080_470d734462cd775639c9.png with policy now 2023-07-11 21:06:45,770 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:06:45,770 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2080_b5332c71d8ad6af0f6cb.png with policy now 2023-07-11 21:06:45,779 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:06:45,779 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2080_c0c4ed778328664db5ef.png with policy now 2023-07-11 21:06:45,816 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:06:45,818 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:06:45,819 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:06:45,820 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:06:46,063 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/f8w8yqxt-media/images/validation_2080_234274db4799f23af905.png 2023-07-11 21:06:46,072 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/824zecfa-media/images/validation_2080_470d734462cd775639c9.png 2023-07-11 21:06:46,083 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/1q1f16gg-media/images/validation_2080_b5332c71d8ad6af0f6cb.png 2023-07-11 21:06:46,111 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/w68nsjvd-media/images/validation_2080_c0c4ed778328664db5ef.png 2023-07-11 21:06:49,369 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:06:49,497 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:06:49,498 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:06:53,078 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:06:53,080 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:06:53,080 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:06:53,082 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:06:54,578 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:06:54,951 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:06:55,403 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:06:55,404 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:06:55,404 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2080_234274db4799f23af905.png 2023-07-11 21:06:55,404 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2080_470d734462cd775639c9.png 2023-07-11 21:06:55,405 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2080_c0c4ed778328664db5ef.png 2023-07-11 21:06:55,406 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2080_b5332c71d8ad6af0f6cb.png 2023-07-11 21:06:55,406 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 21:06:55,772 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/uv9rq6lv-media/images/validation_2080_c0c4ed778328664db5ef.png 2023-07-11 21:06:55,775 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/5jl0qmmo-media/images/validation_2080_234274db4799f23af905.png 2023-07-11 21:06:55,954 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/2yw55kk9-media/images/validation_2080_470d734462cd775639c9.png 2023-07-11 21:06:55,961 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/u8zdqs59-media/images/validation_2080_b5332c71d8ad6af0f6cb.png 2023-07-11 21:06:56,384 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:06:56,386 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:06:56,386 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:06:56,388 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:06:59,388 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:06:59,390 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:06:59,390 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:06:59,392 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:06:59,952 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:07:00,293 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:07:02,320 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:07:02,321 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:07:02,322 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:07:02,323 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:07:05,504 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:07:05,517 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:07:06,319 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:07:06,321 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:07:06,321 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:07:06,323 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:07:06,422 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:07:06,422 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:07:09,210 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:07:09,211 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:07:09,212 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:07:09,213 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:07:09,494 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:07:09,496 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:07:09,497 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:07:09,498 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:07:10,598 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:07:11,115 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:07:11,289 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 21:07:15,634 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:07:16,292 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:07:17,435 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:07:17,436 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:07:20,682 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:07:21,293 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:07:22,110 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:07:22,110 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2088_1177c53f90c63a53fae4.png with policy now 2023-07-11 21:07:22,122 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:07:22,122 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2088_5726b36383a81db55440.png with policy now 2023-07-11 21:07:22,135 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:07:22,135 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2088_6d9f398496e3d9fb86ac.png with policy now 2023-07-11 21:07:22,144 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:07:22,144 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2088_ad4894c9ed610fe1ef61.png with policy now 2023-07-11 21:07:22,182 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:07:22,185 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:07:22,185 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:07:22,192 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:07:22,429 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/m3rgqhwy-media/images/validation_2088_1177c53f90c63a53fae4.png 2023-07-11 21:07:22,437 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/3vogc1ye-media/images/validation_2088_ad4894c9ed610fe1ef61.png 2023-07-11 21:07:22,444 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/fc4xj2j5-media/images/validation_2088_5726b36383a81db55440.png 2023-07-11 21:07:22,448 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/2qe5fv56-media/images/validation_2088_6d9f398496e3d9fb86ac.png 2023-07-11 21:07:25,701 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:07:26,069 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:07:26,634 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:07:28,456 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:07:28,457 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:07:28,457 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2088_ad4894c9ed610fe1ef61.png 2023-07-11 21:07:28,458 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2088_6d9f398496e3d9fb86ac.png 2023-07-11 21:07:28,458 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2088_5726b36383a81db55440.png 2023-07-11 21:07:28,459 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2088_1177c53f90c63a53fae4.png 2023-07-11 21:07:28,459 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 21:07:28,832 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/gagio1y7-media/images/validation_2088_1177c53f90c63a53fae4.png 2023-07-11 21:07:28,898 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ki9ydd0z-media/images/validation_2088_ad4894c9ed610fe1ef61.png 2023-07-11 21:07:28,997 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/nplbugr6-media/images/validation_2088_5726b36383a81db55440.png 2023-07-11 21:07:29,080 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/wizv9ddq-media/images/validation_2088_6d9f398496e3d9fb86ac.png 2023-07-11 21:07:29,680 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:07:29,682 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:07:29,682 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:07:29,684 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:07:30,702 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:07:31,791 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:07:33,206 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:07:33,208 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:07:33,209 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:07:33,213 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:07:36,029 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:07:36,031 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:07:36,032 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:07:36,033 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:07:36,583 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:07:36,583 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:07:37,628 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:07:39,199 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:07:39,201 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:07:39,202 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:07:39,203 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:07:39,477 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:07:39,477 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:07:41,292 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 21:07:42,297 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:07:43,195 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:07:43,466 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:07:43,468 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:07:43,468 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:07:43,469 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:07:46,876 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:07:46,878 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:07:46,878 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:07:46,880 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:07:47,275 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:07:47,277 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:07:47,278 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:07:47,279 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:07:47,495 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:07:48,941 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:07:50,496 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:07:50,496 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:07:52,510 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:07:53,942 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:07:57,593 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:07:58,944 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:07:59,983 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:07:59,983 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2096_3917a475eb07dd7d8014.png with policy now 2023-07-11 21:07:59,997 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:07:59,998 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2096_b3898442a9a4b0221ae6.png with policy now 2023-07-11 21:08:00,007 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:08:00,007 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2096_ab63da1d97e801140f79.png with policy now 2023-07-11 21:08:00,016 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:08:00,016 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2096_52636a9fab6f23c15f25.png with policy now 2023-07-11 21:08:00,057 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:08:00,058 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:08:00,059 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:08:00,064 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:08:00,309 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/jzur6sls-media/images/validation_2096_b3898442a9a4b0221ae6.png 2023-07-11 21:08:00,312 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/kzbuzd4b-media/images/validation_2096_3917a475eb07dd7d8014.png 2023-07-11 21:08:00,331 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/l3j2r7jt-media/images/validation_2096_52636a9fab6f23c15f25.png 2023-07-11 21:08:00,340 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/zezzlhwu-media/images/validation_2096_ab63da1d97e801140f79.png 2023-07-11 21:08:01,521 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:08:01,522 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2096_52636a9fab6f23c15f25.png 2023-07-11 21:08:01,523 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2096_3917a475eb07dd7d8014.png 2023-07-11 21:08:01,523 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2096_b3898442a9a4b0221ae6.png 2023-07-11 21:08:01,523 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2096_ab63da1d97e801140f79.png 2023-07-11 21:08:01,524 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 21:08:01,871 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/fmtl166e-media/images/validation_2096_52636a9fab6f23c15f25.png 2023-07-11 21:08:01,874 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/0sijwbae-media/images/validation_2096_3917a475eb07dd7d8014.png 2023-07-11 21:08:01,881 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/5wg17aq2-media/images/validation_2096_ab63da1d97e801140f79.png 2023-07-11 21:08:02,092 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/6qxshhmz-media/images/validation_2096_b3898442a9a4b0221ae6.png 2023-07-11 21:08:02,624 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:08:03,367 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:08:04,300 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:08:07,454 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:08:07,455 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:08:07,456 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:08:07,458 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:08:07,736 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:08:07,736 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:08:09,857 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:08:10,749 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:08:10,751 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:08:10,751 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:08:10,753 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:08:11,295 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 21:08:12,540 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:08:12,541 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:08:13,059 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:08:14,391 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:08:14,393 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:08:14,393 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:08:14,395 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:08:15,297 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:08:17,698 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:08:17,700 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:08:17,700 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:08:17,702 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:08:18,461 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:08:20,365 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:08:21,107 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:08:21,109 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:08:21,110 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:08:21,111 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:08:23,561 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:08:23,562 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:08:23,893 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:08:23,895 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:08:23,895 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:08:23,897 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:08:24,177 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:08:24,178 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:08:24,179 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:08:24,180 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:08:24,242 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:08:25,733 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:08:29,315 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:08:30,734 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:08:34,335 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:08:34,581 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:08:34,581 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:08:35,735 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:08:36,777 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:08:36,777 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2104_f2ed3d7c9f2ca2721159.png with policy now 2023-07-11 21:08:36,789 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:08:36,789 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2104_8ff4357dcc31e396a35b.png with policy now 2023-07-11 21:08:36,802 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:08:36,802 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2104_17fbe5a0dfd4b8fd48f2.png with policy now 2023-07-11 21:08:36,813 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:08:36,814 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2104_619ebebabd9789748cac.png with policy now 2023-07-11 21:08:36,850 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:08:36,852 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:08:36,852 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:08:36,854 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:08:37,675 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/rp9d3z5q-media/images/validation_2104_f2ed3d7c9f2ca2721159.png 2023-07-11 21:08:38,123 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/aji3veh6-media/images/validation_2104_17fbe5a0dfd4b8fd48f2.png 2023-07-11 21:08:38,263 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/hfmptakb-media/images/validation_2104_619ebebabd9789748cac.png 2023-07-11 21:08:39,529 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:08:39,600 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/paeqm7cn-media/images/validation_2104_8ff4357dcc31e396a35b.png 2023-07-11 21:08:39,897 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:08:40,778 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:08:41,298 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 21:08:43,416 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:08:43,418 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:08:43,418 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:08:43,420 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:08:45,288 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:08:45,603 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:08:45,604 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:08:45,604 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2104_17fbe5a0dfd4b8fd48f2.png 2023-07-11 21:08:45,605 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2104_f2ed3d7c9f2ca2721159.png 2023-07-11 21:08:45,605 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2104_619ebebabd9789748cac.png 2023-07-11 21:08:45,606 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2104_8ff4357dcc31e396a35b.png 2023-07-11 21:08:45,606 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 21:08:45,930 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/shrclnk1-media/images/validation_2104_17fbe5a0dfd4b8fd48f2.png 2023-07-11 21:08:45,954 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/adpldgwv-media/images/validation_2104_f2ed3d7c9f2ca2721159.png 2023-07-11 21:08:45,968 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/urkapob5-media/images/validation_2104_619ebebabd9789748cac.png 2023-07-11 21:08:45,988 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/hbarwhut-media/images/validation_2104_8ff4357dcc31e396a35b.png 2023-07-11 21:08:46,217 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:08:46,218 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:08:46,219 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:08:46,219 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:08:46,221 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:08:50,267 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:08:50,269 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:08:50,269 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:08:50,271 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:08:50,536 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:08:50,536 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:08:51,897 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:08:53,361 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:08:53,362 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:08:53,363 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:08:53,364 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:08:56,061 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:08:56,063 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:08:56,063 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:08:56,065 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:08:56,156 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:08:56,622 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:08:56,623 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:08:57,857 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:08:59,742 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:08:59,744 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:08:59,744 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:08:59,746 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:09:00,027 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:09:00,029 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:09:00,029 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:09:00,031 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:09:01,163 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:09:03,641 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:09:06,181 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:09:07,640 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:09:07,641 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:09:08,642 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:09:11,214 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:09:11,301 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 21:09:12,642 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:09:12,642 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2112_ee9b23aa9877d821342d.png with policy now 2023-07-11 21:09:12,657 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:09:12,657 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2112_de58ff517f7ad93b7980.png with policy now 2023-07-11 21:09:12,667 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:09:12,667 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2112_fdc939f4f2cfbd53196d.png with policy now 2023-07-11 21:09:12,676 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:09:12,676 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2112_99bcea6ecb4b26371b0a.png with policy now 2023-07-11 21:09:12,715 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:09:12,717 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:09:12,717 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:09:12,719 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:09:12,985 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/qhq0mb4x-media/images/validation_2112_ee9b23aa9877d821342d.png 2023-07-11 21:09:12,985 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ql955h6d-media/images/validation_2112_de58ff517f7ad93b7980.png 2023-07-11 21:09:12,990 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/wb2mexi2-media/images/validation_2112_99bcea6ecb4b26371b0a.png 2023-07-11 21:09:12,990 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/fiibfudq-media/images/validation_2112_fdc939f4f2cfbd53196d.png 2023-07-11 21:09:13,719 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:09:16,361 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:09:17,396 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:09:18,663 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:09:18,664 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:09:18,664 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2112_ee9b23aa9877d821342d.png 2023-07-11 21:09:18,665 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2112_99bcea6ecb4b26371b0a.png 2023-07-11 21:09:18,665 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2112_fdc939f4f2cfbd53196d.png 2023-07-11 21:09:18,666 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2112_de58ff517f7ad93b7980.png 2023-07-11 21:09:18,666 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 21:09:18,889 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:09:19,017 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/idf7tbge-media/images/validation_2112_ee9b23aa9877d821342d.png 2023-07-11 21:09:19,038 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/8nre2tg0-media/images/validation_2112_de58ff517f7ad93b7980.png 2023-07-11 21:09:19,211 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/hme8blm3-media/images/validation_2112_99bcea6ecb4b26371b0a.png 2023-07-11 21:09:19,213 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/yjwrme3s-media/images/validation_2112_fdc939f4f2cfbd53196d.png 2023-07-11 21:09:20,824 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:09:20,826 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:09:20,827 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:09:20,828 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:09:22,128 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:09:22,129 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:09:23,729 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:09:23,730 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:09:23,731 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:09:23,732 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:09:24,593 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:09:26,731 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:09:26,734 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:09:26,734 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:09:26,736 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:09:27,130 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:09:29,682 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:09:29,682 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:09:29,873 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:09:29,875 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:09:29,876 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:09:29,876 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:09:29,878 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:09:32,837 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:09:33,752 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:09:33,754 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:09:33,755 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:09:33,756 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:09:35,113 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:09:36,079 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:09:36,081 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:09:36,081 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:09:36,083 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:09:36,364 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:09:36,366 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:09:36,366 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:09:36,368 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:09:38,076 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:09:40,160 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:09:40,701 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:09:40,701 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:09:41,304 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 21:09:43,109 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:09:45,306 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:09:48,144 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:09:49,210 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:09:49,211 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2120_0cc11534506dbd7f7bac.png with policy now 2023-07-11 21:09:49,226 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:09:49,226 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2120_4dd49f0520bc4123cc57.png with policy now 2023-07-11 21:09:49,235 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:09:49,235 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2120_a5f9ed84ca9d264b9439.png with policy now 2023-07-11 21:09:49,245 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:09:49,245 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2120_f685f7aab896c2a636a6.png with policy now 2023-07-11 21:09:49,280 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:09:49,282 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:09:49,282 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:09:49,283 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:09:49,561 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/45az03p4-media/images/validation_2120_0cc11534506dbd7f7bac.png 2023-07-11 21:09:49,596 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/m32yvv8s-media/images/validation_2120_4dd49f0520bc4123cc57.png 2023-07-11 21:09:49,596 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/chn8aq2k-media/images/validation_2120_f685f7aab896c2a636a6.png 2023-07-11 21:09:49,608 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/3m6ozsoe-media/images/validation_2120_a5f9ed84ca9d264b9439.png 2023-07-11 21:09:50,567 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:09:51,724 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:09:51,724 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:09:51,725 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2120_a5f9ed84ca9d264b9439.png 2023-07-11 21:09:51,725 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2120_f685f7aab896c2a636a6.png 2023-07-11 21:09:51,726 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2120_4dd49f0520bc4123cc57.png 2023-07-11 21:09:51,726 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2120_0cc11534506dbd7f7bac.png 2023-07-11 21:09:51,727 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 21:09:52,098 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/6nyjb8a3-media/images/validation_2120_a5f9ed84ca9d264b9439.png 2023-07-11 21:09:52,105 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/vdqeo7p1-media/images/validation_2120_f685f7aab896c2a636a6.png 2023-07-11 21:09:52,119 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/qg565f52-media/images/validation_2120_4dd49f0520bc4123cc57.png 2023-07-11 21:09:52,232 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/7ubcibsk-media/images/validation_2120_0cc11534506dbd7f7bac.png 2023-07-11 21:09:53,168 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:09:53,217 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:09:56,130 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:09:56,882 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:09:56,884 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:09:56,884 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:09:56,886 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:09:58,717 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:10:00,322 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:10:00,324 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:10:00,324 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:10:00,326 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:10:01,215 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:10:02,743 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:10:02,744 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:10:03,702 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:10:03,704 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:10:03,705 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:10:03,706 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:10:04,479 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:10:06,888 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:10:06,890 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:10:06,890 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:10:06,891 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:10:06,893 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:10:09,810 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:10:10,491 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:10:10,493 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:10:10,493 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:10:10,495 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:10:11,307 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 21:10:11,899 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:10:13,762 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:10:13,763 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:10:13,946 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:10:13,948 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:10:13,949 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:10:13,950 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:10:14,231 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:10:14,232 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:10:14,233 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:10:14,234 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:10:14,879 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:10:17,840 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:10:19,935 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:10:22,841 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:10:24,781 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:10:24,782 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:10:24,972 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:10:26,842 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:10:26,842 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2128_40b0d08fdb15bcbe6d76.png with policy now 2023-07-11 21:10:26,856 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:10:26,857 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2128_b90bc0dcc518a41e4b7c.png with policy now 2023-07-11 21:10:26,867 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:10:26,867 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2128_1c5e23a118700525a7fd.png with policy now 2023-07-11 21:10:26,875 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:10:26,875 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2128_88801f29434a2a3ded1a.png with policy now 2023-07-11 21:10:26,912 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:10:26,913 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:10:26,914 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:10:26,915 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:10:27,164 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/4f0m64vz-media/images/validation_2128_40b0d08fdb15bcbe6d76.png 2023-07-11 21:10:27,179 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/zqw28q80-media/images/validation_2128_88801f29434a2a3ded1a.png 2023-07-11 21:10:27,185 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/45t0pgxe-media/images/validation_2128_1c5e23a118700525a7fd.png 2023-07-11 21:10:27,195 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/1f89u285-media/images/validation_2128_b90bc0dcc518a41e4b7c.png 2023-07-11 21:10:28,719 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:10:29,989 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:10:30,395 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:10:33,701 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:10:33,702 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:10:33,702 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:10:33,704 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:10:34,694 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:10:35,574 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:10:35,574 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:10:35,804 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:10:35,804 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:10:35,804 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2128_1c5e23a118700525a7fd.png 2023-07-11 21:10:35,805 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2128_88801f29434a2a3ded1a.png 2023-07-11 21:10:35,806 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2128_40b0d08fdb15bcbe6d76.png 2023-07-11 21:10:35,806 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2128_b90bc0dcc518a41e4b7c.png 2023-07-11 21:10:35,807 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 21:10:36,317 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/f3ykz8ty-media/images/validation_2128_1c5e23a118700525a7fd.png 2023-07-11 21:10:36,327 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/m94a3ywn-media/images/validation_2128_88801f29434a2a3ded1a.png 2023-07-11 21:10:36,330 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/989ku2cu-media/images/validation_2128_40b0d08fdb15bcbe6d76.png 2023-07-11 21:10:36,352 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/xcz8cltx-media/images/validation_2128_b90bc0dcc518a41e4b7c.png 2023-07-11 21:10:37,159 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:10:37,161 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:10:37,162 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:10:37,163 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:10:40,287 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:10:41,066 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:10:41,068 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:10:41,068 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:10:41,070 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:10:41,310 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 21:10:41,765 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:10:44,437 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:10:44,439 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:10:44,439 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:10:44,441 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:10:45,811 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:10:46,823 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:10:46,823 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:10:47,726 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:10:47,796 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:10:47,797 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:10:47,798 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:10:47,799 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:10:51,115 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:10:51,117 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:10:51,117 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:10:51,118 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:10:51,119 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:10:51,400 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:10:51,402 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:10:51,402 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:10:51,403 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:10:52,727 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:10:56,931 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:10:57,753 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:10:57,841 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:10:57,842 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:11:01,932 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:11:02,796 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:11:03,990 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:11:03,990 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2136_5bc831942544a900b570.png with policy now 2023-07-11 21:11:04,005 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:11:04,005 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2136_961ca855c1854a6642a2.png with policy now 2023-07-11 21:11:04,014 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:11:04,014 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2136_b0e68e79f3ba56f234d1.png with policy now 2023-07-11 21:11:04,024 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:11:04,024 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2136_0358ef62f3e922feee73.png with policy now 2023-07-11 21:11:04,060 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:11:04,066 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:11:04,067 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:11:04,069 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:11:04,309 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/iallcocf-media/images/validation_2136_961ca855c1854a6642a2.png 2023-07-11 21:11:04,322 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/74us87od-media/images/validation_2136_5bc831942544a900b570.png 2023-07-11 21:11:04,322 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/d0q5casv-media/images/validation_2136_b0e68e79f3ba56f234d1.png 2023-07-11 21:11:04,332 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/455bomuv-media/images/validation_2136_0358ef62f3e922feee73.png 2023-07-11 21:11:07,421 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:11:07,422 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:11:08,260 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:11:08,260 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:11:08,863 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:11:08,864 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:11:08,864 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2136_b0e68e79f3ba56f234d1.png 2023-07-11 21:11:08,865 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2136_961ca855c1854a6642a2.png 2023-07-11 21:11:08,865 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2136_0358ef62f3e922feee73.png 2023-07-11 21:11:08,866 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2136_5bc831942544a900b570.png 2023-07-11 21:11:08,866 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 21:11:09,228 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/q0192gs7-media/images/validation_2136_961ca855c1854a6642a2.png 2023-07-11 21:11:09,624 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/tp8qtqwm-media/images/validation_2136_5bc831942544a900b570.png 2023-07-11 21:11:09,788 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/oc8p1g71-media/images/validation_2136_b0e68e79f3ba56f234d1.png 2023-07-11 21:11:09,854 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/qitau6bf-media/images/validation_2136_0358ef62f3e922feee73.png 2023-07-11 21:11:10,661 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:11:10,662 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:11:10,663 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:11:10,664 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:11:11,313 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 21:11:12,756 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:11:13,261 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:11:14,814 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:11:14,817 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:11:14,817 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:11:14,819 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:11:17,906 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:11:18,760 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:11:18,762 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:11:18,762 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:11:18,763 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:11:19,691 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:11:19,884 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:11:19,884 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:11:22,030 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:11:22,032 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:11:22,032 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:11:22,034 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:11:23,035 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:11:25,122 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:11:25,517 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:11:25,518 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:11:25,519 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:11:25,520 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:11:28,669 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:11:28,671 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:11:28,671 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:11:28,672 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:11:28,673 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:11:28,953 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:11:28,955 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:11:28,955 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:11:28,957 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:11:30,541 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:11:30,901 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:11:30,902 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:11:34,509 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:11:35,567 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:11:39,510 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:11:40,611 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:11:41,315 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 21:11:41,557 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:11:41,557 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2144_dd8f27bdf40cf2d6cea9.png with policy now 2023-07-11 21:11:41,567 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:11:41,568 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2144_1116dc6091dfb2a9a99b.png with policy now 2023-07-11 21:11:41,577 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:11:41,577 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2144_33e87beb0aaa1b10bc0b.png with policy now 2023-07-11 21:11:41,585 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:11:41,586 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2144_3e34196265dc5d0dea47.png with policy now 2023-07-11 21:11:41,627 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:11:41,629 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:11:41,629 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:11:41,632 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:11:41,857 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/se6ob04r-media/images/validation_2144_dd8f27bdf40cf2d6cea9.png 2023-07-11 21:11:41,881 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/abib74ml-media/images/validation_2144_1116dc6091dfb2a9a99b.png 2023-07-11 21:11:41,884 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/4ytr20d8-media/images/validation_2144_3e34196265dc5d0dea47.png 2023-07-11 21:11:41,923 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:11:41,925 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:11:41,925 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ftgr01s0-media/images/validation_2144_33e87beb0aaa1b10bc0b.png 2023-07-11 21:11:41,925 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2144_3e34196265dc5d0dea47.png 2023-07-11 21:11:41,926 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2144_dd8f27bdf40cf2d6cea9.png 2023-07-11 21:11:41,927 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2144_33e87beb0aaa1b10bc0b.png 2023-07-11 21:11:41,927 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2144_1116dc6091dfb2a9a99b.png 2023-07-11 21:11:41,928 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 21:11:42,416 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/qstnc6qt-media/images/validation_2144_dd8f27bdf40cf2d6cea9.png 2023-07-11 21:11:42,422 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/jklcquby-media/images/validation_2144_3e34196265dc5d0dea47.png 2023-07-11 21:11:42,425 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/b7cszef1-media/images/validation_2144_33e87beb0aaa1b10bc0b.png 2023-07-11 21:11:42,441 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/9x8hggeg-media/images/validation_2144_1116dc6091dfb2a9a99b.png 2023-07-11 21:11:45,323 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:11:46,446 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:11:46,491 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:11:50,212 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:11:50,214 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:11:50,214 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:11:50,216 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:11:50,887 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:11:51,544 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:11:52,939 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:11:52,940 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:11:53,238 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:11:53,240 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:11:53,241 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:11:53,242 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:11:55,937 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:11:56,575 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:11:56,577 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:11:56,577 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:11:56,579 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:11:57,244 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:11:59,742 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:11:59,744 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:11:59,744 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:11:59,745 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:12:01,389 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:12:02,875 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:12:03,004 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:12:03,008 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:12:03,008 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:12:03,010 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:12:03,959 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:12:03,960 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:12:06,132 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:12:06,134 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:12:06,134 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:12:06,136 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:12:06,416 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:12:06,418 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:12:06,418 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:12:06,419 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:12:06,420 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:12:08,005 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:12:11,320 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 21:12:12,322 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:12:13,058 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:12:14,978 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:12:14,978 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:12:17,323 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:12:18,100 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:12:19,043 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:12:19,043 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2152_7c72c782ab604dee6e7d.png with policy now 2023-07-11 21:12:19,053 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:12:19,053 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2152_9afccb4fc160b21562e7.png with policy now 2023-07-11 21:12:19,062 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:12:19,062 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2152_0772544257fba9b99d85.png with policy now 2023-07-11 21:12:19,071 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:12:19,071 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2152_a1a8f442cbc67c6daade.png with policy now 2023-07-11 21:12:19,112 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:12:19,114 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:12:19,115 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:12:19,117 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:12:19,359 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/b2wwxc3d-media/images/validation_2152_7c72c782ab604dee6e7d.png 2023-07-11 21:12:19,372 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/y8gdrb7o-media/images/validation_2152_9afccb4fc160b21562e7.png 2023-07-11 21:12:19,386 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/i3xo24cm-media/images/validation_2152_0772544257fba9b99d85.png 2023-07-11 21:12:19,456 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/jyuzkoq0-media/images/validation_2152_a1a8f442cbc67c6daade.png 2023-07-11 21:12:22,422 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:12:23,130 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:12:23,486 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:12:23,486 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:12:25,815 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:12:25,817 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:12:25,818 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:12:25,819 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:12:26,000 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:12:26,000 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:12:26,001 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2152_9afccb4fc160b21562e7.png 2023-07-11 21:12:26,001 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2152_a1a8f442cbc67c6daade.png 2023-07-11 21:12:26,002 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2152_0772544257fba9b99d85.png 2023-07-11 21:12:26,002 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2152_7c72c782ab604dee6e7d.png 2023-07-11 21:12:26,003 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 21:12:26,354 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/s8126u65-media/images/validation_2152_7c72c782ab604dee6e7d.png 2023-07-11 21:12:26,461 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/bd0rlyzq-media/images/validation_2152_a1a8f442cbc67c6daade.png 2023-07-11 21:12:26,526 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/f9703og5-media/images/validation_2152_9afccb4fc160b21562e7.png 2023-07-11 21:12:26,545 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/hxr3y6ny-media/images/validation_2152_0772544257fba9b99d85.png 2023-07-11 21:12:27,683 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:12:28,707 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:12:29,274 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:12:29,276 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:12:29,276 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:12:29,278 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:12:33,025 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:12:33,026 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:12:33,026 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:12:33,027 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:12:33,028 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:12:33,985 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:12:36,264 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:12:36,266 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:12:36,267 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:12:36,268 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:12:37,067 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:12:37,067 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:12:38,493 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:12:39,044 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:12:39,045 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:12:39,046 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:12:39,047 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:12:39,370 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:12:41,321 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 21:12:42,495 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:12:42,497 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:12:42,497 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:12:42,499 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:12:42,779 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:12:42,781 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:12:42,781 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:12:42,783 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:12:44,406 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:12:44,413 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:12:48,085 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:12:48,085 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:12:49,414 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:12:49,453 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:12:54,415 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:12:54,490 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:12:55,474 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:12:55,474 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2160_df30dfecafc8706acc38.png with policy now 2023-07-11 21:12:55,488 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:12:55,488 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2160_d966f9a37b7475161185.png with policy now 2023-07-11 21:12:55,499 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:12:55,499 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2160_802878029984f839da7a.png with policy now 2023-07-11 21:12:55,507 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:12:55,508 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2160_42ae5bc916492ef02b4e.png with policy now 2023-07-11 21:12:55,543 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:12:55,544 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:12:55,545 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:12:55,546 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:12:55,810 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/nx3fp3kp-media/images/validation_2160_df30dfecafc8706acc38.png 2023-07-11 21:12:55,812 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/imsa395f-media/images/validation_2160_d966f9a37b7475161185.png 2023-07-11 21:12:55,817 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/i0zlwtwf-media/images/validation_2160_802878029984f839da7a.png 2023-07-11 21:12:55,821 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/8ov7zde7-media/images/validation_2160_42ae5bc916492ef02b4e.png 2023-07-11 21:12:58,908 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:12:59,108 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:12:59,108 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:12:59,109 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2160_df30dfecafc8706acc38.png 2023-07-11 21:12:59,109 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2160_d966f9a37b7475161185.png 2023-07-11 21:12:59,110 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2160_802878029984f839da7a.png 2023-07-11 21:12:59,110 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2160_42ae5bc916492ef02b4e.png 2023-07-11 21:12:59,111 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 21:12:59,480 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/gvk3qcei-media/images/validation_2160_d966f9a37b7475161185.png 2023-07-11 21:12:59,530 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:12:59,647 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/3w5f5fqs-media/images/validation_2160_42ae5bc916492ef02b4e.png 2023-07-11 21:12:59,647 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/3zmozsth-media/images/validation_2160_df30dfecafc8706acc38.png 2023-07-11 21:12:59,661 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/t9nwh6cm-media/images/validation_2160_802878029984f839da7a.png 2023-07-11 21:12:59,891 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:13:02,179 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:13:02,180 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:13:02,181 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:13:02,183 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:13:05,049 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:13:05,841 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:13:05,913 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:13:05,915 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:13:05,915 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:13:05,917 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:13:08,676 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:13:08,678 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:13:08,678 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:13:08,680 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:13:10,127 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:13:10,127 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:13:10,161 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:13:10,161 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:13:11,324 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 21:13:11,329 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:13:11,842 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:13:11,844 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:13:11,844 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:13:11,846 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:13:14,506 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:13:14,508 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:13:14,508 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:13:14,509 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:13:15,381 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:13:16,522 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:13:18,419 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:13:18,421 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:13:18,421 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:13:18,423 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:13:18,703 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:13:18,705 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:13:18,705 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:13:18,707 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:13:20,727 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:13:21,145 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:13:21,146 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:13:22,237 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:13:25,749 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:13:27,238 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:13:30,790 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:13:31,348 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:13:31,348 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2168_7f77a8960cb444700fe1.png with policy now 2023-07-11 21:13:31,363 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:13:31,363 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2168_7fa7b07d6e51ffc6ee7c.png with policy now 2023-07-11 21:13:31,373 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:13:31,374 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2168_11433d98fdff95688292.png with policy now 2023-07-11 21:13:31,376 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:13:31,376 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2168_ab42f1cb028adcc36624.png with policy now 2023-07-11 21:13:31,417 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:13:31,419 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:13:31,419 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:13:31,420 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:13:31,660 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/p63sjr6c-media/images/validation_2168_7f77a8960cb444700fe1.png 2023-07-11 21:13:31,689 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/j2pnpagr-media/images/validation_2168_7fa7b07d6e51ffc6ee7c.png 2023-07-11 21:13:31,693 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/cclkx6s5-media/images/validation_2168_ab42f1cb028adcc36624.png 2023-07-11 21:13:31,697 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/j3eqiwki-media/images/validation_2168_11433d98fdff95688292.png 2023-07-11 21:13:32,168 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:13:32,169 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2168_ab42f1cb028adcc36624.png 2023-07-11 21:13:32,169 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2168_11433d98fdff95688292.png 2023-07-11 21:13:32,170 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2168_7fa7b07d6e51ffc6ee7c.png 2023-07-11 21:13:32,170 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2168_7f77a8960cb444700fe1.png 2023-07-11 21:13:32,171 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 21:13:32,421 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:13:32,596 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/fq27asig-media/images/validation_2168_7fa7b07d6e51ffc6ee7c.png 2023-07-11 21:13:32,660 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/apkpt1rd-media/images/validation_2168_ab42f1cb028adcc36624.png 2023-07-11 21:13:32,680 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/kpaloh8u-media/images/validation_2168_11433d98fdff95688292.png 2023-07-11 21:13:32,749 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/tgljjcyz-media/images/validation_2168_7f77a8960cb444700fe1.png 2023-07-11 21:13:35,688 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:13:35,791 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:13:38,024 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:13:38,760 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:13:38,762 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:13:38,763 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:13:38,764 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:13:40,922 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:13:41,327 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 21:13:42,167 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:13:42,169 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:13:42,170 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:13:42,171 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:13:43,123 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:13:43,187 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:13:43,187 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:13:46,011 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:13:46,013 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:13:46,013 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:13:46,014 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:13:46,883 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:13:48,506 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:13:49,152 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:13:49,154 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:13:49,154 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:13:49,156 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:13:52,338 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:13:52,339 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:13:52,339 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:13:52,341 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:13:52,468 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:13:53,703 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:13:54,202 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:13:54,203 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:13:54,914 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:13:54,916 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:13:54,916 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:13:54,918 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:13:55,199 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:13:55,201 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:13:55,201 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:13:55,203 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:13:57,625 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:13:58,753 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:14:02,687 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:14:03,754 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:14:05,221 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:14:05,222 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:14:07,713 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:14:07,784 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:14:07,784 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2176_4743122548c8aa7b69c0.png with policy now 2023-07-11 21:14:07,794 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:14:07,794 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2176_282f811b6189202a6ebc.png with policy now 2023-07-11 21:14:07,807 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:14:07,807 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2176_0917d80c2d5e3650d3c3.png with policy now 2023-07-11 21:14:07,817 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:14:07,817 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2176_91fab3422693e45733b1.png with policy now 2023-07-11 21:14:07,852 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:14:07,854 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:14:07,854 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:14:07,856 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:14:08,094 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/vns86ke9-media/images/validation_2176_4743122548c8aa7b69c0.png 2023-07-11 21:14:08,115 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/dqshkd9c-media/images/validation_2176_282f811b6189202a6ebc.png 2023-07-11 21:14:08,116 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/t1rt42qp-media/images/validation_2176_91fab3422693e45733b1.png 2023-07-11 21:14:08,121 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/t7911zx3-media/images/validation_2176_0917d80c2d5e3650d3c3.png 2023-07-11 21:14:08,856 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:14:11,330 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 21:14:11,623 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:14:13,496 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:14:14,299 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:14:15,243 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:14:15,245 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:14:15,245 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:14:15,247 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:14:16,242 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:14:16,243 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:14:16,243 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2176_282f811b6189202a6ebc.png 2023-07-11 21:14:16,244 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2176_4743122548c8aa7b69c0.png 2023-07-11 21:14:16,244 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2176_91fab3422693e45733b1.png 2023-07-11 21:14:16,245 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2176_0917d80c2d5e3650d3c3.png 2023-07-11 21:14:16,245 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 21:14:16,599 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ioyilaob-media/images/validation_2176_0917d80c2d5e3650d3c3.png 2023-07-11 21:14:16,763 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/sg0w2ei7-media/images/validation_2176_4743122548c8aa7b69c0.png 2023-07-11 21:14:16,776 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/4hukclps-media/images/validation_2176_282f811b6189202a6ebc.png 2023-07-11 21:14:16,900 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/aw2nbbz1-media/images/validation_2176_91fab3422693e45733b1.png 2023-07-11 21:14:18,103 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:14:18,105 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:14:18,105 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:14:18,107 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:14:19,072 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:14:20,194 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:14:22,089 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:14:22,090 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:14:22,091 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:14:22,092 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:14:24,245 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:14:24,662 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:14:24,664 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:14:24,664 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:14:24,666 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:14:25,667 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:14:27,262 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:14:27,263 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:14:28,910 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:14:28,911 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:14:28,911 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:14:28,913 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:14:30,142 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:14:31,023 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:14:31,858 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:14:31,860 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:14:31,861 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:14:31,863 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:14:32,143 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:14:32,145 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:14:32,145 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:14:32,146 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:14:35,266 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:14:36,713 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:14:38,281 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:14:38,281 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:14:40,316 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:14:41,333 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 21:14:42,335 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:14:44,779 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:14:44,780 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2184_1f1d05e973eac5066d87.png with policy now 2023-07-11 21:14:44,790 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:14:44,790 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2184_92af547bd8ce73a91373.png with policy now 2023-07-11 21:14:44,799 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:14:44,799 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2184_f7d901834bbc876e0496.png with policy now 2023-07-11 21:14:44,811 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:14:44,811 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2184_8bea371a51240befcb0c.png with policy now 2023-07-11 21:14:44,847 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:14:44,848 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:14:44,849 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:14:44,850 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:14:45,123 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/7rzfgm4o-media/images/validation_2184_1f1d05e973eac5066d87.png 2023-07-11 21:14:45,140 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/y1dbgc5i-media/images/validation_2184_f7d901834bbc876e0496.png 2023-07-11 21:14:45,159 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/whs3ppj8-media/images/validation_2184_92af547bd8ce73a91373.png 2023-07-11 21:14:45,171 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/nmuqkb56-media/images/validation_2184_8bea371a51240befcb0c.png 2023-07-11 21:14:46,936 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:14:47,806 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:14:49,266 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:14:49,304 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:14:49,304 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:14:49,304 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2184_f7d901834bbc876e0496.png 2023-07-11 21:14:49,305 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2184_1f1d05e973eac5066d87.png 2023-07-11 21:14:49,305 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2184_8bea371a51240befcb0c.png 2023-07-11 21:14:49,306 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2184_92af547bd8ce73a91373.png 2023-07-11 21:14:49,306 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 21:14:49,681 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/snem9m1q-media/images/validation_2184_1f1d05e973eac5066d87.png 2023-07-11 21:14:49,744 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/eggntz3g-media/images/validation_2184_f7d901834bbc876e0496.png 2023-07-11 21:14:49,791 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/jt1kn9r8-media/images/validation_2184_8bea371a51240befcb0c.png 2023-07-11 21:14:49,872 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/xk2kra5t-media/images/validation_2184_92af547bd8ce73a91373.png 2023-07-11 21:14:52,167 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:14:52,169 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:14:52,169 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:14:52,171 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:14:52,642 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:14:52,642 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:14:53,509 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:14:54,784 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:14:54,786 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:14:54,787 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:14:54,788 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:14:57,642 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:14:57,644 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:14:57,645 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:14:57,646 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:14:57,656 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:14:59,248 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:15:00,320 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:15:00,320 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:15:00,558 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:15:00,560 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:15:00,560 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:15:00,562 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:15:02,677 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:15:04,501 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:15:04,503 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:15:04,503 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:15:04,504 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:15:04,506 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:15:07,956 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:15:08,342 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:15:08,343 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:15:08,344 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:15:08,345 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:15:08,624 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:15:08,626 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:15:08,626 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:15:08,628 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:15:10,182 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:15:11,338 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:15:11,338 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 21:15:11,339 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:15:13,069 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:15:15,341 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:15:18,100 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:15:20,342 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:15:21,194 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:15:21,194 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2192_43ded98624cac24bcce2.png with policy now 2023-07-11 21:15:21,204 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:15:21,205 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2192_1a9b8a8abe690731327a.png with policy now 2023-07-11 21:15:21,213 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:15:21,213 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2192_cf71b06b81ea71e6cb38.png with policy now 2023-07-11 21:15:21,222 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:15:21,222 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2192_ba57f9ac069863aa6a16.png with policy now 2023-07-11 21:15:21,264 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:15:21,266 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:15:21,266 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:15:21,269 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:15:21,508 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/s69rmprm-media/images/validation_2192_43ded98624cac24bcce2.png 2023-07-11 21:15:21,526 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/cxjnwmhc-media/images/validation_2192_1a9b8a8abe690731327a.png 2023-07-11 21:15:21,527 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/3n74fd77-media/images/validation_2192_cf71b06b81ea71e6cb38.png 2023-07-11 21:15:21,550 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/dofbrm44-media/images/validation_2192_ba57f9ac069863aa6a16.png 2023-07-11 21:15:22,357 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:15:22,357 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2192_cf71b06b81ea71e6cb38.png 2023-07-11 21:15:22,358 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2192_ba57f9ac069863aa6a16.png 2023-07-11 21:15:22,359 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2192_1a9b8a8abe690731327a.png 2023-07-11 21:15:22,360 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2192_43ded98624cac24bcce2.png 2023-07-11 21:15:22,360 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 21:15:22,717 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/eyirfjyi-media/images/validation_2192_43ded98624cac24bcce2.png 2023-07-11 21:15:22,725 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/h6db8cqz-media/images/validation_2192_cf71b06b81ea71e6cb38.png 2023-07-11 21:15:22,871 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ci7goknz-media/images/validation_2192_ba57f9ac069863aa6a16.png 2023-07-11 21:15:22,904 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/27pxqvha-media/images/validation_2192_1a9b8a8abe690731327a.png 2023-07-11 21:15:23,114 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:15:25,324 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:15:25,801 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:15:28,366 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:15:28,368 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:15:28,368 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:15:28,370 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:15:28,466 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:15:31,266 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:15:31,267 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:15:31,267 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:15:31,268 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:15:31,270 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:15:33,378 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:15:33,378 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:15:33,879 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:15:34,763 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:15:34,765 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:15:34,765 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:15:34,767 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:15:36,349 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:15:38,147 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:15:38,149 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:15:38,149 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:15:38,151 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:15:39,229 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:15:41,341 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 21:15:41,750 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:15:41,754 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:15:41,754 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:15:41,754 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:15:41,756 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:15:44,495 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:15:45,398 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:15:45,398 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:15:45,454 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:15:45,455 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:15:45,456 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:15:45,457 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:15:45,738 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:15:45,739 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:15:45,740 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:15:45,742 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:15:47,297 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:15:49,540 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:15:52,298 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:15:54,571 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:15:57,299 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:15:57,415 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:15:57,415 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:15:58,350 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:15:58,350 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2200_ee3e22833ab7e1d894c4.png with policy now 2023-07-11 21:15:58,366 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:15:58,366 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2200_c93ac716040789e5a0e3.png with policy now 2023-07-11 21:15:58,369 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:15:58,369 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2200_00596fead5bf8fb0065d.png with policy now 2023-07-11 21:15:58,378 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:15:58,378 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2200_4dfaff90212e9e27e4d2.png with policy now 2023-07-11 21:15:58,418 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:15:58,420 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:15:58,420 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:15:58,422 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:15:58,687 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/0j0ael56-media/images/validation_2200_c93ac716040789e5a0e3.png 2023-07-11 21:15:58,687 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/u4k2v9fm-media/images/validation_2200_ee3e22833ab7e1d894c4.png 2023-07-11 21:15:58,709 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/h3f1xerj-media/images/validation_2200_4dfaff90212e9e27e4d2.png 2023-07-11 21:15:58,726 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/05uekkzz-media/images/validation_2200_00596fead5bf8fb0065d.png 2023-07-11 21:15:59,761 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:16:02,392 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:16:02,962 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:16:04,856 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:16:06,166 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:16:06,168 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:16:06,169 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:16:06,170 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:16:08,176 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:16:09,437 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:16:09,437 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:16:09,438 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2200_4dfaff90212e9e27e4d2.png 2023-07-11 21:16:09,438 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2200_c93ac716040789e5a0e3.png 2023-07-11 21:16:09,439 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2200_00596fead5bf8fb0065d.png 2023-07-11 21:16:09,439 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2200_ee3e22833ab7e1d894c4.png 2023-07-11 21:16:09,440 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 21:16:09,765 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/y0j7l09x-media/images/validation_2200_4dfaff90212e9e27e4d2.png 2023-07-11 21:16:09,806 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/sg4tgswc-media/images/validation_2200_ee3e22833ab7e1d894c4.png 2023-07-11 21:16:09,807 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/0g53j903-media/images/validation_2200_00596fead5bf8fb0065d.png 2023-07-11 21:16:09,970 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/f6blsot2-media/images/validation_2200_c93ac716040789e5a0e3.png 2023-07-11 21:16:10,095 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:16:10,097 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:16:10,097 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:16:10,098 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:16:10,176 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:16:10,179 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:16:11,344 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 21:16:12,604 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:16:12,606 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:16:12,606 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:16:12,608 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:16:13,355 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:16:15,901 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:16:16,028 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:16:16,030 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:16:16,030 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:16:16,032 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:16:18,666 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:16:19,367 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:16:19,368 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:16:19,369 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:16:19,370 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:16:21,448 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:16:21,449 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:16:21,640 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:16:23,189 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:16:23,191 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:16:23,191 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:16:23,193 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:16:23,471 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:16:23,473 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:16:23,473 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:16:23,474 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:16:23,881 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:16:26,676 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:16:29,017 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:16:31,720 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:16:33,467 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:16:33,468 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:16:34,018 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:16:36,097 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:16:36,098 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2208_ffbfe4f9aca48bda03c2.png with policy now 2023-07-11 21:16:36,112 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:16:36,113 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2208_eef023764c90ffd3644e.png with policy now 2023-07-11 21:16:36,121 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:16:36,122 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2208_2f73c4c4c50247c1768f.png with policy now 2023-07-11 21:16:36,132 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:16:36,132 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2208_308d8b2bbe36fb5f30fc.png with policy now 2023-07-11 21:16:36,166 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:16:36,167 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:16:36,174 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:16:36,177 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:16:36,429 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/xfmvo6m9-media/images/validation_2208_2f73c4c4c50247c1768f.png 2023-07-11 21:16:36,441 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/i9asam27-media/images/validation_2208_eef023764c90ffd3644e.png 2023-07-11 21:16:36,441 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/uy8ti4y4-media/images/validation_2208_308d8b2bbe36fb5f30fc.png 2023-07-11 21:16:36,449 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/heh56qge-media/images/validation_2208_ffbfe4f9aca48bda03c2.png 2023-07-11 21:16:36,743 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:16:39,081 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:16:39,640 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:16:41,348 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 21:16:42,094 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:16:42,609 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:16:42,611 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:16:42,611 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:16:42,613 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:16:44,726 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:16:45,495 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:16:45,495 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:16:45,496 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2208_308d8b2bbe36fb5f30fc.png 2023-07-11 21:16:45,496 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2208_2f73c4c4c50247c1768f.png 2023-07-11 21:16:45,497 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2208_ffbfe4f9aca48bda03c2.png 2023-07-11 21:16:45,497 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2208_eef023764c90ffd3644e.png 2023-07-11 21:16:45,498 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 21:16:45,838 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/r5ikueh6-media/images/validation_2208_308d8b2bbe36fb5f30fc.png 2023-07-11 21:16:45,862 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/2o19k7wr-media/images/validation_2208_eef023764c90ffd3644e.png 2023-07-11 21:16:45,870 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/b4hgkyfj-media/images/validation_2208_ffbfe4f9aca48bda03c2.png 2023-07-11 21:16:45,928 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/eu0ukpvk-media/images/validation_2208_2f73c4c4c50247c1768f.png 2023-07-11 21:16:46,522 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:16:46,524 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:16:46,524 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:16:46,526 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:16:47,242 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:16:49,772 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:16:49,772 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:16:49,773 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:16:49,773 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:16:49,775 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:16:53,038 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:16:53,038 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:16:53,182 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:16:53,184 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:16:53,184 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:16:53,186 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:16:55,180 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:16:56,549 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:16:56,550 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:16:56,551 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:16:56,552 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:16:57,506 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:16:57,507 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:16:58,900 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:17:00,169 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:17:00,170 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:17:00,171 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:17:00,173 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:17:00,451 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:17:00,453 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:17:00,453 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:17:00,454 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:17:00,455 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:17:04,076 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:17:05,998 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:17:09,117 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:17:09,527 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:17:09,528 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:17:11,000 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:17:11,350 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 21:17:13,014 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:17:13,014 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2216_441d862976c9f51b92d6.png with policy now 2023-07-11 21:17:13,029 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:17:13,029 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2216_5f8a9556edf1586f4491.png with policy now 2023-07-11 21:17:13,039 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:17:13,039 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2216_775468a5cae55b01f9e6.png with policy now 2023-07-11 21:17:13,042 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:17:13,042 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2216_8c1ab982496cf82a8f19.png with policy now 2023-07-11 21:17:13,082 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:17:13,083 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:17:13,083 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:17:13,085 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:17:13,350 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ygftknal-media/images/validation_2216_441d862976c9f51b92d6.png 2023-07-11 21:17:13,366 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/2c1pgdhs-media/images/validation_2216_8c1ab982496cf82a8f19.png 2023-07-11 21:17:13,372 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/9gkrgs6y-media/images/validation_2216_775468a5cae55b01f9e6.png 2023-07-11 21:17:13,377 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ctsqs9ak-media/images/validation_2216_5f8a9556edf1586f4491.png 2023-07-11 21:17:14,206 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:17:16,188 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:17:16,674 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:17:19,555 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:17:20,745 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:17:20,747 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:17:20,747 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:17:20,749 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:17:21,552 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:17:21,553 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:17:21,553 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2216_775468a5cae55b01f9e6.png 2023-07-11 21:17:21,554 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2216_441d862976c9f51b92d6.png 2023-07-11 21:17:21,554 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2216_5f8a9556edf1586f4491.png 2023-07-11 21:17:21,555 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2216_8c1ab982496cf82a8f19.png 2023-07-11 21:17:21,555 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 21:17:21,660 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:17:21,889 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/mwffnwai-media/images/validation_2216_775468a5cae55b01f9e6.png 2023-07-11 21:17:21,910 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/i3lld6ir-media/images/validation_2216_5f8a9556edf1586f4491.png 2023-07-11 21:17:21,910 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/wfwdbegk-media/images/validation_2216_441d862976c9f51b92d6.png 2023-07-11 21:17:22,094 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/om0my1s9-media/images/validation_2216_8c1ab982496cf82a8f19.png 2023-07-11 21:17:24,440 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:17:24,442 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:17:24,443 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:17:24,444 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:17:25,420 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:17:27,177 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:17:27,922 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:17:27,924 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:17:27,925 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:17:27,926 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:17:30,798 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:17:31,332 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:17:31,334 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:17:31,335 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:17:31,336 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:17:32,337 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:17:33,569 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:17:33,569 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:17:34,507 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:17:34,508 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:17:34,508 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:17:34,510 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:17:36,381 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:17:37,866 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:17:37,868 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:17:37,868 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:17:37,868 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:17:37,870 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:17:38,147 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:17:38,149 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:17:38,149 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:17:38,151 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:17:41,353 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 21:17:41,739 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:17:43,355 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:17:45,590 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:17:45,590 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:17:46,768 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:17:48,356 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:17:51,034 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:17:51,034 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2224_13c69897297371dcf8c5.png with policy now 2023-07-11 21:17:51,049 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:17:51,049 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2224_ef5660a5a79987882227.png with policy now 2023-07-11 21:17:51,059 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:17:51,059 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2224_d3babefe131fdc098ad5.png with policy now 2023-07-11 21:17:51,068 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:17:51,068 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2224_9b460613f0ab2e925269.png with policy now 2023-07-11 21:17:51,103 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:17:51,105 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:17:51,105 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:17:51,106 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:17:51,350 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/722g2eu2-media/images/validation_2224_13c69897297371dcf8c5.png 2023-07-11 21:17:51,365 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/b00s4cks-media/images/validation_2224_9b460613f0ab2e925269.png 2023-07-11 21:17:51,372 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/g8vfbbim-media/images/validation_2224_ef5660a5a79987882227.png 2023-07-11 21:17:51,389 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/61j2gjgh-media/images/validation_2224_d3babefe131fdc098ad5.png 2023-07-11 21:17:51,790 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:17:54,227 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:17:55,280 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:17:57,171 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:17:57,171 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:17:57,612 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:17:57,612 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:17:57,613 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2224_13c69897297371dcf8c5.png 2023-07-11 21:17:57,613 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2224_d3babefe131fdc098ad5.png 2023-07-11 21:17:57,614 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2224_9b460613f0ab2e925269.png 2023-07-11 21:17:57,614 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2224_ef5660a5a79987882227.png 2023-07-11 21:17:57,614 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 21:17:57,974 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/bjj3fbpi-media/images/validation_2224_9b460613f0ab2e925269.png 2023-07-11 21:17:57,981 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/9fr3pabc-media/images/validation_2224_d3babefe131fdc098ad5.png 2023-07-11 21:17:57,981 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/33a366s3-media/images/validation_2224_ef5660a5a79987882227.png 2023-07-11 21:17:58,050 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/563nsyoo-media/images/validation_2224_13c69897297371dcf8c5.png 2023-07-11 21:17:59,137 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:17:59,139 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:17:59,139 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:17:59,141 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:17:59,802 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:18:03,033 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:18:03,162 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:18:03,163 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:18:03,163 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:18:03,165 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:18:05,101 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:18:05,861 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:18:05,863 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:18:05,864 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:18:05,865 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:18:08,034 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:18:08,965 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:18:08,967 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:18:08,967 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:18:08,969 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:18:09,623 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:18:09,624 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:18:10,203 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:18:11,355 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 21:18:11,961 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:18:11,963 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:18:11,963 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:18:11,965 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:18:13,081 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:18:15,619 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:18:15,621 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:18:15,621 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:18:15,622 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:18:15,624 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:18:15,901 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:18:15,903 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:18:15,903 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:18:15,905 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:18:18,494 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:18:21,482 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:18:21,642 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:18:21,642 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:18:23,541 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:18:26,483 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:18:28,492 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:18:28,492 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2232_5428aa3c038948a084d2.png with policy now 2023-07-11 21:18:28,507 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:18:28,507 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2232_d019d6eafe82501f45c5.png with policy now 2023-07-11 21:18:28,517 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:18:28,517 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2232_de35cad4622885a2e7e3.png with policy now 2023-07-11 21:18:28,526 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:18:28,526 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2232_810d1b9e7965f4ca6207.png with policy now 2023-07-11 21:18:28,564 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:18:28,567 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:18:28,567 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:18:28,570 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:18:28,577 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:18:28,816 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/3l9h4j1q-media/images/validation_2232_5428aa3c038948a084d2.png 2023-07-11 21:18:28,822 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/xn9d673a-media/images/validation_2232_d019d6eafe82501f45c5.png 2023-07-11 21:18:28,824 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/8b3tse4w-media/images/validation_2232_810d1b9e7965f4ca6207.png 2023-07-11 21:18:28,838 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/thaalkd3-media/images/validation_2232_de35cad4622885a2e7e3.png 2023-07-11 21:18:32,121 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:18:32,122 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:18:33,658 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:18:33,659 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:18:33,659 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2232_810d1b9e7965f4ca6207.png 2023-07-11 21:18:33,659 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2232_de35cad4622885a2e7e3.png 2023-07-11 21:18:33,660 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2232_d019d6eafe82501f45c5.png 2023-07-11 21:18:33,660 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2232_5428aa3c038948a084d2.png 2023-07-11 21:18:33,661 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 21:18:34,093 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/4je1jzp4-media/images/validation_2232_810d1b9e7965f4ca6207.png 2023-07-11 21:18:34,115 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/vat14ukg-media/images/validation_2232_de35cad4622885a2e7e3.png 2023-07-11 21:18:34,173 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:18:34,183 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/w23du0qt-media/images/validation_2232_d019d6eafe82501f45c5.png 2023-07-11 21:18:34,194 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/8jfkdy63-media/images/validation_2232_5428aa3c038948a084d2.png 2023-07-11 21:18:35,039 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:18:35,041 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:18:35,041 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:18:35,043 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:18:37,796 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:18:38,368 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:18:38,370 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:18:38,370 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:18:38,372 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:18:39,760 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:18:39,761 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:18:41,131 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:18:41,133 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:18:41,133 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:18:41,135 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:18:41,358 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 21:18:43,526 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:18:44,432 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:18:44,434 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:18:44,434 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:18:44,435 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:18:45,501 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:18:45,674 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:18:45,675 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:18:48,134 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:18:48,136 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:18:48,136 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:18:48,138 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:18:49,094 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:18:51,199 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:18:51,327 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:18:51,328 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:18:51,329 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:18:51,330 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:18:51,609 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:18:51,611 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:18:51,612 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:18:51,613 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:18:54,159 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:18:56,233 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:18:57,694 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:18:57,695 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:18:59,161 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:19:01,257 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:19:04,162 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:19:04,201 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:19:04,201 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2240_fc9f1f75957ce5fda00b.png with policy now 2023-07-11 21:19:04,216 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:19:04,216 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2240_0be5a7d6e1a87c6281ad.png with policy now 2023-07-11 21:19:04,226 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:19:04,226 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2240_e9d3e01b8fb9ddbcc763.png with policy now 2023-07-11 21:19:04,235 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:19:04,235 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2240_5e39882221c773c2aaeb.png with policy now 2023-07-11 21:19:04,270 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:19:04,275 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:19:04,276 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:19:04,279 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:19:04,522 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/5zdweygj-media/images/validation_2240_fc9f1f75957ce5fda00b.png 2023-07-11 21:19:04,526 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/vuvaqnhv-media/images/validation_2240_e9d3e01b8fb9ddbcc763.png 2023-07-11 21:19:04,533 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/rs7bwxyc-media/images/validation_2240_0be5a7d6e1a87c6281ad.png 2023-07-11 21:19:04,548 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/1b12iypb-media/images/validation_2240_5e39882221c773c2aaeb.png 2023-07-11 21:19:06,303 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:19:07,431 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:19:09,717 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:19:09,718 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:19:09,718 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2240_fc9f1f75957ce5fda00b.png 2023-07-11 21:19:09,718 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2240_e9d3e01b8fb9ddbcc763.png 2023-07-11 21:19:09,719 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2240_5e39882221c773c2aaeb.png 2023-07-11 21:19:09,719 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2240_0be5a7d6e1a87c6281ad.png 2023-07-11 21:19:09,720 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 21:19:09,859 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:19:10,080 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/imdvrury-media/images/validation_2240_fc9f1f75957ce5fda00b.png 2023-07-11 21:19:10,087 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/o4lnqrnu-media/images/validation_2240_e9d3e01b8fb9ddbcc763.png 2023-07-11 21:19:10,255 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/pehqrn94-media/images/validation_2240_5e39882221c773c2aaeb.png 2023-07-11 21:19:10,289 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/hvqi1i0v-media/images/validation_2240_0be5a7d6e1a87c6281ad.png 2023-07-11 21:19:10,733 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:19:10,735 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:19:10,735 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:19:10,737 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:19:11,360 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 21:19:11,376 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:19:11,376 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:19:14,080 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:19:14,082 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:19:14,083 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:19:14,084 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:19:15,826 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:19:16,734 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:19:17,182 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:19:17,184 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:19:17,185 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:19:17,186 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:19:20,790 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:19:20,794 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:19:20,794 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:19:20,795 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:19:21,457 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:19:21,733 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:19:21,733 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:19:21,784 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:19:24,217 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:19:24,219 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:19:24,219 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:19:24,221 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:19:26,689 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:19:27,766 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:19:27,766 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:19:28,502 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:19:28,504 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:19:28,505 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:19:28,506 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:19:28,785 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:19:28,787 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:19:28,787 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:19:28,789 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:19:32,366 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:19:32,881 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:19:33,754 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:19:33,755 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:19:37,367 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:19:37,926 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:19:41,362 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 21:19:41,411 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:19:41,411 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2248_4d6ac12b990c4625492f.png with policy now 2023-07-11 21:19:41,426 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:19:41,426 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2248_60ebe036679521533b7c.png with policy now 2023-07-11 21:19:41,433 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:19:41,433 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2248_2662eb4ad8f0124e3af2.png with policy now 2023-07-11 21:19:41,443 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:19:41,443 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2248_20630ae4a09080a1aebc.png with policy now 2023-07-11 21:19:41,482 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:19:41,485 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:19:41,486 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:19:41,489 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:19:41,748 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/l3hdizdj-media/images/validation_2248_60ebe036679521533b7c.png 2023-07-11 21:19:41,750 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/jidir096-media/images/validation_2248_20630ae4a09080a1aebc.png 2023-07-11 21:19:41,756 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/lb8swzwt-media/images/validation_2248_2662eb4ad8f0124e3af2.png 2023-07-11 21:19:41,772 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/fwn25ym3-media/images/validation_2248_4d6ac12b990c4625492f.png 2023-07-11 21:19:42,491 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:19:42,946 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:19:45,751 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:19:45,777 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:19:45,778 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:19:45,778 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2248_4d6ac12b990c4625492f.png 2023-07-11 21:19:45,779 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2248_2662eb4ad8f0124e3af2.png 2023-07-11 21:19:45,779 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2248_20630ae4a09080a1aebc.png 2023-07-11 21:19:45,780 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2248_60ebe036679521533b7c.png 2023-07-11 21:19:45,780 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 21:19:46,131 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/maw73jr5-media/images/validation_2248_2662eb4ad8f0124e3af2.png 2023-07-11 21:19:46,269 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/eb3x3t5i-media/images/validation_2248_4d6ac12b990c4625492f.png 2023-07-11 21:19:46,306 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/vxkd39tp-media/images/validation_2248_20630ae4a09080a1aebc.png 2023-07-11 21:19:46,343 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/idh6das8-media/images/validation_2248_60ebe036679521533b7c.png 2023-07-11 21:19:47,956 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:19:48,307 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:19:48,661 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:19:48,663 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:19:48,664 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:19:48,665 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:19:52,372 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:19:52,374 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:19:52,374 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:19:52,376 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:19:53,377 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:19:53,613 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:19:55,739 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:19:55,740 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:19:55,741 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:19:55,742 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:19:57,796 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:19:57,796 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:19:58,766 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:19:58,889 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:19:58,946 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:19:58,947 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:19:58,948 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:19:58,949 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:20:02,505 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:20:02,507 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:20:02,507 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:20:02,509 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:20:04,391 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:20:05,138 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:20:05,266 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:20:05,268 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:20:05,268 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:20:05,270 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:20:05,548 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:20:05,550 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:20:05,550 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:20:05,552 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:20:09,815 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:20:09,816 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:20:10,104 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:20:10,152 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:20:11,364 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 21:20:15,174 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:20:15,366 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:20:18,102 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:20:18,102 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2256_f708e90e7fd5277c4fe2.png with policy now 2023-07-11 21:20:18,112 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:20:18,113 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2256_e517c86d4a854fa8f3b6.png with policy now 2023-07-11 21:20:18,125 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:20:18,125 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2256_6feffa88023b5b42629e.png with policy now 2023-07-11 21:20:18,137 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:20:18,137 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2256_6030757c8800e8a83428.png with policy now 2023-07-11 21:20:18,172 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:20:18,173 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:20:18,173 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:20:18,175 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:20:18,419 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/bvme8k4d-media/images/validation_2256_f708e90e7fd5277c4fe2.png 2023-07-11 21:20:18,439 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/srqlgvhj-media/images/validation_2256_6feffa88023b5b42629e.png 2023-07-11 21:20:18,447 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/tpzdgin0-media/images/validation_2256_6030757c8800e8a83428.png 2023-07-11 21:20:18,447 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/9y6j0zti-media/images/validation_2256_e517c86d4a854fa8f3b6.png 2023-07-11 21:20:20,480 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:20:20,481 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:20:21,838 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:20:21,839 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:20:21,839 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2256_f708e90e7fd5277c4fe2.png 2023-07-11 21:20:21,840 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2256_e517c86d4a854fa8f3b6.png 2023-07-11 21:20:21,840 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2256_6feffa88023b5b42629e.png 2023-07-11 21:20:21,841 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2256_6030757c8800e8a83428.png 2023-07-11 21:20:21,841 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 21:20:22,170 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/hlkaldpi-media/images/validation_2256_f708e90e7fd5277c4fe2.png 2023-07-11 21:20:22,185 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/xry4no3b-media/images/validation_2256_e517c86d4a854fa8f3b6.png 2023-07-11 21:20:22,195 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/y0rno00v-media/images/validation_2256_6feffa88023b5b42629e.png 2023-07-11 21:20:22,286 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/te3y0b9r-media/images/validation_2256_6030757c8800e8a83428.png 2023-07-11 21:20:22,287 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:20:25,622 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:20:25,622 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:20:25,745 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:20:25,877 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:20:25,879 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:20:25,879 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:20:25,881 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:20:29,562 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:20:29,564 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:20:29,564 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:20:29,565 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:20:31,029 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:20:31,433 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:20:32,394 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:20:32,396 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:20:32,396 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:20:32,398 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:20:33,860 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:20:33,861 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:20:35,816 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:20:35,818 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:20:35,818 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:20:35,820 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:20:36,813 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:20:36,821 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:20:39,265 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:20:39,267 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:20:39,267 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:20:39,269 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:20:41,367 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 21:20:41,902 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:20:42,062 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:20:42,662 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:20:42,664 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:20:42,664 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:20:42,665 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:20:42,945 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:20:42,946 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:20:42,947 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:20:42,948 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:20:45,876 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:20:45,877 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:20:47,143 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:20:47,587 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:20:52,179 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:20:52,588 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:20:55,626 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:20:55,627 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2264_f57b404ae171a2afb73d.png with policy now 2023-07-11 21:20:55,641 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:20:55,641 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2264_ab601295b277ed2d8585.png with policy now 2023-07-11 21:20:55,651 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:20:55,651 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2264_d0f444003a54ce1ba603.png with policy now 2023-07-11 21:20:55,654 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:20:55,654 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2264_6107b10540fd61346e76.png with policy now 2023-07-11 21:20:55,695 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:20:55,696 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:20:55,697 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:20:55,702 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:20:55,958 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/vzvretmj-media/images/validation_2264_f57b404ae171a2afb73d.png 2023-07-11 21:20:55,967 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/1ycmvguc-media/images/validation_2264_6107b10540fd61346e76.png 2023-07-11 21:20:55,973 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/vuxt7rh6-media/images/validation_2264_ab601295b277ed2d8585.png 2023-07-11 21:20:55,980 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/yvatydgd-media/images/validation_2264_d0f444003a54ce1ba603.png 2023-07-11 21:20:57,203 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:20:57,837 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:20:57,897 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:20:57,898 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2264_6107b10540fd61346e76.png 2023-07-11 21:20:57,898 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2264_d0f444003a54ce1ba603.png 2023-07-11 21:20:57,899 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2264_ab601295b277ed2d8585.png 2023-07-11 21:20:57,899 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2264_f57b404ae171a2afb73d.png 2023-07-11 21:20:57,899 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 21:20:58,235 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/rgwz44pl-media/images/validation_2264_6107b10540fd61346e76.png 2023-07-11 21:20:58,255 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/vrielnki-media/images/validation_2264_d0f444003a54ce1ba603.png 2023-07-11 21:20:58,270 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/z3x80luu-media/images/validation_2264_ab601295b277ed2d8585.png 2023-07-11 21:20:58,373 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/zbryfpf8-media/images/validation_2264_f57b404ae171a2afb73d.png 2023-07-11 21:20:59,257 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:21:02,891 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:21:02,893 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:21:02,893 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:21:02,894 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:21:02,896 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:21:03,340 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:21:05,629 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:21:05,631 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:21:05,631 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:21:05,633 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:21:08,385 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:21:08,613 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:21:09,408 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:21:09,409 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:21:09,410 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:21:09,411 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:21:09,913 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:21:09,914 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:21:11,371 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 21:21:12,815 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:21:12,817 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:21:12,817 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:21:12,819 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:21:13,820 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:21:14,251 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:21:17,004 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:21:17,006 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:21:17,007 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:21:17,008 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:21:19,554 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:21:19,745 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:21:20,961 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:21:20,962 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:21:20,963 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:21:20,965 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:21:21,243 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:21:21,245 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:21:21,246 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:21:21,247 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:21:21,934 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:21:21,935 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:21:24,828 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:21:24,858 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:21:29,829 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:21:29,892 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:21:33,901 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:21:33,902 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2272_5b58952985f4c02da378.png with policy now 2023-07-11 21:21:33,916 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:21:33,916 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2272_1e1c2cc5fe23888d9a0f.png with policy now 2023-07-11 21:21:33,927 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:21:33,927 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2272_5722a72d5ca3dd7d3435.png with policy now 2023-07-11 21:21:33,937 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:21:33,937 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2272_7bfa00fea9619c31d907.png with policy now 2023-07-11 21:21:33,958 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:21:33,959 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2272_1e1c2cc5fe23888d9a0f.png 2023-07-11 21:21:33,959 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2272_5722a72d5ca3dd7d3435.png 2023-07-11 21:21:33,960 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2272_5b58952985f4c02da378.png 2023-07-11 21:21:33,961 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2272_7bfa00fea9619c31d907.png 2023-07-11 21:21:33,961 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 21:21:33,975 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:21:33,976 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:21:33,977 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:21:33,982 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:21:34,215 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/2oqdv9df-media/images/validation_2272_5b58952985f4c02da378.png 2023-07-11 21:21:34,232 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/9dfb7uqx-media/images/validation_2272_1e1c2cc5fe23888d9a0f.png 2023-07-11 21:21:34,262 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ncysnn3h-media/images/validation_2272_5722a72d5ca3dd7d3435.png 2023-07-11 21:21:34,315 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ar13vx63-media/images/validation_2272_7bfa00fea9619c31d907.png 2023-07-11 21:21:34,772 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/naze5dgj-media/images/validation_2272_7bfa00fea9619c31d907.png 2023-07-11 21:21:34,825 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/wxm3z538-media/images/validation_2272_5b58952985f4c02da378.png 2023-07-11 21:21:34,983 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:21:34,998 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:21:35,162 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/sfidmpi2-media/images/validation_2272_1e1c2cc5fe23888d9a0f.png 2023-07-11 21:21:35,342 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/b5laphum-media/images/validation_2272_5722a72d5ca3dd7d3435.png 2023-07-11 21:21:37,703 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:21:40,053 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:21:40,063 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:21:41,085 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:21:41,087 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:21:41,087 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:21:41,089 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:21:41,372 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 21:21:45,278 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:21:45,442 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:21:45,503 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:21:45,505 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:21:45,505 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:21:45,507 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:21:45,978 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:21:45,979 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:21:48,260 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:21:48,262 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:21:48,262 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:21:48,264 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:21:50,444 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:21:50,853 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:21:51,690 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:21:51,692 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:21:51,692 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:21:51,694 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:21:54,009 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:21:54,011 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:21:54,011 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:21:54,013 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:21:55,650 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:21:56,333 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:21:57,124 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:21:57,125 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:21:57,126 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:21:57,128 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:21:57,406 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:21:57,408 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:21:57,408 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:21:57,410 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:21:57,996 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:21:57,997 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:22:00,666 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:22:02,017 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:22:05,697 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:22:07,018 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:22:10,015 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:22:10,055 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:22:10,055 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2280_3dc11ed55953ee19931f.png with policy now 2023-07-11 21:22:10,070 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:22:10,070 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2280_9613c1b3cfb26096117f.png with policy now 2023-07-11 21:22:10,081 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:22:10,081 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2280_ea5a63c05b6dc2e76aad.png with policy now 2023-07-11 21:22:10,089 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:22:10,089 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2280_6d2411d6fdd40f97df6f.png with policy now 2023-07-11 21:22:10,125 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:22:10,131 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:22:10,131 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:22:10,135 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:22:10,384 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/h0cvdjcy-media/images/validation_2280_3dc11ed55953ee19931f.png 2023-07-11 21:22:10,398 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/tut8dfr4-media/images/validation_2280_9613c1b3cfb26096117f.png 2023-07-11 21:22:10,404 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ktmm4dgo-media/images/validation_2280_ea5a63c05b6dc2e76aad.png 2023-07-11 21:22:10,407 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/x6sk2khk-media/images/validation_2280_6d2411d6fdd40f97df6f.png 2023-07-11 21:22:10,745 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:22:11,375 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 21:22:12,377 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:22:14,305 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:22:16,709 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:22:17,832 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:22:17,949 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:22:17,950 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:22:17,951 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:22:17,952 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:22:22,037 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:22:22,038 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:22:22,038 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2280_3dc11ed55953ee19931f.png 2023-07-11 21:22:22,039 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2280_ea5a63c05b6dc2e76aad.png 2023-07-11 21:22:22,039 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2280_6d2411d6fdd40f97df6f.png 2023-07-11 21:22:22,040 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2280_9613c1b3cfb26096117f.png 2023-07-11 21:22:22,041 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 21:22:22,060 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:22:22,063 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:22:22,063 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:22:22,065 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:22:22,369 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:22:22,382 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/etw5rbz8-media/images/validation_2280_6d2411d6fdd40f97df6f.png 2023-07-11 21:22:22,402 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/k4a27yy3-media/images/validation_2280_ea5a63c05b6dc2e76aad.png 2023-07-11 21:22:22,408 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/3baalsv5-media/images/validation_2280_9613c1b3cfb26096117f.png 2023-07-11 21:22:22,471 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ux9dvsnh-media/images/validation_2280_3dc11ed55953ee19931f.png 2023-07-11 21:22:23,058 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:22:25,097 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:22:25,099 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:22:25,099 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:22:25,101 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:22:27,508 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:22:27,508 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:22:28,596 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:22:28,894 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:22:28,896 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:22:28,897 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:22:28,898 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:22:31,792 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:22:31,794 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:22:31,794 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:22:31,795 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:22:32,786 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:22:34,054 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:22:34,054 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:22:34,295 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:22:34,715 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:22:34,717 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:22:34,717 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:22:34,719 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:22:35,000 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:22:35,001 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:22:35,002 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:22:35,003 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:22:38,202 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:22:39,563 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:22:41,378 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 21:22:43,240 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:22:45,380 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:22:46,074 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:22:46,075 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:22:47,650 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:22:47,650 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2288_7286636f68f0a2aa4e30.png with policy now 2023-07-11 21:22:47,664 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:22:47,664 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2288_a5cceb7ef5da03ddb014.png with policy now 2023-07-11 21:22:47,673 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:22:47,674 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2288_572e255d21901e5a6046.png with policy now 2023-07-11 21:22:47,683 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:22:47,683 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2288_67a013a5a370a35a3059.png with policy now 2023-07-11 21:22:47,717 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:22:47,718 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:22:47,719 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:22:47,720 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:22:48,010 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/a6zvl5oe-media/images/validation_2288_572e255d21901e5a6046.png 2023-07-11 21:22:48,010 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/zjnewhjd-media/images/validation_2288_7286636f68f0a2aa4e30.png 2023-07-11 21:22:48,024 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/9ngvqu1a-media/images/validation_2288_a5cceb7ef5da03ddb014.png 2023-07-11 21:22:48,024 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/jrjup11z-media/images/validation_2288_67a013a5a370a35a3059.png 2023-07-11 21:22:48,267 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:22:50,675 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:22:51,587 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:22:53,326 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:22:54,614 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:22:54,616 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:22:54,617 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:22:54,618 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:22:56,041 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:22:57,607 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:22:57,609 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:22:57,610 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:22:57,611 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:22:58,098 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:22:58,098 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:22:58,099 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2288_572e255d21901e5a6046.png 2023-07-11 21:22:58,099 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2288_67a013a5a370a35a3059.png 2023-07-11 21:22:58,100 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2288_a5cceb7ef5da03ddb014.png 2023-07-11 21:22:58,101 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2288_7286636f68f0a2aa4e30.png 2023-07-11 21:22:58,101 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 21:22:58,438 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/91smy13n-media/images/validation_2288_67a013a5a370a35a3059.png 2023-07-11 21:22:58,458 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/0101ea95-media/images/validation_2288_572e255d21901e5a6046.png 2023-07-11 21:22:58,627 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/v0nqwurm-media/images/validation_2288_7286636f68f0a2aa4e30.png 2023-07-11 21:22:58,635 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ju2ead78-media/images/validation_2288_a5cceb7ef5da03ddb014.png 2023-07-11 21:22:59,410 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:23:00,855 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:23:00,856 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:23:00,856 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:23:00,858 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:23:01,813 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:23:04,424 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:23:04,426 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:23:04,426 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:23:04,428 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:23:04,692 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:23:06,921 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:23:08,112 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:23:08,114 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:23:08,114 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:23:08,116 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:23:09,946 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:23:10,117 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:23:10,117 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:23:11,315 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:23:11,317 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:23:11,317 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:23:11,319 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:23:11,381 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 21:23:11,597 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:23:11,599 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:23:11,600 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:23:11,601 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:23:12,123 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:23:15,218 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:23:17,259 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:23:20,259 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:23:22,133 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:23:22,133 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:23:22,260 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:23:24,228 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:23:24,228 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2296_e87577dcb5ac8c5c2a2c.png with policy now 2023-07-11 21:23:24,242 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:23:24,243 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2296_cb7397f7fd1ac20d272d.png with policy now 2023-07-11 21:23:24,252 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:23:24,252 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2296_47afc8d7e2eb0efcc511.png with policy now 2023-07-11 21:23:24,261 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:23:24,261 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2296_250d2d9f0b19e4298842.png with policy now 2023-07-11 21:23:24,298 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:23:24,300 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:23:24,300 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:23:24,302 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:23:24,551 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/rj4pd2xf-media/images/validation_2296_e87577dcb5ac8c5c2a2c.png 2023-07-11 21:23:24,563 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/3tl1j3bz-media/images/validation_2296_47afc8d7e2eb0efcc511.png 2023-07-11 21:23:24,565 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/26c2kxq8-media/images/validation_2296_cb7397f7fd1ac20d272d.png 2023-07-11 21:23:24,588 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/am8a7n0z-media/images/validation_2296_250d2d9f0b19e4298842.png 2023-07-11 21:23:25,284 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:23:27,703 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:23:28,805 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:23:30,702 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:23:31,789 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:23:31,791 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:23:31,791 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:23:31,793 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:23:32,794 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:23:34,157 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:23:34,157 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:23:34,158 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2296_47afc8d7e2eb0efcc511.png 2023-07-11 21:23:34,158 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2296_e87577dcb5ac8c5c2a2c.png 2023-07-11 21:23:34,159 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2296_cb7397f7fd1ac20d272d.png 2023-07-11 21:23:34,160 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2296_250d2d9f0b19e4298842.png 2023-07-11 21:23:34,160 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 21:23:34,488 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/838sbvgd-media/images/validation_2296_e87577dcb5ac8c5c2a2c.png 2023-07-11 21:23:34,517 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/p73rbrqr-media/images/validation_2296_47afc8d7e2eb0efcc511.png 2023-07-11 21:23:34,675 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ggknzxoc-media/images/validation_2296_250d2d9f0b19e4298842.png 2023-07-11 21:23:34,795 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/polul8xl-media/images/validation_2296_cb7397f7fd1ac20d272d.png 2023-07-11 21:23:35,186 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:23:35,188 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:23:35,188 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:23:35,190 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:23:36,177 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:23:37,884 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:23:39,063 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:23:39,065 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:23:39,065 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:23:39,067 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:23:41,383 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 21:23:42,029 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:23:42,029 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:23:43,387 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:23:43,391 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:23:43,391 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:23:43,392 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:23:43,394 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:23:46,173 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:23:46,174 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:23:46,307 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:23:46,309 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:23:46,310 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:23:46,311 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:23:47,267 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:23:49,192 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:23:49,195 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:23:49,195 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:23:49,196 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:23:49,198 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:23:49,476 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:23:49,478 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:23:49,479 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:23:49,480 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:23:52,287 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:23:55,036 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:23:57,308 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:23:58,190 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:23:58,190 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:24:00,037 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:24:02,049 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:24:02,050 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2304_4cd5bd883752c4c9c8c0.png with policy now 2023-07-11 21:24:02,064 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:24:02,065 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2304_e90e75900878a3127495.png with policy now 2023-07-11 21:24:02,074 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:24:02,074 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2304_5c183443d040ea97fa1b.png with policy now 2023-07-11 21:24:02,083 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:24:02,084 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2304_2e6facfca2619f0d7cb5.png with policy now 2023-07-11 21:24:02,121 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:24:02,122 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:24:02,123 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:24:02,126 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:24:02,349 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:24:02,383 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/5rweu2eo-media/images/validation_2304_5c183443d040ea97fa1b.png 2023-07-11 21:24:02,433 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/6uhp952y-media/images/validation_2304_e90e75900878a3127495.png 2023-07-11 21:24:02,435 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/3sehqtzs-media/images/validation_2304_4cd5bd883752c4c9c8c0.png 2023-07-11 21:24:02,440 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ec7e7ywk-media/images/validation_2304_2e6facfca2619f0d7cb5.png 2023-07-11 21:24:05,232 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:24:05,233 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:24:07,892 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:24:08,444 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:24:08,446 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:24:08,447 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:24:08,448 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:24:10,213 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:24:10,213 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:24:10,214 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2304_2e6facfca2619f0d7cb5.png 2023-07-11 21:24:10,214 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2304_e90e75900878a3127495.png 2023-07-11 21:24:10,215 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2304_4cd5bd883752c4c9c8c0.png 2023-07-11 21:24:10,215 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2304_5c183443d040ea97fa1b.png 2023-07-11 21:24:10,216 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 21:24:10,238 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:24:10,570 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/9esbsmjo-media/images/validation_2304_e90e75900878a3127495.png 2023-07-11 21:24:10,572 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/2k6pom49-media/images/validation_2304_2e6facfca2619f0d7cb5.png 2023-07-11 21:24:10,573 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/p2xxj0zm-media/images/validation_2304_4cd5bd883752c4c9c8c0.png 2023-07-11 21:24:10,691 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/r04j8zxz-media/images/validation_2304_5c183443d040ea97fa1b.png 2023-07-11 21:24:11,385 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 21:24:12,464 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:24:12,466 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:24:12,466 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:24:12,468 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:24:13,264 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:24:13,265 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:24:15,841 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:24:15,843 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:24:15,843 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:24:15,844 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:24:15,845 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:24:18,265 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:24:18,952 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:24:18,954 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:24:18,954 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:24:18,956 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:24:21,230 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:24:22,138 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:24:22,140 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:24:22,140 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:24:22,142 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:24:22,234 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:24:22,235 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:24:23,645 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:24:24,801 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:24:24,803 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:24:24,803 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:24:24,805 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:24:25,085 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:24:25,087 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:24:25,088 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:24:25,089 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:24:26,709 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:24:28,673 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:24:31,710 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:24:33,723 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:24:34,255 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:24:34,255 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:24:36,711 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:24:37,770 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:24:37,770 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2312_9c89f883bf5fcc238280.png with policy now 2023-07-11 21:24:37,780 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:24:37,781 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2312_21c7c0440b0b11b2d491.png with policy now 2023-07-11 21:24:37,790 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:24:37,790 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2312_e9126f14486264f97c5b.png with policy now 2023-07-11 21:24:37,798 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:24:37,798 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2312_52029317168919eb0da8.png with policy now 2023-07-11 21:24:37,839 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:24:37,843 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:24:37,843 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:24:37,851 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:24:38,100 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ndeftomj-media/images/validation_2312_9c89f883bf5fcc238280.png 2023-07-11 21:24:38,118 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ndv4wq3r-media/images/validation_2312_21c7c0440b0b11b2d491.png 2023-07-11 21:24:38,123 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/avn2rjgq-media/images/validation_2312_e9126f14486264f97c5b.png 2023-07-11 21:24:38,144 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/n4npxxor-media/images/validation_2312_52029317168919eb0da8.png 2023-07-11 21:24:38,747 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:24:41,388 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 21:24:41,553 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:24:42,520 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:24:44,425 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:24:44,432 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:24:44,435 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:24:44,435 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:24:44,437 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:24:46,278 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:24:46,278 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:24:46,279 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2312_21c7c0440b0b11b2d491.png 2023-07-11 21:24:46,279 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2312_9c89f883bf5fcc238280.png 2023-07-11 21:24:46,280 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2312_e9126f14486264f97c5b.png 2023-07-11 21:24:46,280 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2312_52029317168919eb0da8.png 2023-07-11 21:24:46,280 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 21:24:46,656 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/q954exyy-media/images/validation_2312_21c7c0440b0b11b2d491.png 2023-07-11 21:24:46,818 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/xznf5h7a-media/images/validation_2312_52029317168919eb0da8.png 2023-07-11 21:24:46,836 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/9q8rxugv-media/images/validation_2312_9c89f883bf5fcc238280.png 2023-07-11 21:24:46,861 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/htrupkiy-media/images/validation_2312_e9126f14486264f97c5b.png 2023-07-11 21:24:47,545 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:24:47,547 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:24:47,548 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:24:47,548 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:24:47,550 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:24:50,202 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:24:51,176 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:24:51,178 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:24:51,178 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:24:51,180 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:24:52,855 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:24:54,190 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:24:54,192 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:24:54,192 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:24:54,194 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:24:55,891 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:24:58,006 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:24:58,008 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:24:58,008 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:24:58,009 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:24:58,010 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:24:58,299 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:24:58,299 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:25:01,025 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:25:01,781 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:25:01,783 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:25:01,783 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:25:01,785 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:25:02,063 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:25:02,065 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:25:02,065 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:25:02,067 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:25:03,714 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:25:06,074 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:25:08,715 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:25:10,319 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:25:10,320 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:25:11,097 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:25:11,390 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 21:25:14,392 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:25:14,738 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:25:14,738 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2320_39c700c0233bf9ec1887.png with policy now 2023-07-11 21:25:14,748 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:25:14,748 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2320_68a147e55429db2d8cd5.png with policy now 2023-07-11 21:25:14,761 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:25:14,761 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2320_3e0ad5d834a85111111c.png with policy now 2023-07-11 21:25:14,771 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:25:14,771 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2320_84ca29059a3b9b5905fc.png with policy now 2023-07-11 21:25:14,805 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:25:14,806 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:25:14,808 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:25:14,810 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:25:15,089 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/mrno831l-media/images/validation_2320_68a147e55429db2d8cd5.png 2023-07-11 21:25:15,089 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/8r46m43q-media/images/validation_2320_3e0ad5d834a85111111c.png 2023-07-11 21:25:15,099 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/jwrs64li-media/images/validation_2320_84ca29059a3b9b5905fc.png 2023-07-11 21:25:15,103 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/15pghsgr-media/images/validation_2320_39c700c0233bf9ec1887.png 2023-07-11 21:25:16,186 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:25:19,482 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:25:19,483 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:25:21,375 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:25:22,343 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:25:22,343 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:25:22,344 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2320_68a147e55429db2d8cd5.png 2023-07-11 21:25:22,344 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2320_84ca29059a3b9b5905fc.png 2023-07-11 21:25:22,345 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2320_39c700c0233bf9ec1887.png 2023-07-11 21:25:22,345 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2320_3e0ad5d834a85111111c.png 2023-07-11 21:25:22,345 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 21:25:22,724 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/asny7cqt-media/images/validation_2320_39c700c0233bf9ec1887.png 2023-07-11 21:25:22,748 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/6f4s7r04-media/images/validation_2320_84ca29059a3b9b5905fc.png 2023-07-11 21:25:22,753 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/yn2s9aw1-media/images/validation_2320_3e0ad5d834a85111111c.png 2023-07-11 21:25:22,821 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/3yio43u2-media/images/validation_2320_68a147e55429db2d8cd5.png 2023-07-11 21:25:23,159 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:25:23,161 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:25:23,161 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:25:23,163 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:25:25,157 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:25:26,024 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:25:26,026 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:25:26,026 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:25:26,028 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:25:27,035 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:25:29,341 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:25:29,343 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:25:29,343 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:25:29,353 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:25:30,354 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:25:32,360 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:25:32,835 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:25:32,836 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:25:32,837 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:25:32,838 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:25:34,364 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:25:34,364 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:25:35,618 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:25:36,504 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:25:36,506 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:25:36,507 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:25:36,508 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:25:37,376 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:25:39,250 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:25:39,252 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:25:39,252 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:25:39,254 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:25:39,534 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:25:39,536 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:25:39,536 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:25:39,538 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:25:41,127 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:25:41,393 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 21:25:42,442 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:25:46,380 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:25:46,381 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:25:46,394 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:25:47,486 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:25:51,395 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:25:52,172 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:25:52,172 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2328_0dafda20d7536a74a759.png with policy now 2023-07-11 21:25:52,186 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:25:52,186 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2328_80c49839f74d15e546e2.png with policy now 2023-07-11 21:25:52,196 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:25:52,197 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2328_26e2b28b1b2c040caf0a.png with policy now 2023-07-11 21:25:52,205 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:25:52,205 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2328_62f7b60d0949855bbdda.png with policy now 2023-07-11 21:25:52,239 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:25:52,245 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:25:52,245 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:25:52,248 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:25:52,503 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/9eag6oy9-media/images/validation_2328_0dafda20d7536a74a759.png 2023-07-11 21:25:52,503 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/u79ijmaz-media/images/validation_2328_62f7b60d0949855bbdda.png 2023-07-11 21:25:52,510 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:25:52,520 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/l2zu9ino-media/images/validation_2328_80c49839f74d15e546e2.png 2023-07-11 21:25:52,555 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/5lhbls9n-media/images/validation_2328_26e2b28b1b2c040caf0a.png 2023-07-11 21:25:56,045 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:25:56,706 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:25:57,586 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:25:58,405 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:25:58,406 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:25:58,406 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2328_26e2b28b1b2c040caf0a.png 2023-07-11 21:25:58,407 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2328_0dafda20d7536a74a759.png 2023-07-11 21:25:58,407 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2328_62f7b60d0949855bbdda.png 2023-07-11 21:25:58,408 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2328_80c49839f74d15e546e2.png 2023-07-11 21:25:58,408 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 21:25:58,749 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/0ej5itzq-media/images/validation_2328_62f7b60d0949855bbdda.png 2023-07-11 21:25:58,757 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/a5544r69-media/images/validation_2328_26e2b28b1b2c040caf0a.png 2023-07-11 21:25:58,762 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/jvjcdlqd-media/images/validation_2328_0dafda20d7536a74a759.png 2023-07-11 21:25:58,772 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/idmzx77i-media/images/validation_2328_80c49839f74d15e546e2.png 2023-07-11 21:26:00,193 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:26:00,195 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:26:00,195 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:26:00,197 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:26:02,228 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:26:02,847 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:26:02,849 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:26:02,849 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:26:02,851 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:26:03,864 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:26:06,136 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:26:06,138 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:26:06,139 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:26:06,140 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:26:07,640 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:26:09,012 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:26:09,082 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:26:09,084 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:26:09,084 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:26:09,086 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:26:10,424 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:26:10,424 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:26:11,396 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 21:26:11,609 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:26:11,611 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:26:11,611 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:26:11,612 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:26:13,299 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:26:14,483 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:26:15,409 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:26:15,411 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:26:15,412 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:26:15,413 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:26:15,694 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:26:15,696 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:26:15,696 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:26:15,698 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:26:19,252 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:26:19,513 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:26:22,444 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:26:22,445 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:26:24,253 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:26:24,549 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:26:28,288 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:26:28,288 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2336_ae700d32677535960099.png with policy now 2023-07-11 21:26:28,299 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:26:28,300 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2336_301a887d407c4ac4a9e8.png with policy now 2023-07-11 21:26:28,313 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:26:28,313 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2336_a5b295c1cf8890d267c2.png with policy now 2023-07-11 21:26:28,322 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:26:28,322 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2336_354d7f9111304540102a.png with policy now 2023-07-11 21:26:28,358 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:26:28,359 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:26:28,359 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:26:28,361 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:26:28,603 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/fbwcffid-media/images/validation_2336_ae700d32677535960099.png 2023-07-11 21:26:28,627 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/0bodh1jc-media/images/validation_2336_301a887d407c4ac4a9e8.png 2023-07-11 21:26:28,634 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/j97l56y5-media/images/validation_2336_a5b295c1cf8890d267c2.png 2023-07-11 21:26:28,638 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/vgac4jxp-media/images/validation_2336_354d7f9111304540102a.png 2023-07-11 21:26:29,361 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:26:29,573 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:26:32,206 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:26:34,468 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:26:34,468 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:26:34,469 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2336_301a887d407c4ac4a9e8.png 2023-07-11 21:26:34,469 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2336_ae700d32677535960099.png 2023-07-11 21:26:34,470 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2336_a5b295c1cf8890d267c2.png 2023-07-11 21:26:34,470 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2336_354d7f9111304540102a.png 2023-07-11 21:26:34,471 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 21:26:34,704 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:26:34,840 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ctpvjphi-media/images/validation_2336_301a887d407c4ac4a9e8.png 2023-07-11 21:26:34,864 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/kxsj81fg-media/images/validation_2336_a5b295c1cf8890d267c2.png 2023-07-11 21:26:35,009 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/etkdwz3e-media/images/validation_2336_ae700d32677535960099.png 2023-07-11 21:26:35,049 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/h70cvupp-media/images/validation_2336_354d7f9111304540102a.png 2023-07-11 21:26:35,531 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:26:35,660 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:26:35,662 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:26:35,662 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:26:35,663 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:26:39,010 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:26:39,012 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:26:39,013 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:26:39,014 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:26:40,015 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:26:40,882 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:26:41,397 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 21:26:42,399 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:26:42,401 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:26:42,401 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:26:42,403 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:26:45,145 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:26:45,858 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:26:45,860 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:26:45,860 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:26:45,862 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:26:46,074 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:26:46,488 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:26:46,488 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:26:48,996 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:26:48,997 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:26:48,998 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:26:48,999 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:26:50,480 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:26:51,075 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:26:51,469 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:26:51,471 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:26:51,471 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:26:51,473 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:26:51,751 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:26:51,753 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:26:51,753 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:26:51,754 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:26:56,101 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:26:56,342 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:26:58,543 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:26:58,543 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:27:01,139 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:27:01,343 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:27:04,383 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:27:04,383 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2344_d627b8b33c228c221574.png with policy now 2023-07-11 21:27:04,398 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:27:04,398 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2344_e060b93c7cc983117765.png with policy now 2023-07-11 21:27:04,406 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:27:04,406 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2344_f26688d7e9d47ebeba3b.png with policy now 2023-07-11 21:27:04,416 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:27:04,417 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2344_b0129eec4a00fb5d2469.png with policy now 2023-07-11 21:27:04,451 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:27:04,454 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:27:04,454 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:27:04,456 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:27:04,691 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/99w2vpgd-media/images/validation_2344_d627b8b33c228c221574.png 2023-07-11 21:27:04,722 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/d8m8jjtb-media/images/validation_2344_b0129eec4a00fb5d2469.png 2023-07-11 21:27:04,725 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/zbc7xptc-media/images/validation_2344_f26688d7e9d47ebeba3b.png 2023-07-11 21:27:04,833 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/hzl06h6a-media/images/validation_2344_e060b93c7cc983117765.png 2023-07-11 21:27:06,154 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:27:06,606 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:27:08,245 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:27:10,566 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:27:10,567 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:27:10,567 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2344_b0129eec4a00fb5d2469.png 2023-07-11 21:27:10,568 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2344_f26688d7e9d47ebeba3b.png 2023-07-11 21:27:10,568 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2344_d627b8b33c228c221574.png 2023-07-11 21:27:10,569 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2344_e060b93c7cc983117765.png 2023-07-11 21:27:10,569 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 21:27:10,965 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/9qrkvrwq-media/images/validation_2344_b0129eec4a00fb5d2469.png 2023-07-11 21:27:10,982 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/opiarlch-media/images/validation_2344_d627b8b33c228c221574.png 2023-07-11 21:27:10,997 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/zm71g25n-media/images/validation_2344_f26688d7e9d47ebeba3b.png 2023-07-11 21:27:11,100 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/zr4qeoac-media/images/validation_2344_e060b93c7cc983117765.png 2023-07-11 21:27:11,400 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 21:27:11,671 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:27:11,671 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:27:11,797 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:27:11,799 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:27:11,799 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:27:11,799 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:27:11,801 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:27:15,042 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:27:15,044 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:27:15,044 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:27:15,046 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:27:17,748 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:27:18,179 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:27:19,532 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:27:19,534 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:27:19,534 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:27:19,536 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:27:22,587 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:27:22,588 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:27:22,853 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:27:22,854 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:27:22,855 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:27:22,855 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:27:22,857 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:27:23,897 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:27:26,172 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:27:26,174 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:27:26,175 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:27:26,176 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:27:28,635 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:27:29,525 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:27:29,532 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:27:29,533 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:27:29,534 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:27:29,535 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:27:29,814 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:27:29,815 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:27:29,816 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:27:29,817 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:27:34,383 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:27:34,541 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:27:34,608 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:27:34,609 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:27:39,384 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:27:39,558 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:27:41,403 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 21:27:42,419 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:27:42,419 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2352_ce593cd883ca40dee3d7.png with policy now 2023-07-11 21:27:42,429 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:27:42,429 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2352_e9cbdb785c02cd01bf26.png with policy now 2023-07-11 21:27:42,438 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:27:42,438 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2352_c61b0d4f5e5ee8adb21f.png with policy now 2023-07-11 21:27:42,447 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:27:42,447 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2352_c77b9bc3c6982cc22106.png with policy now 2023-07-11 21:27:42,488 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:27:42,490 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:27:42,490 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:27:42,496 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:27:42,749 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/409wkr7k-media/images/validation_2352_e9cbdb785c02cd01bf26.png 2023-07-11 21:27:42,751 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/y45ppq1j-media/images/validation_2352_ce593cd883ca40dee3d7.png 2023-07-11 21:27:42,751 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/l6fw318t-media/images/validation_2352_c77b9bc3c6982cc22106.png 2023-07-11 21:27:42,812 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/jvz5rwge-media/images/validation_2352_c61b0d4f5e5ee8adb21f.png 2023-07-11 21:27:44,698 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:27:44,784 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:27:45,804 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:27:46,631 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:27:46,632 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:27:46,632 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2352_ce593cd883ca40dee3d7.png 2023-07-11 21:27:46,633 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2352_c77b9bc3c6982cc22106.png 2023-07-11 21:27:46,633 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2352_c61b0d4f5e5ee8adb21f.png 2023-07-11 21:27:46,634 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2352_e9cbdb785c02cd01bf26.png 2023-07-11 21:27:46,634 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 21:27:46,987 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/gzegp1yt-media/images/validation_2352_c77b9bc3c6982cc22106.png 2023-07-11 21:27:46,992 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/8vw79ufs-media/images/validation_2352_e9cbdb785c02cd01bf26.png 2023-07-11 21:27:47,019 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/o7v3aao3-media/images/validation_2352_c61b0d4f5e5ee8adb21f.png 2023-07-11 21:27:47,154 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/m6hr4sc8-media/images/validation_2352_ce593cd883ca40dee3d7.png 2023-07-11 21:27:49,157 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:27:49,159 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:27:49,160 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:27:49,161 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:27:50,029 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:27:50,109 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:27:53,516 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:27:53,518 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:27:53,518 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:27:53,520 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:27:55,328 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:27:55,434 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:27:57,195 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:27:57,197 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:27:57,198 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:27:57,199 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:27:58,653 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:27:58,654 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:27:59,926 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:27:59,928 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:27:59,928 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:27:59,930 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:28:00,640 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:28:00,758 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:28:03,419 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:28:03,421 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:28:03,422 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:28:03,423 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:28:05,876 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:28:06,294 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:28:06,921 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:28:06,923 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:28:06,924 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:28:06,925 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:28:07,203 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:28:07,205 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:28:07,205 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:28:07,207 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:28:10,675 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:28:10,675 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:28:10,983 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:28:11,321 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:28:11,406 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 21:28:16,367 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:28:16,408 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:28:20,047 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:28:20,047 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2360_4d37f738f3ac19d459df.png with policy now 2023-07-11 21:28:20,061 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:28:20,062 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2360_0a11cdad43876d67f6b1.png with policy now 2023-07-11 21:28:20,071 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:28:20,071 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2360_945a8790f028d3a3a4ac.png with policy now 2023-07-11 21:28:20,080 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:28:20,080 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2360_9df37ef1c70edfc470f2.png with policy now 2023-07-11 21:28:20,117 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:28:20,120 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:28:20,120 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:28:20,124 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:28:20,383 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/g5ekgw2u-media/images/validation_2360_945a8790f028d3a3a4ac.png 2023-07-11 21:28:20,385 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/x9xvplpm-media/images/validation_2360_0a11cdad43876d67f6b1.png 2023-07-11 21:28:20,401 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/fidj5rc7-media/images/validation_2360_9df37ef1c70edfc470f2.png 2023-07-11 21:28:20,466 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/908tm0ou-media/images/validation_2360_4d37f738f3ac19d459df.png 2023-07-11 21:28:21,414 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:28:21,729 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:28:22,698 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:28:22,698 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:28:22,699 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2360_4d37f738f3ac19d459df.png 2023-07-11 21:28:22,699 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2360_0a11cdad43876d67f6b1.png 2023-07-11 21:28:22,700 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2360_9df37ef1c70edfc470f2.png 2023-07-11 21:28:22,700 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2360_945a8790f028d3a3a4ac.png 2023-07-11 21:28:22,701 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 21:28:23,052 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/22sa5uko-media/images/validation_2360_9df37ef1c70edfc470f2.png 2023-07-11 21:28:23,066 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/sugyd20t-media/images/validation_2360_0a11cdad43876d67f6b1.png 2023-07-11 21:28:23,070 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/npk9d00c-media/images/validation_2360_945a8790f028d3a3a4ac.png 2023-07-11 21:28:23,203 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/pjmftbnm-media/images/validation_2360_4d37f738f3ac19d459df.png 2023-07-11 21:28:23,479 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:28:26,464 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:28:26,466 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:28:26,466 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:28:26,468 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:28:26,511 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:28:27,363 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:28:30,000 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:28:30,002 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:28:30,002 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:28:30,004 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:28:31,872 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:28:32,908 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:28:32,909 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:28:32,909 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:28:32,910 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:28:32,912 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:28:34,717 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:28:34,718 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:28:36,883 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:28:36,884 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:28:36,885 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:28:36,886 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:28:37,644 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:28:38,768 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:28:40,540 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:28:40,542 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:28:40,542 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:28:40,544 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:28:41,408 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 21:28:43,514 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:28:44,246 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:28:44,298 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:28:44,300 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:28:44,300 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:28:44,388 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:28:44,580 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:28:44,582 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:28:44,582 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:28:44,584 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:28:46,739 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:28:46,739 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:28:48,678 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:28:50,115 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:28:53,707 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:28:55,116 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:28:57,161 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:28:57,161 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2368_0b60b02ca6831735feee.png with policy now 2023-07-11 21:28:57,172 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:28:57,173 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2368_e455e3cf2c345fdc0326.png with policy now 2023-07-11 21:28:57,185 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:28:57,185 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2368_342e28a8b292c6288351.png with policy now 2023-07-11 21:28:57,195 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:28:57,195 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2368_d3688889a7a8ef78609d.png with policy now 2023-07-11 21:28:57,232 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:28:57,233 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:28:57,233 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:28:57,236 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:28:57,461 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/w2kdbcd7-media/images/validation_2368_0b60b02ca6831735feee.png 2023-07-11 21:28:57,502 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/etnwurwh-media/images/validation_2368_d3688889a7a8ef78609d.png 2023-07-11 21:28:57,503 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/3r8pw7v1-media/images/validation_2368_e455e3cf2c345fdc0326.png 2023-07-11 21:28:57,511 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/crgo9pgd-media/images/validation_2368_342e28a8b292c6288351.png 2023-07-11 21:28:58,729 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:28:58,764 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:28:58,765 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2368_e455e3cf2c345fdc0326.png 2023-07-11 21:28:58,765 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2368_342e28a8b292c6288351.png 2023-07-11 21:28:58,766 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2368_0b60b02ca6831735feee.png 2023-07-11 21:28:58,766 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2368_d3688889a7a8ef78609d.png 2023-07-11 21:28:58,767 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 21:28:59,134 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/t6eaktbd-media/images/validation_2368_0b60b02ca6831735feee.png 2023-07-11 21:28:59,156 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/bdrafiev-media/images/validation_2368_342e28a8b292c6288351.png 2023-07-11 21:28:59,161 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/wyozqfh1-media/images/validation_2368_e455e3cf2c345fdc0326.png 2023-07-11 21:28:59,170 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/nc0oy3pr-media/images/validation_2368_d3688889a7a8ef78609d.png 2023-07-11 21:29:00,180 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:29:00,842 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:29:03,455 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:29:03,457 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:29:03,458 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:29:03,459 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:29:04,675 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:29:05,776 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:29:07,767 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:29:07,769 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:29:07,769 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:29:07,772 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:29:09,912 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:29:10,786 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:29:10,786 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:29:10,999 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:29:11,001 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:29:11,002 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:29:11,002 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:29:11,004 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:29:11,412 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 21:29:13,955 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:29:13,957 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:29:13,957 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:29:13,959 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:29:15,096 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:29:16,455 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:29:17,661 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:29:17,663 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:29:17,663 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:29:17,665 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:29:20,128 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:29:20,939 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:29:20,941 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:29:20,942 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:29:20,943 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:29:21,223 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:29:21,225 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:29:21,225 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:29:21,227 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:29:21,726 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:29:22,808 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:29:22,808 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:29:25,132 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:29:26,861 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:29:30,174 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:29:31,861 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:29:33,915 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:29:33,915 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2376_cc2cb6db5aeb4f4ce3e7.png with policy now 2023-07-11 21:29:33,930 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:29:33,930 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2376_bc22907973133c98cdde.png with policy now 2023-07-11 21:29:33,939 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:29:33,940 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2376_e2203a4cdfa94cd450dc.png with policy now 2023-07-11 21:29:33,948 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:29:33,948 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2376_11c406e93bca8b9b3bca.png with policy now 2023-07-11 21:29:33,984 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:29:33,986 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:29:33,986 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:29:33,988 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:29:34,247 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/4dkv5coy-media/images/validation_2376_cc2cb6db5aeb4f4ce3e7.png 2023-07-11 21:29:34,249 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/s118zvp6-media/images/validation_2376_11c406e93bca8b9b3bca.png 2023-07-11 21:29:34,256 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/xv9hsoz6-media/images/validation_2376_bc22907973133c98cdde.png 2023-07-11 21:29:34,262 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/zlf1f0x0-media/images/validation_2376_e2203a4cdfa94cd450dc.png 2023-07-11 21:29:34,831 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:29:34,831 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:29:34,832 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2376_cc2cb6db5aeb4f4ce3e7.png 2023-07-11 21:29:34,832 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2376_bc22907973133c98cdde.png 2023-07-11 21:29:34,833 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2376_e2203a4cdfa94cd450dc.png 2023-07-11 21:29:34,833 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2376_11c406e93bca8b9b3bca.png 2023-07-11 21:29:34,833 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 21:29:35,202 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/d6o3misn-media/images/validation_2376_bc22907973133c98cdde.png 2023-07-11 21:29:35,206 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/gx7c6cma-media/images/validation_2376_e2203a4cdfa94cd450dc.png 2023-07-11 21:29:35,209 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/qrf6tryq-media/images/validation_2376_11c406e93bca8b9b3bca.png 2023-07-11 21:29:35,306 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:29:35,365 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/lncmxj4n-media/images/validation_2376_cc2cb6db5aeb4f4ce3e7.png 2023-07-11 21:29:37,155 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:29:37,660 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:29:39,856 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:29:39,858 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:29:39,858 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:29:39,860 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:29:40,407 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:29:41,413 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 21:29:42,415 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:29:43,632 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:29:43,634 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:29:43,634 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:29:43,636 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:29:45,519 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:29:46,704 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:29:46,706 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:29:46,706 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:29:46,708 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:29:46,852 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:29:46,852 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:29:48,176 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:29:50,141 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:29:50,143 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:29:50,143 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:29:50,194 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:29:51,135 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:29:53,439 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:29:53,971 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:29:53,972 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:29:53,973 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:29:53,975 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:29:56,917 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:29:57,927 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:29:57,929 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:29:57,929 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:29:57,931 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:29:58,211 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:29:58,213 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:29:58,213 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:29:58,215 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:29:58,620 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:29:58,924 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:29:58,924 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:30:01,957 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:30:03,800 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:30:07,001 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:30:08,801 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:30:10,822 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:30:10,822 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2384_81c5a876097ae538156d.png with policy now 2023-07-11 21:30:10,837 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:30:10,838 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2384_34e253f269b378eb15d6.png with policy now 2023-07-11 21:30:10,847 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:30:10,848 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2384_d4b3165970b466047f71.png with policy now 2023-07-11 21:30:10,851 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:30:10,851 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2384_0b1a1c25ca7c935944d8.png with policy now 2023-07-11 21:30:10,896 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:30:10,900 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:30:10,901 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:30:10,907 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:30:10,945 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:30:10,946 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:30:10,946 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2384_0b1a1c25ca7c935944d8.png 2023-07-11 21:30:10,946 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2384_d4b3165970b466047f71.png 2023-07-11 21:30:10,947 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2384_34e253f269b378eb15d6.png 2023-07-11 21:30:10,947 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2384_81c5a876097ae538156d.png 2023-07-11 21:30:10,948 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 21:30:11,144 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/3lm72r8i-media/images/validation_2384_81c5a876097ae538156d.png 2023-07-11 21:30:11,158 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/pe011dpg-media/images/validation_2384_34e253f269b378eb15d6.png 2023-07-11 21:30:11,170 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/3luv1mmf-media/images/validation_2384_d4b3165970b466047f71.png 2023-07-11 21:30:11,178 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/6jyjmk0y-media/images/validation_2384_0b1a1c25ca7c935944d8.png 2023-07-11 21:30:11,416 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 21:30:11,525 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/a02fr620-media/images/validation_2384_34e253f269b378eb15d6.png 2023-07-11 21:30:11,685 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/440ustuh-media/images/validation_2384_81c5a876097ae538156d.png 2023-07-11 21:30:11,850 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/os6npab3-media/images/validation_2384_0b1a1c25ca7c935944d8.png 2023-07-11 21:30:12,024 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:30:12,237 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/q5vc1sg5-media/images/validation_2384_d4b3165970b466047f71.png 2023-07-11 21:30:13,931 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:30:14,674 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:30:17,289 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:30:17,843 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:30:17,845 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:30:17,845 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:30:17,846 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:30:19,569 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:30:21,447 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:30:21,449 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:30:21,449 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:30:21,451 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:30:22,564 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:30:22,963 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:30:22,964 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:30:24,677 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:30:24,679 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:30:24,679 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:30:24,680 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:30:24,681 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:30:27,599 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:30:28,396 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:30:28,398 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:30:28,398 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:30:28,400 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:30:30,206 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:30:31,370 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:30:31,372 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:30:31,372 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:30:31,374 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:30:33,066 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:30:34,835 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:30:34,837 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:30:34,837 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:30:34,839 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:30:34,984 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:30:34,984 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:30:35,119 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:30:35,121 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:30:35,122 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:30:35,123 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:30:35,577 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:30:38,248 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:30:40,713 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:30:41,418 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 21:30:43,288 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:30:46,420 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:30:47,004 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:30:47,004 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:30:47,749 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:30:47,749 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2392_ab6620fbc899727dc772.png with policy now 2023-07-11 21:30:47,763 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:30:47,763 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2392_15295da0dc5bd19625bd.png with policy now 2023-07-11 21:30:47,774 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:30:47,774 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2392_1c61e21e1cd4c686eaaf.png with policy now 2023-07-11 21:30:47,782 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:30:47,782 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2392_884c2d28486ad0ad23ae.png with policy now 2023-07-11 21:30:47,823 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:30:47,825 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:30:47,825 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:30:47,828 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:30:48,069 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/vcvz8xgn-media/images/validation_2392_15295da0dc5bd19625bd.png 2023-07-11 21:30:48,072 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ps8zy6r4-media/images/validation_2392_ab6620fbc899727dc772.png 2023-07-11 21:30:48,084 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/vb3kio53-media/images/validation_2392_884c2d28486ad0ad23ae.png 2023-07-11 21:30:48,093 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/we08qe1r-media/images/validation_2392_1c61e21e1cd4c686eaaf.png 2023-07-11 21:30:48,669 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:30:51,243 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:30:51,991 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:30:53,836 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:30:54,804 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:30:54,806 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:30:54,806 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:30:54,808 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:30:57,719 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:30:57,875 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:30:57,877 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:30:57,878 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:30:57,879 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:30:59,029 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:30:59,029 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:30:59,030 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2392_1c61e21e1cd4c686eaaf.png 2023-07-11 21:30:59,030 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2392_15295da0dc5bd19625bd.png 2023-07-11 21:30:59,031 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2392_884c2d28486ad0ad23ae.png 2023-07-11 21:30:59,031 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2392_ab6620fbc899727dc772.png 2023-07-11 21:30:59,031 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 21:30:59,382 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/abcmbytc-media/images/validation_2392_15295da0dc5bd19625bd.png 2023-07-11 21:30:59,391 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ic4haljt-media/images/validation_2392_1c61e21e1cd4c686eaaf.png 2023-07-11 21:30:59,403 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/rf11lc8r-media/images/validation_2392_ab6620fbc899727dc772.png 2023-07-11 21:30:59,403 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/t1hrp8it-media/images/validation_2392_884c2d28486ad0ad23ae.png 2023-07-11 21:30:59,553 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:30:59,554 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:31:01,489 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:31:01,491 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:31:01,491 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:31:01,493 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:31:03,532 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:31:04,568 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:31:05,336 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:31:05,338 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:31:05,338 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:31:05,340 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:31:09,334 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:31:09,385 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:31:09,387 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:31:09,388 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:31:09,389 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:31:09,979 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:31:11,050 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:31:11,051 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:31:11,420 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 21:31:12,341 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:31:12,343 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:31:12,343 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:31:12,345 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:31:12,624 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:31:12,626 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:31:12,626 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:31:12,628 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:31:15,259 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:31:15,335 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:31:20,260 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:31:20,380 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:31:24,072 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:31:24,072 INFO Thread-12 :2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:31:25,244 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:31:25,244 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2400_edf7e219837bb0398a66.png with policy now 2023-07-11 21:31:25,259 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:31:25,259 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2400_8744e2e9594e95a56a40.png with policy now 2023-07-11 21:31:25,268 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:31:25,268 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2400_689387d208c87e82a9d4.png with policy now 2023-07-11 21:31:25,269 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:31:25,271 DEBUG SenderThread:2302110 [sender.py:send():369] send: files 2023-07-11 21:31:25,277 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file media/images/validation_2400_550889b0c84a4a159425.png with policy now 2023-07-11 21:31:25,314 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: partial_history 2023-07-11 21:31:25,315 DEBUG SenderThread:2302110 [sender.py:send():369] send: history 2023-07-11 21:31:25,315 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: summary_record 2023-07-11 21:31:25,316 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:31:25,425 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:31:25,551 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/ilh24klx-media/images/validation_2400_edf7e219837bb0398a66.png 2023-07-11 21:31:25,583 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/4c56edtb-media/images/validation_2400_689387d208c87e82a9d4.png 2023-07-11 21:31:25,590 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/v51xhien-media/images/validation_2400_8744e2e9594e95a56a40.png 2023-07-11 21:31:25,609 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/q5wh8dyq-media/images/validation_2400_550889b0c84a4a159425.png 2023-07-11 21:31:30,271 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:31:30,426 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: keepalive 2023-07-11 21:31:32,808 DEBUG SenderThread:2302110 [sender.py:send():369] send: exit 2023-07-11 21:31:32,808 INFO SenderThread:2302110 [sender.py:send_exit():574] handling exit code: 1 2023-07-11 21:31:32,808 INFO SenderThread:2302110 [sender.py:send_exit():576] handling runtime: 11332 2023-07-11 21:31:32,813 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:31:32,814 INFO SenderThread:2302110 [sender.py:send_exit():582] send defer 2023-07-11 21:31:32,814 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: defer 2023-07-11 21:31:32,814 INFO HandlerThread:2302110 [handler.py:handle_request_defer():170] handle defer: 0 2023-07-11 21:31:32,815 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: defer 2023-07-11 21:31:32,815 INFO SenderThread:2302110 [sender.py:send_request_defer():598] handle sender defer: 0 2023-07-11 21:31:32,815 INFO SenderThread:2302110 [sender.py:transition_state():602] send defer: 1 2023-07-11 21:31:32,815 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: defer 2023-07-11 21:31:32,815 INFO HandlerThread:2302110 [handler.py:handle_request_defer():170] handle defer: 1 2023-07-11 21:31:32,816 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: defer 2023-07-11 21:31:32,816 INFO SenderThread:2302110 [sender.py:send_request_defer():598] handle sender defer: 1 2023-07-11 21:31:32,816 INFO SenderThread:2302110 [sender.py:transition_state():602] send defer: 2 2023-07-11 21:31:32,816 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: defer 2023-07-11 21:31:32,816 INFO HandlerThread:2302110 [handler.py:handle_request_defer():170] handle defer: 2 2023-07-11 21:31:32,816 INFO HandlerThread:2302110 [system_monitor.py:finish():190] Stopping system monitor 2023-07-11 21:31:32,817 INFO HandlerThread:2302110 [interfaces.py:finish():202] Joined cpu monitor 2023-07-11 21:31:32,817 DEBUG SystemMonitor:2302110 [system_monitor.py:_start():166] Finished system metrics aggregation loop 2023-07-11 21:31:32,817 DEBUG SystemMonitor:2302110 [system_monitor.py:_start():170] Publishing last batch of metrics 2023-07-11 21:31:32,820 INFO HandlerThread:2302110 [interfaces.py:finish():202] Joined disk monitor 2023-07-11 21:31:33,271 INFO HandlerThread:2302110 [interfaces.py:finish():202] Joined gpu monitor 2023-07-11 21:31:33,272 INFO HandlerThread:2302110 [interfaces.py:finish():202] Joined memory monitor 2023-07-11 21:31:33,272 INFO HandlerThread:2302110 [interfaces.py:finish():202] Joined network monitor 2023-07-11 21:31:33,273 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: defer 2023-07-11 21:31:33,273 INFO SenderThread:2302110 [sender.py:send_request_defer():598] handle sender defer: 2 2023-07-11 21:31:33,273 INFO SenderThread:2302110 [sender.py:transition_state():602] send defer: 3 2023-07-11 21:31:33,274 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: defer 2023-07-11 21:31:33,274 INFO HandlerThread:2302110 [handler.py:handle_request_defer():170] handle defer: 3 2023-07-11 21:31:33,275 DEBUG SenderThread:2302110 [sender.py:send():369] send: stats 2023-07-11 21:31:33,277 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: defer 2023-07-11 21:31:33,277 INFO SenderThread:2302110 [sender.py:send_request_defer():598] handle sender defer: 3 2023-07-11 21:31:33,277 INFO SenderThread:2302110 [sender.py:transition_state():602] send defer: 4 2023-07-11 21:31:33,277 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: defer 2023-07-11 21:31:33,278 INFO HandlerThread:2302110 [handler.py:handle_request_defer():170] handle defer: 4 2023-07-11 21:31:33,278 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: defer 2023-07-11 21:31:33,278 INFO SenderThread:2302110 [sender.py:send_request_defer():598] handle sender defer: 4 2023-07-11 21:31:33,278 INFO SenderThread:2302110 [sender.py:transition_state():602] send defer: 5 2023-07-11 21:31:33,278 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: defer 2023-07-11 21:31:33,278 INFO HandlerThread:2302110 [handler.py:handle_request_defer():170] handle defer: 5 2023-07-11 21:31:33,279 DEBUG SenderThread:2302110 [sender.py:send():369] send: summary 2023-07-11 21:31:33,282 INFO SenderThread:2302110 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end 2023-07-11 21:31:33,282 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: defer 2023-07-11 21:31:33,282 INFO SenderThread:2302110 [sender.py:send_request_defer():598] handle sender defer: 5 2023-07-11 21:31:33,282 INFO SenderThread:2302110 [sender.py:transition_state():602] send defer: 6 2023-07-11 21:31:33,283 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: defer 2023-07-11 21:31:33,283 INFO HandlerThread:2302110 [handler.py:handle_request_defer():170] handle defer: 6 2023-07-11 21:31:33,283 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: defer 2023-07-11 21:31:33,283 INFO SenderThread:2302110 [sender.py:send_request_defer():598] handle sender defer: 6 2023-07-11 21:31:33,284 INFO SenderThread:2302110 [sender.py:transition_state():602] send defer: 7 2023-07-11 21:31:33,284 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: status_report 2023-07-11 21:31:33,284 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: defer 2023-07-11 21:31:33,284 INFO HandlerThread:2302110 [handler.py:handle_request_defer():170] handle defer: 7 2023-07-11 21:31:33,284 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: defer 2023-07-11 21:31:33,285 INFO SenderThread:2302110 [sender.py:send_request_defer():598] handle sender defer: 7 2023-07-11 21:31:33,808 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: poll_exit 2023-07-11 21:31:34,458 INFO SenderThread:2302110 [sender.py:transition_state():602] send defer: 8 2023-07-11 21:31:34,459 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: poll_exit 2023-07-11 21:31:34,459 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: defer 2023-07-11 21:31:34,462 INFO HandlerThread:2302110 [handler.py:handle_request_defer():170] handle defer: 8 2023-07-11 21:31:34,463 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: defer 2023-07-11 21:31:34,463 INFO SenderThread:2302110 [sender.py:send_request_defer():598] handle sender defer: 8 2023-07-11 21:31:34,463 INFO SenderThread:2302110 [job_builder.py:build():232] Attempting to build job artifact 2023-07-11 21:31:34,465 INFO SenderThread:2302110 [job_builder.py:build():266] no source found 2023-07-11 21:31:34,465 INFO SenderThread:2302110 [sender.py:transition_state():602] send defer: 9 2023-07-11 21:31:34,465 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: defer 2023-07-11 21:31:34,465 INFO HandlerThread:2302110 [handler.py:handle_request_defer():170] handle defer: 9 2023-07-11 21:31:34,466 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: defer 2023-07-11 21:31:34,466 INFO SenderThread:2302110 [sender.py:send_request_defer():598] handle sender defer: 9 2023-07-11 21:31:34,466 INFO SenderThread:2302110 [dir_watcher.py:finish():359] shutting down directory watcher 2023-07-11 21:31:34,809 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: poll_exit 2023-07-11 21:31:35,099 INFO SenderThread:2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:31:35,099 INFO SenderThread:2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:31:35,099 INFO SenderThread:2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2400_edf7e219837bb0398a66.png 2023-07-11 21:31:35,099 INFO SenderThread:2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2400_550889b0c84a4a159425.png 2023-07-11 21:31:35,100 INFO SenderThread:2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2400_8744e2e9594e95a56a40.png 2023-07-11 21:31:35,100 INFO SenderThread:2302110 [dir_watcher.py:_on_file_created():272] file/dir created: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2400_689387d208c87e82a9d4.png 2023-07-11 21:31:35,100 INFO SenderThread:2302110 [dir_watcher.py:_on_file_modified():289] file/dir modified: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images 2023-07-11 21:31:35,101 INFO SenderThread:2302110 [dir_watcher.py:finish():389] scan: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files 2023-07-11 21:31:35,101 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/config.yaml config.yaml 2023-07-11 21:31:35,101 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json wandb-summary.json 2023-07-11 21:31:35,101 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/requirements.txt requirements.txt 2023-07-11 21:31:35,102 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-metadata.json wandb-metadata.json 2023-07-11 21:31:35,102 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log output.log 2023-07-11 21:31:35,116 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_8_4f431957962c49867234.png media/images/validation_8_4f431957962c49867234.png 2023-07-11 21:31:35,116 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_8_43f961633a45946caf61.png media/images/validation_8_43f961633a45946caf61.png 2023-07-11 21:31:35,116 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_8_018b5a46c57a4e62ee91.png media/images/validation_8_018b5a46c57a4e62ee91.png 2023-07-11 21:31:35,116 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_8_b1af5834223b938d847b.png media/images/validation_8_b1af5834223b938d847b.png 2023-07-11 21:31:35,116 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_16_0cf3546cb22517953a42.png media/images/validation_16_0cf3546cb22517953a42.png 2023-07-11 21:31:35,117 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_16_ba6f49bf4d359df02f2b.png media/images/validation_16_ba6f49bf4d359df02f2b.png 2023-07-11 21:31:35,117 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_16_53e698e35096606e0dae.png media/images/validation_16_53e698e35096606e0dae.png 2023-07-11 21:31:35,117 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_16_e935b3df8a409038c0e2.png media/images/validation_16_e935b3df8a409038c0e2.png 2023-07-11 21:31:35,117 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_24_5a5b54a523e7e8a202d4.png media/images/validation_24_5a5b54a523e7e8a202d4.png 2023-07-11 21:31:35,117 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_24_68fac8d4cdcb87c0351c.png media/images/validation_24_68fac8d4cdcb87c0351c.png 2023-07-11 21:31:35,117 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_24_e0d04b7701f5a5d35a98.png media/images/validation_24_e0d04b7701f5a5d35a98.png 2023-07-11 21:31:35,117 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_24_0684dbe8f6eb82a2da7f.png media/images/validation_24_0684dbe8f6eb82a2da7f.png 2023-07-11 21:31:35,117 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_32_caeb7e11117ca44da4c8.png media/images/validation_32_caeb7e11117ca44da4c8.png 2023-07-11 21:31:35,117 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_32_7ece7c36041231bd9830.png media/images/validation_32_7ece7c36041231bd9830.png 2023-07-11 21:31:35,118 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_32_603e1acd8de551001b1b.png media/images/validation_32_603e1acd8de551001b1b.png 2023-07-11 21:31:35,118 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_32_d076df0f0d397a793093.png media/images/validation_32_d076df0f0d397a793093.png 2023-07-11 21:31:35,118 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_40_18d5333919a6d64e6d60.png media/images/validation_40_18d5333919a6d64e6d60.png 2023-07-11 21:31:35,118 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_40_86b79bd8a5f65f5b321b.png media/images/validation_40_86b79bd8a5f65f5b321b.png 2023-07-11 21:31:35,118 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_40_549de1ccb1807d11a4b8.png media/images/validation_40_549de1ccb1807d11a4b8.png 2023-07-11 21:31:35,118 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_40_144fef68c2f0b44f2e84.png media/images/validation_40_144fef68c2f0b44f2e84.png 2023-07-11 21:31:35,118 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_48_9c214adc0d7f41183109.png media/images/validation_48_9c214adc0d7f41183109.png 2023-07-11 21:31:35,118 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_48_d2e4462cea1aed1f2937.png media/images/validation_48_d2e4462cea1aed1f2937.png 2023-07-11 21:31:35,119 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_48_a907fc518966b66deb3c.png media/images/validation_48_a907fc518966b66deb3c.png 2023-07-11 21:31:35,119 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_48_0262af6aae483d433db7.png media/images/validation_48_0262af6aae483d433db7.png 2023-07-11 21:31:35,119 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_56_664030382ee576a5c9b2.png media/images/validation_56_664030382ee576a5c9b2.png 2023-07-11 21:31:35,119 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_56_1536eefdcfae05b945b9.png media/images/validation_56_1536eefdcfae05b945b9.png 2023-07-11 21:31:35,119 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_56_12bc42303d2e7628fd3f.png media/images/validation_56_12bc42303d2e7628fd3f.png 2023-07-11 21:31:35,119 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_56_ef4ee0a1a460a56f4751.png media/images/validation_56_ef4ee0a1a460a56f4751.png 2023-07-11 21:31:35,119 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_64_3e87611fd915ec96e580.png media/images/validation_64_3e87611fd915ec96e580.png 2023-07-11 21:31:35,119 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_64_2537a2652dff56481399.png media/images/validation_64_2537a2652dff56481399.png 2023-07-11 21:31:35,119 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_64_9bd0267d729986b95a5d.png media/images/validation_64_9bd0267d729986b95a5d.png 2023-07-11 21:31:35,120 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_64_75bb28ff057271dedef7.png media/images/validation_64_75bb28ff057271dedef7.png 2023-07-11 21:31:35,120 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_72_8f41b5c8c4403cc49f21.png media/images/validation_72_8f41b5c8c4403cc49f21.png 2023-07-11 21:31:35,120 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_72_8f7cadaa97753cdcfd66.png media/images/validation_72_8f7cadaa97753cdcfd66.png 2023-07-11 21:31:35,120 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_72_c4f043b99e28a96d715b.png media/images/validation_72_c4f043b99e28a96d715b.png 2023-07-11 21:31:35,120 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_72_9b536587af12e637b86b.png media/images/validation_72_9b536587af12e637b86b.png 2023-07-11 21:31:35,120 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_80_642de8ca4837518430ae.png media/images/validation_80_642de8ca4837518430ae.png 2023-07-11 21:31:35,120 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_80_002146d58462dcf1138f.png media/images/validation_80_002146d58462dcf1138f.png 2023-07-11 21:31:35,120 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_80_7fce802a62c03f349904.png media/images/validation_80_7fce802a62c03f349904.png 2023-07-11 21:31:35,120 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_80_4b64689fc286b148b39a.png media/images/validation_80_4b64689fc286b148b39a.png 2023-07-11 21:31:35,120 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_88_b86a7ba09e3469d620dc.png media/images/validation_88_b86a7ba09e3469d620dc.png 2023-07-11 21:31:35,121 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_88_a4422b8ecb356c655d3a.png media/images/validation_88_a4422b8ecb356c655d3a.png 2023-07-11 21:31:35,121 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_88_d1da96f2064c4210048a.png media/images/validation_88_d1da96f2064c4210048a.png 2023-07-11 21:31:35,121 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_88_88384511781630fcde4e.png media/images/validation_88_88384511781630fcde4e.png 2023-07-11 21:31:35,121 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_96_0d776d95eff027dba77c.png media/images/validation_96_0d776d95eff027dba77c.png 2023-07-11 21:31:35,121 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_96_f4754be0e0fb99c57d60.png media/images/validation_96_f4754be0e0fb99c57d60.png 2023-07-11 21:31:35,121 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_96_298b674dbb75f0e2d399.png media/images/validation_96_298b674dbb75f0e2d399.png 2023-07-11 21:31:35,121 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_96_632ab0608582a5d13e60.png media/images/validation_96_632ab0608582a5d13e60.png 2023-07-11 21:31:35,121 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_104_34b936bfe047d298afd0.png media/images/validation_104_34b936bfe047d298afd0.png 2023-07-11 21:31:35,121 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_104_a3c6437c2be1111726ba.png media/images/validation_104_a3c6437c2be1111726ba.png 2023-07-11 21:31:35,121 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_104_d7a07b12de591dfc784b.png media/images/validation_104_d7a07b12de591dfc784b.png 2023-07-11 21:31:35,121 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_104_85d0efb91c0e48ceb8a6.png media/images/validation_104_85d0efb91c0e48ceb8a6.png 2023-07-11 21:31:35,122 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_112_77f47ee5ca4647b53e91.png media/images/validation_112_77f47ee5ca4647b53e91.png 2023-07-11 21:31:35,122 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_112_58efbe7f01b6a5ce6ada.png media/images/validation_112_58efbe7f01b6a5ce6ada.png 2023-07-11 21:31:35,122 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_112_37039f7d8b61bffb22b3.png media/images/validation_112_37039f7d8b61bffb22b3.png 2023-07-11 21:31:35,122 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_112_3994094417c23d4c4a21.png media/images/validation_112_3994094417c23d4c4a21.png 2023-07-11 21:31:35,122 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_120_e11e426abf94dc089311.png media/images/validation_120_e11e426abf94dc089311.png 2023-07-11 21:31:35,122 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_120_70875a697c0992f9c209.png media/images/validation_120_70875a697c0992f9c209.png 2023-07-11 21:31:35,122 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_120_10e52d49b55c29d4c70a.png media/images/validation_120_10e52d49b55c29d4c70a.png 2023-07-11 21:31:35,122 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_120_bf2536cdbcf9021e038a.png media/images/validation_120_bf2536cdbcf9021e038a.png 2023-07-11 21:31:35,122 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_128_f12975048cb9e24609f6.png media/images/validation_128_f12975048cb9e24609f6.png 2023-07-11 21:31:35,122 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_128_c6d1e00286ea24ea39ba.png media/images/validation_128_c6d1e00286ea24ea39ba.png 2023-07-11 21:31:35,122 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_128_f4dfbac2cbc6682335d5.png media/images/validation_128_f4dfbac2cbc6682335d5.png 2023-07-11 21:31:35,122 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_128_c9fb58eadba85e11521d.png media/images/validation_128_c9fb58eadba85e11521d.png 2023-07-11 21:31:35,123 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_136_4a6f2aa08c542f4f3d3b.png media/images/validation_136_4a6f2aa08c542f4f3d3b.png 2023-07-11 21:31:35,123 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_136_e3e73b728661ef994342.png media/images/validation_136_e3e73b728661ef994342.png 2023-07-11 21:31:35,123 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_136_36bb53c164993a8cf8d8.png media/images/validation_136_36bb53c164993a8cf8d8.png 2023-07-11 21:31:35,123 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_136_3b8a8548b12f4fe561b0.png media/images/validation_136_3b8a8548b12f4fe561b0.png 2023-07-11 21:31:35,123 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_144_bf1d4866e27ceb3b9a7c.png media/images/validation_144_bf1d4866e27ceb3b9a7c.png 2023-07-11 21:31:35,123 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_144_2a98bd32e667f5d3ffa0.png media/images/validation_144_2a98bd32e667f5d3ffa0.png 2023-07-11 21:31:35,123 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_144_9571275e76adeef75d31.png media/images/validation_144_9571275e76adeef75d31.png 2023-07-11 21:31:35,123 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_144_21da72f5de55ac2c601b.png media/images/validation_144_21da72f5de55ac2c601b.png 2023-07-11 21:31:35,123 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_152_630cc9c3ad8164670f80.png media/images/validation_152_630cc9c3ad8164670f80.png 2023-07-11 21:31:35,123 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_152_eb8b309f2e7d0c0900fe.png media/images/validation_152_eb8b309f2e7d0c0900fe.png 2023-07-11 21:31:35,124 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_152_e43b38eafcadeda16c8c.png media/images/validation_152_e43b38eafcadeda16c8c.png 2023-07-11 21:31:35,124 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_152_e20fdb183cf6dd559eb8.png media/images/validation_152_e20fdb183cf6dd559eb8.png 2023-07-11 21:31:35,124 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_160_fd584a28b09e2fa88b66.png media/images/validation_160_fd584a28b09e2fa88b66.png 2023-07-11 21:31:35,124 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_160_268e9a732b517d634b0f.png media/images/validation_160_268e9a732b517d634b0f.png 2023-07-11 21:31:35,124 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_160_f9cc3e65bbd82b475a59.png media/images/validation_160_f9cc3e65bbd82b475a59.png 2023-07-11 21:31:35,124 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_160_e8dd26e2169153e2c665.png media/images/validation_160_e8dd26e2169153e2c665.png 2023-07-11 21:31:35,124 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_168_7a6f1b9b97ae1a405c3d.png media/images/validation_168_7a6f1b9b97ae1a405c3d.png 2023-07-11 21:31:35,124 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_168_770dcafceef728f19c85.png media/images/validation_168_770dcafceef728f19c85.png 2023-07-11 21:31:35,124 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_168_01bbee240219c25af928.png media/images/validation_168_01bbee240219c25af928.png 2023-07-11 21:31:35,124 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_168_fe40b1e9c3e598d742f2.png media/images/validation_168_fe40b1e9c3e598d742f2.png 2023-07-11 21:31:35,124 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_176_7c3251a3bc6842a14eaf.png media/images/validation_176_7c3251a3bc6842a14eaf.png 2023-07-11 21:31:35,124 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_176_ddb9188a2ecb3ba1a967.png media/images/validation_176_ddb9188a2ecb3ba1a967.png 2023-07-11 21:31:35,125 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_176_b57338304a381b55774f.png media/images/validation_176_b57338304a381b55774f.png 2023-07-11 21:31:35,125 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_176_77496af6afd4fcfc7802.png media/images/validation_176_77496af6afd4fcfc7802.png 2023-07-11 21:31:35,125 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_184_973f6a3c3ee07eda891b.png media/images/validation_184_973f6a3c3ee07eda891b.png 2023-07-11 21:31:35,125 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_184_e6bdd88d9dc7c75d4ca3.png media/images/validation_184_e6bdd88d9dc7c75d4ca3.png 2023-07-11 21:31:35,125 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_184_b07a08bc9ab86b402e2a.png media/images/validation_184_b07a08bc9ab86b402e2a.png 2023-07-11 21:31:35,125 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_184_7c9b69b4f13b5c0abece.png media/images/validation_184_7c9b69b4f13b5c0abece.png 2023-07-11 21:31:35,125 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_192_d1e1f9c8aa7873e56eed.png media/images/validation_192_d1e1f9c8aa7873e56eed.png 2023-07-11 21:31:35,125 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_192_63bb647b973f8f813f9e.png media/images/validation_192_63bb647b973f8f813f9e.png 2023-07-11 21:31:35,125 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_192_9846167229bd54634f68.png media/images/validation_192_9846167229bd54634f68.png 2023-07-11 21:31:35,125 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_192_5fc6b92733f1b72454a9.png media/images/validation_192_5fc6b92733f1b72454a9.png 2023-07-11 21:31:35,126 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_200_b47039a85898acafffb2.png media/images/validation_200_b47039a85898acafffb2.png 2023-07-11 21:31:35,126 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_200_3dc0cb79df01ab7c84bc.png media/images/validation_200_3dc0cb79df01ab7c84bc.png 2023-07-11 21:31:35,126 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_200_7b675596c6407752ba34.png media/images/validation_200_7b675596c6407752ba34.png 2023-07-11 21:31:35,126 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_200_96aa6aebfb42c1649c43.png media/images/validation_200_96aa6aebfb42c1649c43.png 2023-07-11 21:31:35,126 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_208_b6b72350a5e90be236b8.png media/images/validation_208_b6b72350a5e90be236b8.png 2023-07-11 21:31:35,126 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_208_b0925536eb78c8a0506f.png media/images/validation_208_b0925536eb78c8a0506f.png 2023-07-11 21:31:35,126 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_208_8f8d4515856693522ab8.png media/images/validation_208_8f8d4515856693522ab8.png 2023-07-11 21:31:35,126 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_208_75c3a361aae943fc98c0.png media/images/validation_208_75c3a361aae943fc98c0.png 2023-07-11 21:31:35,126 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_216_4a85055a3c02c45c760a.png media/images/validation_216_4a85055a3c02c45c760a.png 2023-07-11 21:31:35,126 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_216_4d5eab540def9d6b25f3.png media/images/validation_216_4d5eab540def9d6b25f3.png 2023-07-11 21:31:35,126 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_216_cfc28146873310f734c3.png media/images/validation_216_cfc28146873310f734c3.png 2023-07-11 21:31:35,126 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_216_7a3a532a14ba9a1b34c2.png media/images/validation_216_7a3a532a14ba9a1b34c2.png 2023-07-11 21:31:35,127 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_224_693eb3a2354f1b571e7c.png media/images/validation_224_693eb3a2354f1b571e7c.png 2023-07-11 21:31:35,127 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_224_301d4cf0e8bf493be61e.png media/images/validation_224_301d4cf0e8bf493be61e.png 2023-07-11 21:31:35,127 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_224_c41cb95d77436452e8b0.png media/images/validation_224_c41cb95d77436452e8b0.png 2023-07-11 21:31:35,127 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_224_e2e0e2a8e4633ed2e014.png media/images/validation_224_e2e0e2a8e4633ed2e014.png 2023-07-11 21:31:35,127 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_232_a060116adb492f4b9a59.png media/images/validation_232_a060116adb492f4b9a59.png 2023-07-11 21:31:35,127 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_232_c7a422f0918a7e63fef3.png media/images/validation_232_c7a422f0918a7e63fef3.png 2023-07-11 21:31:35,127 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_232_c57b0c82f494de77f729.png media/images/validation_232_c57b0c82f494de77f729.png 2023-07-11 21:31:35,133 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_232_713378cc3efc25bf233a.png media/images/validation_232_713378cc3efc25bf233a.png 2023-07-11 21:31:35,133 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_240_cf699d9c0706f5fbf1d1.png media/images/validation_240_cf699d9c0706f5fbf1d1.png 2023-07-11 21:31:35,133 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_240_b17212d3811e99b0c92e.png media/images/validation_240_b17212d3811e99b0c92e.png 2023-07-11 21:31:35,133 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_240_4363a80dfcc1b540ae2c.png media/images/validation_240_4363a80dfcc1b540ae2c.png 2023-07-11 21:31:35,134 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_240_131f48b2cfb4ebcfb628.png media/images/validation_240_131f48b2cfb4ebcfb628.png 2023-07-11 21:31:35,134 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_248_9e9ae0e3b6539cacbcfd.png media/images/validation_248_9e9ae0e3b6539cacbcfd.png 2023-07-11 21:31:35,134 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_248_84e81851db303d973735.png media/images/validation_248_84e81851db303d973735.png 2023-07-11 21:31:35,134 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_248_57792db981d24437505d.png media/images/validation_248_57792db981d24437505d.png 2023-07-11 21:31:35,134 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_248_981e491cb9b69cf79bc6.png media/images/validation_248_981e491cb9b69cf79bc6.png 2023-07-11 21:31:35,134 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_256_e0fd66e23b03e8fb0137.png media/images/validation_256_e0fd66e23b03e8fb0137.png 2023-07-11 21:31:35,134 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_256_003e58e75a4b35087fc3.png media/images/validation_256_003e58e75a4b35087fc3.png 2023-07-11 21:31:35,134 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_256_6e977e09b8e28e89d88b.png media/images/validation_256_6e977e09b8e28e89d88b.png 2023-07-11 21:31:35,134 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_256_398994785799033e6506.png media/images/validation_256_398994785799033e6506.png 2023-07-11 21:31:35,134 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_264_df9f8e90426ed7bffb77.png media/images/validation_264_df9f8e90426ed7bffb77.png 2023-07-11 21:31:35,134 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_264_be323385875893b62402.png media/images/validation_264_be323385875893b62402.png 2023-07-11 21:31:35,135 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_264_50b6b68312c4f0c0f919.png media/images/validation_264_50b6b68312c4f0c0f919.png 2023-07-11 21:31:35,135 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_264_0ba4a6c56127f845b572.png media/images/validation_264_0ba4a6c56127f845b572.png 2023-07-11 21:31:35,135 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_272_c2159fc2ce0f2913430b.png media/images/validation_272_c2159fc2ce0f2913430b.png 2023-07-11 21:31:35,135 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_272_cd804946c37522db1906.png media/images/validation_272_cd804946c37522db1906.png 2023-07-11 21:31:35,135 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_272_5f6299d67a2c504b2dcc.png media/images/validation_272_5f6299d67a2c504b2dcc.png 2023-07-11 21:31:35,135 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_272_2787282524c47ad5cb18.png media/images/validation_272_2787282524c47ad5cb18.png 2023-07-11 21:31:35,135 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_280_a60cb3ddac08d073d30c.png media/images/validation_280_a60cb3ddac08d073d30c.png 2023-07-11 21:31:35,136 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_280_4af7c3c5f9585a30429c.png media/images/validation_280_4af7c3c5f9585a30429c.png 2023-07-11 21:31:35,136 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_280_558ec026a5f3b0c54aab.png media/images/validation_280_558ec026a5f3b0c54aab.png 2023-07-11 21:31:35,136 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_280_c9a2c308a53c060e2f9e.png media/images/validation_280_c9a2c308a53c060e2f9e.png 2023-07-11 21:31:35,136 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_288_d322f4dde2e9d2d4c2d0.png media/images/validation_288_d322f4dde2e9d2d4c2d0.png 2023-07-11 21:31:35,136 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_288_e8cc62c8cd0593a12258.png media/images/validation_288_e8cc62c8cd0593a12258.png 2023-07-11 21:31:35,136 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_288_2a46f6447f317761e631.png media/images/validation_288_2a46f6447f317761e631.png 2023-07-11 21:31:35,136 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_288_7092f5d1e02e7ebba8fb.png media/images/validation_288_7092f5d1e02e7ebba8fb.png 2023-07-11 21:31:35,136 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_296_9e30465455fb3e14420c.png media/images/validation_296_9e30465455fb3e14420c.png 2023-07-11 21:31:35,136 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_296_ce669110321002a765f4.png media/images/validation_296_ce669110321002a765f4.png 2023-07-11 21:31:35,136 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_296_1ed73a63f24513e30257.png media/images/validation_296_1ed73a63f24513e30257.png 2023-07-11 21:31:35,137 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_296_88dc791f343412ad2af1.png media/images/validation_296_88dc791f343412ad2af1.png 2023-07-11 21:31:35,137 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_304_ce2d8ab42d2884a6a507.png media/images/validation_304_ce2d8ab42d2884a6a507.png 2023-07-11 21:31:35,137 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_304_acbea64c9e5d9aa9013b.png media/images/validation_304_acbea64c9e5d9aa9013b.png 2023-07-11 21:31:35,137 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_304_408941d3a9aad78e3825.png media/images/validation_304_408941d3a9aad78e3825.png 2023-07-11 21:31:35,137 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_304_8cb43723394c9698f98b.png media/images/validation_304_8cb43723394c9698f98b.png 2023-07-11 21:31:35,137 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_312_d97fb722dbbfe3980d7a.png media/images/validation_312_d97fb722dbbfe3980d7a.png 2023-07-11 21:31:35,137 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_312_ae0fbaa0b9e169ccfaf7.png media/images/validation_312_ae0fbaa0b9e169ccfaf7.png 2023-07-11 21:31:35,137 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_312_3f17f5fe28a713544173.png media/images/validation_312_3f17f5fe28a713544173.png 2023-07-11 21:31:35,137 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_312_8d287303a5c0f6c9b182.png media/images/validation_312_8d287303a5c0f6c9b182.png 2023-07-11 21:31:35,137 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_320_ca6b8df2ff2f328c13ab.png media/images/validation_320_ca6b8df2ff2f328c13ab.png 2023-07-11 21:31:35,137 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_320_787669ccfd6f2189c2e9.png media/images/validation_320_787669ccfd6f2189c2e9.png 2023-07-11 21:31:35,137 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_320_6fac76e86a9c94c7429f.png media/images/validation_320_6fac76e86a9c94c7429f.png 2023-07-11 21:31:35,138 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_320_59eb061cba639d911944.png media/images/validation_320_59eb061cba639d911944.png 2023-07-11 21:31:35,138 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_328_9a95e0071e58dbf2382e.png media/images/validation_328_9a95e0071e58dbf2382e.png 2023-07-11 21:31:35,138 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_328_a295737d90d34f9422e2.png media/images/validation_328_a295737d90d34f9422e2.png 2023-07-11 21:31:35,138 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_328_d261d0f406cd6d609cb4.png media/images/validation_328_d261d0f406cd6d609cb4.png 2023-07-11 21:31:35,138 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_328_5ee531180049cce5568d.png media/images/validation_328_5ee531180049cce5568d.png 2023-07-11 21:31:35,138 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_336_0ffc6176fb6d7e1c0839.png media/images/validation_336_0ffc6176fb6d7e1c0839.png 2023-07-11 21:31:35,138 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_336_d3ae5109418d504e15a5.png media/images/validation_336_d3ae5109418d504e15a5.png 2023-07-11 21:31:35,138 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_336_a39fd3fa08d829b242c5.png media/images/validation_336_a39fd3fa08d829b242c5.png 2023-07-11 21:31:35,138 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_336_db973e73f7779e6592b8.png media/images/validation_336_db973e73f7779e6592b8.png 2023-07-11 21:31:35,138 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_344_f5043cd63cbfcc5b803e.png media/images/validation_344_f5043cd63cbfcc5b803e.png 2023-07-11 21:31:35,138 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_344_f73ffb4d2e046ab554c4.png media/images/validation_344_f73ffb4d2e046ab554c4.png 2023-07-11 21:31:35,139 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_344_cea16b4401e675df5e86.png media/images/validation_344_cea16b4401e675df5e86.png 2023-07-11 21:31:35,139 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_344_6aa625d41ec15aca550a.png media/images/validation_344_6aa625d41ec15aca550a.png 2023-07-11 21:31:35,139 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_352_816057c5787d207de5ff.png media/images/validation_352_816057c5787d207de5ff.png 2023-07-11 21:31:35,139 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_352_6788b6099bc68d956e71.png media/images/validation_352_6788b6099bc68d956e71.png 2023-07-11 21:31:35,139 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_352_8c3b42ad97f04b9342e2.png media/images/validation_352_8c3b42ad97f04b9342e2.png 2023-07-11 21:31:35,139 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_352_3a044ff8368fa9c583c0.png media/images/validation_352_3a044ff8368fa9c583c0.png 2023-07-11 21:31:35,139 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_360_c20dcd4a948eaf4fa197.png media/images/validation_360_c20dcd4a948eaf4fa197.png 2023-07-11 21:31:35,139 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_360_9a31898c9077cdd9a41d.png media/images/validation_360_9a31898c9077cdd9a41d.png 2023-07-11 21:31:35,139 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_360_e2e4e89dbfed8848cd88.png media/images/validation_360_e2e4e89dbfed8848cd88.png 2023-07-11 21:31:35,139 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_360_c8519aaa69a9bd44cd79.png media/images/validation_360_c8519aaa69a9bd44cd79.png 2023-07-11 21:31:35,139 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_368_e3360ff6b5d76e8dac77.png media/images/validation_368_e3360ff6b5d76e8dac77.png 2023-07-11 21:31:35,140 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_368_f4da615e9afd2c2cd8b4.png media/images/validation_368_f4da615e9afd2c2cd8b4.png 2023-07-11 21:31:35,140 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_368_718865898338ad2310bd.png media/images/validation_368_718865898338ad2310bd.png 2023-07-11 21:31:35,140 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_368_72cbe5ebdd0210e3d6fe.png media/images/validation_368_72cbe5ebdd0210e3d6fe.png 2023-07-11 21:31:35,140 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_376_c945a66b43ba7bfe4fc4.png media/images/validation_376_c945a66b43ba7bfe4fc4.png 2023-07-11 21:31:35,140 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_376_1c6e0b1c9c4a02e2487e.png media/images/validation_376_1c6e0b1c9c4a02e2487e.png 2023-07-11 21:31:35,140 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_376_e09462e3bcde1058d20b.png media/images/validation_376_e09462e3bcde1058d20b.png 2023-07-11 21:31:35,140 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_376_05d73dd8573520eeccd6.png media/images/validation_376_05d73dd8573520eeccd6.png 2023-07-11 21:31:35,140 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_384_155d4ae5c7fbde37d2d4.png media/images/validation_384_155d4ae5c7fbde37d2d4.png 2023-07-11 21:31:35,140 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_384_11cb9d1bc6f6a97d0955.png media/images/validation_384_11cb9d1bc6f6a97d0955.png 2023-07-11 21:31:35,140 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_384_7a702633095b41268383.png media/images/validation_384_7a702633095b41268383.png 2023-07-11 21:31:35,140 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_384_905222ac5248c3db9c7f.png media/images/validation_384_905222ac5248c3db9c7f.png 2023-07-11 21:31:35,141 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_392_86c67c37eda7ead98c90.png media/images/validation_392_86c67c37eda7ead98c90.png 2023-07-11 21:31:35,141 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_392_6c8f25f05cf96b671ec4.png media/images/validation_392_6c8f25f05cf96b671ec4.png 2023-07-11 21:31:35,141 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_392_7d995d67c424df992085.png media/images/validation_392_7d995d67c424df992085.png 2023-07-11 21:31:35,141 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_392_cba110247f071a63aa27.png media/images/validation_392_cba110247f071a63aa27.png 2023-07-11 21:31:35,141 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_400_ed68c4b1954c2b559c72.png media/images/validation_400_ed68c4b1954c2b559c72.png 2023-07-11 21:31:35,141 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_400_b541dd5a4efd2e7f476e.png media/images/validation_400_b541dd5a4efd2e7f476e.png 2023-07-11 21:31:35,141 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_400_1a34069e82abe9147618.png media/images/validation_400_1a34069e82abe9147618.png 2023-07-11 21:31:35,141 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_400_d81856892dae7f37a9ef.png media/images/validation_400_d81856892dae7f37a9ef.png 2023-07-11 21:31:35,141 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_408_dea2f3fa8cc40610700c.png media/images/validation_408_dea2f3fa8cc40610700c.png 2023-07-11 21:31:35,141 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_408_8049b1e57a48ac722fb3.png media/images/validation_408_8049b1e57a48ac722fb3.png 2023-07-11 21:31:35,141 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_408_db3da578241ccb488fb6.png media/images/validation_408_db3da578241ccb488fb6.png 2023-07-11 21:31:35,141 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_408_5100cba967fc4cc1cdf6.png media/images/validation_408_5100cba967fc4cc1cdf6.png 2023-07-11 21:31:35,142 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_416_d31ae94219028126481b.png media/images/validation_416_d31ae94219028126481b.png 2023-07-11 21:31:35,142 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_416_0fb15379db5c9ff2e0fc.png media/images/validation_416_0fb15379db5c9ff2e0fc.png 2023-07-11 21:31:35,142 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_416_cb07e3b163172a97c565.png media/images/validation_416_cb07e3b163172a97c565.png 2023-07-11 21:31:35,142 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_416_33b6f1678b8b6a41d991.png media/images/validation_416_33b6f1678b8b6a41d991.png 2023-07-11 21:31:35,142 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_424_33240f6708272a52eea8.png media/images/validation_424_33240f6708272a52eea8.png 2023-07-11 21:31:35,142 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_424_3e2d62ab8a779b41aad1.png media/images/validation_424_3e2d62ab8a779b41aad1.png 2023-07-11 21:31:35,142 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_424_85fe6cca04ba628c90f1.png media/images/validation_424_85fe6cca04ba628c90f1.png 2023-07-11 21:31:35,142 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_424_ff3c9534a6bae344d14f.png media/images/validation_424_ff3c9534a6bae344d14f.png 2023-07-11 21:31:35,142 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_432_18313983b36cde50dd3b.png media/images/validation_432_18313983b36cde50dd3b.png 2023-07-11 21:31:35,142 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_432_9aaaaf8c3f6ad3ebcd99.png media/images/validation_432_9aaaaf8c3f6ad3ebcd99.png 2023-07-11 21:31:35,142 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_432_c7dc0e4c561f71eb5b3e.png media/images/validation_432_c7dc0e4c561f71eb5b3e.png 2023-07-11 21:31:35,142 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_432_0dfe0505f8d9adcc6ad1.png media/images/validation_432_0dfe0505f8d9adcc6ad1.png 2023-07-11 21:31:35,143 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_440_bf3806856d1001a26a5b.png media/images/validation_440_bf3806856d1001a26a5b.png 2023-07-11 21:31:35,143 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_440_12692f8ccbda27de07b9.png media/images/validation_440_12692f8ccbda27de07b9.png 2023-07-11 21:31:35,143 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_440_d5960ae197569bb66752.png media/images/validation_440_d5960ae197569bb66752.png 2023-07-11 21:31:35,143 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_440_b23e879f7839905b9cab.png media/images/validation_440_b23e879f7839905b9cab.png 2023-07-11 21:31:35,143 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_448_9e5ba8c3ea8c4ba6489b.png media/images/validation_448_9e5ba8c3ea8c4ba6489b.png 2023-07-11 21:31:35,143 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_448_a781cbefdd5dbd10be81.png media/images/validation_448_a781cbefdd5dbd10be81.png 2023-07-11 21:31:35,143 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_448_9d3f7736aabd23e4c942.png media/images/validation_448_9d3f7736aabd23e4c942.png 2023-07-11 21:31:35,143 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_448_b685a88a96b2c1a8a665.png media/images/validation_448_b685a88a96b2c1a8a665.png 2023-07-11 21:31:35,143 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_456_7274f3777b238c5f0628.png media/images/validation_456_7274f3777b238c5f0628.png 2023-07-11 21:31:35,143 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_456_3c82fec3d730872e6a5e.png media/images/validation_456_3c82fec3d730872e6a5e.png 2023-07-11 21:31:35,143 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_456_8e44c66b80c9ca0412ba.png media/images/validation_456_8e44c66b80c9ca0412ba.png 2023-07-11 21:31:35,144 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_456_ed31315a0d1d059dae78.png media/images/validation_456_ed31315a0d1d059dae78.png 2023-07-11 21:31:35,144 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_464_b52266a063208eac90d1.png media/images/validation_464_b52266a063208eac90d1.png 2023-07-11 21:31:35,144 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_464_b0bdbc989e686d54eacb.png media/images/validation_464_b0bdbc989e686d54eacb.png 2023-07-11 21:31:35,144 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_464_66cbaba9a39ca8bfc815.png media/images/validation_464_66cbaba9a39ca8bfc815.png 2023-07-11 21:31:35,144 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_464_97d923b01964cb8ebb37.png media/images/validation_464_97d923b01964cb8ebb37.png 2023-07-11 21:31:35,144 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_472_cfb31d01ff9d8fcc9935.png media/images/validation_472_cfb31d01ff9d8fcc9935.png 2023-07-11 21:31:35,144 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_472_820adba40c6ce80d2f4a.png media/images/validation_472_820adba40c6ce80d2f4a.png 2023-07-11 21:31:35,144 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_472_059ba63ee59da583f8b0.png media/images/validation_472_059ba63ee59da583f8b0.png 2023-07-11 21:31:35,144 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_472_ce05e5603e88c594f7a0.png media/images/validation_472_ce05e5603e88c594f7a0.png 2023-07-11 21:31:35,144 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_480_cccdf5a911d086c92c1b.png media/images/validation_480_cccdf5a911d086c92c1b.png 2023-07-11 21:31:35,144 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_480_ccca929b7019e723af09.png media/images/validation_480_ccca929b7019e723af09.png 2023-07-11 21:31:35,144 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_480_be6c1fc77e931ea5eccf.png media/images/validation_480_be6c1fc77e931ea5eccf.png 2023-07-11 21:31:35,145 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_480_a98a50da4795b7ed390e.png media/images/validation_480_a98a50da4795b7ed390e.png 2023-07-11 21:31:35,145 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_488_edd688b52163221b6553.png media/images/validation_488_edd688b52163221b6553.png 2023-07-11 21:31:35,145 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_488_446494043ddef0089999.png media/images/validation_488_446494043ddef0089999.png 2023-07-11 21:31:35,145 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_488_f79a2569df6f42330b22.png media/images/validation_488_f79a2569df6f42330b22.png 2023-07-11 21:31:35,145 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_488_1f7f3d27cf4d0d695023.png media/images/validation_488_1f7f3d27cf4d0d695023.png 2023-07-11 21:31:35,145 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_496_68e06148c0b530b292ef.png media/images/validation_496_68e06148c0b530b292ef.png 2023-07-11 21:31:35,145 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_496_2a1d91ee7dad3aca3377.png media/images/validation_496_2a1d91ee7dad3aca3377.png 2023-07-11 21:31:35,145 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_496_37d891b4d6c71e28eca9.png media/images/validation_496_37d891b4d6c71e28eca9.png 2023-07-11 21:31:35,145 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_496_ab5da33b4fab6d97badb.png media/images/validation_496_ab5da33b4fab6d97badb.png 2023-07-11 21:31:35,145 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_504_f98f85886287d586fb24.png media/images/validation_504_f98f85886287d586fb24.png 2023-07-11 21:31:35,145 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_504_aab7b24d7b2a7c51475f.png media/images/validation_504_aab7b24d7b2a7c51475f.png 2023-07-11 21:31:35,146 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_504_8d503921a8e36638c133.png media/images/validation_504_8d503921a8e36638c133.png 2023-07-11 21:31:35,146 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_504_4cd9dbf50122f56d08c9.png media/images/validation_504_4cd9dbf50122f56d08c9.png 2023-07-11 21:31:35,146 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_512_95c88d4ee00e6dceedec.png media/images/validation_512_95c88d4ee00e6dceedec.png 2023-07-11 21:31:35,146 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_512_c7af5aea69ad1b787bdd.png media/images/validation_512_c7af5aea69ad1b787bdd.png 2023-07-11 21:31:35,146 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_512_b6b8325d8fe282e8ee86.png media/images/validation_512_b6b8325d8fe282e8ee86.png 2023-07-11 21:31:35,146 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_512_87b03e3c08874339a548.png media/images/validation_512_87b03e3c08874339a548.png 2023-07-11 21:31:35,146 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_520_2111edf7d29b2a9e9ffa.png media/images/validation_520_2111edf7d29b2a9e9ffa.png 2023-07-11 21:31:35,146 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_520_b5b4ceac10276cd5b030.png media/images/validation_520_b5b4ceac10276cd5b030.png 2023-07-11 21:31:35,146 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_520_3a4cf4d788de75533c67.png media/images/validation_520_3a4cf4d788de75533c67.png 2023-07-11 21:31:35,146 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_520_78d0488f99cba790913e.png media/images/validation_520_78d0488f99cba790913e.png 2023-07-11 21:31:35,146 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_528_875885eb3355d2fc1a7a.png media/images/validation_528_875885eb3355d2fc1a7a.png 2023-07-11 21:31:35,146 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_528_b7baa092f64c64dba260.png media/images/validation_528_b7baa092f64c64dba260.png 2023-07-11 21:31:35,147 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_528_5348c071dbdd44e6d366.png media/images/validation_528_5348c071dbdd44e6d366.png 2023-07-11 21:31:35,147 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_528_767c54c5dd821b8fd78d.png media/images/validation_528_767c54c5dd821b8fd78d.png 2023-07-11 21:31:35,147 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_536_a8e5af09990ec6f7b046.png media/images/validation_536_a8e5af09990ec6f7b046.png 2023-07-11 21:31:35,147 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_536_08ad374c5434931379b1.png media/images/validation_536_08ad374c5434931379b1.png 2023-07-11 21:31:35,147 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_536_e6b2848e4a4a5a1fa3b2.png media/images/validation_536_e6b2848e4a4a5a1fa3b2.png 2023-07-11 21:31:35,147 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_536_915b0997964674e19d30.png media/images/validation_536_915b0997964674e19d30.png 2023-07-11 21:31:35,147 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_544_d527e1f00fb003991478.png media/images/validation_544_d527e1f00fb003991478.png 2023-07-11 21:31:35,147 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_544_b99da3c0fcb7ededf7b0.png media/images/validation_544_b99da3c0fcb7ededf7b0.png 2023-07-11 21:31:35,147 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_544_f145c478050d42496e6e.png media/images/validation_544_f145c478050d42496e6e.png 2023-07-11 21:31:35,147 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_544_ec40722860c852345c23.png media/images/validation_544_ec40722860c852345c23.png 2023-07-11 21:31:35,147 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_552_8c78fd5084c9eef35e50.png media/images/validation_552_8c78fd5084c9eef35e50.png 2023-07-11 21:31:35,147 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_552_41181c5e1fbbaa3688f6.png media/images/validation_552_41181c5e1fbbaa3688f6.png 2023-07-11 21:31:35,148 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_552_257d3c0b8f2426466221.png media/images/validation_552_257d3c0b8f2426466221.png 2023-07-11 21:31:35,148 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_552_8812f45a50bdc79a8517.png media/images/validation_552_8812f45a50bdc79a8517.png 2023-07-11 21:31:35,148 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_560_f76cf11600d692be2502.png media/images/validation_560_f76cf11600d692be2502.png 2023-07-11 21:31:35,148 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_560_bbef43cf934160e04cfd.png media/images/validation_560_bbef43cf934160e04cfd.png 2023-07-11 21:31:35,148 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_560_8a39c93dbe4043d18b4f.png media/images/validation_560_8a39c93dbe4043d18b4f.png 2023-07-11 21:31:35,148 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_560_b956f5c40cffd0dcba98.png media/images/validation_560_b956f5c40cffd0dcba98.png 2023-07-11 21:31:35,148 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_568_dba4f198a349b1343cdd.png media/images/validation_568_dba4f198a349b1343cdd.png 2023-07-11 21:31:35,148 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_568_04d62c7653c5d162a2e3.png media/images/validation_568_04d62c7653c5d162a2e3.png 2023-07-11 21:31:35,148 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_568_e297ab1a0ce4b9855bb5.png media/images/validation_568_e297ab1a0ce4b9855bb5.png 2023-07-11 21:31:35,148 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_568_68596aff73764ec36f54.png media/images/validation_568_68596aff73764ec36f54.png 2023-07-11 21:31:35,148 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_576_3b7a05a66d1f89e67bb8.png media/images/validation_576_3b7a05a66d1f89e67bb8.png 2023-07-11 21:31:35,149 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_576_b1cd8c64caeaede72534.png media/images/validation_576_b1cd8c64caeaede72534.png 2023-07-11 21:31:35,149 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_576_0fa5d5fa63f3526989d2.png media/images/validation_576_0fa5d5fa63f3526989d2.png 2023-07-11 21:31:35,149 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_576_b8f228173e9910efda0e.png media/images/validation_576_b8f228173e9910efda0e.png 2023-07-11 21:31:35,149 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_584_fea9e895a6bad4c69a30.png media/images/validation_584_fea9e895a6bad4c69a30.png 2023-07-11 21:31:35,149 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_584_b49048e90ace4b32dca7.png media/images/validation_584_b49048e90ace4b32dca7.png 2023-07-11 21:31:35,149 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_584_434c1624a967a3f3a962.png media/images/validation_584_434c1624a967a3f3a962.png 2023-07-11 21:31:35,149 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_584_29997c31305c3912cd04.png media/images/validation_584_29997c31305c3912cd04.png 2023-07-11 21:31:35,149 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_592_3924670710a9e29d06d3.png media/images/validation_592_3924670710a9e29d06d3.png 2023-07-11 21:31:35,149 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_592_0e50e9bfb15ce39c112d.png media/images/validation_592_0e50e9bfb15ce39c112d.png 2023-07-11 21:31:35,149 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_592_9a03f64388706e1447b6.png media/images/validation_592_9a03f64388706e1447b6.png 2023-07-11 21:31:35,149 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_592_cd080a9de17f2873d239.png media/images/validation_592_cd080a9de17f2873d239.png 2023-07-11 21:31:35,150 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_600_8300fd15af7363a95498.png media/images/validation_600_8300fd15af7363a95498.png 2023-07-11 21:31:35,150 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_600_4dddb5352b02ea5893dc.png media/images/validation_600_4dddb5352b02ea5893dc.png 2023-07-11 21:31:35,150 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_600_1d97a6cab0fe06041ca1.png media/images/validation_600_1d97a6cab0fe06041ca1.png 2023-07-11 21:31:35,150 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_600_917bd69bfc0758359482.png media/images/validation_600_917bd69bfc0758359482.png 2023-07-11 21:31:35,150 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_608_2104758725886dd01ccd.png media/images/validation_608_2104758725886dd01ccd.png 2023-07-11 21:31:35,150 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_608_e93612b41299320c0daf.png media/images/validation_608_e93612b41299320c0daf.png 2023-07-11 21:31:35,150 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_608_6863da4ca0ed68a9855f.png media/images/validation_608_6863da4ca0ed68a9855f.png 2023-07-11 21:31:35,150 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_608_0f027a7a2f974076b6e9.png media/images/validation_608_0f027a7a2f974076b6e9.png 2023-07-11 21:31:35,150 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_616_a89cec9bd984af45d04e.png media/images/validation_616_a89cec9bd984af45d04e.png 2023-07-11 21:31:35,150 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_616_3cf079583c2c3172a1af.png media/images/validation_616_3cf079583c2c3172a1af.png 2023-07-11 21:31:35,150 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_616_7dcb623099bcf2fb9044.png media/images/validation_616_7dcb623099bcf2fb9044.png 2023-07-11 21:31:35,150 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_616_b8d74a65b6fc0e8293ef.png media/images/validation_616_b8d74a65b6fc0e8293ef.png 2023-07-11 21:31:35,151 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_624_870de5f70c0e3c35f00a.png media/images/validation_624_870de5f70c0e3c35f00a.png 2023-07-11 21:31:35,151 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_624_5194e21fa06d24dfa3a2.png media/images/validation_624_5194e21fa06d24dfa3a2.png 2023-07-11 21:31:35,151 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_624_7988a620d724c156773b.png media/images/validation_624_7988a620d724c156773b.png 2023-07-11 21:31:35,151 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_624_9cd574bf60106cbeb7f7.png media/images/validation_624_9cd574bf60106cbeb7f7.png 2023-07-11 21:31:35,151 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_632_9db58ceffde2ac6026b2.png media/images/validation_632_9db58ceffde2ac6026b2.png 2023-07-11 21:31:35,151 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_632_90d4a1e76489230b3d0e.png media/images/validation_632_90d4a1e76489230b3d0e.png 2023-07-11 21:31:35,151 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_632_cbe1621c20b110b72424.png media/images/validation_632_cbe1621c20b110b72424.png 2023-07-11 21:31:35,151 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_632_e3edcf43076ac8fbe703.png media/images/validation_632_e3edcf43076ac8fbe703.png 2023-07-11 21:31:35,151 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_640_b0840e49c29d03b310d6.png media/images/validation_640_b0840e49c29d03b310d6.png 2023-07-11 21:31:35,151 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_640_926cd9184da41f18f64c.png media/images/validation_640_926cd9184da41f18f64c.png 2023-07-11 21:31:35,151 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_640_988f0fde8c0b0bda7e16.png media/images/validation_640_988f0fde8c0b0bda7e16.png 2023-07-11 21:31:35,152 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_640_cf8925a87c42c9b29e53.png media/images/validation_640_cf8925a87c42c9b29e53.png 2023-07-11 21:31:35,152 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_648_728dd91334abb53b86d0.png media/images/validation_648_728dd91334abb53b86d0.png 2023-07-11 21:31:35,152 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_648_55287d9a3ca04b88ba22.png media/images/validation_648_55287d9a3ca04b88ba22.png 2023-07-11 21:31:35,152 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_648_7c2b88efee87b1a3d3bc.png media/images/validation_648_7c2b88efee87b1a3d3bc.png 2023-07-11 21:31:35,152 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_648_5b836be338a558b667f9.png media/images/validation_648_5b836be338a558b667f9.png 2023-07-11 21:31:35,152 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_656_44ad3c8fcf80a92c74c0.png media/images/validation_656_44ad3c8fcf80a92c74c0.png 2023-07-11 21:31:35,152 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_656_c5399b2a25bca627fffe.png media/images/validation_656_c5399b2a25bca627fffe.png 2023-07-11 21:31:35,152 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_656_09858d6b1f00f78cae5c.png media/images/validation_656_09858d6b1f00f78cae5c.png 2023-07-11 21:31:35,152 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_656_c7768aa4e07faefe0067.png media/images/validation_656_c7768aa4e07faefe0067.png 2023-07-11 21:31:35,152 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_664_158f555f9864c3964f75.png media/images/validation_664_158f555f9864c3964f75.png 2023-07-11 21:31:35,152 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_664_95c67446531a911a320b.png media/images/validation_664_95c67446531a911a320b.png 2023-07-11 21:31:35,152 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_664_092cf810659ff0a3ad9b.png media/images/validation_664_092cf810659ff0a3ad9b.png 2023-07-11 21:31:35,153 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_664_6d9e47d176e82143fca1.png media/images/validation_664_6d9e47d176e82143fca1.png 2023-07-11 21:31:35,153 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_672_e6e648fc0801971ed27f.png media/images/validation_672_e6e648fc0801971ed27f.png 2023-07-11 21:31:35,153 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_672_640323be00d22421969d.png media/images/validation_672_640323be00d22421969d.png 2023-07-11 21:31:35,153 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_672_e7cccd2e24c981238fe9.png media/images/validation_672_e7cccd2e24c981238fe9.png 2023-07-11 21:31:35,153 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_672_0a35aec0325a8b5931bd.png media/images/validation_672_0a35aec0325a8b5931bd.png 2023-07-11 21:31:35,153 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_680_aa194dccdb4cde13bf28.png media/images/validation_680_aa194dccdb4cde13bf28.png 2023-07-11 21:31:35,153 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_680_5296d50b33663e449a0e.png media/images/validation_680_5296d50b33663e449a0e.png 2023-07-11 21:31:35,153 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_680_72ef33de170d84e10fda.png media/images/validation_680_72ef33de170d84e10fda.png 2023-07-11 21:31:35,153 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_680_590049620b94f7f699ff.png media/images/validation_680_590049620b94f7f699ff.png 2023-07-11 21:31:35,153 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_688_dc7a2400adb3ddfda5f1.png media/images/validation_688_dc7a2400adb3ddfda5f1.png 2023-07-11 21:31:35,153 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_688_9f4c23b3d6799b0996b4.png media/images/validation_688_9f4c23b3d6799b0996b4.png 2023-07-11 21:31:35,154 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_688_1987fb0f39545442f7d1.png media/images/validation_688_1987fb0f39545442f7d1.png 2023-07-11 21:31:35,154 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_688_59f2ed6f733f8149780a.png media/images/validation_688_59f2ed6f733f8149780a.png 2023-07-11 21:31:35,154 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_696_bf57326caaf8c0b6ad64.png media/images/validation_696_bf57326caaf8c0b6ad64.png 2023-07-11 21:31:35,154 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_696_b61dd12befa2c9cd68d2.png media/images/validation_696_b61dd12befa2c9cd68d2.png 2023-07-11 21:31:35,154 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_696_72197a206eb0dae2a9b9.png media/images/validation_696_72197a206eb0dae2a9b9.png 2023-07-11 21:31:35,154 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_696_597ae9e605560c042f4a.png media/images/validation_696_597ae9e605560c042f4a.png 2023-07-11 21:31:35,154 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_704_c190c2a29844123428a8.png media/images/validation_704_c190c2a29844123428a8.png 2023-07-11 21:31:35,154 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_704_ff8ed0b57b72882d609c.png media/images/validation_704_ff8ed0b57b72882d609c.png 2023-07-11 21:31:35,155 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_704_072c318a44d7c7157563.png media/images/validation_704_072c318a44d7c7157563.png 2023-07-11 21:31:35,155 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_704_81266f0e45c6c5305caa.png media/images/validation_704_81266f0e45c6c5305caa.png 2023-07-11 21:31:35,155 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_712_6ba33f9b1438e729f3e3.png media/images/validation_712_6ba33f9b1438e729f3e3.png 2023-07-11 21:31:35,155 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_712_7fff33d2cf3b6bfe2855.png media/images/validation_712_7fff33d2cf3b6bfe2855.png 2023-07-11 21:31:35,155 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_712_a41798bd5ba67711765f.png media/images/validation_712_a41798bd5ba67711765f.png 2023-07-11 21:31:35,155 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_712_e66ad705dd270fae09fa.png media/images/validation_712_e66ad705dd270fae09fa.png 2023-07-11 21:31:35,155 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_720_a76739d06259d72923e3.png media/images/validation_720_a76739d06259d72923e3.png 2023-07-11 21:31:35,155 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_720_3ac906f6178bc7cc5030.png media/images/validation_720_3ac906f6178bc7cc5030.png 2023-07-11 21:31:35,155 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_720_18dae1edaa995fefc02c.png media/images/validation_720_18dae1edaa995fefc02c.png 2023-07-11 21:31:35,155 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_720_f473f15d1fa6dae04183.png media/images/validation_720_f473f15d1fa6dae04183.png 2023-07-11 21:31:35,155 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_728_85f42fd5b9d94ea9077f.png media/images/validation_728_85f42fd5b9d94ea9077f.png 2023-07-11 21:31:35,160 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_728_6d3391d06d6901eb149d.png media/images/validation_728_6d3391d06d6901eb149d.png 2023-07-11 21:31:35,161 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_728_b82c35a5387d49e858c0.png media/images/validation_728_b82c35a5387d49e858c0.png 2023-07-11 21:31:35,161 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_728_7db45a659309d31efe67.png media/images/validation_728_7db45a659309d31efe67.png 2023-07-11 21:31:35,161 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_736_ac4e29b11ce1d305f530.png media/images/validation_736_ac4e29b11ce1d305f530.png 2023-07-11 21:31:35,161 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_736_fae683dc9806394cb2e9.png media/images/validation_736_fae683dc9806394cb2e9.png 2023-07-11 21:31:35,161 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_736_f0caac32194909cb1981.png media/images/validation_736_f0caac32194909cb1981.png 2023-07-11 21:31:35,161 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_736_d6436e676d16768a25aa.png media/images/validation_736_d6436e676d16768a25aa.png 2023-07-11 21:31:35,161 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_744_88c24683fdcc6de68876.png media/images/validation_744_88c24683fdcc6de68876.png 2023-07-11 21:31:35,161 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_744_835fc7ba76fe5bdd463b.png media/images/validation_744_835fc7ba76fe5bdd463b.png 2023-07-11 21:31:35,161 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_744_54915f91d21e93f22f28.png media/images/validation_744_54915f91d21e93f22f28.png 2023-07-11 21:31:35,161 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_744_24c539112b53ecad6996.png media/images/validation_744_24c539112b53ecad6996.png 2023-07-11 21:31:35,161 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_752_aa82701d58c5e4be3efd.png media/images/validation_752_aa82701d58c5e4be3efd.png 2023-07-11 21:31:35,161 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_752_e8178d5464133df3775d.png media/images/validation_752_e8178d5464133df3775d.png 2023-07-11 21:31:35,162 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_752_95d256cb61eaa36920d3.png media/images/validation_752_95d256cb61eaa36920d3.png 2023-07-11 21:31:35,162 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_752_ec4a84736de32ae6ae3c.png media/images/validation_752_ec4a84736de32ae6ae3c.png 2023-07-11 21:31:35,167 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_760_8c7815ce3a6257b58e00.png media/images/validation_760_8c7815ce3a6257b58e00.png 2023-07-11 21:31:35,173 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_760_cf617b6b91a851ec265b.png media/images/validation_760_cf617b6b91a851ec265b.png 2023-07-11 21:31:35,176 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_760_a1a1c2ba06a8ef07f019.png media/images/validation_760_a1a1c2ba06a8ef07f019.png 2023-07-11 21:31:35,182 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_760_fb0a52e92cd7ee579cca.png media/images/validation_760_fb0a52e92cd7ee579cca.png 2023-07-11 21:31:35,188 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_768_693e8b6430b9351126e2.png media/images/validation_768_693e8b6430b9351126e2.png 2023-07-11 21:31:35,189 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_768_917890c98c8478bc01dd.png media/images/validation_768_917890c98c8478bc01dd.png 2023-07-11 21:31:35,190 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_768_373eddb1acba3661e232.png media/images/validation_768_373eddb1acba3661e232.png 2023-07-11 21:31:35,190 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_768_282437d359e52d34f4f8.png media/images/validation_768_282437d359e52d34f4f8.png 2023-07-11 21:31:35,191 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_776_2aac3dd33ac8c1b799cb.png media/images/validation_776_2aac3dd33ac8c1b799cb.png 2023-07-11 21:31:35,191 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_776_eb54d867993cb09e2b53.png media/images/validation_776_eb54d867993cb09e2b53.png 2023-07-11 21:31:35,191 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_776_06b4410d2195a28ee2e4.png media/images/validation_776_06b4410d2195a28ee2e4.png 2023-07-11 21:31:35,191 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_776_938b1a9310105c2c30a8.png media/images/validation_776_938b1a9310105c2c30a8.png 2023-07-11 21:31:35,191 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_784_731753b83553f4858c73.png media/images/validation_784_731753b83553f4858c73.png 2023-07-11 21:31:35,191 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_784_07f7d7557663601508eb.png media/images/validation_784_07f7d7557663601508eb.png 2023-07-11 21:31:35,191 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_784_489eed56c83ee92b8e9c.png media/images/validation_784_489eed56c83ee92b8e9c.png 2023-07-11 21:31:35,192 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_784_0fa68e4c56abe4cd2e0d.png media/images/validation_784_0fa68e4c56abe4cd2e0d.png 2023-07-11 21:31:35,192 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_792_e061fd9e41fdfb8c934a.png media/images/validation_792_e061fd9e41fdfb8c934a.png 2023-07-11 21:31:35,192 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_792_2db0a6b3dbfebedf2d8e.png media/images/validation_792_2db0a6b3dbfebedf2d8e.png 2023-07-11 21:31:35,192 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_792_3f75331b78a95b5b40b7.png media/images/validation_792_3f75331b78a95b5b40b7.png 2023-07-11 21:31:35,192 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_792_eac5499ee475cd33206f.png media/images/validation_792_eac5499ee475cd33206f.png 2023-07-11 21:31:35,192 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_800_31cd8fae17813d7d0e65.png media/images/validation_800_31cd8fae17813d7d0e65.png 2023-07-11 21:31:35,192 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_800_d905700551ce75840705.png media/images/validation_800_d905700551ce75840705.png 2023-07-11 21:31:35,193 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_800_3d6397867b321dddf262.png media/images/validation_800_3d6397867b321dddf262.png 2023-07-11 21:31:35,193 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_800_2ed775ff9a30d99a0a43.png media/images/validation_800_2ed775ff9a30d99a0a43.png 2023-07-11 21:31:35,193 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_808_198cfae2d5879abcba46.png media/images/validation_808_198cfae2d5879abcba46.png 2023-07-11 21:31:35,193 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_808_7aaebfffee54696e50f8.png media/images/validation_808_7aaebfffee54696e50f8.png 2023-07-11 21:31:35,193 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_808_f9c3bd83a1bb11eb4cff.png media/images/validation_808_f9c3bd83a1bb11eb4cff.png 2023-07-11 21:31:35,193 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_808_b053e35f6e9aac4bea20.png media/images/validation_808_b053e35f6e9aac4bea20.png 2023-07-11 21:31:35,193 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_816_061281f0958189cb5d31.png media/images/validation_816_061281f0958189cb5d31.png 2023-07-11 21:31:35,194 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_816_e9e3f32b220e23c697cf.png media/images/validation_816_e9e3f32b220e23c697cf.png 2023-07-11 21:31:35,194 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_816_5b6b74a7d7fd49b3636b.png media/images/validation_816_5b6b74a7d7fd49b3636b.png 2023-07-11 21:31:35,194 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_816_69329d4e2a0a472e0d04.png media/images/validation_816_69329d4e2a0a472e0d04.png 2023-07-11 21:31:35,194 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_824_27f999fd2d618c705cb1.png media/images/validation_824_27f999fd2d618c705cb1.png 2023-07-11 21:31:35,195 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_824_9c2a756ba8f9a949c00e.png media/images/validation_824_9c2a756ba8f9a949c00e.png 2023-07-11 21:31:35,195 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_824_e40da72508d9965ce259.png media/images/validation_824_e40da72508d9965ce259.png 2023-07-11 21:31:35,195 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_824_40304dd8f904f4d4b176.png media/images/validation_824_40304dd8f904f4d4b176.png 2023-07-11 21:31:35,195 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_832_7645690b8d91851e7d9d.png media/images/validation_832_7645690b8d91851e7d9d.png 2023-07-11 21:31:35,195 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_832_430e5d6128d45a25cb6f.png media/images/validation_832_430e5d6128d45a25cb6f.png 2023-07-11 21:31:35,195 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_832_306e72d0667b956b2311.png media/images/validation_832_306e72d0667b956b2311.png 2023-07-11 21:31:35,195 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_832_262acb7fdbd40f4b9740.png media/images/validation_832_262acb7fdbd40f4b9740.png 2023-07-11 21:31:35,196 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_840_2be7449ad230441219d7.png media/images/validation_840_2be7449ad230441219d7.png 2023-07-11 21:31:35,196 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_840_b38629e31548571aa3be.png media/images/validation_840_b38629e31548571aa3be.png 2023-07-11 21:31:35,196 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_840_e7fcf8abc3886e356bb8.png media/images/validation_840_e7fcf8abc3886e356bb8.png 2023-07-11 21:31:35,197 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_840_ffd650d33cbd150f9d5a.png media/images/validation_840_ffd650d33cbd150f9d5a.png 2023-07-11 21:31:35,198 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_848_9fd46f5e0289db110e5a.png media/images/validation_848_9fd46f5e0289db110e5a.png 2023-07-11 21:31:35,198 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_848_0da7b9c733de297e7940.png media/images/validation_848_0da7b9c733de297e7940.png 2023-07-11 21:31:35,198 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_848_d3851e1b7aee0085f878.png media/images/validation_848_d3851e1b7aee0085f878.png 2023-07-11 21:31:35,198 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_848_7e99547529310b4b5dbe.png media/images/validation_848_7e99547529310b4b5dbe.png 2023-07-11 21:31:35,198 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_856_b93f1ea746e64027d9cf.png media/images/validation_856_b93f1ea746e64027d9cf.png 2023-07-11 21:31:35,198 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_856_a9c51bde36c7b91e7157.png media/images/validation_856_a9c51bde36c7b91e7157.png 2023-07-11 21:31:35,199 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_856_34456aced927b6fdcddf.png media/images/validation_856_34456aced927b6fdcddf.png 2023-07-11 21:31:35,199 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_856_738cae12b93e3de8b6ad.png media/images/validation_856_738cae12b93e3de8b6ad.png 2023-07-11 21:31:35,199 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_864_950acf7e39a1d4e12f7d.png media/images/validation_864_950acf7e39a1d4e12f7d.png 2023-07-11 21:31:35,199 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_864_8ecbb274e51eb741dfb5.png media/images/validation_864_8ecbb274e51eb741dfb5.png 2023-07-11 21:31:35,199 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_864_8aa45d21612bc1191b24.png media/images/validation_864_8aa45d21612bc1191b24.png 2023-07-11 21:31:35,199 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_864_5a6f78f0d182d46a8beb.png media/images/validation_864_5a6f78f0d182d46a8beb.png 2023-07-11 21:31:35,199 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_872_ca57733755532b74591b.png media/images/validation_872_ca57733755532b74591b.png 2023-07-11 21:31:35,200 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_872_92a3a20861d1d9a07c22.png media/images/validation_872_92a3a20861d1d9a07c22.png 2023-07-11 21:31:35,200 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_872_72610369def04cbca766.png media/images/validation_872_72610369def04cbca766.png 2023-07-11 21:31:35,200 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_872_a69cb09077cee2f55c62.png media/images/validation_872_a69cb09077cee2f55c62.png 2023-07-11 21:31:35,200 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_880_2ab57feb81125c56c260.png media/images/validation_880_2ab57feb81125c56c260.png 2023-07-11 21:31:35,200 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_880_18fe769afdeec594e0f0.png media/images/validation_880_18fe769afdeec594e0f0.png 2023-07-11 21:31:35,200 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_880_3bf88939232b65d3898d.png media/images/validation_880_3bf88939232b65d3898d.png 2023-07-11 21:31:35,200 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_880_4ca27ee28456bc10a80b.png media/images/validation_880_4ca27ee28456bc10a80b.png 2023-07-11 21:31:35,200 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_888_a7440f71b9ffcf04c59e.png media/images/validation_888_a7440f71b9ffcf04c59e.png 2023-07-11 21:31:35,200 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_888_8729ef5a4b0fe41acc66.png media/images/validation_888_8729ef5a4b0fe41acc66.png 2023-07-11 21:31:35,200 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_888_19b900daab11785f4793.png media/images/validation_888_19b900daab11785f4793.png 2023-07-11 21:31:35,201 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_888_aec0fa4930afae7a7541.png media/images/validation_888_aec0fa4930afae7a7541.png 2023-07-11 21:31:35,201 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_896_1195075f26c9c85c17be.png media/images/validation_896_1195075f26c9c85c17be.png 2023-07-11 21:31:35,201 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_896_8b01143d063fb3d088dd.png media/images/validation_896_8b01143d063fb3d088dd.png 2023-07-11 21:31:35,201 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_896_a33d92e49bb3819c9398.png media/images/validation_896_a33d92e49bb3819c9398.png 2023-07-11 21:31:35,201 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_896_2d49f7aa4f79dc70f56e.png media/images/validation_896_2d49f7aa4f79dc70f56e.png 2023-07-11 21:31:35,202 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_904_d08a528486fd9d24317e.png media/images/validation_904_d08a528486fd9d24317e.png 2023-07-11 21:31:35,202 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_904_74a58ccedcd0aba02d9d.png media/images/validation_904_74a58ccedcd0aba02d9d.png 2023-07-11 21:31:35,202 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_904_01099aeefecc508a7281.png media/images/validation_904_01099aeefecc508a7281.png 2023-07-11 21:31:35,202 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_904_d9f4aaa16030ef084b93.png media/images/validation_904_d9f4aaa16030ef084b93.png 2023-07-11 21:31:35,202 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_912_7c4eaf576d54dc0892c4.png media/images/validation_912_7c4eaf576d54dc0892c4.png 2023-07-11 21:31:35,202 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_912_4a51dc0e598ef75ade74.png media/images/validation_912_4a51dc0e598ef75ade74.png 2023-07-11 21:31:35,202 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_912_87ef29d0d2134d35550f.png media/images/validation_912_87ef29d0d2134d35550f.png 2023-07-11 21:31:35,202 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_912_ac2d97b761361e2c7f95.png media/images/validation_912_ac2d97b761361e2c7f95.png 2023-07-11 21:31:35,202 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_920_dd362beabef9f0f07f1c.png media/images/validation_920_dd362beabef9f0f07f1c.png 2023-07-11 21:31:35,203 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_920_ceb5dd8574ce0588ad3d.png media/images/validation_920_ceb5dd8574ce0588ad3d.png 2023-07-11 21:31:35,203 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_920_cb802cc8bb9421190135.png media/images/validation_920_cb802cc8bb9421190135.png 2023-07-11 21:31:35,203 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_920_5b7b240cdae3212a828d.png media/images/validation_920_5b7b240cdae3212a828d.png 2023-07-11 21:31:35,203 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_928_35f2e6d732e429b27cef.png media/images/validation_928_35f2e6d732e429b27cef.png 2023-07-11 21:31:35,203 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_928_c7d072dca160d6de598d.png media/images/validation_928_c7d072dca160d6de598d.png 2023-07-11 21:31:35,203 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_928_6c25bd7611e598bbfbe5.png media/images/validation_928_6c25bd7611e598bbfbe5.png 2023-07-11 21:31:35,203 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_928_0ccf3153354added95d1.png media/images/validation_928_0ccf3153354added95d1.png 2023-07-11 21:31:35,203 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_936_14f68a769ca53b61b6be.png media/images/validation_936_14f68a769ca53b61b6be.png 2023-07-11 21:31:35,203 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_936_fbfe70b4fd3580e4922a.png media/images/validation_936_fbfe70b4fd3580e4922a.png 2023-07-11 21:31:35,203 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_936_190b7cee3f4ce931d581.png media/images/validation_936_190b7cee3f4ce931d581.png 2023-07-11 21:31:35,203 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_936_427cd756a6fbf28d7994.png media/images/validation_936_427cd756a6fbf28d7994.png 2023-07-11 21:31:35,204 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_944_131e7280660d61568c15.png media/images/validation_944_131e7280660d61568c15.png 2023-07-11 21:31:35,204 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_944_41a5dce9cfa5cf744e71.png media/images/validation_944_41a5dce9cfa5cf744e71.png 2023-07-11 21:31:35,204 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_944_d7a08791c37a0d7e716d.png media/images/validation_944_d7a08791c37a0d7e716d.png 2023-07-11 21:31:35,204 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_944_8d33167a8809bc5e2e10.png media/images/validation_944_8d33167a8809bc5e2e10.png 2023-07-11 21:31:35,204 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_952_fc214fd3c543e1062168.png media/images/validation_952_fc214fd3c543e1062168.png 2023-07-11 21:31:35,204 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_952_49c73fa7e966791f5390.png media/images/validation_952_49c73fa7e966791f5390.png 2023-07-11 21:31:35,204 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_952_c88dcfd62ec0e16b152a.png media/images/validation_952_c88dcfd62ec0e16b152a.png 2023-07-11 21:31:35,204 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_952_e9c245f08754c65b88da.png media/images/validation_952_e9c245f08754c65b88da.png 2023-07-11 21:31:35,204 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_960_bbcff34d61dc915cfffa.png media/images/validation_960_bbcff34d61dc915cfffa.png 2023-07-11 21:31:35,205 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_960_d943310ca37254daf167.png media/images/validation_960_d943310ca37254daf167.png 2023-07-11 21:31:35,205 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_960_1f1bb4f5e11489979814.png media/images/validation_960_1f1bb4f5e11489979814.png 2023-07-11 21:31:35,205 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_960_dab10b88bdf06be619c2.png media/images/validation_960_dab10b88bdf06be619c2.png 2023-07-11 21:31:35,205 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_968_d57c92e936cc34db3b66.png media/images/validation_968_d57c92e936cc34db3b66.png 2023-07-11 21:31:35,205 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_968_37b53ef8f7963631c78a.png media/images/validation_968_37b53ef8f7963631c78a.png 2023-07-11 21:31:35,205 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_968_d874c01a95067c21bdbb.png media/images/validation_968_d874c01a95067c21bdbb.png 2023-07-11 21:31:35,205 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_968_73e3db3d1e8039707146.png media/images/validation_968_73e3db3d1e8039707146.png 2023-07-11 21:31:35,205 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_976_1c6ecb09cf45729ec02d.png media/images/validation_976_1c6ecb09cf45729ec02d.png 2023-07-11 21:31:35,205 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_976_96467eb797af222d5127.png media/images/validation_976_96467eb797af222d5127.png 2023-07-11 21:31:35,205 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_976_b1d4d79c192e807ea9ea.png media/images/validation_976_b1d4d79c192e807ea9ea.png 2023-07-11 21:31:35,206 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_976_24834f9350949b5f8e1c.png media/images/validation_976_24834f9350949b5f8e1c.png 2023-07-11 21:31:35,206 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_984_8ce2b6b9e57abff6242d.png media/images/validation_984_8ce2b6b9e57abff6242d.png 2023-07-11 21:31:35,206 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_984_6babdd3a11961e590892.png media/images/validation_984_6babdd3a11961e590892.png 2023-07-11 21:31:35,206 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_984_8655855a41d6a76b3f50.png media/images/validation_984_8655855a41d6a76b3f50.png 2023-07-11 21:31:35,206 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_984_858c44737753f571d83a.png media/images/validation_984_858c44737753f571d83a.png 2023-07-11 21:31:35,206 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_992_81d31deea5f321a879ea.png media/images/validation_992_81d31deea5f321a879ea.png 2023-07-11 21:31:35,206 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_992_d3be1fc1429f13f91ba7.png media/images/validation_992_d3be1fc1429f13f91ba7.png 2023-07-11 21:31:35,206 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_992_b832ae939713ff831840.png media/images/validation_992_b832ae939713ff831840.png 2023-07-11 21:31:35,206 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_992_297cfc9d19c6bf373211.png media/images/validation_992_297cfc9d19c6bf373211.png 2023-07-11 21:31:35,206 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1000_ffc1a9f27bcd6f2a61ac.png media/images/validation_1000_ffc1a9f27bcd6f2a61ac.png 2023-07-11 21:31:35,207 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1000_3bac780bdad763dfe748.png media/images/validation_1000_3bac780bdad763dfe748.png 2023-07-11 21:31:35,207 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1000_3df8df281127a97b5d1a.png media/images/validation_1000_3df8df281127a97b5d1a.png 2023-07-11 21:31:35,207 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1000_fe76546257030d738699.png media/images/validation_1000_fe76546257030d738699.png 2023-07-11 21:31:35,207 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1008_386a62fe7e44ee995bbb.png media/images/validation_1008_386a62fe7e44ee995bbb.png 2023-07-11 21:31:35,207 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1008_8bde6a294d23990b588f.png media/images/validation_1008_8bde6a294d23990b588f.png 2023-07-11 21:31:35,207 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1008_808168a3292399e3b0a3.png media/images/validation_1008_808168a3292399e3b0a3.png 2023-07-11 21:31:35,207 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1008_2e83a2626ff883d16244.png media/images/validation_1008_2e83a2626ff883d16244.png 2023-07-11 21:31:35,207 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1016_49858d5caa724841d560.png media/images/validation_1016_49858d5caa724841d560.png 2023-07-11 21:31:35,207 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1016_b2e26ce825c90767327d.png media/images/validation_1016_b2e26ce825c90767327d.png 2023-07-11 21:31:35,208 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1016_30d94ad44f6b1b049026.png media/images/validation_1016_30d94ad44f6b1b049026.png 2023-07-11 21:31:35,208 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1016_219d5bff72b5592df3b9.png media/images/validation_1016_219d5bff72b5592df3b9.png 2023-07-11 21:31:35,208 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1024_06f67fa746d45752b1c0.png media/images/validation_1024_06f67fa746d45752b1c0.png 2023-07-11 21:31:35,208 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1024_c8826549b51bc824f1a7.png media/images/validation_1024_c8826549b51bc824f1a7.png 2023-07-11 21:31:35,208 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1024_de6a26e8e064b3d4c07b.png media/images/validation_1024_de6a26e8e064b3d4c07b.png 2023-07-11 21:31:35,208 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1024_78be3e36643b4f4e5aeb.png media/images/validation_1024_78be3e36643b4f4e5aeb.png 2023-07-11 21:31:35,208 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1032_4440c94c8c5b21619372.png media/images/validation_1032_4440c94c8c5b21619372.png 2023-07-11 21:31:35,208 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1032_e0b5d6b0134b15f1acc9.png media/images/validation_1032_e0b5d6b0134b15f1acc9.png 2023-07-11 21:31:35,209 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1032_fa3287622e2f185b3f97.png media/images/validation_1032_fa3287622e2f185b3f97.png 2023-07-11 21:31:35,209 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1032_f343983400b249dec4cb.png media/images/validation_1032_f343983400b249dec4cb.png 2023-07-11 21:31:35,209 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1040_a3486a6a586a730d4368.png media/images/validation_1040_a3486a6a586a730d4368.png 2023-07-11 21:31:35,209 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1040_0a890667226ffaf5b704.png media/images/validation_1040_0a890667226ffaf5b704.png 2023-07-11 21:31:35,209 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1040_6fd15f77048c66e45ab2.png media/images/validation_1040_6fd15f77048c66e45ab2.png 2023-07-11 21:31:35,209 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1040_d91b47aa5350d4e8bc34.png media/images/validation_1040_d91b47aa5350d4e8bc34.png 2023-07-11 21:31:35,210 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1048_f14ff48f5c6ee9389577.png media/images/validation_1048_f14ff48f5c6ee9389577.png 2023-07-11 21:31:35,210 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1048_04bf65c9392f974c75b8.png media/images/validation_1048_04bf65c9392f974c75b8.png 2023-07-11 21:31:35,210 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1048_bd23a5c1898f8da42d93.png media/images/validation_1048_bd23a5c1898f8da42d93.png 2023-07-11 21:31:35,210 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1048_8c9f97026c019506833a.png media/images/validation_1048_8c9f97026c019506833a.png 2023-07-11 21:31:35,210 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1056_30c717b2b7e4a916ae89.png media/images/validation_1056_30c717b2b7e4a916ae89.png 2023-07-11 21:31:35,210 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1056_1479ab3105077bd77424.png media/images/validation_1056_1479ab3105077bd77424.png 2023-07-11 21:31:35,211 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1056_d16d0398bcc7dbf04bbf.png media/images/validation_1056_d16d0398bcc7dbf04bbf.png 2023-07-11 21:31:35,211 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1056_0ac8c42ee2433dffe939.png media/images/validation_1056_0ac8c42ee2433dffe939.png 2023-07-11 21:31:35,211 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1064_ae3016b396ea28947987.png media/images/validation_1064_ae3016b396ea28947987.png 2023-07-11 21:31:35,211 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1064_b69fc5a0e25eb7503924.png media/images/validation_1064_b69fc5a0e25eb7503924.png 2023-07-11 21:31:35,211 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1064_e704101016c7c230c717.png media/images/validation_1064_e704101016c7c230c717.png 2023-07-11 21:31:35,211 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1064_1cc5791fea4c65f0fde3.png media/images/validation_1064_1cc5791fea4c65f0fde3.png 2023-07-11 21:31:35,211 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1072_fed73b1b21a38bd14ea8.png media/images/validation_1072_fed73b1b21a38bd14ea8.png 2023-07-11 21:31:35,211 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1072_4118ab19403ed7a6f06d.png media/images/validation_1072_4118ab19403ed7a6f06d.png 2023-07-11 21:31:35,211 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1072_3bd02124d49d0b8ff154.png media/images/validation_1072_3bd02124d49d0b8ff154.png 2023-07-11 21:31:35,212 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1072_41ce97897430def0ca20.png media/images/validation_1072_41ce97897430def0ca20.png 2023-07-11 21:31:35,212 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1080_c32133924b5d95fd238b.png media/images/validation_1080_c32133924b5d95fd238b.png 2023-07-11 21:31:35,212 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1080_5c4fecd9dcdd1e11f95a.png media/images/validation_1080_5c4fecd9dcdd1e11f95a.png 2023-07-11 21:31:35,212 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1080_1d776c3f3b5c17211029.png media/images/validation_1080_1d776c3f3b5c17211029.png 2023-07-11 21:31:35,212 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1080_9db4cb0a67fe9ae5cb6b.png media/images/validation_1080_9db4cb0a67fe9ae5cb6b.png 2023-07-11 21:31:35,212 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1088_68176a9b6a649b7a461d.png media/images/validation_1088_68176a9b6a649b7a461d.png 2023-07-11 21:31:35,212 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1088_cba58a2d69dc3368c094.png media/images/validation_1088_cba58a2d69dc3368c094.png 2023-07-11 21:31:35,212 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1088_1676e030f39d2264623e.png media/images/validation_1088_1676e030f39d2264623e.png 2023-07-11 21:31:35,212 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1088_fd83b25b70ba4257c9b1.png media/images/validation_1088_fd83b25b70ba4257c9b1.png 2023-07-11 21:31:35,212 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1096_bd01b42ca6177ea47c5a.png media/images/validation_1096_bd01b42ca6177ea47c5a.png 2023-07-11 21:31:35,213 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1096_333e99e73a03c30fe42d.png media/images/validation_1096_333e99e73a03c30fe42d.png 2023-07-11 21:31:35,213 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1096_befbd5340fa0acbaf37e.png media/images/validation_1096_befbd5340fa0acbaf37e.png 2023-07-11 21:31:35,213 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1096_c88fd5a40a05f4bac5c8.png media/images/validation_1096_c88fd5a40a05f4bac5c8.png 2023-07-11 21:31:35,213 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1104_bee308209e779d45bd8b.png media/images/validation_1104_bee308209e779d45bd8b.png 2023-07-11 21:31:35,219 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1104_1f83a28d25fff40729e7.png media/images/validation_1104_1f83a28d25fff40729e7.png 2023-07-11 21:31:35,219 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1104_d8dc6a37b8589618b062.png media/images/validation_1104_d8dc6a37b8589618b062.png 2023-07-11 21:31:35,221 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1104_637101446275b24ba9b1.png media/images/validation_1104_637101446275b24ba9b1.png 2023-07-11 21:31:35,221 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1112_9e12455b1c6a5c5317bb.png media/images/validation_1112_9e12455b1c6a5c5317bb.png 2023-07-11 21:31:35,221 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1112_535e129af2802072858a.png media/images/validation_1112_535e129af2802072858a.png 2023-07-11 21:31:35,221 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1112_e455625d1982cc47ad44.png media/images/validation_1112_e455625d1982cc47ad44.png 2023-07-11 21:31:35,221 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1112_fb29bfacf919ea5e7503.png media/images/validation_1112_fb29bfacf919ea5e7503.png 2023-07-11 21:31:35,221 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1120_d29b97b60ac55d690e1b.png media/images/validation_1120_d29b97b60ac55d690e1b.png 2023-07-11 21:31:35,221 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1120_09953f69ca2d54775598.png media/images/validation_1120_09953f69ca2d54775598.png 2023-07-11 21:31:35,222 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1120_c92093d2b13b886281aa.png media/images/validation_1120_c92093d2b13b886281aa.png 2023-07-11 21:31:35,222 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1120_7383de8551e064f6ad43.png media/images/validation_1120_7383de8551e064f6ad43.png 2023-07-11 21:31:35,222 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1128_3302c80927e3dde2eb3d.png media/images/validation_1128_3302c80927e3dde2eb3d.png 2023-07-11 21:31:35,222 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1128_d8f656ea71301403f7bf.png media/images/validation_1128_d8f656ea71301403f7bf.png 2023-07-11 21:31:35,222 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1128_9459ebef2c19962e4b34.png media/images/validation_1128_9459ebef2c19962e4b34.png 2023-07-11 21:31:35,222 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1128_0f91e55a8a98dd2e8ef8.png media/images/validation_1128_0f91e55a8a98dd2e8ef8.png 2023-07-11 21:31:35,223 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1136_7271d56ca24970e2e44b.png media/images/validation_1136_7271d56ca24970e2e44b.png 2023-07-11 21:31:35,223 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1136_6b3d6e01567109ccdd4f.png media/images/validation_1136_6b3d6e01567109ccdd4f.png 2023-07-11 21:31:35,223 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1136_1b0f4331091bfdab39b6.png media/images/validation_1136_1b0f4331091bfdab39b6.png 2023-07-11 21:31:35,223 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1136_ed5070884043b0e9388b.png media/images/validation_1136_ed5070884043b0e9388b.png 2023-07-11 21:31:35,223 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1144_7debe473f9ee916b15a7.png media/images/validation_1144_7debe473f9ee916b15a7.png 2023-07-11 21:31:35,223 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1144_aeed065d00db7ef520a7.png media/images/validation_1144_aeed065d00db7ef520a7.png 2023-07-11 21:31:35,223 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1144_11f4342cbfa59cc9ac0e.png media/images/validation_1144_11f4342cbfa59cc9ac0e.png 2023-07-11 21:31:35,223 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1144_0f579e23872ee306cab9.png media/images/validation_1144_0f579e23872ee306cab9.png 2023-07-11 21:31:35,223 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1152_75f582b863cdbbe4a504.png media/images/validation_1152_75f582b863cdbbe4a504.png 2023-07-11 21:31:35,223 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1152_ac6be0e73608e31f23e9.png media/images/validation_1152_ac6be0e73608e31f23e9.png 2023-07-11 21:31:35,223 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1152_f44e05a7e3e21014ac08.png media/images/validation_1152_f44e05a7e3e21014ac08.png 2023-07-11 21:31:35,224 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1152_9ee0451c0db170f16218.png media/images/validation_1152_9ee0451c0db170f16218.png 2023-07-11 21:31:35,224 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1160_d1c53487db656ecbc8a8.png media/images/validation_1160_d1c53487db656ecbc8a8.png 2023-07-11 21:31:35,224 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1160_b5862260f9dfb527cb9e.png media/images/validation_1160_b5862260f9dfb527cb9e.png 2023-07-11 21:31:35,224 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1160_2f1ca38ee1b0c5d85255.png media/images/validation_1160_2f1ca38ee1b0c5d85255.png 2023-07-11 21:31:35,224 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1160_bde0a03bf1ad8419d9c3.png media/images/validation_1160_bde0a03bf1ad8419d9c3.png 2023-07-11 21:31:35,225 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1168_7a5ab95838f1b5896fdb.png media/images/validation_1168_7a5ab95838f1b5896fdb.png 2023-07-11 21:31:35,225 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1168_e6f6e202cb1d86be97bc.png media/images/validation_1168_e6f6e202cb1d86be97bc.png 2023-07-11 21:31:35,225 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1168_083b57a36fb9b48aa915.png media/images/validation_1168_083b57a36fb9b48aa915.png 2023-07-11 21:31:35,225 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1168_d980d83f952d5c30a538.png media/images/validation_1168_d980d83f952d5c30a538.png 2023-07-11 21:31:35,225 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1176_b76e250604e885c4bd55.png media/images/validation_1176_b76e250604e885c4bd55.png 2023-07-11 21:31:35,225 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1176_d028389a725dd737ece1.png media/images/validation_1176_d028389a725dd737ece1.png 2023-07-11 21:31:35,225 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1176_df0c9ae2823ce7d96461.png media/images/validation_1176_df0c9ae2823ce7d96461.png 2023-07-11 21:31:35,225 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1176_e97469ade650ac1ab331.png media/images/validation_1176_e97469ade650ac1ab331.png 2023-07-11 21:31:35,225 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1184_974d5757846f0694ffb4.png media/images/validation_1184_974d5757846f0694ffb4.png 2023-07-11 21:31:35,225 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1184_9984128afeb104db29a1.png media/images/validation_1184_9984128afeb104db29a1.png 2023-07-11 21:31:35,225 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1184_09da65422971f700e437.png media/images/validation_1184_09da65422971f700e437.png 2023-07-11 21:31:35,226 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1184_05a4d2cabac98cd4d1d4.png media/images/validation_1184_05a4d2cabac98cd4d1d4.png 2023-07-11 21:31:35,226 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1192_14bd1888a41987672706.png media/images/validation_1192_14bd1888a41987672706.png 2023-07-11 21:31:35,226 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1192_0b8f02a4056f08fbfc01.png media/images/validation_1192_0b8f02a4056f08fbfc01.png 2023-07-11 21:31:35,226 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1192_62796f1f83e7ccdc060a.png media/images/validation_1192_62796f1f83e7ccdc060a.png 2023-07-11 21:31:35,226 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1192_d7bfbf7aea35b7bed028.png media/images/validation_1192_d7bfbf7aea35b7bed028.png 2023-07-11 21:31:35,226 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1200_16f4a0c9ef346f446c13.png media/images/validation_1200_16f4a0c9ef346f446c13.png 2023-07-11 21:31:35,226 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1200_e93ddab6d31d607831c0.png media/images/validation_1200_e93ddab6d31d607831c0.png 2023-07-11 21:31:35,226 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1200_edcc4e9f600fa42e7eff.png media/images/validation_1200_edcc4e9f600fa42e7eff.png 2023-07-11 21:31:35,226 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1200_acc1bde7ddec3fe3577c.png media/images/validation_1200_acc1bde7ddec3fe3577c.png 2023-07-11 21:31:35,226 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1208_9fe00c3f196682ce8aed.png media/images/validation_1208_9fe00c3f196682ce8aed.png 2023-07-11 21:31:35,227 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1208_2403b0bdec188a248ed1.png media/images/validation_1208_2403b0bdec188a248ed1.png 2023-07-11 21:31:35,227 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1208_595eef34e4d21c6eb4a9.png media/images/validation_1208_595eef34e4d21c6eb4a9.png 2023-07-11 21:31:35,227 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1208_0a244457848c676d98e6.png media/images/validation_1208_0a244457848c676d98e6.png 2023-07-11 21:31:35,227 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1216_71a11268d66c7c3c3964.png media/images/validation_1216_71a11268d66c7c3c3964.png 2023-07-11 21:31:35,227 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1216_ce73942bbae6d5990516.png media/images/validation_1216_ce73942bbae6d5990516.png 2023-07-11 21:31:35,227 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1216_6f69187eb5366064d111.png media/images/validation_1216_6f69187eb5366064d111.png 2023-07-11 21:31:35,229 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1216_0ce2aa47670fb06b9041.png media/images/validation_1216_0ce2aa47670fb06b9041.png 2023-07-11 21:31:35,230 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1224_e37d195d179a05497868.png media/images/validation_1224_e37d195d179a05497868.png 2023-07-11 21:31:35,231 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1224_39b1e250a34221277567.png media/images/validation_1224_39b1e250a34221277567.png 2023-07-11 21:31:35,231 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1224_c5fa5e45c4aa753c17c9.png media/images/validation_1224_c5fa5e45c4aa753c17c9.png 2023-07-11 21:31:35,231 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1224_2628069d13a5bea9e3df.png media/images/validation_1224_2628069d13a5bea9e3df.png 2023-07-11 21:31:35,231 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1232_4c55aad4840c63f78577.png media/images/validation_1232_4c55aad4840c63f78577.png 2023-07-11 21:31:35,231 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1232_d39d5a329f94acd4f6a4.png media/images/validation_1232_d39d5a329f94acd4f6a4.png 2023-07-11 21:31:35,231 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1232_e6497da0525aea382fbe.png media/images/validation_1232_e6497da0525aea382fbe.png 2023-07-11 21:31:35,231 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1232_87e2c345946d0460c998.png media/images/validation_1232_87e2c345946d0460c998.png 2023-07-11 21:31:35,232 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1240_9502bd66906821a48a2f.png media/images/validation_1240_9502bd66906821a48a2f.png 2023-07-11 21:31:35,232 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1240_c1ba7526a4470eb03bb6.png media/images/validation_1240_c1ba7526a4470eb03bb6.png 2023-07-11 21:31:35,232 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1240_8441de0e32308471748a.png media/images/validation_1240_8441de0e32308471748a.png 2023-07-11 21:31:35,232 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1240_4a6eef5fa6860d20cbc1.png media/images/validation_1240_4a6eef5fa6860d20cbc1.png 2023-07-11 21:31:35,232 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1248_d2b71281b721da5f3901.png media/images/validation_1248_d2b71281b721da5f3901.png 2023-07-11 21:31:35,232 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1248_d8552d083a3c70763db2.png media/images/validation_1248_d8552d083a3c70763db2.png 2023-07-11 21:31:35,232 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1248_f192f7502e6f7f5405e5.png media/images/validation_1248_f192f7502e6f7f5405e5.png 2023-07-11 21:31:35,232 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1248_5c1a4d1cff1b24bda178.png media/images/validation_1248_5c1a4d1cff1b24bda178.png 2023-07-11 21:31:35,232 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1256_910aaa0aeb996f2bff52.png media/images/validation_1256_910aaa0aeb996f2bff52.png 2023-07-11 21:31:35,232 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1256_6eaa59f4c776007b4577.png media/images/validation_1256_6eaa59f4c776007b4577.png 2023-07-11 21:31:35,233 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1256_008ae0ebd6c28b34501c.png media/images/validation_1256_008ae0ebd6c28b34501c.png 2023-07-11 21:31:35,233 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1256_4d5909a879197a9d6947.png media/images/validation_1256_4d5909a879197a9d6947.png 2023-07-11 21:31:35,233 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1264_a34f676857c79531f476.png media/images/validation_1264_a34f676857c79531f476.png 2023-07-11 21:31:35,233 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1264_12e521ee796cd4abe6b4.png media/images/validation_1264_12e521ee796cd4abe6b4.png 2023-07-11 21:31:35,233 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1264_41d26f1ebd22f8010867.png media/images/validation_1264_41d26f1ebd22f8010867.png 2023-07-11 21:31:35,233 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1264_3c294db4afe2a101d5c2.png media/images/validation_1264_3c294db4afe2a101d5c2.png 2023-07-11 21:31:35,233 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1272_1d9217684c014aa87e19.png media/images/validation_1272_1d9217684c014aa87e19.png 2023-07-11 21:31:35,233 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1272_217b42b3f8761d16b428.png media/images/validation_1272_217b42b3f8761d16b428.png 2023-07-11 21:31:35,233 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1272_0fe5a4f175505061d59f.png media/images/validation_1272_0fe5a4f175505061d59f.png 2023-07-11 21:31:35,233 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1272_dd42d9becd27cc9d54fe.png media/images/validation_1272_dd42d9becd27cc9d54fe.png 2023-07-11 21:31:35,233 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1280_5c8b2ecfb16916c0242c.png media/images/validation_1280_5c8b2ecfb16916c0242c.png 2023-07-11 21:31:35,234 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1280_d62f57953ca77cb73b3f.png media/images/validation_1280_d62f57953ca77cb73b3f.png 2023-07-11 21:31:35,234 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1280_1cc9b7ef9b70d011ac71.png media/images/validation_1280_1cc9b7ef9b70d011ac71.png 2023-07-11 21:31:35,234 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1280_f43945339b235e0f01c6.png media/images/validation_1280_f43945339b235e0f01c6.png 2023-07-11 21:31:35,234 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1288_ce6487d4a69d3d6621e0.png media/images/validation_1288_ce6487d4a69d3d6621e0.png 2023-07-11 21:31:35,234 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1288_401eb7d8a6e899c795fc.png media/images/validation_1288_401eb7d8a6e899c795fc.png 2023-07-11 21:31:35,234 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1288_477cc6da95a873dd86c6.png media/images/validation_1288_477cc6da95a873dd86c6.png 2023-07-11 21:31:35,234 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1288_d7f23e91466de6777b6a.png media/images/validation_1288_d7f23e91466de6777b6a.png 2023-07-11 21:31:35,234 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1296_db936d1e5ef08167a9b9.png media/images/validation_1296_db936d1e5ef08167a9b9.png 2023-07-11 21:31:35,234 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1296_42e08916c70b880ffc82.png media/images/validation_1296_42e08916c70b880ffc82.png 2023-07-11 21:31:35,234 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1296_2a52a9d5fd9d9e666456.png media/images/validation_1296_2a52a9d5fd9d9e666456.png 2023-07-11 21:31:35,234 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1296_23cc3b9e181168d6ed71.png media/images/validation_1296_23cc3b9e181168d6ed71.png 2023-07-11 21:31:35,235 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1304_5455e556b43dc90874e7.png media/images/validation_1304_5455e556b43dc90874e7.png 2023-07-11 21:31:35,235 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1304_bb39219de51d2ce5bad3.png media/images/validation_1304_bb39219de51d2ce5bad3.png 2023-07-11 21:31:35,235 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1304_500c611fcafe0a66ed45.png media/images/validation_1304_500c611fcafe0a66ed45.png 2023-07-11 21:31:35,235 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1304_1d25ec0bcdc8cc75bf15.png media/images/validation_1304_1d25ec0bcdc8cc75bf15.png 2023-07-11 21:31:35,235 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1312_8123dc447f99405e7c54.png media/images/validation_1312_8123dc447f99405e7c54.png 2023-07-11 21:31:35,235 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1312_3ec5f179893340066c13.png media/images/validation_1312_3ec5f179893340066c13.png 2023-07-11 21:31:35,235 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1312_c4b40ef60ee62f599312.png media/images/validation_1312_c4b40ef60ee62f599312.png 2023-07-11 21:31:35,235 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1312_2f48e342607d482d7214.png media/images/validation_1312_2f48e342607d482d7214.png 2023-07-11 21:31:35,235 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1320_9444d2a3d4d06dc520ad.png media/images/validation_1320_9444d2a3d4d06dc520ad.png 2023-07-11 21:31:35,236 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1320_808df0a2d3159d683778.png media/images/validation_1320_808df0a2d3159d683778.png 2023-07-11 21:31:35,236 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1320_8cc1e0a2f932c625950d.png media/images/validation_1320_8cc1e0a2f932c625950d.png 2023-07-11 21:31:35,236 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1320_bc6cc209488d5a0dd512.png media/images/validation_1320_bc6cc209488d5a0dd512.png 2023-07-11 21:31:35,236 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1328_d5ddf8aec98ceaf31a6d.png media/images/validation_1328_d5ddf8aec98ceaf31a6d.png 2023-07-11 21:31:35,236 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1328_2a8e50b9a2f0ea3c709b.png media/images/validation_1328_2a8e50b9a2f0ea3c709b.png 2023-07-11 21:31:35,236 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1328_3644a270a89ab0e7f320.png media/images/validation_1328_3644a270a89ab0e7f320.png 2023-07-11 21:31:35,236 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1328_66a0e97f0510c0041e71.png media/images/validation_1328_66a0e97f0510c0041e71.png 2023-07-11 21:31:35,236 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1336_1e36f4db41c7e9a383a2.png media/images/validation_1336_1e36f4db41c7e9a383a2.png 2023-07-11 21:31:35,236 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1336_bd08d38f68d5827ddd30.png media/images/validation_1336_bd08d38f68d5827ddd30.png 2023-07-11 21:31:35,237 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1336_1c736decbf969e547f0e.png media/images/validation_1336_1c736decbf969e547f0e.png 2023-07-11 21:31:35,237 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1336_5c456d466b0c671052e3.png media/images/validation_1336_5c456d466b0c671052e3.png 2023-07-11 21:31:35,237 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1344_fdc245de4607a1ec4488.png media/images/validation_1344_fdc245de4607a1ec4488.png 2023-07-11 21:31:35,237 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1344_365a301a6ed3b1ea3869.png media/images/validation_1344_365a301a6ed3b1ea3869.png 2023-07-11 21:31:35,237 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1344_48edd71e0c11119f6254.png media/images/validation_1344_48edd71e0c11119f6254.png 2023-07-11 21:31:35,237 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1344_1b37e5405be2f6007744.png media/images/validation_1344_1b37e5405be2f6007744.png 2023-07-11 21:31:35,237 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1352_4092798bf460bac8f7a8.png media/images/validation_1352_4092798bf460bac8f7a8.png 2023-07-11 21:31:35,237 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1352_8fd1600004e6bf64927d.png media/images/validation_1352_8fd1600004e6bf64927d.png 2023-07-11 21:31:35,237 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1352_eca533871e5a572dfd60.png media/images/validation_1352_eca533871e5a572dfd60.png 2023-07-11 21:31:35,237 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1352_9573be96ed0d456f9e97.png media/images/validation_1352_9573be96ed0d456f9e97.png 2023-07-11 21:31:35,237 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1360_61604253d2d4fc950a9b.png media/images/validation_1360_61604253d2d4fc950a9b.png 2023-07-11 21:31:35,238 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1360_1346866ddc9966bbc550.png media/images/validation_1360_1346866ddc9966bbc550.png 2023-07-11 21:31:35,238 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1360_905182c9f1a3424d43d9.png media/images/validation_1360_905182c9f1a3424d43d9.png 2023-07-11 21:31:35,240 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1360_130544406f3b3910ce09.png media/images/validation_1360_130544406f3b3910ce09.png 2023-07-11 21:31:35,241 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1368_26163caf1003b87c0917.png media/images/validation_1368_26163caf1003b87c0917.png 2023-07-11 21:31:35,241 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1368_ed8be6871a61d2c59c72.png media/images/validation_1368_ed8be6871a61d2c59c72.png 2023-07-11 21:31:35,241 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1368_8ccd1f2533b3ff46050a.png media/images/validation_1368_8ccd1f2533b3ff46050a.png 2023-07-11 21:31:35,241 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1368_c0a22e845a09ff0cd3c5.png media/images/validation_1368_c0a22e845a09ff0cd3c5.png 2023-07-11 21:31:35,241 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1376_9731a8706e6ed0046c32.png media/images/validation_1376_9731a8706e6ed0046c32.png 2023-07-11 21:31:35,241 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1376_a8c45d7ccc7b5c5f5a87.png media/images/validation_1376_a8c45d7ccc7b5c5f5a87.png 2023-07-11 21:31:35,241 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1376_1bed837a55c2e8de98f6.png media/images/validation_1376_1bed837a55c2e8de98f6.png 2023-07-11 21:31:35,241 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1376_a1dbd4bbb6ca33baecc1.png media/images/validation_1376_a1dbd4bbb6ca33baecc1.png 2023-07-11 21:31:35,241 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1384_52b65704857359088b8a.png media/images/validation_1384_52b65704857359088b8a.png 2023-07-11 21:31:35,241 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1384_ca0a6b436b4b86c17ede.png media/images/validation_1384_ca0a6b436b4b86c17ede.png 2023-07-11 21:31:35,241 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1384_a0a8d6a6682689911e9c.png media/images/validation_1384_a0a8d6a6682689911e9c.png 2023-07-11 21:31:35,241 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1384_91c90db7ff100b331d46.png media/images/validation_1384_91c90db7ff100b331d46.png 2023-07-11 21:31:35,242 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1392_a4ad4ffc5207f63facbc.png media/images/validation_1392_a4ad4ffc5207f63facbc.png 2023-07-11 21:31:35,243 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1392_eca6ba6691c330e7631e.png media/images/validation_1392_eca6ba6691c330e7631e.png 2023-07-11 21:31:35,243 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1392_ca30155883656ac69b8a.png media/images/validation_1392_ca30155883656ac69b8a.png 2023-07-11 21:31:35,243 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1392_f0607d0b692156045759.png media/images/validation_1392_f0607d0b692156045759.png 2023-07-11 21:31:35,243 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1400_4c95c187c0312a8c87f2.png media/images/validation_1400_4c95c187c0312a8c87f2.png 2023-07-11 21:31:35,243 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1400_a6f4367b333bbbefc831.png media/images/validation_1400_a6f4367b333bbbefc831.png 2023-07-11 21:31:35,243 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1400_8023481f74cd92744fdd.png media/images/validation_1400_8023481f74cd92744fdd.png 2023-07-11 21:31:35,243 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1400_5bf1b64dcf28d3495823.png media/images/validation_1400_5bf1b64dcf28d3495823.png 2023-07-11 21:31:35,243 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1408_64af9f65d337fc4f7fc0.png media/images/validation_1408_64af9f65d337fc4f7fc0.png 2023-07-11 21:31:35,244 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1408_a9dee32d07729bbe5ab6.png media/images/validation_1408_a9dee32d07729bbe5ab6.png 2023-07-11 21:31:35,244 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1408_673f682e941b2195ef5f.png media/images/validation_1408_673f682e941b2195ef5f.png 2023-07-11 21:31:35,244 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1408_d9062fb94fef4cfadbd2.png media/images/validation_1408_d9062fb94fef4cfadbd2.png 2023-07-11 21:31:35,244 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1416_42fdf771bb26d71377ce.png media/images/validation_1416_42fdf771bb26d71377ce.png 2023-07-11 21:31:35,244 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1416_3aead0846726e7166380.png media/images/validation_1416_3aead0846726e7166380.png 2023-07-11 21:31:35,244 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1416_8e95d8013f7d1cb53ee7.png media/images/validation_1416_8e95d8013f7d1cb53ee7.png 2023-07-11 21:31:35,245 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1416_9a51711877bf807947e8.png media/images/validation_1416_9a51711877bf807947e8.png 2023-07-11 21:31:35,245 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1424_7ce5c6417cd370810ef9.png media/images/validation_1424_7ce5c6417cd370810ef9.png 2023-07-11 21:31:35,245 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1424_53e42d50be50609b55e6.png media/images/validation_1424_53e42d50be50609b55e6.png 2023-07-11 21:31:35,245 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1424_bc80783a000c38fd4ffb.png media/images/validation_1424_bc80783a000c38fd4ffb.png 2023-07-11 21:31:35,245 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1424_ac49a027c2b42beeb91d.png media/images/validation_1424_ac49a027c2b42beeb91d.png 2023-07-11 21:31:35,245 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1432_c09b2e9e85aa46b447a1.png media/images/validation_1432_c09b2e9e85aa46b447a1.png 2023-07-11 21:31:35,246 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1432_bd3dcb8ce785525d5487.png media/images/validation_1432_bd3dcb8ce785525d5487.png 2023-07-11 21:31:35,246 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1432_a5ace0cb3df2538fb498.png media/images/validation_1432_a5ace0cb3df2538fb498.png 2023-07-11 21:31:35,246 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1432_43ab5d4df5c3f9be760e.png media/images/validation_1432_43ab5d4df5c3f9be760e.png 2023-07-11 21:31:35,246 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1440_c3be6299a9293bc4d02e.png media/images/validation_1440_c3be6299a9293bc4d02e.png 2023-07-11 21:31:35,246 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1440_e6f44d029048b1c1f306.png media/images/validation_1440_e6f44d029048b1c1f306.png 2023-07-11 21:31:35,246 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1440_f4b00c37b355d60cde54.png media/images/validation_1440_f4b00c37b355d60cde54.png 2023-07-11 21:31:35,246 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1440_36f9aa3dad42c2857b92.png media/images/validation_1440_36f9aa3dad42c2857b92.png 2023-07-11 21:31:35,246 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1448_fa3457d0fce8bba4f078.png media/images/validation_1448_fa3457d0fce8bba4f078.png 2023-07-11 21:31:35,246 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1448_4314f1796f83f2dc2195.png media/images/validation_1448_4314f1796f83f2dc2195.png 2023-07-11 21:31:35,246 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1448_74b0779cd28b2472d148.png media/images/validation_1448_74b0779cd28b2472d148.png 2023-07-11 21:31:35,246 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1448_268b0859bbaca6723e6a.png media/images/validation_1448_268b0859bbaca6723e6a.png 2023-07-11 21:31:35,246 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1456_1ea2e8f00cd9d7016b4c.png media/images/validation_1456_1ea2e8f00cd9d7016b4c.png 2023-07-11 21:31:35,247 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1456_2cc115b8258f18162f1a.png media/images/validation_1456_2cc115b8258f18162f1a.png 2023-07-11 21:31:35,247 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1456_d34a21e6b56035864f15.png media/images/validation_1456_d34a21e6b56035864f15.png 2023-07-11 21:31:35,247 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1456_79e985401ee07adaf747.png media/images/validation_1456_79e985401ee07adaf747.png 2023-07-11 21:31:35,247 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1464_3e948f3d4a9690e3e832.png media/images/validation_1464_3e948f3d4a9690e3e832.png 2023-07-11 21:31:35,247 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1464_ef2794b66f8354aeb131.png media/images/validation_1464_ef2794b66f8354aeb131.png 2023-07-11 21:31:35,247 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1464_19080724f5041cc8877a.png media/images/validation_1464_19080724f5041cc8877a.png 2023-07-11 21:31:35,247 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1464_0ca40d1889e8c981f83b.png media/images/validation_1464_0ca40d1889e8c981f83b.png 2023-07-11 21:31:35,247 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1472_e88bd6f6b5daeddb588f.png media/images/validation_1472_e88bd6f6b5daeddb588f.png 2023-07-11 21:31:35,247 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1472_f3df87bdd58ea75008e9.png media/images/validation_1472_f3df87bdd58ea75008e9.png 2023-07-11 21:31:35,247 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1472_ec9e711d039b6031a67b.png media/images/validation_1472_ec9e711d039b6031a67b.png 2023-07-11 21:31:35,247 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1472_97171e5e266bd9576202.png media/images/validation_1472_97171e5e266bd9576202.png 2023-07-11 21:31:35,248 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1480_c13f5759d561a664b48e.png media/images/validation_1480_c13f5759d561a664b48e.png 2023-07-11 21:31:35,248 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1480_03ac52985174dcc37f37.png media/images/validation_1480_03ac52985174dcc37f37.png 2023-07-11 21:31:35,248 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1480_b5501bb70ba4c344c2a7.png media/images/validation_1480_b5501bb70ba4c344c2a7.png 2023-07-11 21:31:35,248 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1480_389503b866d34e567119.png media/images/validation_1480_389503b866d34e567119.png 2023-07-11 21:31:35,248 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1488_d8f7bbf4bace619f94c8.png media/images/validation_1488_d8f7bbf4bace619f94c8.png 2023-07-11 21:31:35,248 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1488_bad17279d6c78d388689.png media/images/validation_1488_bad17279d6c78d388689.png 2023-07-11 21:31:35,249 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1488_945435b42cdef1f698ac.png media/images/validation_1488_945435b42cdef1f698ac.png 2023-07-11 21:31:35,249 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1488_7a327cfc0f0d14000d66.png media/images/validation_1488_7a327cfc0f0d14000d66.png 2023-07-11 21:31:35,249 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1496_861caf343730e243e96c.png media/images/validation_1496_861caf343730e243e96c.png 2023-07-11 21:31:35,249 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1496_792b9ac80f3c91021509.png media/images/validation_1496_792b9ac80f3c91021509.png 2023-07-11 21:31:35,249 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1496_5999e1ed8ed0b37baa88.png media/images/validation_1496_5999e1ed8ed0b37baa88.png 2023-07-11 21:31:35,249 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1496_f3cbc6942eee44231d8a.png media/images/validation_1496_f3cbc6942eee44231d8a.png 2023-07-11 21:31:35,249 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1504_1629e05d318559a66a72.png media/images/validation_1504_1629e05d318559a66a72.png 2023-07-11 21:31:35,249 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1504_53154f693252da98516a.png media/images/validation_1504_53154f693252da98516a.png 2023-07-11 21:31:35,250 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1504_fb85857165e5c2b83d35.png media/images/validation_1504_fb85857165e5c2b83d35.png 2023-07-11 21:31:35,250 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1504_b30643dac647697eb3e3.png media/images/validation_1504_b30643dac647697eb3e3.png 2023-07-11 21:31:35,250 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1512_76d0a37e65b519005e16.png media/images/validation_1512_76d0a37e65b519005e16.png 2023-07-11 21:31:35,250 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1512_9628924d5a8c1579e38e.png media/images/validation_1512_9628924d5a8c1579e38e.png 2023-07-11 21:31:35,250 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1512_ac8f638669bb1938fe1e.png media/images/validation_1512_ac8f638669bb1938fe1e.png 2023-07-11 21:31:35,250 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1512_dc90df180ea054ba86a6.png media/images/validation_1512_dc90df180ea054ba86a6.png 2023-07-11 21:31:35,250 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1520_759fd5f584726cd89a08.png media/images/validation_1520_759fd5f584726cd89a08.png 2023-07-11 21:31:35,250 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1520_cbf2c6d45cd68fb8ce0d.png media/images/validation_1520_cbf2c6d45cd68fb8ce0d.png 2023-07-11 21:31:35,250 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1520_da08c97000f72b3ea979.png media/images/validation_1520_da08c97000f72b3ea979.png 2023-07-11 21:31:35,250 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1520_4acf2882ee3fdf5b3a10.png media/images/validation_1520_4acf2882ee3fdf5b3a10.png 2023-07-11 21:31:35,250 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1528_27dd1112c1ee345cd5a7.png media/images/validation_1528_27dd1112c1ee345cd5a7.png 2023-07-11 21:31:35,250 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1528_175b3055df42cae41485.png media/images/validation_1528_175b3055df42cae41485.png 2023-07-11 21:31:35,251 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1528_823ba070d344cad76878.png media/images/validation_1528_823ba070d344cad76878.png 2023-07-11 21:31:35,251 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1528_2e0f5eda4c827c82d01f.png media/images/validation_1528_2e0f5eda4c827c82d01f.png 2023-07-11 21:31:35,251 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1536_08a161065bf6a4856204.png media/images/validation_1536_08a161065bf6a4856204.png 2023-07-11 21:31:35,253 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1536_d45ac8dd546b5c439b82.png media/images/validation_1536_d45ac8dd546b5c439b82.png 2023-07-11 21:31:35,253 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1536_71bad97180edd0fb5ddd.png media/images/validation_1536_71bad97180edd0fb5ddd.png 2023-07-11 21:31:35,254 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1536_3fc0a6e0fd102232715a.png media/images/validation_1536_3fc0a6e0fd102232715a.png 2023-07-11 21:31:35,254 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1544_3c1a45e0401e280d0a21.png media/images/validation_1544_3c1a45e0401e280d0a21.png 2023-07-11 21:31:35,254 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1544_de785629b33cb381b74b.png media/images/validation_1544_de785629b33cb381b74b.png 2023-07-11 21:31:35,254 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1544_f2506e5ffc80bfd166ad.png media/images/validation_1544_f2506e5ffc80bfd166ad.png 2023-07-11 21:31:35,254 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1544_bf2932c9751754389a5c.png media/images/validation_1544_bf2932c9751754389a5c.png 2023-07-11 21:31:35,254 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1552_8d1e832ac023537c101c.png media/images/validation_1552_8d1e832ac023537c101c.png 2023-07-11 21:31:35,254 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1552_75bb106df0919e3df204.png media/images/validation_1552_75bb106df0919e3df204.png 2023-07-11 21:31:35,254 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1552_aa4cc079d3d9191a8770.png media/images/validation_1552_aa4cc079d3d9191a8770.png 2023-07-11 21:31:35,254 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1552_b723e293e89f77fe239f.png media/images/validation_1552_b723e293e89f77fe239f.png 2023-07-11 21:31:35,254 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1560_080450d9e4c3f1b32250.png media/images/validation_1560_080450d9e4c3f1b32250.png 2023-07-11 21:31:35,254 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1560_c97c1d61c8724e14c00b.png media/images/validation_1560_c97c1d61c8724e14c00b.png 2023-07-11 21:31:35,255 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1560_7ff55320d57bf225ae68.png media/images/validation_1560_7ff55320d57bf225ae68.png 2023-07-11 21:31:35,255 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1560_92d3ec020ffb72174fba.png media/images/validation_1560_92d3ec020ffb72174fba.png 2023-07-11 21:31:35,255 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1568_67b6d1876f24f6e8c3e4.png media/images/validation_1568_67b6d1876f24f6e8c3e4.png 2023-07-11 21:31:35,255 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1568_b803421655b7dc310cf8.png media/images/validation_1568_b803421655b7dc310cf8.png 2023-07-11 21:31:35,255 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1568_45dc21f5d3d178d54339.png media/images/validation_1568_45dc21f5d3d178d54339.png 2023-07-11 21:31:35,255 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1568_a041609e0e3c44c0a850.png media/images/validation_1568_a041609e0e3c44c0a850.png 2023-07-11 21:31:35,256 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1576_2104b767ef4f17684879.png media/images/validation_1576_2104b767ef4f17684879.png 2023-07-11 21:31:35,256 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1576_c1fc87dd1314b4adf472.png media/images/validation_1576_c1fc87dd1314b4adf472.png 2023-07-11 21:31:35,256 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1576_4f1e1d5fc289cbc9fa89.png media/images/validation_1576_4f1e1d5fc289cbc9fa89.png 2023-07-11 21:31:35,256 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1576_a69d5aa10180a8853110.png media/images/validation_1576_a69d5aa10180a8853110.png 2023-07-11 21:31:35,256 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1584_fc7f4c75961dd33ba081.png media/images/validation_1584_fc7f4c75961dd33ba081.png 2023-07-11 21:31:35,256 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1584_6d080ebd872af391cc12.png media/images/validation_1584_6d080ebd872af391cc12.png 2023-07-11 21:31:35,256 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1584_5060670cca612a4e440f.png media/images/validation_1584_5060670cca612a4e440f.png 2023-07-11 21:31:35,256 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1584_4047a2d99cc326c5a57b.png media/images/validation_1584_4047a2d99cc326c5a57b.png 2023-07-11 21:31:35,256 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1592_f48579df1612af3504f2.png media/images/validation_1592_f48579df1612af3504f2.png 2023-07-11 21:31:35,256 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1592_97278c134a95be731717.png media/images/validation_1592_97278c134a95be731717.png 2023-07-11 21:31:35,258 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1592_92ac374f16616a3f52d5.png media/images/validation_1592_92ac374f16616a3f52d5.png 2023-07-11 21:31:35,259 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1592_00402f5cbaef44c29659.png media/images/validation_1592_00402f5cbaef44c29659.png 2023-07-11 21:31:35,259 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1600_699d0df245df08e0830f.png media/images/validation_1600_699d0df245df08e0830f.png 2023-07-11 21:31:35,259 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1600_0f6bf7c1ef77ecb39e6c.png media/images/validation_1600_0f6bf7c1ef77ecb39e6c.png 2023-07-11 21:31:35,259 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1600_deb755fc1e7c3773dac4.png media/images/validation_1600_deb755fc1e7c3773dac4.png 2023-07-11 21:31:35,260 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1600_1759c1e65a73f7f13991.png media/images/validation_1600_1759c1e65a73f7f13991.png 2023-07-11 21:31:35,260 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1608_da4f2d17fb83c3abf79b.png media/images/validation_1608_da4f2d17fb83c3abf79b.png 2023-07-11 21:31:35,260 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1608_6d2196a2c690a306717d.png media/images/validation_1608_6d2196a2c690a306717d.png 2023-07-11 21:31:35,260 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1608_7ae5a05f8fc5b0d59741.png media/images/validation_1608_7ae5a05f8fc5b0d59741.png 2023-07-11 21:31:35,260 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1608_47f23ae6bc7899d2f737.png media/images/validation_1608_47f23ae6bc7899d2f737.png 2023-07-11 21:31:35,260 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1616_4162b0a1073cede299c3.png media/images/validation_1616_4162b0a1073cede299c3.png 2023-07-11 21:31:35,260 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1616_23f6c16a47fdbf946082.png media/images/validation_1616_23f6c16a47fdbf946082.png 2023-07-11 21:31:35,260 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1616_08b89fa693ea9f058799.png media/images/validation_1616_08b89fa693ea9f058799.png 2023-07-11 21:31:35,260 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1616_5cc14ce0bb3ca15eff14.png media/images/validation_1616_5cc14ce0bb3ca15eff14.png 2023-07-11 21:31:35,261 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1624_c5c2b67c71ebaa893e3d.png media/images/validation_1624_c5c2b67c71ebaa893e3d.png 2023-07-11 21:31:35,261 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1624_87fd4f364cf3c1a5d743.png media/images/validation_1624_87fd4f364cf3c1a5d743.png 2023-07-11 21:31:35,261 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1624_2d21b8cf80456a450416.png media/images/validation_1624_2d21b8cf80456a450416.png 2023-07-11 21:31:35,261 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1624_f2a4008163e3b0de1c67.png media/images/validation_1624_f2a4008163e3b0de1c67.png 2023-07-11 21:31:35,261 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1632_87c3258c1b9a48214036.png media/images/validation_1632_87c3258c1b9a48214036.png 2023-07-11 21:31:35,261 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1632_4f352d796eb262af149d.png media/images/validation_1632_4f352d796eb262af149d.png 2023-07-11 21:31:35,261 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1632_147308a3bc862cc2ef12.png media/images/validation_1632_147308a3bc862cc2ef12.png 2023-07-11 21:31:35,261 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1632_ea4f6f4c802c31284d78.png media/images/validation_1632_ea4f6f4c802c31284d78.png 2023-07-11 21:31:35,261 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1640_3d4aeb5bd69764009277.png media/images/validation_1640_3d4aeb5bd69764009277.png 2023-07-11 21:31:35,261 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1640_0dd87108f2fea6cda5da.png media/images/validation_1640_0dd87108f2fea6cda5da.png 2023-07-11 21:31:35,262 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1640_134a552e8977908b132a.png media/images/validation_1640_134a552e8977908b132a.png 2023-07-11 21:31:35,262 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1640_41d647d9e8df50b6d062.png media/images/validation_1640_41d647d9e8df50b6d062.png 2023-07-11 21:31:35,262 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1648_ae897ffe1c6081601953.png media/images/validation_1648_ae897ffe1c6081601953.png 2023-07-11 21:31:35,262 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1648_3801cc005fba469011fe.png media/images/validation_1648_3801cc005fba469011fe.png 2023-07-11 21:31:35,262 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1648_cab21230e1e43c4dd2f2.png media/images/validation_1648_cab21230e1e43c4dd2f2.png 2023-07-11 21:31:35,262 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1648_1de3e9f61db08c1fa525.png media/images/validation_1648_1de3e9f61db08c1fa525.png 2023-07-11 21:31:35,262 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1656_8c3aa89e0dc3c0cfd217.png media/images/validation_1656_8c3aa89e0dc3c0cfd217.png 2023-07-11 21:31:35,262 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1656_e4d7864777608c38c24e.png media/images/validation_1656_e4d7864777608c38c24e.png 2023-07-11 21:31:35,262 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1656_e5c0e9634965058135d3.png media/images/validation_1656_e5c0e9634965058135d3.png 2023-07-11 21:31:35,263 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1656_55996a575df11d966c0a.png media/images/validation_1656_55996a575df11d966c0a.png 2023-07-11 21:31:35,263 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1664_66dbe96487807e61c639.png media/images/validation_1664_66dbe96487807e61c639.png 2023-07-11 21:31:35,263 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1664_97cb7ab131b642c3b7bc.png media/images/validation_1664_97cb7ab131b642c3b7bc.png 2023-07-11 21:31:35,263 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1664_2ae67e741780fdc97d63.png media/images/validation_1664_2ae67e741780fdc97d63.png 2023-07-11 21:31:35,263 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1664_ef66c85637afc9c7e435.png media/images/validation_1664_ef66c85637afc9c7e435.png 2023-07-11 21:31:35,263 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1672_f950a91f8f5d756c5764.png media/images/validation_1672_f950a91f8f5d756c5764.png 2023-07-11 21:31:35,263 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1672_379652fb1beafd6c8e09.png media/images/validation_1672_379652fb1beafd6c8e09.png 2023-07-11 21:31:35,263 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1672_02de7ad6917c63f45a9a.png media/images/validation_1672_02de7ad6917c63f45a9a.png 2023-07-11 21:31:35,263 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1672_32cff90f0713a31af381.png media/images/validation_1672_32cff90f0713a31af381.png 2023-07-11 21:31:35,263 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1680_e0d685807d82b1fe3333.png media/images/validation_1680_e0d685807d82b1fe3333.png 2023-07-11 21:31:35,264 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1680_34a40e5cb3d7a0052b50.png media/images/validation_1680_34a40e5cb3d7a0052b50.png 2023-07-11 21:31:35,264 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1680_3d0c633b59f3f6e54d50.png media/images/validation_1680_3d0c633b59f3f6e54d50.png 2023-07-11 21:31:35,264 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1680_3015d5a2cea98cfefb90.png media/images/validation_1680_3015d5a2cea98cfefb90.png 2023-07-11 21:31:35,264 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1688_59c336d7ac9a01cc96ed.png media/images/validation_1688_59c336d7ac9a01cc96ed.png 2023-07-11 21:31:35,264 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1688_78b53a833655e5e7ddfd.png media/images/validation_1688_78b53a833655e5e7ddfd.png 2023-07-11 21:31:35,264 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1688_47be1430323289df7f7b.png media/images/validation_1688_47be1430323289df7f7b.png 2023-07-11 21:31:35,264 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1688_afc0af5c54864aff9b73.png media/images/validation_1688_afc0af5c54864aff9b73.png 2023-07-11 21:31:35,264 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1696_50897643f723faf8a1d8.png media/images/validation_1696_50897643f723faf8a1d8.png 2023-07-11 21:31:35,264 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1696_e4fbe946ca939c5f2907.png media/images/validation_1696_e4fbe946ca939c5f2907.png 2023-07-11 21:31:35,264 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1696_ac0d4f4b95333e8b92f1.png media/images/validation_1696_ac0d4f4b95333e8b92f1.png 2023-07-11 21:31:35,265 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1696_d73227af624327a56f89.png media/images/validation_1696_d73227af624327a56f89.png 2023-07-11 21:31:35,265 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1704_e1dcbbc29cdb423960a2.png media/images/validation_1704_e1dcbbc29cdb423960a2.png 2023-07-11 21:31:35,265 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1704_314bebb79cb782edfd8f.png media/images/validation_1704_314bebb79cb782edfd8f.png 2023-07-11 21:31:35,265 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1704_42a268d779b7667ccc2e.png media/images/validation_1704_42a268d779b7667ccc2e.png 2023-07-11 21:31:35,265 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1704_e0d8a90f9602209761f6.png media/images/validation_1704_e0d8a90f9602209761f6.png 2023-07-11 21:31:35,265 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1712_50df1a6368b684b0eabd.png media/images/validation_1712_50df1a6368b684b0eabd.png 2023-07-11 21:31:35,265 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1712_bb78706228e3f64446a2.png media/images/validation_1712_bb78706228e3f64446a2.png 2023-07-11 21:31:35,265 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1712_27d02e0fde32d79961e2.png media/images/validation_1712_27d02e0fde32d79961e2.png 2023-07-11 21:31:35,266 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1712_b2393282cb27ff5aaf28.png media/images/validation_1712_b2393282cb27ff5aaf28.png 2023-07-11 21:31:35,266 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1720_679437e5e8fa85b1b9fa.png media/images/validation_1720_679437e5e8fa85b1b9fa.png 2023-07-11 21:31:35,266 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1720_973ea290aaf93bd8b3d2.png media/images/validation_1720_973ea290aaf93bd8b3d2.png 2023-07-11 21:31:35,266 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1720_5a01ec75660827b76fe7.png media/images/validation_1720_5a01ec75660827b76fe7.png 2023-07-11 21:31:35,266 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1720_c173e21a79da12ee1883.png media/images/validation_1720_c173e21a79da12ee1883.png 2023-07-11 21:31:35,266 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1728_590c097e334a7ac52f79.png media/images/validation_1728_590c097e334a7ac52f79.png 2023-07-11 21:31:35,266 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1728_f006b6af0fba27faf76e.png media/images/validation_1728_f006b6af0fba27faf76e.png 2023-07-11 21:31:35,266 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1728_89f458f901461e553af6.png media/images/validation_1728_89f458f901461e553af6.png 2023-07-11 21:31:35,266 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1728_61df962c93eb4afff8f0.png media/images/validation_1728_61df962c93eb4afff8f0.png 2023-07-11 21:31:35,266 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1736_23c66688cd6e5a6be908.png media/images/validation_1736_23c66688cd6e5a6be908.png 2023-07-11 21:31:35,267 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1736_81a3ccf88a1ed0a25cef.png media/images/validation_1736_81a3ccf88a1ed0a25cef.png 2023-07-11 21:31:35,267 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1736_1029fe384454cadd60b1.png media/images/validation_1736_1029fe384454cadd60b1.png 2023-07-11 21:31:35,267 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1736_9f5d69e25ec444d8827e.png media/images/validation_1736_9f5d69e25ec444d8827e.png 2023-07-11 21:31:35,267 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1744_d7f69c8955653068af1c.png media/images/validation_1744_d7f69c8955653068af1c.png 2023-07-11 21:31:35,267 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1744_5a12380a065557e23f86.png media/images/validation_1744_5a12380a065557e23f86.png 2023-07-11 21:31:35,267 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1744_9f5c0597dd918b1fd1cd.png media/images/validation_1744_9f5c0597dd918b1fd1cd.png 2023-07-11 21:31:35,267 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1744_ce9d3bc192f9ffe8e69b.png media/images/validation_1744_ce9d3bc192f9ffe8e69b.png 2023-07-11 21:31:35,267 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1752_41340b7ef5a230ba2c1d.png media/images/validation_1752_41340b7ef5a230ba2c1d.png 2023-07-11 21:31:35,267 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1752_2d2c4c4207dded6ac310.png media/images/validation_1752_2d2c4c4207dded6ac310.png 2023-07-11 21:31:35,267 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1752_1fca70034e43e1efd16b.png media/images/validation_1752_1fca70034e43e1efd16b.png 2023-07-11 21:31:35,267 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1752_c0be450d736d315a7f7f.png media/images/validation_1752_c0be450d736d315a7f7f.png 2023-07-11 21:31:35,268 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1760_f9e33a1e2eb7c8bf304f.png media/images/validation_1760_f9e33a1e2eb7c8bf304f.png 2023-07-11 21:31:35,268 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1760_963ce470d918d6ff96d4.png media/images/validation_1760_963ce470d918d6ff96d4.png 2023-07-11 21:31:35,268 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1760_2b0a2a64666aeb10c8d5.png media/images/validation_1760_2b0a2a64666aeb10c8d5.png 2023-07-11 21:31:35,268 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1760_1a916468c5f244822b28.png media/images/validation_1760_1a916468c5f244822b28.png 2023-07-11 21:31:35,268 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1768_2dae9d2ec2b54e62fa2a.png media/images/validation_1768_2dae9d2ec2b54e62fa2a.png 2023-07-11 21:31:35,268 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1768_cfcc80a8ccc0659ac78e.png media/images/validation_1768_cfcc80a8ccc0659ac78e.png 2023-07-11 21:31:35,268 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1768_bfde8140b1fc4f9db23e.png media/images/validation_1768_bfde8140b1fc4f9db23e.png 2023-07-11 21:31:35,268 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1768_bad2b0cfefb3891be2e4.png media/images/validation_1768_bad2b0cfefb3891be2e4.png 2023-07-11 21:31:35,268 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1776_7c1cf0a21ce762d708cc.png media/images/validation_1776_7c1cf0a21ce762d708cc.png 2023-07-11 21:31:35,268 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1776_a6253244e0fcd16b6309.png media/images/validation_1776_a6253244e0fcd16b6309.png 2023-07-11 21:31:35,269 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1776_1dfeb68dada29a861d6d.png media/images/validation_1776_1dfeb68dada29a861d6d.png 2023-07-11 21:31:35,269 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1776_145e5db4ddf1294e74ca.png media/images/validation_1776_145e5db4ddf1294e74ca.png 2023-07-11 21:31:35,269 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1784_2f7a93c89a8e73332a71.png media/images/validation_1784_2f7a93c89a8e73332a71.png 2023-07-11 21:31:35,269 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1784_40c1334312f0eaa4a409.png media/images/validation_1784_40c1334312f0eaa4a409.png 2023-07-11 21:31:35,269 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1784_0b7130c5c030a08cc234.png media/images/validation_1784_0b7130c5c030a08cc234.png 2023-07-11 21:31:35,269 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1784_7e32e1185f32b992ee09.png media/images/validation_1784_7e32e1185f32b992ee09.png 2023-07-11 21:31:35,269 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1792_6c8c3c85979fc0112886.png media/images/validation_1792_6c8c3c85979fc0112886.png 2023-07-11 21:31:35,269 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1792_03a513699d32547cfcce.png media/images/validation_1792_03a513699d32547cfcce.png 2023-07-11 21:31:35,269 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1792_e03fe724b53ff007f509.png media/images/validation_1792_e03fe724b53ff007f509.png 2023-07-11 21:31:35,269 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1792_5119d4d7a2257dd71e9b.png media/images/validation_1792_5119d4d7a2257dd71e9b.png 2023-07-11 21:31:35,269 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1800_29829117e355f84867a9.png media/images/validation_1800_29829117e355f84867a9.png 2023-07-11 21:31:35,270 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1800_1d80862a2e619ee149af.png media/images/validation_1800_1d80862a2e619ee149af.png 2023-07-11 21:31:35,270 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1800_ba06bc64108c60c3d249.png media/images/validation_1800_ba06bc64108c60c3d249.png 2023-07-11 21:31:35,270 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1800_595d783a674b31607abe.png media/images/validation_1800_595d783a674b31607abe.png 2023-07-11 21:31:35,270 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1808_2c08b8408a4ed78c3242.png media/images/validation_1808_2c08b8408a4ed78c3242.png 2023-07-11 21:31:35,270 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1808_4e784e29fddbb72791ca.png media/images/validation_1808_4e784e29fddbb72791ca.png 2023-07-11 21:31:35,270 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1808_60edf5874ce5e54fff8a.png media/images/validation_1808_60edf5874ce5e54fff8a.png 2023-07-11 21:31:35,270 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1808_e42f3a5cff465c0326eb.png media/images/validation_1808_e42f3a5cff465c0326eb.png 2023-07-11 21:31:35,270 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1816_6ce5b89f865c878ca7ce.png media/images/validation_1816_6ce5b89f865c878ca7ce.png 2023-07-11 21:31:35,270 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1816_db34427fe5dc5f158ded.png media/images/validation_1816_db34427fe5dc5f158ded.png 2023-07-11 21:31:35,270 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1816_0b84c7e55854de6abb35.png media/images/validation_1816_0b84c7e55854de6abb35.png 2023-07-11 21:31:35,270 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1816_c0afacd8e639ac88c950.png media/images/validation_1816_c0afacd8e639ac88c950.png 2023-07-11 21:31:35,271 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1824_de127351776d646ede0d.png media/images/validation_1824_de127351776d646ede0d.png 2023-07-11 21:31:35,271 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1824_2018a3ead09df8a1c987.png media/images/validation_1824_2018a3ead09df8a1c987.png 2023-07-11 21:31:35,271 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1824_9cf90d8af441b007efe6.png media/images/validation_1824_9cf90d8af441b007efe6.png 2023-07-11 21:31:35,271 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1824_ffbfc753df3dec5c3208.png media/images/validation_1824_ffbfc753df3dec5c3208.png 2023-07-11 21:31:35,271 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1832_9cd399d776df586c9c02.png media/images/validation_1832_9cd399d776df586c9c02.png 2023-07-11 21:31:35,271 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1832_5e23f48d1d42652a0b10.png media/images/validation_1832_5e23f48d1d42652a0b10.png 2023-07-11 21:31:35,271 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1832_7a8f27b94cbeac050b37.png media/images/validation_1832_7a8f27b94cbeac050b37.png 2023-07-11 21:31:35,271 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1832_1bfea247631943876000.png media/images/validation_1832_1bfea247631943876000.png 2023-07-11 21:31:35,271 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1840_2e537d6ae5539024016c.png media/images/validation_1840_2e537d6ae5539024016c.png 2023-07-11 21:31:35,271 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1840_eacff8e0f39e8da08765.png media/images/validation_1840_eacff8e0f39e8da08765.png 2023-07-11 21:31:35,271 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1840_beb7c5efe82f32f2c89f.png media/images/validation_1840_beb7c5efe82f32f2c89f.png 2023-07-11 21:31:35,272 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1840_4fb89ef66c812201003f.png media/images/validation_1840_4fb89ef66c812201003f.png 2023-07-11 21:31:35,272 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1848_08204171975de6aa75a9.png media/images/validation_1848_08204171975de6aa75a9.png 2023-07-11 21:31:35,272 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1848_98bfc990a3604dddcca6.png media/images/validation_1848_98bfc990a3604dddcca6.png 2023-07-11 21:31:35,272 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1848_d9322c759e78020c14f9.png media/images/validation_1848_d9322c759e78020c14f9.png 2023-07-11 21:31:35,272 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1848_1be533c9b8642ccbd69e.png media/images/validation_1848_1be533c9b8642ccbd69e.png 2023-07-11 21:31:35,272 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1856_fc9e1244d8b7441727e9.png media/images/validation_1856_fc9e1244d8b7441727e9.png 2023-07-11 21:31:35,272 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1856_27b928ab82c5575c233c.png media/images/validation_1856_27b928ab82c5575c233c.png 2023-07-11 21:31:35,272 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1856_acd7add95c524a43efc7.png media/images/validation_1856_acd7add95c524a43efc7.png 2023-07-11 21:31:35,272 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1856_9f028c76839d73eab2ab.png media/images/validation_1856_9f028c76839d73eab2ab.png 2023-07-11 21:31:35,272 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1864_ec496ac094d6a7c53ab6.png media/images/validation_1864_ec496ac094d6a7c53ab6.png 2023-07-11 21:31:35,272 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1864_2cb8e9d19416aa855ae3.png media/images/validation_1864_2cb8e9d19416aa855ae3.png 2023-07-11 21:31:35,273 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1864_bb9b59dfd7940d706691.png media/images/validation_1864_bb9b59dfd7940d706691.png 2023-07-11 21:31:35,273 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1864_3af6cca038957854954d.png media/images/validation_1864_3af6cca038957854954d.png 2023-07-11 21:31:35,273 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1872_cdeb27bcabdc8cb919a1.png media/images/validation_1872_cdeb27bcabdc8cb919a1.png 2023-07-11 21:31:35,273 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1872_65231e5433fd842a9973.png media/images/validation_1872_65231e5433fd842a9973.png 2023-07-11 21:31:35,273 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1872_31e589344f5ef8978c9f.png media/images/validation_1872_31e589344f5ef8978c9f.png 2023-07-11 21:31:35,273 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1872_c36006f7e46b670362b4.png media/images/validation_1872_c36006f7e46b670362b4.png 2023-07-11 21:31:35,273 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1880_59266af13de06f4cd472.png media/images/validation_1880_59266af13de06f4cd472.png 2023-07-11 21:31:35,273 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1880_75325ad11ccd36e3bca1.png media/images/validation_1880_75325ad11ccd36e3bca1.png 2023-07-11 21:31:35,273 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1880_c848d9be75adb5960650.png media/images/validation_1880_c848d9be75adb5960650.png 2023-07-11 21:31:35,273 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1880_16f97c54584e8c176782.png media/images/validation_1880_16f97c54584e8c176782.png 2023-07-11 21:31:35,274 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1888_7b75911ccdb678b3a976.png media/images/validation_1888_7b75911ccdb678b3a976.png 2023-07-11 21:31:35,274 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1888_86d54203ecfaca6291d0.png media/images/validation_1888_86d54203ecfaca6291d0.png 2023-07-11 21:31:35,274 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1888_8ae9cfb1af9ccf7c1a1d.png media/images/validation_1888_8ae9cfb1af9ccf7c1a1d.png 2023-07-11 21:31:35,274 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1888_3adb0b07a95c158b84ad.png media/images/validation_1888_3adb0b07a95c158b84ad.png 2023-07-11 21:31:35,274 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1896_a000f6436ad45ec3f551.png media/images/validation_1896_a000f6436ad45ec3f551.png 2023-07-11 21:31:35,274 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1896_43076fbb48e6ed29a61c.png media/images/validation_1896_43076fbb48e6ed29a61c.png 2023-07-11 21:31:35,274 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1896_27c5036394cf299adb47.png media/images/validation_1896_27c5036394cf299adb47.png 2023-07-11 21:31:35,274 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1896_29f4e682c32068be6ee6.png media/images/validation_1896_29f4e682c32068be6ee6.png 2023-07-11 21:31:35,274 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1904_a8928482b1db701915ac.png media/images/validation_1904_a8928482b1db701915ac.png 2023-07-11 21:31:35,274 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1904_234baf3f279d560d92d2.png media/images/validation_1904_234baf3f279d560d92d2.png 2023-07-11 21:31:35,274 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1904_c1886dcd749cbc60adfd.png media/images/validation_1904_c1886dcd749cbc60adfd.png 2023-07-11 21:31:35,274 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1904_35be072a700b769038db.png media/images/validation_1904_35be072a700b769038db.png 2023-07-11 21:31:35,275 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1912_cdb5377e850db598c266.png media/images/validation_1912_cdb5377e850db598c266.png 2023-07-11 21:31:35,275 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1912_3af03303490f4370396f.png media/images/validation_1912_3af03303490f4370396f.png 2023-07-11 21:31:35,275 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1912_3adb61482ec02a0583d5.png media/images/validation_1912_3adb61482ec02a0583d5.png 2023-07-11 21:31:35,275 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1912_9da4544d95a30d67cb32.png media/images/validation_1912_9da4544d95a30d67cb32.png 2023-07-11 21:31:35,275 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1920_d5a4ff7de7a7153f2941.png media/images/validation_1920_d5a4ff7de7a7153f2941.png 2023-07-11 21:31:35,275 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1920_0d10a7dbf7251bbada04.png media/images/validation_1920_0d10a7dbf7251bbada04.png 2023-07-11 21:31:35,275 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1920_d289710871a92ba999e8.png media/images/validation_1920_d289710871a92ba999e8.png 2023-07-11 21:31:35,275 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1920_6d4d45fd2048c6be252f.png media/images/validation_1920_6d4d45fd2048c6be252f.png 2023-07-11 21:31:35,275 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1928_a008fe9b5be650d7d99f.png media/images/validation_1928_a008fe9b5be650d7d99f.png 2023-07-11 21:31:35,275 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1928_4bd865ae05d0074a2375.png media/images/validation_1928_4bd865ae05d0074a2375.png 2023-07-11 21:31:35,276 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1928_78729b8aaac1e6b08697.png media/images/validation_1928_78729b8aaac1e6b08697.png 2023-07-11 21:31:35,276 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1928_af8fa5edbca9692cffc2.png media/images/validation_1928_af8fa5edbca9692cffc2.png 2023-07-11 21:31:35,276 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1936_35429031d120f2b96126.png media/images/validation_1936_35429031d120f2b96126.png 2023-07-11 21:31:35,276 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1936_f3a3fadee46766d2470a.png media/images/validation_1936_f3a3fadee46766d2470a.png 2023-07-11 21:31:35,276 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1936_dcec368a7622b3d8c0be.png media/images/validation_1936_dcec368a7622b3d8c0be.png 2023-07-11 21:31:35,276 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1936_852acc36599591503eef.png media/images/validation_1936_852acc36599591503eef.png 2023-07-11 21:31:35,276 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1944_78808717363c62ee0546.png media/images/validation_1944_78808717363c62ee0546.png 2023-07-11 21:31:35,276 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1944_8177942062418e290d7d.png media/images/validation_1944_8177942062418e290d7d.png 2023-07-11 21:31:35,276 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1944_35adf9bd55486e1c564c.png media/images/validation_1944_35adf9bd55486e1c564c.png 2023-07-11 21:31:35,276 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1944_fa0c0398f5b77951fef8.png media/images/validation_1944_fa0c0398f5b77951fef8.png 2023-07-11 21:31:35,277 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1952_01c4c69e0d5b531f80da.png media/images/validation_1952_01c4c69e0d5b531f80da.png 2023-07-11 21:31:35,277 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1952_b9e6d46bbf3ea0d7f03d.png media/images/validation_1952_b9e6d46bbf3ea0d7f03d.png 2023-07-11 21:31:35,277 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1952_6ad546e10ab053b15547.png media/images/validation_1952_6ad546e10ab053b15547.png 2023-07-11 21:31:35,277 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1952_9eff640395fdb57cd9c7.png media/images/validation_1952_9eff640395fdb57cd9c7.png 2023-07-11 21:31:35,277 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1960_f69ac92a12eeba479470.png media/images/validation_1960_f69ac92a12eeba479470.png 2023-07-11 21:31:35,277 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1960_d0a34bec2dd9fe1ba19d.png media/images/validation_1960_d0a34bec2dd9fe1ba19d.png 2023-07-11 21:31:35,277 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1960_b35c1f83e65423b07bda.png media/images/validation_1960_b35c1f83e65423b07bda.png 2023-07-11 21:31:35,277 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1960_9263ee13eaef00dc177f.png media/images/validation_1960_9263ee13eaef00dc177f.png 2023-07-11 21:31:35,277 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1968_ff19fe998502f08ef355.png media/images/validation_1968_ff19fe998502f08ef355.png 2023-07-11 21:31:35,277 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1968_b94497f03241b794f556.png media/images/validation_1968_b94497f03241b794f556.png 2023-07-11 21:31:35,277 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1968_de75a859cd98957c62e9.png media/images/validation_1968_de75a859cd98957c62e9.png 2023-07-11 21:31:35,278 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1968_212445186f3f2a7ed98e.png media/images/validation_1968_212445186f3f2a7ed98e.png 2023-07-11 21:31:35,278 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1976_ca58948c6f5e8d1e6c38.png media/images/validation_1976_ca58948c6f5e8d1e6c38.png 2023-07-11 21:31:35,278 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1976_2891304fc5901c3c1e71.png media/images/validation_1976_2891304fc5901c3c1e71.png 2023-07-11 21:31:35,278 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1976_0df481310c9adbf409ba.png media/images/validation_1976_0df481310c9adbf409ba.png 2023-07-11 21:31:35,278 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1976_ec9a8a5c1f221d207e64.png media/images/validation_1976_ec9a8a5c1f221d207e64.png 2023-07-11 21:31:35,278 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1984_cf4f7819de6a9d73ca74.png media/images/validation_1984_cf4f7819de6a9d73ca74.png 2023-07-11 21:31:35,278 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1984_597e25f12127dac13b2a.png media/images/validation_1984_597e25f12127dac13b2a.png 2023-07-11 21:31:35,278 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1984_c4207687ea8ecd4f4a45.png media/images/validation_1984_c4207687ea8ecd4f4a45.png 2023-07-11 21:31:35,278 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1984_61b9f23963df182bd12d.png media/images/validation_1984_61b9f23963df182bd12d.png 2023-07-11 21:31:35,278 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1992_f8c5d5ec8577a44a670e.png media/images/validation_1992_f8c5d5ec8577a44a670e.png 2023-07-11 21:31:35,278 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1992_cf518b56981a6ad11247.png media/images/validation_1992_cf518b56981a6ad11247.png 2023-07-11 21:31:35,278 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1992_92aeaed1bdf499b11f79.png media/images/validation_1992_92aeaed1bdf499b11f79.png 2023-07-11 21:31:35,279 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_1992_281a32d10f5b0365fb14.png media/images/validation_1992_281a32d10f5b0365fb14.png 2023-07-11 21:31:35,279 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2000_56be1cccfe16e31d1eda.png media/images/validation_2000_56be1cccfe16e31d1eda.png 2023-07-11 21:31:35,279 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2000_5747cd6b7dfc06af840d.png media/images/validation_2000_5747cd6b7dfc06af840d.png 2023-07-11 21:31:35,279 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2000_11513f1b9a12e97321f9.png media/images/validation_2000_11513f1b9a12e97321f9.png 2023-07-11 21:31:35,279 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2000_4ee817ff55a50642bd59.png media/images/validation_2000_4ee817ff55a50642bd59.png 2023-07-11 21:31:35,279 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2008_edeeab747d59b9ede014.png media/images/validation_2008_edeeab747d59b9ede014.png 2023-07-11 21:31:35,279 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2008_a0785c61ee79cf436c16.png media/images/validation_2008_a0785c61ee79cf436c16.png 2023-07-11 21:31:35,279 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2008_9c383de17af136bf0d80.png media/images/validation_2008_9c383de17af136bf0d80.png 2023-07-11 21:31:35,279 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2008_67662513522c4b9f8fd5.png media/images/validation_2008_67662513522c4b9f8fd5.png 2023-07-11 21:31:35,279 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2016_7b8a29624a01f6e3b5aa.png media/images/validation_2016_7b8a29624a01f6e3b5aa.png 2023-07-11 21:31:35,279 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2016_42552ac0dc72fef56307.png media/images/validation_2016_42552ac0dc72fef56307.png 2023-07-11 21:31:35,280 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2016_caa85d659cf2405bcefd.png media/images/validation_2016_caa85d659cf2405bcefd.png 2023-07-11 21:31:35,280 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2016_b666077c67c4ed2ef995.png media/images/validation_2016_b666077c67c4ed2ef995.png 2023-07-11 21:31:35,280 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2024_39ba76f44c6320920da5.png media/images/validation_2024_39ba76f44c6320920da5.png 2023-07-11 21:31:35,280 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2024_42449f3583b7d0ee5299.png media/images/validation_2024_42449f3583b7d0ee5299.png 2023-07-11 21:31:35,280 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2024_58e7ab91d438b3e02f2c.png media/images/validation_2024_58e7ab91d438b3e02f2c.png 2023-07-11 21:31:35,280 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2024_5cd6da1a61c49184c69c.png media/images/validation_2024_5cd6da1a61c49184c69c.png 2023-07-11 21:31:35,280 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2032_e9dca02fa7653317e9d3.png media/images/validation_2032_e9dca02fa7653317e9d3.png 2023-07-11 21:31:35,280 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2032_0966289b7bf679cabff2.png media/images/validation_2032_0966289b7bf679cabff2.png 2023-07-11 21:31:35,280 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2032_ab427828257bfc2f48cd.png media/images/validation_2032_ab427828257bfc2f48cd.png 2023-07-11 21:31:35,280 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2032_f92644e9ab4ae89453ab.png media/images/validation_2032_f92644e9ab4ae89453ab.png 2023-07-11 21:31:35,281 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2040_1b5809dee200dda60230.png media/images/validation_2040_1b5809dee200dda60230.png 2023-07-11 21:31:35,281 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2040_f6836c5b36ab58e6c0fb.png media/images/validation_2040_f6836c5b36ab58e6c0fb.png 2023-07-11 21:31:35,281 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2040_11ffc1e2966f22ed98b2.png media/images/validation_2040_11ffc1e2966f22ed98b2.png 2023-07-11 21:31:35,281 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2040_c4c3d69e91379eeb2517.png media/images/validation_2040_c4c3d69e91379eeb2517.png 2023-07-11 21:31:35,281 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2048_123cf925aafba429d41f.png media/images/validation_2048_123cf925aafba429d41f.png 2023-07-11 21:31:35,281 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2048_2fac4432914c0d06816d.png media/images/validation_2048_2fac4432914c0d06816d.png 2023-07-11 21:31:35,281 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2048_b964563afa4fd9125c79.png media/images/validation_2048_b964563afa4fd9125c79.png 2023-07-11 21:31:35,281 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2048_9132f798001f5c0ccc6d.png media/images/validation_2048_9132f798001f5c0ccc6d.png 2023-07-11 21:31:35,281 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2056_f2d52946f625bac82075.png media/images/validation_2056_f2d52946f625bac82075.png 2023-07-11 21:31:35,281 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2056_ae649897728e92928855.png media/images/validation_2056_ae649897728e92928855.png 2023-07-11 21:31:35,282 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2056_c30db4d32c0339da583c.png media/images/validation_2056_c30db4d32c0339da583c.png 2023-07-11 21:31:35,282 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2056_119cb34b947beaab2613.png media/images/validation_2056_119cb34b947beaab2613.png 2023-07-11 21:31:35,282 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2064_ea44735eac8f71037fce.png media/images/validation_2064_ea44735eac8f71037fce.png 2023-07-11 21:31:35,282 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2064_b712e152cd356cd14c7a.png media/images/validation_2064_b712e152cd356cd14c7a.png 2023-07-11 21:31:35,282 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2064_519a4aa69108743c7ec6.png media/images/validation_2064_519a4aa69108743c7ec6.png 2023-07-11 21:31:35,282 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2064_31a98ecd5357f46a64be.png media/images/validation_2064_31a98ecd5357f46a64be.png 2023-07-11 21:31:35,282 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2072_ddac369b4ded67607317.png media/images/validation_2072_ddac369b4ded67607317.png 2023-07-11 21:31:35,282 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2072_1bc4c894c72890f665a0.png media/images/validation_2072_1bc4c894c72890f665a0.png 2023-07-11 21:31:35,282 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2072_e0f0cd526f1fbf142c07.png media/images/validation_2072_e0f0cd526f1fbf142c07.png 2023-07-11 21:31:35,282 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2072_a7d9f847844b98f32c62.png media/images/validation_2072_a7d9f847844b98f32c62.png 2023-07-11 21:31:35,282 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2080_234274db4799f23af905.png media/images/validation_2080_234274db4799f23af905.png 2023-07-11 21:31:35,282 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2080_470d734462cd775639c9.png media/images/validation_2080_470d734462cd775639c9.png 2023-07-11 21:31:35,283 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2080_b5332c71d8ad6af0f6cb.png media/images/validation_2080_b5332c71d8ad6af0f6cb.png 2023-07-11 21:31:35,283 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2080_c0c4ed778328664db5ef.png media/images/validation_2080_c0c4ed778328664db5ef.png 2023-07-11 21:31:35,283 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2088_1177c53f90c63a53fae4.png media/images/validation_2088_1177c53f90c63a53fae4.png 2023-07-11 21:31:35,283 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2088_5726b36383a81db55440.png media/images/validation_2088_5726b36383a81db55440.png 2023-07-11 21:31:35,283 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2088_6d9f398496e3d9fb86ac.png media/images/validation_2088_6d9f398496e3d9fb86ac.png 2023-07-11 21:31:35,283 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2088_ad4894c9ed610fe1ef61.png media/images/validation_2088_ad4894c9ed610fe1ef61.png 2023-07-11 21:31:35,283 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2096_3917a475eb07dd7d8014.png media/images/validation_2096_3917a475eb07dd7d8014.png 2023-07-11 21:31:35,283 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2096_b3898442a9a4b0221ae6.png media/images/validation_2096_b3898442a9a4b0221ae6.png 2023-07-11 21:31:35,283 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2096_ab63da1d97e801140f79.png media/images/validation_2096_ab63da1d97e801140f79.png 2023-07-11 21:31:35,283 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2096_52636a9fab6f23c15f25.png media/images/validation_2096_52636a9fab6f23c15f25.png 2023-07-11 21:31:35,283 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2104_f2ed3d7c9f2ca2721159.png media/images/validation_2104_f2ed3d7c9f2ca2721159.png 2023-07-11 21:31:35,284 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2104_8ff4357dcc31e396a35b.png media/images/validation_2104_8ff4357dcc31e396a35b.png 2023-07-11 21:31:35,284 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2104_17fbe5a0dfd4b8fd48f2.png media/images/validation_2104_17fbe5a0dfd4b8fd48f2.png 2023-07-11 21:31:35,284 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2104_619ebebabd9789748cac.png media/images/validation_2104_619ebebabd9789748cac.png 2023-07-11 21:31:35,284 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2112_ee9b23aa9877d821342d.png media/images/validation_2112_ee9b23aa9877d821342d.png 2023-07-11 21:31:35,284 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2112_de58ff517f7ad93b7980.png media/images/validation_2112_de58ff517f7ad93b7980.png 2023-07-11 21:31:35,284 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2112_fdc939f4f2cfbd53196d.png media/images/validation_2112_fdc939f4f2cfbd53196d.png 2023-07-11 21:31:35,284 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2112_99bcea6ecb4b26371b0a.png media/images/validation_2112_99bcea6ecb4b26371b0a.png 2023-07-11 21:31:35,284 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2120_0cc11534506dbd7f7bac.png media/images/validation_2120_0cc11534506dbd7f7bac.png 2023-07-11 21:31:35,285 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2120_4dd49f0520bc4123cc57.png media/images/validation_2120_4dd49f0520bc4123cc57.png 2023-07-11 21:31:35,285 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2120_a5f9ed84ca9d264b9439.png media/images/validation_2120_a5f9ed84ca9d264b9439.png 2023-07-11 21:31:35,285 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2120_f685f7aab896c2a636a6.png media/images/validation_2120_f685f7aab896c2a636a6.png 2023-07-11 21:31:35,285 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2128_40b0d08fdb15bcbe6d76.png media/images/validation_2128_40b0d08fdb15bcbe6d76.png 2023-07-11 21:31:35,285 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2128_b90bc0dcc518a41e4b7c.png media/images/validation_2128_b90bc0dcc518a41e4b7c.png 2023-07-11 21:31:35,285 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2128_1c5e23a118700525a7fd.png media/images/validation_2128_1c5e23a118700525a7fd.png 2023-07-11 21:31:35,285 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2128_88801f29434a2a3ded1a.png media/images/validation_2128_88801f29434a2a3ded1a.png 2023-07-11 21:31:35,285 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2136_5bc831942544a900b570.png media/images/validation_2136_5bc831942544a900b570.png 2023-07-11 21:31:35,285 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2136_961ca855c1854a6642a2.png media/images/validation_2136_961ca855c1854a6642a2.png 2023-07-11 21:31:35,285 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2136_b0e68e79f3ba56f234d1.png media/images/validation_2136_b0e68e79f3ba56f234d1.png 2023-07-11 21:31:35,285 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2136_0358ef62f3e922feee73.png media/images/validation_2136_0358ef62f3e922feee73.png 2023-07-11 21:31:35,286 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2144_dd8f27bdf40cf2d6cea9.png media/images/validation_2144_dd8f27bdf40cf2d6cea9.png 2023-07-11 21:31:35,286 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2144_1116dc6091dfb2a9a99b.png media/images/validation_2144_1116dc6091dfb2a9a99b.png 2023-07-11 21:31:35,286 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2144_33e87beb0aaa1b10bc0b.png media/images/validation_2144_33e87beb0aaa1b10bc0b.png 2023-07-11 21:31:35,286 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2144_3e34196265dc5d0dea47.png media/images/validation_2144_3e34196265dc5d0dea47.png 2023-07-11 21:31:35,286 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2152_7c72c782ab604dee6e7d.png media/images/validation_2152_7c72c782ab604dee6e7d.png 2023-07-11 21:31:35,286 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2152_9afccb4fc160b21562e7.png media/images/validation_2152_9afccb4fc160b21562e7.png 2023-07-11 21:31:35,286 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2152_0772544257fba9b99d85.png media/images/validation_2152_0772544257fba9b99d85.png 2023-07-11 21:31:35,286 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2152_a1a8f442cbc67c6daade.png media/images/validation_2152_a1a8f442cbc67c6daade.png 2023-07-11 21:31:35,286 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2160_df30dfecafc8706acc38.png media/images/validation_2160_df30dfecafc8706acc38.png 2023-07-11 21:31:35,286 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2160_d966f9a37b7475161185.png media/images/validation_2160_d966f9a37b7475161185.png 2023-07-11 21:31:35,286 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2160_802878029984f839da7a.png media/images/validation_2160_802878029984f839da7a.png 2023-07-11 21:31:35,286 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2160_42ae5bc916492ef02b4e.png media/images/validation_2160_42ae5bc916492ef02b4e.png 2023-07-11 21:31:35,287 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2168_7f77a8960cb444700fe1.png media/images/validation_2168_7f77a8960cb444700fe1.png 2023-07-11 21:31:35,287 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2168_7fa7b07d6e51ffc6ee7c.png media/images/validation_2168_7fa7b07d6e51ffc6ee7c.png 2023-07-11 21:31:35,287 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2168_11433d98fdff95688292.png media/images/validation_2168_11433d98fdff95688292.png 2023-07-11 21:31:35,287 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2168_ab42f1cb028adcc36624.png media/images/validation_2168_ab42f1cb028adcc36624.png 2023-07-11 21:31:35,287 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2176_4743122548c8aa7b69c0.png media/images/validation_2176_4743122548c8aa7b69c0.png 2023-07-11 21:31:35,287 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2176_282f811b6189202a6ebc.png media/images/validation_2176_282f811b6189202a6ebc.png 2023-07-11 21:31:35,287 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2176_0917d80c2d5e3650d3c3.png media/images/validation_2176_0917d80c2d5e3650d3c3.png 2023-07-11 21:31:35,287 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2176_91fab3422693e45733b1.png media/images/validation_2176_91fab3422693e45733b1.png 2023-07-11 21:31:35,287 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2184_1f1d05e973eac5066d87.png media/images/validation_2184_1f1d05e973eac5066d87.png 2023-07-11 21:31:35,287 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2184_92af547bd8ce73a91373.png media/images/validation_2184_92af547bd8ce73a91373.png 2023-07-11 21:31:35,287 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2184_f7d901834bbc876e0496.png media/images/validation_2184_f7d901834bbc876e0496.png 2023-07-11 21:31:35,288 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2184_8bea371a51240befcb0c.png media/images/validation_2184_8bea371a51240befcb0c.png 2023-07-11 21:31:35,288 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2192_43ded98624cac24bcce2.png media/images/validation_2192_43ded98624cac24bcce2.png 2023-07-11 21:31:35,288 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2192_1a9b8a8abe690731327a.png media/images/validation_2192_1a9b8a8abe690731327a.png 2023-07-11 21:31:35,288 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2192_cf71b06b81ea71e6cb38.png media/images/validation_2192_cf71b06b81ea71e6cb38.png 2023-07-11 21:31:35,288 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2192_ba57f9ac069863aa6a16.png media/images/validation_2192_ba57f9ac069863aa6a16.png 2023-07-11 21:31:35,288 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2200_ee3e22833ab7e1d894c4.png media/images/validation_2200_ee3e22833ab7e1d894c4.png 2023-07-11 21:31:35,288 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2200_c93ac716040789e5a0e3.png media/images/validation_2200_c93ac716040789e5a0e3.png 2023-07-11 21:31:35,288 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2200_00596fead5bf8fb0065d.png media/images/validation_2200_00596fead5bf8fb0065d.png 2023-07-11 21:31:35,288 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2200_4dfaff90212e9e27e4d2.png media/images/validation_2200_4dfaff90212e9e27e4d2.png 2023-07-11 21:31:35,288 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2208_ffbfe4f9aca48bda03c2.png media/images/validation_2208_ffbfe4f9aca48bda03c2.png 2023-07-11 21:31:35,289 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2208_eef023764c90ffd3644e.png media/images/validation_2208_eef023764c90ffd3644e.png 2023-07-11 21:31:35,289 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2208_2f73c4c4c50247c1768f.png media/images/validation_2208_2f73c4c4c50247c1768f.png 2023-07-11 21:31:35,289 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2208_308d8b2bbe36fb5f30fc.png media/images/validation_2208_308d8b2bbe36fb5f30fc.png 2023-07-11 21:31:35,289 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2216_441d862976c9f51b92d6.png media/images/validation_2216_441d862976c9f51b92d6.png 2023-07-11 21:31:35,289 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2216_5f8a9556edf1586f4491.png media/images/validation_2216_5f8a9556edf1586f4491.png 2023-07-11 21:31:35,289 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2216_775468a5cae55b01f9e6.png media/images/validation_2216_775468a5cae55b01f9e6.png 2023-07-11 21:31:35,289 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2216_8c1ab982496cf82a8f19.png media/images/validation_2216_8c1ab982496cf82a8f19.png 2023-07-11 21:31:35,289 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2224_13c69897297371dcf8c5.png media/images/validation_2224_13c69897297371dcf8c5.png 2023-07-11 21:31:35,289 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2224_ef5660a5a79987882227.png media/images/validation_2224_ef5660a5a79987882227.png 2023-07-11 21:31:35,289 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2224_d3babefe131fdc098ad5.png media/images/validation_2224_d3babefe131fdc098ad5.png 2023-07-11 21:31:35,290 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2224_9b460613f0ab2e925269.png media/images/validation_2224_9b460613f0ab2e925269.png 2023-07-11 21:31:35,290 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2232_5428aa3c038948a084d2.png media/images/validation_2232_5428aa3c038948a084d2.png 2023-07-11 21:31:35,290 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2232_d019d6eafe82501f45c5.png media/images/validation_2232_d019d6eafe82501f45c5.png 2023-07-11 21:31:35,290 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2232_de35cad4622885a2e7e3.png media/images/validation_2232_de35cad4622885a2e7e3.png 2023-07-11 21:31:35,290 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2232_810d1b9e7965f4ca6207.png media/images/validation_2232_810d1b9e7965f4ca6207.png 2023-07-11 21:31:35,290 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2240_fc9f1f75957ce5fda00b.png media/images/validation_2240_fc9f1f75957ce5fda00b.png 2023-07-11 21:31:35,290 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2240_0be5a7d6e1a87c6281ad.png media/images/validation_2240_0be5a7d6e1a87c6281ad.png 2023-07-11 21:31:35,290 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2240_e9d3e01b8fb9ddbcc763.png media/images/validation_2240_e9d3e01b8fb9ddbcc763.png 2023-07-11 21:31:35,290 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2240_5e39882221c773c2aaeb.png media/images/validation_2240_5e39882221c773c2aaeb.png 2023-07-11 21:31:35,290 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2248_4d6ac12b990c4625492f.png media/images/validation_2248_4d6ac12b990c4625492f.png 2023-07-11 21:31:35,290 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2248_60ebe036679521533b7c.png media/images/validation_2248_60ebe036679521533b7c.png 2023-07-11 21:31:35,290 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2248_2662eb4ad8f0124e3af2.png media/images/validation_2248_2662eb4ad8f0124e3af2.png 2023-07-11 21:31:35,291 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2248_20630ae4a09080a1aebc.png media/images/validation_2248_20630ae4a09080a1aebc.png 2023-07-11 21:31:35,291 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2256_f708e90e7fd5277c4fe2.png media/images/validation_2256_f708e90e7fd5277c4fe2.png 2023-07-11 21:31:35,291 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2256_e517c86d4a854fa8f3b6.png media/images/validation_2256_e517c86d4a854fa8f3b6.png 2023-07-11 21:31:35,291 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2256_6feffa88023b5b42629e.png media/images/validation_2256_6feffa88023b5b42629e.png 2023-07-11 21:31:35,291 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2256_6030757c8800e8a83428.png media/images/validation_2256_6030757c8800e8a83428.png 2023-07-11 21:31:35,291 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2264_f57b404ae171a2afb73d.png media/images/validation_2264_f57b404ae171a2afb73d.png 2023-07-11 21:31:35,291 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2264_ab601295b277ed2d8585.png media/images/validation_2264_ab601295b277ed2d8585.png 2023-07-11 21:31:35,291 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2264_d0f444003a54ce1ba603.png media/images/validation_2264_d0f444003a54ce1ba603.png 2023-07-11 21:31:35,291 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2264_6107b10540fd61346e76.png media/images/validation_2264_6107b10540fd61346e76.png 2023-07-11 21:31:35,292 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2272_5b58952985f4c02da378.png media/images/validation_2272_5b58952985f4c02da378.png 2023-07-11 21:31:35,292 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2272_1e1c2cc5fe23888d9a0f.png media/images/validation_2272_1e1c2cc5fe23888d9a0f.png 2023-07-11 21:31:35,292 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2272_5722a72d5ca3dd7d3435.png media/images/validation_2272_5722a72d5ca3dd7d3435.png 2023-07-11 21:31:35,292 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2272_7bfa00fea9619c31d907.png media/images/validation_2272_7bfa00fea9619c31d907.png 2023-07-11 21:31:35,292 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2280_3dc11ed55953ee19931f.png media/images/validation_2280_3dc11ed55953ee19931f.png 2023-07-11 21:31:35,292 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2280_9613c1b3cfb26096117f.png media/images/validation_2280_9613c1b3cfb26096117f.png 2023-07-11 21:31:35,292 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2280_ea5a63c05b6dc2e76aad.png media/images/validation_2280_ea5a63c05b6dc2e76aad.png 2023-07-11 21:31:35,292 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2280_6d2411d6fdd40f97df6f.png media/images/validation_2280_6d2411d6fdd40f97df6f.png 2023-07-11 21:31:35,292 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2288_7286636f68f0a2aa4e30.png media/images/validation_2288_7286636f68f0a2aa4e30.png 2023-07-11 21:31:35,292 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2288_a5cceb7ef5da03ddb014.png media/images/validation_2288_a5cceb7ef5da03ddb014.png 2023-07-11 21:31:35,292 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2288_572e255d21901e5a6046.png media/images/validation_2288_572e255d21901e5a6046.png 2023-07-11 21:31:35,293 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2288_67a013a5a370a35a3059.png media/images/validation_2288_67a013a5a370a35a3059.png 2023-07-11 21:31:35,293 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2296_e87577dcb5ac8c5c2a2c.png media/images/validation_2296_e87577dcb5ac8c5c2a2c.png 2023-07-11 21:31:35,293 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2296_cb7397f7fd1ac20d272d.png media/images/validation_2296_cb7397f7fd1ac20d272d.png 2023-07-11 21:31:35,293 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2296_47afc8d7e2eb0efcc511.png media/images/validation_2296_47afc8d7e2eb0efcc511.png 2023-07-11 21:31:35,293 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2296_250d2d9f0b19e4298842.png media/images/validation_2296_250d2d9f0b19e4298842.png 2023-07-11 21:31:35,293 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2304_4cd5bd883752c4c9c8c0.png media/images/validation_2304_4cd5bd883752c4c9c8c0.png 2023-07-11 21:31:35,293 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2304_e90e75900878a3127495.png media/images/validation_2304_e90e75900878a3127495.png 2023-07-11 21:31:35,293 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2304_5c183443d040ea97fa1b.png media/images/validation_2304_5c183443d040ea97fa1b.png 2023-07-11 21:31:35,293 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2304_2e6facfca2619f0d7cb5.png media/images/validation_2304_2e6facfca2619f0d7cb5.png 2023-07-11 21:31:35,293 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2312_9c89f883bf5fcc238280.png media/images/validation_2312_9c89f883bf5fcc238280.png 2023-07-11 21:31:35,293 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2312_21c7c0440b0b11b2d491.png media/images/validation_2312_21c7c0440b0b11b2d491.png 2023-07-11 21:31:35,294 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2312_e9126f14486264f97c5b.png media/images/validation_2312_e9126f14486264f97c5b.png 2023-07-11 21:31:35,294 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2312_52029317168919eb0da8.png media/images/validation_2312_52029317168919eb0da8.png 2023-07-11 21:31:35,294 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2320_39c700c0233bf9ec1887.png media/images/validation_2320_39c700c0233bf9ec1887.png 2023-07-11 21:31:35,294 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2320_68a147e55429db2d8cd5.png media/images/validation_2320_68a147e55429db2d8cd5.png 2023-07-11 21:31:35,294 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2320_3e0ad5d834a85111111c.png media/images/validation_2320_3e0ad5d834a85111111c.png 2023-07-11 21:31:35,294 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2320_84ca29059a3b9b5905fc.png media/images/validation_2320_84ca29059a3b9b5905fc.png 2023-07-11 21:31:35,294 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2328_0dafda20d7536a74a759.png media/images/validation_2328_0dafda20d7536a74a759.png 2023-07-11 21:31:35,294 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2328_80c49839f74d15e546e2.png media/images/validation_2328_80c49839f74d15e546e2.png 2023-07-11 21:31:35,294 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2328_26e2b28b1b2c040caf0a.png media/images/validation_2328_26e2b28b1b2c040caf0a.png 2023-07-11 21:31:35,294 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2328_62f7b60d0949855bbdda.png media/images/validation_2328_62f7b60d0949855bbdda.png 2023-07-11 21:31:35,294 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2336_ae700d32677535960099.png media/images/validation_2336_ae700d32677535960099.png 2023-07-11 21:31:35,294 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2336_301a887d407c4ac4a9e8.png media/images/validation_2336_301a887d407c4ac4a9e8.png 2023-07-11 21:31:35,295 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2336_a5b295c1cf8890d267c2.png media/images/validation_2336_a5b295c1cf8890d267c2.png 2023-07-11 21:31:35,295 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2336_354d7f9111304540102a.png media/images/validation_2336_354d7f9111304540102a.png 2023-07-11 21:31:35,295 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2344_d627b8b33c228c221574.png media/images/validation_2344_d627b8b33c228c221574.png 2023-07-11 21:31:35,295 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2344_e060b93c7cc983117765.png media/images/validation_2344_e060b93c7cc983117765.png 2023-07-11 21:31:35,295 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2344_f26688d7e9d47ebeba3b.png media/images/validation_2344_f26688d7e9d47ebeba3b.png 2023-07-11 21:31:35,295 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2344_b0129eec4a00fb5d2469.png media/images/validation_2344_b0129eec4a00fb5d2469.png 2023-07-11 21:31:35,295 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2352_ce593cd883ca40dee3d7.png media/images/validation_2352_ce593cd883ca40dee3d7.png 2023-07-11 21:31:35,295 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2352_e9cbdb785c02cd01bf26.png media/images/validation_2352_e9cbdb785c02cd01bf26.png 2023-07-11 21:31:35,295 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2352_c61b0d4f5e5ee8adb21f.png media/images/validation_2352_c61b0d4f5e5ee8adb21f.png 2023-07-11 21:31:35,295 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2352_c77b9bc3c6982cc22106.png media/images/validation_2352_c77b9bc3c6982cc22106.png 2023-07-11 21:31:35,295 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2360_4d37f738f3ac19d459df.png media/images/validation_2360_4d37f738f3ac19d459df.png 2023-07-11 21:31:35,296 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2360_0a11cdad43876d67f6b1.png media/images/validation_2360_0a11cdad43876d67f6b1.png 2023-07-11 21:31:35,296 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2360_945a8790f028d3a3a4ac.png media/images/validation_2360_945a8790f028d3a3a4ac.png 2023-07-11 21:31:35,296 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2360_9df37ef1c70edfc470f2.png media/images/validation_2360_9df37ef1c70edfc470f2.png 2023-07-11 21:31:35,296 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2368_0b60b02ca6831735feee.png media/images/validation_2368_0b60b02ca6831735feee.png 2023-07-11 21:31:35,296 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2368_e455e3cf2c345fdc0326.png media/images/validation_2368_e455e3cf2c345fdc0326.png 2023-07-11 21:31:35,296 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2368_342e28a8b292c6288351.png media/images/validation_2368_342e28a8b292c6288351.png 2023-07-11 21:31:35,296 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2368_d3688889a7a8ef78609d.png media/images/validation_2368_d3688889a7a8ef78609d.png 2023-07-11 21:31:35,296 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2376_cc2cb6db5aeb4f4ce3e7.png media/images/validation_2376_cc2cb6db5aeb4f4ce3e7.png 2023-07-11 21:31:35,296 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2376_bc22907973133c98cdde.png media/images/validation_2376_bc22907973133c98cdde.png 2023-07-11 21:31:35,296 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2376_e2203a4cdfa94cd450dc.png media/images/validation_2376_e2203a4cdfa94cd450dc.png 2023-07-11 21:31:35,296 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2376_11c406e93bca8b9b3bca.png media/images/validation_2376_11c406e93bca8b9b3bca.png 2023-07-11 21:31:35,296 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2384_81c5a876097ae538156d.png media/images/validation_2384_81c5a876097ae538156d.png 2023-07-11 21:31:35,297 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2384_34e253f269b378eb15d6.png media/images/validation_2384_34e253f269b378eb15d6.png 2023-07-11 21:31:35,297 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2384_d4b3165970b466047f71.png media/images/validation_2384_d4b3165970b466047f71.png 2023-07-11 21:31:35,297 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2384_0b1a1c25ca7c935944d8.png media/images/validation_2384_0b1a1c25ca7c935944d8.png 2023-07-11 21:31:35,297 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2392_ab6620fbc899727dc772.png media/images/validation_2392_ab6620fbc899727dc772.png 2023-07-11 21:31:35,297 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2392_15295da0dc5bd19625bd.png media/images/validation_2392_15295da0dc5bd19625bd.png 2023-07-11 21:31:35,297 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2392_1c61e21e1cd4c686eaaf.png media/images/validation_2392_1c61e21e1cd4c686eaaf.png 2023-07-11 21:31:35,297 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2392_884c2d28486ad0ad23ae.png media/images/validation_2392_884c2d28486ad0ad23ae.png 2023-07-11 21:31:35,297 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2400_edf7e219837bb0398a66.png media/images/validation_2400_edf7e219837bb0398a66.png 2023-07-11 21:31:35,297 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2400_8744e2e9594e95a56a40.png media/images/validation_2400_8744e2e9594e95a56a40.png 2023-07-11 21:31:35,297 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2400_689387d208c87e82a9d4.png media/images/validation_2400_689387d208c87e82a9d4.png 2023-07-11 21:31:35,297 INFO SenderThread:2302110 [dir_watcher.py:finish():403] scan save: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/media/images/validation_2400_550889b0c84a4a159425.png media/images/validation_2400_550889b0c84a4a159425.png 2023-07-11 21:31:35,298 INFO SenderThread:2302110 [sender.py:transition_state():602] send defer: 10 2023-07-11 21:31:35,298 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: poll_exit 2023-07-11 21:31:35,300 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: defer 2023-07-11 21:31:35,300 INFO HandlerThread:2302110 [handler.py:handle_request_defer():170] handle defer: 10 2023-07-11 21:31:35,301 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: defer 2023-07-11 21:31:35,301 INFO SenderThread:2302110 [sender.py:send_request_defer():598] handle sender defer: 10 2023-07-11 21:31:35,301 INFO SenderThread:2302110 [file_pusher.py:finish():159] shutting down file pusher 2023-07-11 21:31:35,514 INFO wandb-upload_0:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/cvv9ycwl-media/images/validation_2400_550889b0c84a4a159425.png 2023-07-11 21:31:35,552 INFO wandb-upload_2:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/j3w4vq1e-media/images/validation_2400_8744e2e9594e95a56a40.png 2023-07-11 21:31:35,605 INFO wandb-upload_5:2302110 [upload_job.py:push():133] Uploaded file /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/wandb-summary.json 2023-07-11 21:31:35,610 INFO wandb-upload_4:2302110 [upload_job.py:push():133] Uploaded file /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/config.yaml 2023-07-11 21:31:35,618 INFO wandb-upload_3:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/p72wx9am-media/images/validation_2400_edf7e219837bb0398a66.png 2023-07-11 21:31:35,631 INFO wandb-upload_7:2302110 [upload_job.py:push():133] Uploaded file /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/output.log 2023-07-11 21:31:35,671 INFO wandb-upload_6:2302110 [upload_job.py:push():133] Uploaded file /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/files/requirements.txt 2023-07-11 21:31:35,725 INFO wandb-upload_1:2302110 [upload_job.py:push():133] Uploaded file /tmp/tmpuvh15xivwandb/dd2w1cfj-media/images/validation_2400_689387d208c87e82a9d4.png 2023-07-11 21:31:35,810 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: poll_exit 2023-07-11 21:31:35,811 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: poll_exit 2023-07-11 21:31:35,926 INFO Thread-11 :2302110 [sender.py:transition_state():602] send defer: 11 2023-07-11 21:31:35,926 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: defer 2023-07-11 21:31:35,927 INFO HandlerThread:2302110 [handler.py:handle_request_defer():170] handle defer: 11 2023-07-11 21:31:35,928 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: defer 2023-07-11 21:31:35,928 INFO SenderThread:2302110 [sender.py:send_request_defer():598] handle sender defer: 11 2023-07-11 21:31:35,928 INFO SenderThread:2302110 [file_pusher.py:join():164] waiting for file pusher 2023-07-11 21:31:35,928 INFO SenderThread:2302110 [sender.py:transition_state():602] send defer: 12 2023-07-11 21:31:35,928 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: defer 2023-07-11 21:31:35,929 INFO HandlerThread:2302110 [handler.py:handle_request_defer():170] handle defer: 12 2023-07-11 21:31:35,929 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: defer 2023-07-11 21:31:35,929 INFO SenderThread:2302110 [sender.py:send_request_defer():598] handle sender defer: 12 2023-07-11 21:31:36,341 INFO SenderThread:2302110 [sender.py:transition_state():602] send defer: 13 2023-07-11 21:31:36,342 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: defer 2023-07-11 21:31:36,342 INFO HandlerThread:2302110 [handler.py:handle_request_defer():170] handle defer: 13 2023-07-11 21:31:36,343 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: defer 2023-07-11 21:31:36,343 INFO SenderThread:2302110 [sender.py:send_request_defer():598] handle sender defer: 13 2023-07-11 21:31:36,343 INFO SenderThread:2302110 [sender.py:transition_state():602] send defer: 14 2023-07-11 21:31:36,344 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: defer 2023-07-11 21:31:36,344 INFO HandlerThread:2302110 [handler.py:handle_request_defer():170] handle defer: 14 2023-07-11 21:31:36,344 DEBUG SenderThread:2302110 [sender.py:send():369] send: final 2023-07-11 21:31:36,345 DEBUG SenderThread:2302110 [sender.py:send():369] send: footer 2023-07-11 21:31:36,345 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: defer 2023-07-11 21:31:36,345 INFO SenderThread:2302110 [sender.py:send_request_defer():598] handle sender defer: 14 2023-07-11 21:31:36,346 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: poll_exit 2023-07-11 21:31:36,347 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: poll_exit 2023-07-11 21:31:36,351 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: poll_exit 2023-07-11 21:31:36,351 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: poll_exit 2023-07-11 21:31:36,355 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: server_info 2023-07-11 21:31:36,356 DEBUG SenderThread:2302110 [sender.py:send_request():396] send_request: server_info 2023-07-11 21:31:36,356 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: get_summary 2023-07-11 21:31:36,362 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: sampled_history 2023-07-11 21:31:36,449 INFO MainThread:2302110 [wandb_run.py:_footer_history_summary_info():3467] rendering history 2023-07-11 21:31:36,450 INFO MainThread:2302110 [wandb_run.py:_footer_history_summary_info():3499] rendering summary 2023-07-11 21:31:36,450 INFO MainThread:2302110 [wandb_run.py:_footer_sync_info():3426] logging synced files 2023-07-11 21:31:36,451 DEBUG HandlerThread:2302110 [handler.py:handle_request():144] handle_request: shutdown 2023-07-11 21:31:36,451 INFO HandlerThread:2302110 [handler.py:finish():854] shutting down handler 2023-07-11 21:31:37,356 INFO WriterThread:2302110 [datastore.py:close():298] close: /l/vision/v5/sragas/easel_ai/models_plain/wandb/run-20230711_182238-k29fqnbt/run-k29fqnbt.wandb 2023-07-11 21:31:37,449 INFO SenderThread:2302110 [sender.py:finish():1526] shutting down sender 2023-07-11 21:31:37,450 INFO SenderThread:2302110 [file_pusher.py:finish():159] shutting down file pusher 2023-07-11 21:31:37,450 INFO SenderThread:2302110 [file_pusher.py:join():164] waiting for file pusher